The Producer-Consumer Mismatch

Modern smartphone cameras capture frames at 30–240 fps. Meanwhile, even quantized MobileNet variants on an A15 Bionic might process only 15–20 fps. This gap creates a classic backpressure problem: when downstream consumers (your ML inference pipeline) cannot keep pace with upstream producers (the camera HAL), you must decide how to handle the overflow.

The naive approach—unbounded queuing—leads to memory bloat and stale predictions. A 60fps camera feeding a 15fps model accumulates 45 frames per second. Within three seconds you've queued 135 frames, consuming 200+ MB of RAM and introducing two-second latency between capture and result. For live OCR, gesture recognition, or PPG analysis, this delay renders the system unusable.

Strategy One: Drop Frames

The simplest backpressure strategy is to skip frames when the model is busy. In pseudocode:

if (!modelBusy) {
  modelBusy = true
  runInference(frame)
    .finally(() => modelBusy = false)
} else {
  // drop frame
}

This approach guarantees constant memory usage and minimal latency. When shipping HearingAid Pro, we processed audio-synchronized video at camera frame rate but only ran lip-detection inference when the previous pass completed. The result: 12–18fps effective throughput with zero queue buildup.

Advantages

  • Predictable memory: Only one frame in flight at any time
  • Low latency: Results reflect near-current state
  • Thermal stability: Natural rate limiting prevents sustained CPU/GPU saturation

Tradeoffs

You sacrifice temporal continuity. If your model processes frames at 15fps but the camera runs at 60fps, you analyze only 25% of available data. For object tracking, this means you might miss fast-moving subjects. For PPG heart rate estimation, you lose signal fidelity—critical 240Hz photoplethysmogram peaks may fall between sampled frames.

Strategy Two: Bounded Queue with FIFO Eviction

A fixed-size ring buffer decouples producer and consumer while capping memory. When the queue fills, you evict the oldest frame:

class BoundedFrameQueue {
  private frames: Frame[] = []
  private maxSize = 3

  enqueue(frame: Frame) {
    if (frames.length >= maxSize) {
      frames.shift() // drop oldest
    }
    frames.push(frame)
  }

  dequeue(): Frame | null {
    return frames.shift() ?? null
  }
}

This pattern appeared in GlucoScan AI, where we queued up to four PPG frames (16ms each at 60fps) before inference. The buffer smoothed out iOS priority inversion spikes—when the UI thread briefly starved the inference thread—without unbounded growth.

Why FIFO Eviction?

Dropping the oldest frame preserves recency. If your queue holds frames at t=0ms, t=16ms, t=32ms and a new frame arrives at t=48ms, you discard t=0ms. The model always sees the three most recent samples, maintaining temporal relevance.

Measuring Queue Depth

Instrument your pipeline to log queue occupancy. In production, we observed:

  • Idle state: Queue empty 40% of the time (model faster than camera intermittently)
  • Steady state: 1-2 frames queued (model slightly slower on average)
  • Thermal throttle: Queue saturated at max depth (CPU frequency scaled down)

If you see sustained saturation, your buffer is too small or your model is too slow. A queue that never fills suggests you're over-provisioned—you could reduce buffer size or increase model complexity.

Strategy Three: Priority-Based Dropping

Not all frames carry equal information. In KidzCare's speech therapy module, we ran real-time phoneme detection on video. Frames where the child's mouth was closed contributed little to articulation analysis. We assigned each frame a saliency score based on optical flow magnitude and dropped low-priority frames first:

enqueue(frame: Frame, priority: number) {
  if (frames.length >= maxSize) {
    const minIdx = frames.reduce(
      (min, f, i) => f.priority < frames[min].priority ? i : min,
      0
    )
    if (priority > frames[minIdx].priority) {
      frames[minIdx] = { frame, priority }
    }
  } else {
    frames.push({ frame, priority })
  }
}

This heuristic improved phoneme detection accuracy by 11% compared to blind FIFO, because we retained frames with visible mouth motion during queue pressure.

Computing Priority

Saliency scoring must be cheaper than the ML model itself, or you've simply moved the bottleneck. We used a lightweight Sobel edge detector on a downsampled 80×60 crop around the detected mouth region. Total cost: 0.3ms on iPhone 12, vs. 65ms for the full phoneme classifier.

Hybrid: Adaptive Dropping

The most sophisticated approach dynamically adjusts behavior based on system state. Monitor three signals:

  1. Queue occupancy: Current depth vs. max capacity
  2. Inference latency: Rolling P95 over last 30 frames
  3. Thermal state: iOS ProcessInfo.thermalState or Android PowerManager.getThermalHeadroom

Transition between strategies:

if (thermalState === 'critical' || queueDepth > 0.8 * maxSize) {
  strategy = 'drop'
} else if (inferenceP95 < targetLatency * 0.7) {
  strategy = 'queue'
} else {
  strategy = 'priority'
}

In OfflineAI, our on-device LLM chat app, we queued user input tokens during normal operation but switched to dropping intermediate decoding steps when thermal throttling kicked in. This maintained UI responsiveness (users saw tokens streaming) while preventing thermal shutdown.

Measuring Real-World Impact

Instrument four metrics:

  • End-to-end latency: Camera timestamp to result display (P50, P95, P99)
  • Frame utilization: Percentage of captured frames that reach inference
  • Memory high-water mark: Peak allocated bytes during 60-second session
  • Thermal events: Count of thermal state transitions

On iPhone 13 Pro running continuous object detection, we measured:

StrategyP95 LatencyUtilizationPeak MemoryDrop68ms22%45MBQueue (n=5)310ms100%180MBPriority95ms54%72MBAdaptive88ms61%68MB

The adaptive approach delivered 77% of the queuing strategy's utilization at 28% of its latency, with controlled memory growth.

Implementation Notes

In Swift, use DispatchQueue with .userInitiated QoS for the inference thread. Avoid .background—iOS may deprioritize it below system daemons, causing unbounded queue growth. In Kotlin, prefer Dispatchers.Default over Dispatchers.IO for CPU-bound ML work.

For camera frames, use AVCaptureVideoDataOutput's alwaysDiscardsLateVideoFrames on iOS or ImageReader with maxImages=1 on Android to let the OS handle upstream backpressure before frames reach your queue.

Choosing Your Strategy

Select based on workload characteristics:

  • Latency-critical (AR, gesture control): Drop frames
  • High-fidelity signal (PPG, audio sync): Small bounded queue (n=2–4)
  • Variable importance (OCR, scene understanding): Priority dropping
  • Long sessions with thermal risk: Adaptive hybrid

The wrong choice compounds. An OCR app that queues frames will show users stale text from seconds ago, training them to hold the camera still—exactly the opposite of the fluid scanning experience you want. Meanwhile, a PPG heart rate monitor that drops 80% of frames will produce noisy, unreliable BPM estimates.

Backpressure is not a performance optimization—it's a product design decision that directly shapes user experience. Choose deliberately.