Sampling Video Without Decoding Every Frame

While adding video import to Ayumi, I wanted to reduce how much video I sent to Gemini, especially when uploading over a mobile connection. Sending the original camera video meant transferring far more data than I wanted for a transcription-oriented journaling workflow.

Gemini’s default static video processing samples one frame per second. If Gemini looks at one frame per second anyway, uploading thirty frames per second mostly transfers pictures it does not use. Supported models also offer an agentic mode that dynamically explores the timeline and inspects content as needed, as described in Google’s video-understanding documentation; a one-frame-per-second proxy would leave that mode less to explore. For my transcription-oriented use case, static processing seemed sufficient, although I have not compared the output quality of the two modes.

That led me to explore sampling and compressing the video on the device before sending it. Having Gemini sample an uploaded original would still require transferring that original. I wanted to reduce upload traffic and mobile-data costs by sending a smaller video in the first place. The preprocessing happens on the device; the resulting proxy still goes to Gemini for cloud processing.

My first AVFoundation pipeline decoded a 4K HEVC track, kept one frame per second, and wrote those frames as H.264. I also needed still images for the resulting entry. The output was small, but preparing it still took several seconds for a short clip.

I questioned why dropping so many frames was still slow. The drop happened after AVAssetReader had decoded each frame. Writing one frame per second had not removed the work of decoding thirty.

Remove the writer to find the cost

I tested the read paths separately on a Mac with a synthetic 4K HEVC, 30 fps clip: six generated scenes with text-to-speech narration, lasting about 44.6 seconds. A second input repeated that sequence eight times, lasting about six minutes. Neither input was a private journal recording.

The microbenchmark did no video writing. It compared decoded reader output, compressed reader output, and AVAssetImageGenerator requests at one-second intervals. The image generator’s maximum size was 640 × 640.

The decoded reader returned 1,338 frames for the short clip. The tolerant image generator returned 45 images. This isolated the expensive part: reading compressed samples was cheap in this run, while turning all of them into pixels was not.

These are single-run microbenchmarks, not complete proxy-export times or an iPhone speed claim, and they do not predict the cost for a different encoding or longer keyframe intervals. The sparse path also reduces image dimensions, so the table compares work I could omit for this importer rather than identical outputs. The zero-tolerance path returned 44 images on the short input and 356 on the long one, one fewer than the tolerant path in each case. The harness skipped failed requests, so I did not establish that exact-time requests succeed at every timestamp.

Request sparse images before encoding

I replaced the full-track video reader with AVAssetImageGenerator, then fed the generated images to AVAssetWriter. Audio still goes through a separate reader. The same generated images also supply candidate stills, so preparing the proxy and preparing those stills share one extraction pass.

The relevant setup in the builder is:

let generator = AVAssetImageGenerator(asset: asset)
generator.appliesPreferredTrackTransform = true
generator.dynamicRangePolicy = .forceSDR
generator.maximumSize = CGSize(width: options.stillLongSide, height: options.stillLongSide)
let tolerance = CMTime(seconds: 0.5 / options.framesPerSecond, preferredTimescale: 600)
generator.requestedTimeToleranceBefore = tolerance
generator.requestedTimeToleranceAfter = tolerance

The production builder requests larger images than the microbenchmark, 1920 pixels on the long side by default, because they also become stills. It scales those pixels down for the proxy, whose default short side is 360 pixels. The benchmark’s 0.46 seconds therefore does not describe this whole builder.

The tolerance is half the sampling interval, 0.5 / fps, which is ±0.5 seconds at the default one frame per second. Apple documents that the default tolerance is unbounded, and that setting both tolerances to zero requests frame-accurate generation and may add decoding delay. Allowing half an interval on either side leaves room to use a nearby image that is cheaper to produce, while keeping each picture no farther from its own sample time than from the next one. In the extraction-only benchmark, going from zero to ±0.5 seconds cut the time from 0.76 to 0.46 seconds on the 44.6-second clip and from 6.13 to 3.84 seconds on the six-minute one, though the two paths did not return the same number of images. A nonzero tolerance still does not promise that every request uses a keyframe or avoids all intermediate decoding.

I have not shown that ±0.5 seconds is the best value. I did not compare it with ±0.1 or ±0.25 seconds, or measure how the time offset affects recognition quality. I took it as a compromise between speed and timing accuracy.

That choice has a visible boundary. The builder writes each image at the requested presentation time and records that requested time for the still. The picture can come from a nearby source time. This is a sampled proxy, not a frame-accurate editing timeline.

Check the pixels as well as the duration

A smaller output file would not catch a sideways or cropped picture. The builder’s regression test creates a landscape-encoded red-and-blue clip with a portrait display transform and a synthetic audio tone. It checks that the proxy is physically 360 × 640 with an identity transform, red above blue, three candidate stills for a three-second input, and a 16 kHz mono AAC track. A separate test covers video with no audio.

Those tests establish the synthetic geometry and audio paths. They do not establish HDR appearance, Gemini’s interpretation of arbitrary video, or long-video performance on a phone. The frame-generation loop also skips failed image requests, leaving the preceding proxy frame visible longer; successful export alone is not a check that every requested image was produced.

For this importer I kept the tolerant, sparse-image path. Decoding fewer pictures changed the work before the encoder, while dropping pictures afterward had only changed what I wrote.