Preface:

It all started with a misunderstanding.

I noticed a new page in the Gemini API documentation called Omni, introducing a model named Gemini Omni Flash, described as "natively multimodal, processing text, images, audio, and video simultaneously." My first thought was straightforward: if I throw a whole folder of videos and photos from my phone into it, let it understand what each asset is about, and then tell it in one sentence to edit them into a short video—isn't that a video editing app?

After reading the documentation, I realized I had misunderstood, and the misunderstanding happened to be at the most critical point. However, after bypassing that limitation, the rest was actually feasible. The result is ReelCraft: a Python CLI where you feed in a bunch of videos and photos, Gemini 3.7 Flash understands the assets one by one and provides editing suggestions. Once I confirm the edit list, ffmpeg cuts it into a 9:16 vertical short video, background music is generated using Lyria 3, and subtitles are automatically burned in.

Along the way, there were three issues where both ffmpeg and Gemini reported success, but the output was wrong—the kind of errors you only discover by actually playing the video.

TL;DR

This article will cover:

• Omni Flash is not what I thought it was
• Bypassing limitations: Per-file understanding, then text aggregation
• Using edl.yaml as a human confirmation point
• The difference after switching to Gemini 3.7 Flash
• Background music: Lyria 3 uses a different API
• ffmpeg will silently fail your edits
• Subtitles: Two issues only visible after burning them in
• Other pitfalls
• Conclusion
• Reference links

Omni Flash is not what I thought it was

Gemini Omni Flash (gemini-omni-flash-preview) is a video generation and editing model that uses the Interactions API. It allows you to use natural language to apply effects to a single video, such as "when the person touches the mirror, make the mirror ripple beautifully like liquid." It is not a tool for "understanding a bunch of videos."

The limitation section states clearly:

Referencing or reasoning across multiple videos is not supported. Attempting multi-video prompting may result in degraded model performance or unexpected outputs.

Additionally:

Video references up to 3 seconds in duration are accepted by the API schema but are not correctly processed by the model at this time.

So the path of "throwing a bunch of videos in and letting it understand and edit them" was blocked for Omni Flash. The models that can actually perform multi-video understanding are the standard Gemini models: starting from version 2.5, a single request can include up to 10 videos. With a 1M context window, it can handle about an hour of footage at default resolution, tokenize it second-by-second, and output scene descriptions with timestamps.

The time spent on this misunderstanding wasn't wasted. The verification process helped clarify "which task should be handled by which model," and the architecture followed naturally.

Bypassing limitations: Per-file understanding, then text aggregation

The entire pipeline is split into five stages, with states stored in files:

[Asset Folder]
     │ poc ingest: Scan videos/photos → catalog.json
     ▼
     │ poc analyze: Call Gemini for each file individually → analysis/*.json
     ▼
     │ poc plan: Aggregate all analysis results, call once for editing suggestions
     ▼ → summary.md (for humans) + edl.yaml (for machine execution)
     ⏸ Human inspection and editing of edl.yaml
     ▼
     │ poc render: ffmpeg editing, 9:16 cropping, xfade transitions
     ▼
output/final.mp4

The key design decision is in the second and third steps: call Gemini once for each video to get precise internal timestamps and descriptions; then feed these text results (not the raw videos) into a second call for cross-asset aggregation, sequencing, and editing suggestions.

This approach has two benefits. First, it completely avoids the "multi-video reasoning not supported" issue because the second call only sees text, not ten videos. Second, it isn't limited by the 10 videos/request cap; no matter how many assets there are, it just means more independent calls in the analyze phase. Those calls can be retried or fail individually without affecting each other.

Testing also proved that timestamps are more reliable when processed separately. When asking about ten videos in a single prompt ("which seconds are the highlights?"), the model easily confuses the timelines of different videos.

Failure handling in the analyze phase is recorded separately: if a file fails after three retries, it's logged in analysis/_errors.json, while other files continue. This later revealed a loophole during review, which I'll discuss later.

Using edl.yaml as a human confirmation point

I decided from the start not to make it "one-click fully automatic." Between inputting assets and outputting the final product, there must be a place where I can manually intervene, because LLM-provided edit points will inevitably have some irrationalities, and re-running the entire pipeline incurs API costs again.

That interface is a YAML file:

target_duration_sec: 23
aspect_ratio: '9:16'
clips:
- source: /abs/path/808327978.mp4
  note: Opening shot: Showing the COSCUP x UbuCon Asia main visual backdrop.
  in: '00:00.000'
  out: '00:02.500'
- source: /abs/path/S__1908753.jpg
  note: Fun venue easter egg: Creative semiconductor chip snacks distributed on-site.
  duration_sec: 4.0
transitions: crossfade 0.3s
mood_tags: [Professional, Joyful, Community Cohesion]

Videos use in/out to mark the range, photos use duration_sec for duration, and note is the reason for selection written by Gemini (this field was later used for subtitles, see below). To change an edit point, just change the numbers; to change the order, move the clip; after saving, run poc render.

The outputs of each stage remain in the project directory, so any step can be re-run individually. analyze also skips files that already have analysis results, so re-running doesn't incur double charges—this is very helpful when iterating on prompts.

poc plan --theme was added later: you can provide a sentence as the editing theme, e.g., --theme "Participating in the COSCUP open source community". This affects the narrative angle of the summary, the priority of clip selection, and the wording of each clip's note. Since it only affects the plan stage, changing the theme doesn't require re-analyzing assets, making it very cheap to try different narratives on the same set of materials.

The difference after switching to Gemini 3.7 Flash

The understanding and aggregation stages initially used gemini-2.5-flash, then switched to gemini-3.7-flash. This is the GA stable version, not a preview:

For this project, the most important features are structured outputs and video input, as the analyze stage involves feeding in a video and requesting a JSON with a fixed schema.

After switching, I didn't just change the string and call it a day; I verified it with actual API calls, running analyze_file on real assets. For the same lecture video, the difference in descriptions between the two models was quite noticeable.

gemini-2.5-flash version:

At the start of the video, a woman on stage uses a microphone to introduce herself to the audience. The large screen behind her shows her name "Zona Wang" and her job description.

gemini-3.7-flash version:

In the video, a female speaker (Zona Wang, LINE Technology Evangelist) is giving a self-introduction and presentation on a stage in a lecture hall, followed by a camera pan across the audience listening intently.

The difference lies in "job description" vs. "LINE Technology Evangelist." The latter actually read the small text on the slide, while the former only knew there was some job information there.

The gap in the aggregation stage was even larger. For the same set of COSCUP assets and the same --theme, 2.5's summary was: "This short video aims to showcase the vitality and diversity of the COSCUP open source community. From professional knowledge sharing and deep technical exchange to warm interaction and inclusion among community members"—the whole thing stayed at an abstract level. 3.7 recognized the full event name "COSCUP x UbuCon Asia," booth names like "FOSS for All" and "Kubernetes," and even described a photo as "Creative semiconductor chip snacks distributed on-site." These details weren't in my prompt; they all came from the text and objects in the photos.

For an application where "asset understanding quality directly determines editing quality," the benefit of switching models was greater than I expected. The editing suggestions improved because it actually understood more, not because the prompt was written better.

By the way, 3.7's note style also changed to a "Short Label: Detailed Description" format. This change later broke all my subtitles, as discussed below.

Background music: Lyria 3 uses a different API

Background music is generated using Lyria 3. There are two models: lyria-3-clip-preview for 30-second clips, and lyria-3-pro-preview for full songs. My output is about 20 seconds, so the clip version is perfect.

It doesn't require a separate Vertex AI application or allowlisting; the same Gemini API key works. However, the calling method is completely different from generate_content, using client.interactions.create():

interaction = client.interactions.create(
    model="lyria-3-clip-preview",
    input="An instrumental background music track for a short social-media video, "
          "about 20 seconds long. Mood: Professional, Joyful, Community Cohesion, Happy. "
          "No vocals, no lyrics, loopable.",
)
audio_bytes = base64.b64decode(interaction.output_audio.data)

Several things were different from what I imagined.

It has no structured parameters. Length, BPM, genre, and mood must all be written in the natural language prompt, rather than passing a field like bpm=120. So the generate_score(mood_tags, duration_sec) function's job is actually to concatenate mood tags and seconds into an English sentence. Mood tags are aggregated from asset analysis results during the plan stage, and poc render --mood "Happy, Joyful, Celebration" can further overlay desired directions.

It is single-turn generation and cannot be iteratively modified. Unlike Omni Flash's video editing, once the music is generated, it's set; if you're not satisfied, you have to submit a new prompt. All generated audio includes a SynthID watermark.

When the music is shorter than the video, you have to handle it yourself. The clip version is max 30 seconds, but the video might be longer. So during mixing, I use -stream_loop -1 to loop the audio infinitely and -shortest to trim it to the video length:

cmd.extend(["-stream_loop", "-1", "-i", str(audio_path)])
# ... filter_complex, map video ...
cmd.extend(["-map", f"{audio_index}:a", "-c:a", "aac", "-b:a", "128k", "-shortest"])

Music generation failure (quota, network, safety filters) won't crash the entire render; it prints a warning and falls back to silent output. This principle was later added to the project's CLAUDE.md: any value-added feature calling an external generative API must degrade gracefully and not let the main process die because of a secondary feature.

ffmpeg will silently fail your edits

The render stage uses ffmpeg's xfade filter to connect clips. Each xfade requires an offset parameter, which is "at which second in the output timeline to start this transition." The logic for accumulation is: the sum of all previous clip lengths minus the seconds overlapped by each transition.

After writing the first version, unit tests were all green, and real assets produced normal videos. Then review identified two scenarios where ffmpeg returns exit code 0, but the output file is wrong.

Scenario one: The transition is longer than the clip, causing the clip to be silently swallowed. For two 1-second clips with transitions: "crossfade 2s", the calculated offset is -1.000. ffmpeg accepts this negative number, doesn't report an error, and finishes normally. The output is a 1-second video containing only the first clip; the second one disappears entirely. Since EDL.transitions is a free-text field, it's entirely possible for me to type 3s instead of 0.3s when manually editing the YAML, and it won't tell me in any way.

Scenario two: out exceeds the actual asset length, causing everything following it to be truncated. For a 10-second video, if the EDL says in: 8.0 / out: 15.0, only 2 seconds can actually be taken. If a 1.5-second photo follows, the offset is calculated as 6.700, which falls after the end of the first stream. The result is a 2-second output where the photo is completely missing, and the exit code is still 0. This scenario is even more important to prevent because the EDL is generated by an LLM, and hallucinating an out-of-bounds end time is quite natural.

I added explicit checks for both: if a negative offset is calculated, a ValueError is thrown specifying which clip and transition length; before rendering, ffprobe is used to read the actual length of each video asset, and if out exceeds it, an error is reported clearly stating the requested vs. actual duration.

I care so much because a "successful" but incorrect output is much worse than a crash. If it crashes, I know to fix it immediately. With exit code 0 and a seemingly normal mp4, I might not notice until I watch the whole video and think "wait, a segment is missing," and then have no idea where to start investigating.

Subtitles: Two issues only visible after burning them in

The source for subtitles is the note for each clip in the EDL—the editing reason written by Gemini. Since it already wrote a description for each segment, using it as an on-screen title is perfect.

The implementation doesn't use drawtext; instead, it generates an SRT file and burns it in using libass's subtitles filter. The reason is that drawtext requires manual handling of Chinese font paths and escaping characters; colons, commas, and single quotes all clash with filtergraph syntax. SRT with force_style is much cleaner, and specifying FontName=Noto Sans TC lets fontconfig find the Chinese font.

The first issue was two subtitles appearing on screen simultaneously. In the first version, each subtitle's display interval was just the clip's own start and end times. But with a 0.3s crossfade overlap between adjacent clips, those 0.3 seconds would have two lines of white text on a black background stacked together, which looked ugly. The fix was to change each subtitle's end time to "when the next clip starts" rather than its own end time, ensuring at most one subtitle is visible at any moment. Unit tests couldn't catch this because the SRT was perfectly valid and ffmpeg burned it successfully; I only found it by looking at the frames.

The second issue was subtitles all trailing with an ellipsis. The note is a full sentence description, which would fill the screen if burned directly, so it's truncated into a short title: cut at the first comma or period, or use a character limit if no punctuation is found, adding "..." if truncated.

After switching to Gemini 3.7 Flash, this rule fell apart. 3.7 tends to write notes in a "Opening shot: Showing the 2024 COSCUP x UbuCon Asia main visual backdrop" format—a "Short Label: Detailed Description" style. Since colons weren't in my sentence-breaking character list, the whole sentence fell into the character-limit truncation path, and all eight subtitles ended with "...".

Hard truncation had a second flaw: it ignored word boundaries. "Presenting the female speaker sharing presentation content about ChatGPT and Antigravity" cut at the 20th character resulted in "...and An...", a halved English word.

I fixed both: colons are now treated as label separators, and the label itself is used as the full title without an ellipsis; when hard truncation is necessary, if the cut point falls in the middle of a continuous string of English letters/numbers, it backtracks to before that string started, discarding the whole segment rather than cutting it in half. I also relaxed the character limit from 20 to 24.

After re-burning, the eight subtitles became clean short titles like "Opening Shot," "Session Hall Live," "Technical Sharing Close-up," "Venue Easter Egg," and "Community Booth Interaction," without a single ellipsis.

Other pitfalls

files.upload() returning doesn't mean the file is ready. This was caught by digging into the SDK source code during review, and it would crash on the real API while never showing up in tests. client.files.upload() returns as soon as the bytes are transferred, without waiting for server-side processing. After a video is uploaded, it stays in a PROCESSING state for several seconds; trying to use it for generate_content during this time results in a 400 FAILED_PRECONDITION.

Worse, my original retry loop made things worse: analyze_file was wrapped in a retry, so each retry re-uploaded the entire video and immediately failed again, with only about 3 seconds of backoff across three tries. After three tries, the asset went into _errors.json, and the plan stage didn't read that file at the time, so the asset silently disappeared from the final product. The fix was adding a wait_for_active(), polling client.files.get() after upload until the state is ACTIVE before proceeding, and moving the upload out of the retry loop.

_errors.json was written but not read. As mentioned, analyze diligently recorded failed assets, but plan didn't read them, and summary.md wouldn't mention them. The only way a user could notice was by counting the segments in the final product. Now plan attaches the failure list to the end of the summary, explicitly stating which assets were not included.

Re-running ingest can bite you with old analysis results. This was encountered during actual use, not review. I changed the contents of the asset folder, adding new photos and deleting old ones, then re-ran poc ingest. catalog.json was updated, but analysis results for deleted files were still sitting in analysis/. When plan read the analysis results, it didn't cross-reference them with the current catalog, so it fed outdated assets to Gemini. The model reasonably picked a segment from them, but since the file no longer existed, the whole plan failed. Now load_analyses() filters by the catalog and prints which outdated records are ignored.

Timestamp precision. format_timestamp initially used :04.1f, keeping only one decimal place. Every time an EDL went in and out of YAML, it lost up to 0.05 seconds, which is about 1.5 frames at 30fps, causing edit points to drift. I changed it to :06.3f to keep millisecond precision.

Looking back, these problems fall into two categories. files.upload and ffmpeg silent errors were caught by reading the code line-by-line during review. Subtitle overlapping, ellipsis issues, and stale analysis results only surfaced by actually running the code, playing the videos, and trying different sets of assets. When the tests were all green, those three issues were still lurking in the code.

Conclusion

What ReelCraft does now is simple: a folder of videos and photos goes in, and a 9:16 short video with music and subtitles comes out, with a YAML file in the middle that I can manually edit.

Architecturally, what really makes this work is the "per-file understanding, text aggregation" split. It was conceived to bypass Omni Flash's lack of multi-video reasoning, but it ended up solving timestamp precision and asset count limits as well. After switching to Gemini 3.7 Flash, the granularity of asset understanding significantly increased, and the editing suggestions improved accordingly—the gains here were greater than what I got from tuning prompts.

Two areas remain untouched: Omni Flash's single-clip generative touch-up has an empty touch_up_clip interface, and subtitles are currently derived automatically from note, with the text_overlays field still empty. Neither music nor subtitles are cached; they are re-generated every time render is run.

Reference Links:

• kkdai/reelcraft
• Gemini 3.7 Flash Model Documentation
• Gemini Omni Flash (Video Generation and Editing)
• Gemini Video Understanding
• Lyria Music Generation
• ffmpeg xfade filter
• FFmpeg subtitles filter and libass