Build a long-video to vertical-clips pipeline: metadata in, ranked 9:16 clips out

TL;DR

We'll upload a long recording with AI metadata flags on, catch the webhook, write a ranking function that picks clip boundaries from chapters + scene changes + transcript, and cut vertical clips. The ranking function is the part that matters; everything else is plumbing.

Here is the naive version of this project:

ffmpeg -ss 00:14:22 -to 00:14:52 -i webinar.mp4 -c copy clip.mp4

That's the cutting solved. Now watch the output: it starts four words into a sentence, opens on a hard cut to a slide, and ends mid-word. Six of those and you've built something nobody will use.

The actual problem is choosing 00:14:22. Let's build that. You'll need node 20.x or newer and ffmpeg 7.x or 8.x.

1. 🎯 What a good clip boundary actually requires

Watch a human do this and they're checking three things:

1. Does it start on a complete thought? Needs word-level transcript timestamps so you can snap to a sentence start.
2. Does it start on a shot boundary? Needs scene-change detection so the clip doesn't open mid-cut.
3. Does the subject stay in frame? Needs to survive a 9:16 crop, which a two-person shot won't.

Three signals. Assemble them from three tools and you get three different clocks: Whisper timestamps drift against container PTS, PySceneDetect reports frame indices you convert with a frame rate that may be variable, and a tracker samples on its own interval. Joining them is where the week goes.

So the design decision up front is: do you own the extraction, or get it pre-joined?

2. Get the metadata (the flag-on-upload version)

I've been using FastPix for this because the AI capabilities are boolean flags on the upload call rather than a second pipeline you orchestrate. Same idea works with any provider that returns time-aligned metadata; the shape below is what to look for.

curl -X POST 'https://api.fastpix.com/v1/on-demand' \
  --user "$FASTPIX_TOKEN_ID:$FASTPIX_SECRET" \
  -H 'Content-Type: application/json' \
  -d '{
    "inputs": [
      { "type": "video", "url": "https://cdn.example.com/webinar.mp4" }
    ],
    "chapters": true,
    "accessPolicy": "private",
    "maxResolution": "1080p"
  }'

{
  "success": true,
  "data": {
    "id": "a1d1acdd-8f4e-4add-b498-6b398cf349d9",
    "status": "Created",
    "createdAt": "2026-08-17T10:50:34.594302Z",
    "playbackIds": [
      { "id": "6ta85f64-5717-4562-b3fc-2c963f66afa6", "accessPolicy": "private" }
    ],
    "maxResolution": "1080p",
    "mediaQuality": "standard"
  }
}

Auth is Basic: Access Token ID as username, Secret Key as password. Playback is https://stream.fastpix.com/<playbackId>.m3u8, with a JWT appended as ?token= for private assets.

⚠️ data is an object, not an array. If you're porting from an older integration that did data[0].id, that's your first bug.

💡 Tip: chapters is one flag in a family. summary, namedEntities, moderation and subtitles are siblings on the same create call, and because they run inside the same processing pass they land referenced to the same timeline. That property is why I stopped maintaining my own extraction stack.

⚠️ The API field is chapters, but the dashboard's custom media settings JSON calls it generateChapters. Two names for one feature.

Same settings for a direct (push) upload, nested under pushMediaSettings:

{
  "corsOrigin": "*",
  "pushMediaSettings": {
    "chapters": true,
    "accessPolicy": "private",
    "maxResolution": "1080p",
    "metadata": { "source": "weekly-webinar" }
  }
}

And for assets you uploaded before you thought of this, it's a PATCH rather than a re-upload:

curl -X PATCH "https://api.fastpix.com/v1/on-demand/$MEDIA_ID/chapters" \
  --user "$FASTPIX_TOKEN_ID:$FASTPIX_SECRET" \
  -H 'Content-Type: application/json' \
  -d '{ "chapters": true }'

3. 🛠️ Catch the webhook

Results arrive asynchronously. Handle the lifecycle events and the AI event separately, because they fire at different times.

// webhook.js
import express from 'express';

const app = express();
app.use(express.json());

app.post('/webhooks/video', async (req, res) => {
  // ack fast, work later
  res.sendStatus(200);

  const { type, data, object } = req.body;

  switch (type) {
    case 'video.media.created':
      await db.assets.upsert({ id: object.id, status: 'processing' });
      break;

    case 'video.media.ready':
      await db.assets.update(object.id, { status: 'ready' });
      break;

    case 'video.media.failed':
      await db.assets.update(object.id, { status: 'failed' });
      break;

    case 'video.mediaAI.chapters.ready':
      await onChapters(object.id, data.chapters);
      break;
  }
});

app.listen(3000);

The chapters payload looks like this:

{
  "type": "video.mediaAI.chapters.ready",
  "object": { "type": "media", "id": "f081fd53-6a9a-43ae-9d64-9974ef243dbd" },
  "id": "51a61b56-0197-4127-8a61-472e4d3fa59a",
  "workspace": { "name": "clips-pipeline", "id": "f7a13f50-7f5c-48f4-b7b2-c901dcff61c6" },
  "status": "ready",
  "data": {
    "isChaptersGenerated": true,
    "chapters": [
      {
        "chapter": "1",
        "startTime": "00:00:00",
        "endTime": "00:03:59",
        "title": "The Circle Challenge Begins",
        "summary": "Contestants start stacking items in a circle for a chance to win."
      }
    ]
  },
  "createdAt": "2026-08-17T11:52:29.526588692Z",
  "attempts": []
}

Three field names worth pinning down, because they are easy to guess wrong: the sequence number is chapter and it is a string, the prose field is summary (not description), and object.type is media even though the event is a mediaAI event.

⚠️ Times come back as hh:mm:ss strings. Convert once, at the boundary, and keep seconds internally. Mixing string times and float seconds in the same codebase is a bug generator.

const toSeconds = (hms) => {
  const [h, m, s] = hms.split(':').map(Number);
  return h * 3600 + m * 60 + s;
};

4. Write the ranking function (this is the product)

Providers will rank clips for you. FastPix's AI clipping produces ranked short clips scored on hook, pacing and narrative, and it's a fine candidate generator. But "which 30 seconds are worth posting" depends on your audience, not on the video, so treat vendor ranking as candidate generation and put your own opinion on top.

// rank.js
const MIN_LEN = 15;
const MAX_LEN = 60;
const SHOT_TOLERANCE = 0.4; // seconds

export function rankCandidates({ chapters, scenes, words }) {
  const candidates = [];

  for (const ch of chapters) {
    const start = toSeconds(ch.startTime);
    const end = toSeconds(ch.endTime);

    // slide a window through the chapter, snapping to sentence starts
    for (const w of words) {
      if (w.start < start || w.start > end) continue;
      if (!w.isSentenceStart) continue;

      const closeWord = lastWordBefore(words, w.start + MAX_LEN);
      const length = closeWord.end - w.start;
      if (length < MIN_LEN) continue;

      candidates.push({
        start: w.start,
        end: closeWord.end,
        chapter: ch.title,
        score: score({ start: w.start, end: closeWord.end, words, scenes }),
      });
    }
  }

  return dedupeOverlapping(candidates.sort((a, b) => b.score - a.score));
}

function score({ start, end, words, scenes }) {
  const inWindow = words.filter((w) => w.start >= start && w.end <= end);
  if (!inWindow.length) return 0;

  const endsClean = inWindow.at(-1).endsSentence ? 1 : 0;
  const shotSafe = scenes.some((s) => Math.abs(s - start) < SHOT_TOLERANCE) ? 1 : 0;

  // words per second: dead air is not a clip, but neither is a firehose
  const density = inWindow.length / (end - start);
  const densityScore = density < 1.2 ? 0 : Math.min(density / 3, 1.5);

  // single-speaker stretches survive a vertical crop; crosstalk does not
  const speakers = new Set(inWindow.map((w) => w.speaker));
  const speakerScore = speakers.size === 1 ? 1 : 0;

  return 2 + endsClean + shotSafe + densityScore + speakerScore;
}

The base 2 is for starting on a sentence, a hard filter rather than a score. Everything else is tunable. Tune it by watching output, not by reasoning about it. I got the density floor wrong twice before sitting down and watching twenty rejected candidates.

Doing your own scene detection instead? FFmpeg gives you boundaries directly:

ffmpeg -i webinar.mp4 -filter:v "select='gt(scene,0.4)',showinfo" \
  -f null - 2>&1 | grep showinfo | sed -n 's/.*pts_time:\([0-9.]*\).*/\1/p'

14.220
62.480
119.100
187.760

💡 The 0.4 threshold is a starting point, not a constant. Talking-head footage with a static camera needs it lower; a heavily-cut promo needs it higher.

5. Cut and reframe

Cutting is the easy half, but there's one gotcha:

# fast, no re-encode, but snaps to the nearest keyframe
ffmpeg -ss 14.22 -to 44.51 -i clip-source.mp4 -c copy clip.mp4

# frame-accurate: -ss after -i, and you re-encode
ffmpeg -i clip-source.mp4 -ss 14.22 -to 44.51 \
  -c:v libx264 -preset veryfast -crf 20 -c:a aac -b:a 128k clip.mp4

Stream copy is fast and wrong for this job. Your ranking function worked hard to land on a sentence start, and -c copy will slide that to the nearest keyframe, which is up to a GOP away. Re-encode.

Now the crop. Here's the version everybody writes first:

ffmpeg -i clip.mp4 -vf "crop=ih*9/16:ih,scale=1080:1920" -c:a copy vertical.mp4

That's a centre crop, and it's correct exactly when your subject is centred. On a two-person interview it produces a beautiful vertical clip of the gap between them. On a stage recording where the speaker walks, it produces a lectern.

Subject-tracking reframe is the fix, and it's real work to build: detect the subject per frame, smooth the crop path so it doesn't jitter, reset on cuts rather than panning across them. FastPix's auto-reframe does 16:9 to 9:16, 1:1 or 4:5 with subject tracking rather than centre-cropping, which is what made the output shippable instead of something I had to eyeball every time.

Know which one you need:

6. What it costs you to run

Billing is per minute, encoding is free on the standard plan (you pay for delivery, storage and add-ons), and signup gives you $25 in credits with no credit card for the free tier. Early-stage teams can get $600 through the Startup Program (under four years old, under $10M raised), which covers running this over a real back catalogue while you decide.

One honest caveat: this is API-first, not a no-code CMS. If what your team actually wants is a timeline UI where a marketer picks moments by hand, you're building that front end. That was fine here because removing the human from the loop was the point.

What's next

• Build the ranking function before anything else. Watch twenty clips a human picked, write down why each works, and turn that into the score. Your criteria will be almost entirely about sentence and shot boundaries.
• Add burned-in captions. You already have word-level timestamps, so this is a subtitles filter away, and it's what drives silent-autoplay watch time.
• Feed the same metadata into search and chapter navigation on the long-form asset. The extraction is already paid for.

The documented AI flags (chapters, summary, namedEntities, moderation, subtitles) and their webhook payloads are in the In-Video AI docs; the clipping and reframe capabilities are on the product side, so check the current reference for their parameter names before you wire them in. The upload-and-webhook pattern here transfers to any managed video API, so that integration work isn't wasted if you switch. What varies a lot between providers is the AI capability set itself, so check yours against your actual list before you commit.