Open Source Project #154: claude-video — Give Claude Eyes to Watch Videos, One Command to Analyze YouTube/Loom/Local Files

Introduction

"Claude can read code, analyze images — but couldn't watch a video. Until this."

This is article #154 in the "One Open Source Project a Day" series. Today's project is claude-video — a Claude skill plugin by Brad Bonanno (Solaris Automation) built around the /watch command, giving Claude the ability to actually watch videos and answer questions about their content.

/watch https://youtube.com/watch?v=xxx  What's the core argument in this talk?
/watch bug-repro.mov  What error is happening in this recording?
/watch product-demo.mp4  Extract all features into a list

15,200 Stars. MIT license.

What You'll Learn

• The seven-step video analysis pipeline and its technical design
• Subtitle-first strategy: why this design decision cuts costs dramatically
• The MAD dedup algorithm: how 16×16 thumbnails catch slow fades
• Automatic token budget management: dynamic frame scaling by video duration
• Four detail modes and available parameters
• Installation on Claude Code and other agent hosts

Prerequisites

• Experience with Claude Code or a similar AI coding agent
• Basic video concepts (frames, transcription)
• No Python or ffmpeg knowledge needed

Background: The Problem

Claude supports image input but not video natively. Video files are too large to inject directly into context, and even if they weren't, a pure frame sequence without transcript text loses most of the information.

Existing "make AI watch video" approaches typically do: full download → fixed-fps uniform sampling → all frames to the model. Three problems:

• Wasted tokens: static scenes produce many near-identical frames, each consuming tokens
• Missing audio: no transcription means speech, narration, and dialogue are lost
• Unpredictable costs: a 50-minute video with fixed-fps sampling might produce hundreds of images

claude-video optimizes at every stage.

Seven-Step Pipeline

Video URL / local path
    ↓ [1] yt-dlp attempts subtitle retrieval
    ↓ [2] Subtitles found? Use them, skip download
    ↓ [3] No subtitles? Download video, extract frames via ffmpeg
    ↓ [4] MAD dedup algorithm, scene-change aware
    ↓ [5] Whisper transcription (Groq preferred)
    ↓ [6] Frames + transcript sent to Claude context
    ↓ [7] Claude answers, temp files cleaned up

Key Optimization 1: Subtitle-First, Zero Download

Many video platforms (YouTube especially) have native or CC subtitles. yt-dlp checks for subtitles before downloading the video.

If subtitles are found:

• No video download, no frame extraction, no Whisper call
• Parse the VTT subtitle file, use it directly as the transcript
• Total time drops from ~37 seconds (downloading a 76MB video) to ~4.5 seconds

A 49-minute YouTube lecture in transcript mode: 4.5 seconds, near-zero token cost.

Key Optimization 2: Scene-Aware Frame Extraction

Instead of uniform fps sampling, the tool detects scene changes — frames are only captured when the visual content actually changes.

How it works:

1. ffmpeg outputs a raw frame sequence
2. Each frame shrinks to a 16×16 grayscale thumbnail (standard library only, no Pillow needed)
3. Compute the Mean Absolute Difference (MAD, 0–255) against the last kept frame
4. MAD ≤ 2.0: near-duplicate, discard
5. Frame budget cap applies after deduplication

One design detail worth noting: comparing against the last kept frame, not the previous frame.

This handles slow fades:

Frame A → Frame B → Frame C → Frame D (full scene cut)

A vs B: MAD = 0.5 (similar, B discarded)
A vs C: MAD = 1.0 (still similar, C discarded)
A vs D: MAD = 8.0 (significant change, D kept)

If you compared against the previous frame, each step's delta would be small even through a slow fade — only the final frame would cross the threshold. Comparing against the last kept frame accumulates the difference over time and catches the transition correctly.

Automatic Token Budget Management

Frame count isn't fixed — it scales with video duration:

Token estimate per frame: (width × height) / 750. At default 512px width, each frame is roughly 197 tokens. 100 frames ≈ ~20,000 visual tokens — manageable when combined with the transcript.

Videos over 10 minutes trigger a "sparse scan" warning, suggesting the user use --start/--end to focus on the section they care about.

Four Detail Modes

Controlled via the --detail parameter:

# Summarize a lecture (subtitles only, no frames)
/watch talk.mp4 --detail transcript  Summarize the key arguments

# Quick scan of a demo
/watch demo.mp4 --detail efficient  What features are shown?

# Analyze a bug recording with full frame coverage
/watch bug.mov --detail token-burner  What error occurred and when?

Focus Mode

For long videos, analyze a specific time range — higher frame density, lower cost:

# Focus on 5:30 to 8:00 only
/watch lecture.mp4 --start 5:30 --end 8:00  What's covered in this segment?

# Capture specific timestamps
/watch product.mp4 --timestamps 1:20,3:45,6:10  What's shown at each moment?

# Higher resolution (needed for reading on-screen text)
/watch screen-recording.mov --resolution 1024  Read the code on screen

Supported Video Sources

Via yt-dlp: 50+ platforms including YouTube, Loom, TikTok, X (Twitter), Instagram, Bilibili, Vimeo, and more.

Local files: .mp4, .mov, .mkv, .webm.

Installation

Claude Code (simplest):

/plugin marketplace add bradautomates/claude-video
/plugin install watch@claude-video

Codex / Cursor / GitHub Copilot / Gemini CLI / others:

npx skills add bradautomates/claude-video -g

claude.ai web:
Download the watch.skill file, upload it in Settings → Features → Skills.

First run auto-detects and installs system dependencies:

• macOS: auto-installs yt-dlp and ffmpeg via brew
• Linux: prints the apt / dnf commands to run
• Windows: prints winget commands

API Key Requirements

Without a Groq or OpenAI key, --no-whisper still lets you analyze visual content — you just won't get audio transcription.

Common Use Cases

Competitor analysis:

/watch competitor-demo.mp4  List all features and UI design patterns

Bug diagnosis:

/watch bug-repro.mov  Describe the full sequence of events and what the UI shows

Video notes:

/watch training-video.mp4 --detail balanced  Extract key points with timestamps

Cut through marketing fluff:

/watch product-ad.mp4 --detail transcript  Strip the marketing language, what does it actually do?

Project Links

• GitHub: bradautomates/claude-video
• Author: Brad Bonanno, @bradbonanno
• Company: Solaris Automation

Summary

claude-video's engineering reflects a clear judgment: find the lowest-cost path at each stage rather than brute-forcing the problem.

Subtitle-first reduces the most common case (YouTube videos with subtitles) to near-zero cost. MAD frame deduplication eliminates redundant tokens from static scenes. Dynamic frame budget scaling gives short videos dense coverage without blowing the budget on long ones. These three optimizations stack to make actual token consumption an order of magnitude lower than fixed-fps sampling.

15,200 Stars tells you "let AI watch video" is a real need, not a demo project. The 50+ agent host support shows the author cared about ecosystem compatibility rather than just Claude Code users.

Explore PrimeSkills — A marketplace for handpicked AI Agents and skills. Each is validated in real enterprise workflows, stripping away hype and keeping only what truly works.

Welcome to my Homepage for more useful insights and interesting products.