This is part two of a four-part series on media editing for live performance. Start with part one.
In part one we set up the project, talked about what "performance-ready" actually means, and introduced the four scripts. Now we need to find the moments worth keeping.
From inside the show folder, the first job is to give the detector something sensible to chew on.
Make a mezzanine if the source is awkward
If your source is already a sensible H.264 MP4, leave it alone. scene-cut re-encodes on the way out anyway, so there's no point converting it twice. If it's a variable-frame-rate phone clip, a bizarre codec, or an interlaced capture, make a mezzanine first. For interlaced tape, add yadif at the start of the filter chain.
ffmpeg -i 00-source/source.mov \
-vf "fps=30,format=yuv420p" \
-c:v libx264 -crf 18 -preset slow \
-c:a aac -b:a 192k -movflags +faststart \
00-source/mezzanine.mp4
Then use the mezzanine for everything else. It's a bigger file than you need and that's fine; it's the version you'll be asking ffmpeg to seek around in hundreds of times. The examples below assume the working file is 00-source/source.mp4; swap in mezzanine.mp4 if you made one.
Point the detector at it
scene-detect -i 00-source/source.mp4 -t 0.25 -o 01-scenes/scenes.txt
The script compares consecutive frames, gives each frame a score between 0 and 1 for how different it is from the one before, and writes out the timestamps where that score crosses the threshold. The output is one timestamp per line, starting at zero and ending at the duration of the source: 00:00:00, 00:00:11.875, 00:00:15.750, and so on.
The threshold is the whole argument. Lower it and you get more cuts; raise it and you get fewer.
| Threshold | What it tends to catch | Good starting point for |
|---|---|---|
| 0.10–0.15 | Almost every change, including camera movement | Fast-cut montage, animation, glitch material |
| 0.20–0.30 | Clear cuts, some movement | A first pass on edited footage |
| 0.35–0.50 | Only big changes | Slow films, long takes, fade-heavy material |
These are starting points, not rules. The detector doesn't understand what's in the frame. It only knows that the picture changed. A cut to a title card and a cut to a beautiful landscape score the same. A whip pan can score high enough to look like a cut. A slow dissolve may score lower than both, because there's no single frame where the picture suddenly becomes something else.
If your source is a five-minute shot of a room, the detector will correctly tell you that you have one scene. That's useful information, even if it isn't a clip.
Tune on a slice, not the whole thing
Don't run the detector over two hours while you're still deciding whether 0.25 is right. Pick a representative five minutes and run it there:
scene-detect -s 00:12:00 -e 00:17:00 \
-i 00-source/source.mp4 -t 0.18 \
-o 01-scenes/test.txt
Both -s and -e are required for a range; the script won't accept one on its own. Try a couple of thresholds, compare the number of cut points, and look at a few of the timestamps. Once the slice looks right, run the full thing and go and do something else for a while. Scene detection is a full decode, not a metadata read.
Look before you cut
A list of timestamps is not a library. scene-image grabs the first frame at each timestamp so you can see what the detector actually found. It needs the detection file, not the cutlist.
cd 01-scenes
scene-image -i ../00-source/source.mp4 -c scenes.txt
ffmpeg -y -pattern_type glob -i 'source-*.png' \
-vf "scale=320:180:force_original_aspect_ratio=decrease,pad=320:180:(ow-iw)/2:(oh-ih)/2,tile=5x4" \
-frames:v 1 contact.jpg
open contact.jpg
Match the glob to your input filename. On Linux, use xdg-open contact.jpg. The tile is 5 by 4, so it shows the first twenty thumbnails; adjust the grid or make a second sheet if you've got more.
This is the step that stops the workflow from being a random number generator. You're looking for two things:
- Images that read at a glance. If you can't tell what it is in a thumbnail, you probably can't use it in a dark room with one eye on the crowd.
- Scenes that are long enough to do something with. A one-second flash can be wonderful. Twenty of them in a row is a different kind of show.
Delete the thumbnails you hate; the cutlist doesn't care. Keep the contact sheet as a map of the source.
What you should have now
By the end of this part you should have:
01-scenes/scenes.txt— every cut point the detector found.01-scenes/— a pile of PNGs, one per cut point.01-scenes/contact.jpg— a contact sheet you can scan in a few seconds.
In part three we'll turn those timestamps into clip lengths, cut the clips, and start the real work: culling and naming.
The series: 1. Media Editing · 2. Finding the Cuts · 3. Cutting, Culling and Naming · 4. Performance-Ready Clips