Week 6: Image to Video

The fourth generator — keyframe-first control, locking a first and last frame, and the difference between directing motion and losing identity.

29 March 2027

Image to video reverses the order of the last two weeks: rather than describing a subject in words and hoping the camera cooperates, a finished keyframe locks composition, framing, and art direction first, and the prompt’s only remaining job is to say what happens next. This is the last of the four generators, and the strongest control the semester offers over a single shot — which is exactly why Phase 2 ends here.

Keyframe-first control

t3finished key frame

Starting from a keyframe fixes everything text to video leaves loose: who’s in frame, how they’re dressed, where the light falls, what the background looks like. The motion prompt that follows only has to describe change — what moves, and how — because everything that isn’t supposed to change is already locked in the frame the model was given.

First and last frame

t5first frame + last frame (FL2VA)

Some workflows accept two keyframes instead of one: a starting frame and an ending frame, with the model generating everything in between. That constrains motion far more tightly than a first frame and a text prompt alone, at the cost of needing a second finished keyframe to already exist before the shot can be generated — a trade worth making when the ending pose matters as much as the starting one.

Motion versus identity

t4finished key frame + four sentences

The risk specific to this generator is a shot that drifts identity while chasing motion: a face or costume detail changing partway through the clip because the motion prompt asked for more movement than the model could deliver while holding the keyframe’s details steady. When a take drifts, the fix is almost never a stronger motion prompt — it’s a calmer one, asking for less change per second so identity has room to hold.

The ladder

The ladder for image to video runs five tiers, climbing through how finished the starting keyframe is and how much the motion prompt is asked to do. The lowest tier starts from a rough sketch. The next tier starts from a flat, unlit keyframe. The next tier starts from a finished keyframe with lighting resolved. The next tier keeps that finished keyframe and adds a motion prompt written as the four sentences from earlier in the phase. The top tier keeps all of that and adds either a last frame or a small reference set to hold identity through more motion than a single keyframe alone can guarantee.

sketch as first frame

A line sketch as the first frame: watch the model paint the whole look from nothing.

flat colour as first frame

Flat colour fixes the palette and the silhouette; the face is still up for grabs.

finished key frame

A finished frame holds the identity - compare the face here with t1 and t2.

finished key frame + four sentences

Same first frame as t3, richer prompt: the picture holds the look, the words hold the motion.

first frame + last frame (FL2VA)

Two anchors instead of one: watch the turn land exactly on the tail frame at 5.17 s.

Recorded on the convener's rig with MiniMax H3 (local weights), keyframes with NoobAI-XL, reference stills for The Ninth Head with ChatGPT's image generator. Nothing is generated in the browser. Run it in the Studio.

Before class

Bring one finished keyframe from earlier in the phase, and one rougher version of it to use as this week’s lowest ladder tier.

This week’s exercise

Build all five ladder tiers on the same subject, ending with either a first-and-last-frame pass or a reference-set-backed one. Bring all five clips, in ladder order, and a one-line note on where identity held or drifted, to Wednesday’s Dailies.

Reading

Teaching team