Image to Video

Week 6 · SLOP8760

Phase 2 of 4 · Four Generators

This week adds

Image to video: a finished keyframe locks the shot, and the prompt’s only remaining job is to say what happens next.

Keyframe-first control

The keyframe fixes who is in frame, how they are dressed, where the light falls. The motion prompt only has to describe change.

Recorded image to video ladder, tier t3: finished key frame — Ladder t3 · finished key frame

One frame in, a fixed number of frames out

25frames at 576 x 1024, from one context frame

Stable Video Diffusion’s published shape for a single still. The base model generates fourteen; the extended one generates twenty-five, at the same resolution as the frame it was given.

Source: stable-video-diffusion-img2vid-xt model card — https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt

text-to-image pretraining, video pretraining, and high-quality video finetuning
Blattmann et al., Stable Video Diffusion, 2023

Source: Stable Video Diffusion — https://arxiv.org/abs/2311.15127

First and last frame

Input to tier t5: Picture 1, at 0.00 s — Picture 1, at 0.00 s

Input to tier t5: Picture 2, at 5.17 s — Picture 2, at 5.17 s

Two frames constrain motion far more tightly than one frame and a prompt. The node takes start_image, end_image and a length defaulting to 81; H3’s FL2VA accepts zero, one, or two images.

Source: ComfyUI WanFirstLastFrameToVideo — https://docs.comfy.org/built-in-nodes/WanFirstLastFrameToVideo
Source: MiniMax-H3 card — https://huggingface.co/MiniMaxAI/MiniMax-H3

Motion versus identity

When a take drifts, the fix is a calmer motion prompt, not a stronger one. VBench scores the two separately: Subject Consistency from DINO feature similarity across frames, Background Consistency from CLIP feature similarity, Temporal Flickering from mean absolute difference.

Source: VBench — https://arxiv.org/abs/2311.17982

The risk specific to this generator is a shot that drifts identity while chasing motion: a face or costume detail changing partway through the clip because the motion prompt asked for more movement than the model could deliver while holding the keyframe’s details steady. When a take drifts, the fix is almost never a stronger motion prompt — it’s a calmer one, asking for less change per second so identity has room to hold.

VBench is useful here less as a leaderboard than as a vocabulary: it splits what a viewer calls “it looks wrong” into sixteen dimensions, and the four that matter to this week are the three named on the slide plus Motion Smoothness. Dynamic Degree sits opposite them — a clip that holds perfectly still scores well on consistency and badly on movement, which is the trade this slide is about.

The ladder

A rough sketch as the starting frame
A flat, unlit keyframe
A finished keyframe with lighting resolved
That keyframe plus a motion prompt written as the four sentences
Plus a last frame, or a small reference set, to hold identity

Before class

Bring one finished keyframe from earlier in the phase, and one rougher version of it to use as this week’s lowest ladder tier.

This week’s exercise

Build all five ladder tiers on the same subject, ending with either a first-and-last-frame pass or a reference-set-backed one. Bring all five clips, in ladder order, and a one-line note on where identity held or drifted, to Wednesday’s Dailies.

Reading

Stable Video Diffusion
Blattmann et al.2023
text-to-image pretraining, video pretraining, and high-quality video finetuning
Wan2.1
Wan-Video2025
Currently, only 720P is supported
MiniMax-H3 model card
MiniMaxAI2026
maximum number of files across all input types is 12
VBench: Comprehensive Benchmark Suite for Video Generative Models
Huang et al.2023
DINO feature similarity across frames

Image to Video#

Week 6 · SLOP8760#

This week adds#

Keyframe-first control#

One frame in, a fixed number of frames out#

First and last frame#

Motion versus identity#

The ladder#

Before class#

This week’s exercise#

Reading#