Text to Video

Week 5 · SLOP8760

Phase 2 of 4 · Four Generators

This week adds

Text to video: motion generated from words alone, with the prompt as the only thing anchoring the shot.

The native clip length is a published number

It is not a style choice and not a limit to argue with. It is in the config file, and the prompt is written inside it.

ModelNative clip
Wan2.1 shared config81 frames at 16 fps — 5.06 s
Wan2.2 TI2V-5B121 frames at 24 fps
MiniMax H34 to 15 s, at 24 fps

Source: Wan configs, shared_config.py and wan_ti2v_5B.py — https://github.com/Wan-Video/Wan2.2
Source: MiniMax-H3 model card, Output — https://huggingface.co/MiniMaxAI/MiniMax-H3

spacetime patches which act as transformer tokens

OpenAI, Video generation models as world simulators, 2024

Source: Video generation models as world simulators — https://openai.com/index/video-generation-models-as-world-simulators/

The prompt has fields, not paragraphs

The recorded ladder’s middle tier is written into three: integrated_multimodal_description carries visuals, actions, shots, speakers and dialogue along the timeline; overall_soundscape takes one to four English sentences; non_diegetic_music is the music the characters cannot hear.

Source: MiniMax H3 Video Prompt Writing Guide — https://huggingface.co/MiniMaxAI/MiniMax-H3/raw/main/docs/VIDEO_PROMPT_WRITING_GUIDE_base_en.md

The four sentences

Subject and action. Shot size. What the camera is doing. What the light is doing. In that order, inside the clip length.

Reading a bad take

Wrong physics, a quiet change of subject partway through, or a camera move that contradicts the prompt. Each points at a different sentence.

Recorded text to video ladder, tier t5: contradictory prompt (counter-example)
Ladder t5 · contradictory prompt (counter-example)

The ladder

  1. One line naming only the subject
  2. Plus the subject’s action
  3. The full four sentences — subject, shot size, camera, light
  4. Plus a specific beat the action turns on, and a negative prompt

And a counter-example, not a fifth rung: the same four sentences buried under contradictory camera and style demands, generated once on purpose.

Before class

Pick one action you want to see performed and write it as a single sentence, with no camera or light information yet.

This week’s exercise

Build the ladder above for that action: one line, subject and action, the full four sentences, and the four sentences with a beat and a negative prompt. Generate the counter-example too, on purpose. Bring all five clips, in ladder order with the counter-example marked, to Wednesday’s Dailies.

Reading