Image to Image

Week 4 · SLOP8760

Phase 2 of 4 · Four Generators

This week adds

Image to image: pushing something you already have toward a finished keyframe, with one dial deciding how much of it survives.

One number decides it

Input to tier t3: In: flat-coloured lineart — In: flat-coloured lineart

One prompt and one strength across all five tiers. At denoise 1 the docs say the latent becomes completely random noise, losing all characteristics of the reference image.

Source: ComfyUI docs, image-to-image — https://docs.comfy.org/tutorials/basic/image-to-image

Denoising strength runs from barely touching the input to nearly regenerating from scratch, and the useful range for turning a rough input into a finished keyframe sits well below the top of that scale — high enough to fix proportions and clean up linework, low enough that the composition the input already got right survives the pass. Pushing the number up because a result looks unfinished usually erases the one thing worth keeping.

The tutorial’s own sentence is worth reading twice: “If denoise is 1, the latent space image will become completely random noise … losing all characteristics of the reference image.” The dial does not blend the input with the prompt. It decides how much of the input is still there when the sampler starts — which is why the pair on this slide holds its composition while everything on its surface is repainted.

A pose signal is a list of coordinates

25body keypoints, plus 21 per hand and 70 on the face

ControlNet does not read your picture. It reads a signal extracted from it, and a pose signal is numbers: x0,y0,c0,x1,y1,c1,..., starting at the nose and the neck.

Source: OpenPose output documentation — https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/02_output.md

first adds noise to the input, then subsequently denoises the resulting image
Meng et al., SDEdit, 2021

Source: SDEdit — https://arxiv.org/abs/2108.01073

Structure and identity

Input to tier t5: In: the pose signal alone — In: the pose signal alone

ControlNet holds one extracted signal fixed — edges, depth, pose — through what it calls zero convolutions. IP-Adapter does the other job: identity carried from a reference, in 22M parameters.

Source: ControlNet — https://arxiv.org/abs/2302.05543
Source: IP-Adapter — https://arxiv.org/abs/2308.06721

The picture on the left is the whole of what the generator was given for structure on the top ladder tier: a skeleton on black, no face, no costume, no light. Everything else in the frame beside it came from the prompt and the reference. That is the difference between “keep this composition” and “keep this exact picture.”

IP-Adapter does the opposite job: it carries identity or style forward from a reference image into the new one, independent of the pose or depth signal entirely, through what the paper calls a decoupled cross-attention mechanism. Used together, a pose comes from one input and a face or style anchor comes from another, and the generator reconciles both.

Three numbers worth knowing before you touch the node: ControlNetApplyAdvanced defaults to strength 1.0, start_percent 0.0 and end_percent 1.0, stepping in thousandths — so the control can be released partway through the sampling run rather than held for all of it. The ControlNet paper also reports the method staying “robust with small (<50k) and large (>1m) datasets”, which is why so many community control models exist.

The ladder

One prompt, one strength, five inputs.

A rough sketch
Clean lineart
Flat-coloured lineart
An already finished keyframe, refined rather than built
That same keyframe plus a ControlNet pose signal locking the composition

The ladder this week holds the prompt and the denoising strength fixed and climbs by improving the input image instead — five tiers, one prompt, one strength, across all five. The lowest tier starts from a rough sketch. The next tier starts from clean lineart instead of a rough one. The next tier starts from a flat-coloured version of that lineart. The next tier starts from an already finished keyframe, refined rather than built. The top tier starts from that same finished keyframe plus a ControlNet pose signal locking its composition in place.

A production turnaround is the distance between a rough input and a keyframe good enough to hand to the next stage, and this course measures it the same way a studio does: not by how good the first pass looks, but by how few passes it took to get somewhere usable. A short turnaround built on a clean sketch and one controlled pass beats a long one spent fighting a vague input with prompt changes alone.

Before class

Bring one keyframe from Phase 2 so far, and sketch three rougher versions of it at decreasing levels of finish, to use as this week’s ladder of inputs.

This week’s exercise

Run the same prompt and denoising strength against your four rough inputs plus one ControlNet-locked pass, producing all five ladder tiers. Bring all five outputs, their shared prompt, and the strength value used to Wednesday’s Dailies.

Reading

SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
Meng et al.2021
first adds noise to the input, then subsequently denoises the resulting image
Adding Conditional Control to Text-to-Image Diffusion Models
Zhang, Rao & Agrawala2023
robust with small (<50k) and large (>1m) datasets
IP-Adapter: Text Compatible Image Prompt Adapter
Ye et al.2023
decoupled cross-attention mechanism
ComfyUI: image to image
Comfy Org
the latent space image will become completely random noise

Image to Image#

Week 4 · SLOP8760#

This week adds#

One number decides it#

A pose signal is a list of coordinates#

Structure and identity#

The ladder#

Before class#

This week’s exercise#

Reading#