What is Image-to-Video?

Image-to-video models extend the diffusion process used in text-to-video by conditioning on a visual input rather than starting from pure noise. The source image is encoded into a latent representation that serves as the anchor for frame generation. The model then predicts how the scene should evolve over time — objects move, lighting shifts, cameras pan.

An optional text prompt further guides the animation. For example, uploading a photo of a waterfall with the prompt "slow zoom out, mist rising" tells the model both what the scene looks like (from the image) and how it should move (from the text). This dual conditioning produces more controllable results than either input alone.

Temporal coherence is critical. The model uses temporal attention mechanisms to ensure the subject maintains consistent identity, proportions, and lighting across all generated frames. Advanced models like Sora 2 and Wan 2.6 can handle complex motion — a person walking, hair blowing in wind — while keeping the face and clothing stable.

The pipeline typically includes an image encoder (VAE or CLIP vision), a denoising backbone (DiT or U-Net with temporal layers), and a decoder that converts latent frames back to pixel space. Some models add a super-resolution pass for the final output.