MSALab/LoomVideo

LoomVideo: Unifying Multimodal Inputs into
Video Generation and Editing

Peking University · Alibaba Group

🔥 News

• [2026-06-05] We release LoomVideo paper on Arxiv!
• [2026-06-02] We release the codebase and model weights of LoomVideo!
• [2026-06-02] We release the project page of LoomVideo!

📌 TL;DR

The Problem: Existing unified video generation & editing models are massive (13B+) and rely on token concatenation for source conditioning — doubling sequence length and quadrupling attention cost.

The Method: We present LoomVideo, a compact 5B-parameter unified architecture built on MLLM + DiT that introduces three key designs:

• Deepstack Injection — extracts features from every MLLM layer and injects them into corresponding DiT layers via cross-attention, enabling rich multi-granular semantic guidance.
• Scale-and-Add Conditioning — a zero-overhead approach that scales the clean source latent by the current timestep and directly adds it to the noised target, completely bypassing token concatenation.
• Negative Temporal RoPE — assigns negative temporal indices to reference images, seamlessly integrating multi-image conditions without architectural modification.

The Result: Our 5B model achieves state-of-the-art or highly competitive performance across comprehensive benchmarks, with at least 5.41× inference speedup over models of similar capabilities — demonstrating that efficiency and quality can coexist.

🎯 Supported Tasks

LoomVideo supports four unified video generation and editing tasks within a single model:

Task	Input	Output	Description
Text-to-Video	Text 📝	Video 🎬	Generate a video from a text prompt
Instruction Editing	Video 🎬 + Text 📝	Video 🎬	Edit a video following text instructions
Instruction-Image Editing	Video 🎬 + Image 🖼 + Text 📝	Video 🎬	Edit a video with a reference image as guidance
Multi-Image-to-Video	Images 🖼 + Text 📝	Video 🎬	Compose multiple reference images into a coherent video

🔧 Preparation

Step 1: Clone the Repository

git clone https://github.com/MSALab-PKU/LoomVideo
cd LoomVideo

Step 2: Install Dependencies

We recommend using uv for a fast and fully reproducible environment setup.

uv sync
source .venv/bin/activate

# (Optional) Include evaluation dependencies
uv sync --extra eval

Additionally, install Flash Attention for faster inference and reduced GPU memory consumption. (for reference, our environment uses v2.7.4)

Step 3: Download Model Weights

Download the pretrained LoomVideo checkpoint from Hugging Face and place it under checkpoints/LoomVideo/:

checkpoints/LoomVideo/
└── gen_model.pth

We provide a helper script to download the weights automatically:

'''bash python hf_download.py '''

You can also specify a custom path via the --ckpt_path argument at inference time.

💡 Stage 3 model weights are now available. Higher-performance post-trained weights will be released as soon as possible!

🎬 Inference

LoomVideo provides a unified inference script that supports four generation tasks through a single entry point. Each task is selected via the --task flag.

1. Text-to-Video / Text-to-Image (t2v)

Generate a video from a text description. Default resolution is 480×832 at 81 frames. When --num_frames is set to 1, the pipeline automatically switches to image generation mode and saves the output as a .jpg file.

Required: --prompt

NUM_GPUS=1

accelerate launch --num_processes=${NUM_GPUS} \
    scripts/inference/generate.py \
    --config_path configs/inference/generation.yaml \
    --ckpt_path checkpoints/LoomVideo \
    --task t2v \
    --prompt "Your prompt here" \
    --height 480 \
    --width 832 \
    --num_frames 97 \
    --num_inference_steps 50 \
    --seed 0 \
    --output_path outputs/t2v.mp4

2. Instruction Editing (edit)

Edit an existing image or video based on a text instruction. The source can be either an image file (.jpg, .png, etc.) or a video file (.mp4). Resolution and frame count are automatically inferred from the source when not specified.

Required: --prompt --source_video_path

NUM_GPUS=1

accelerate launch --num_processes=${NUM_GPUS} \
    scripts/inference/generate.py \
    --config_path configs/inference/generation.yaml \
    --ckpt_path checkpoints/LoomVideo \
    --task edit \
    --prompt "Your editing instruction here" \
    --source_video_path /path/to/source_video.mp4 \
    --num_inference_steps 50 \
    --seed 0 \
    --output_path outputs/edit.mp4

3. Instruction-Image Editing (ref_edit)

Edit a source video with guidance from one or more reference images along with a text instruction.

Required: --prompt --source_video_path --ref_image_paths

NUM_GPUS=1

accelerate launch --num_processes=${NUM_GPUS} \
    scripts/inference/generate.py \
    --config_path configs/inference/generation.yaml \
    --ckpt_path checkpoints/LoomVideo \
    --task ref_edit \
    --prompt "Your editing instruction" \
    --source_video_path /path/to/source_video.mp4 \
    --ref_image_paths /path/to/ref1.jpg /path/to/ref2.jpg \
    --num_inference_steps 50 \
    --seed 0 \
    --output_path outputs/ref_edit.mp4

4. Multi-Image-to-Video (mi2v)

Generate a video conditioned on multiple reference images and a text prompt. We recommend using @Image N in the prompt to reference specific input images.

Required: --prompt --ref_image_paths

NUM_GPUS=1

accelerate launch --num_processes=${NUM_GPUS} \
    scripts/inference/generate.py \
    --config_path configs/inference/generation.yaml \
    --ckpt_path checkpoints/LoomVideo \
    --task mi2v \
    --prompt "Your prompt here" \
    --ref_image_paths /path/to/img1.jpg /path/to/img2.jpg /path/to/img3.jpg \
    --num_frames 97 \
    --num_inference_steps 50 \
    --seed 0 \
    --output_path outputs/mi2v.mp4

Additional Arguments

The following arguments can be appended to any task command for further customization:

Generation Control

Argument	Type	Default	Description
--num_inference_steps	int	50	Number of denoising steps.
--guidance_scale	float	5.0 / 2.5	Text CFG scale. 5.0 for t2v/mi2v, 2.5 for edit/ref_edit.
--guidance_scale_visual	float	1.5	Visual CFG scale for source/reference conditioning.
--negative_prompt	str	(from config)	Negative prompt for quality improvement.
--seed	int	0	Random seed. Set to -1 for random generation.

Resolution & Frames

Argument	Type	Default	Description
--height	int	auto	Output height. 480 for t2v; inferred from source for edit.
--width	int	auto	Output width. 832 for t2v; inferred from source for edit.
--num_frames	int	auto	Output frames. 81 for t2v/mi2v; inferred for edit.
--fps	int	24	Output video FPS.

📦 Data Preparation

Since our training relies heavily on proprietary datasets, we are unable to release the original data directly. However, we provide a flexible data organization framework that makes it easy to plug in your own data or publicly available datasets.

Open-Source Datasets

Below are the open-source datasets used in our training. You can download them or substitute with your own data:

Category	Dataset
Video Generation	Koala-36M, OpenVid-1M
Image Editing	CrispEdit-2M, OmniGen-2-Edit, GPT-Image-Edit-1.5M, NHR-Edit, Pico-Banana, ShareGPT-4o-Image
Video Editing	KIWI-Edit
Video Ref Editing / MI2V	RefVIE, Phantom-Data

Organize Data as Single JSON Files

Each data sample should be stored as an individual JSON file, placed in a single directory (e.g., single_jsons/), and named sequentially starting from 0.json:

your_dataset/
└── single_jsons/
    ├── 0.json
    ├── 1.json
    ├── 2.json
    ├── ...

JSON Format for Each Task

Each task type expects a specific set of keys in its JSON file. Below are the templates — fill in according to your data:

Text-to-Video (process_t2v_data):

{
    "text": "A caption describing the video content.",
    "path": "relative/path/to/video.mp4"
}

Text-to-Image (process_t2i_data):

{
    "caption": "A caption describing the image content.",
    "image_path": "relative/path/to/image.jpg"
}

Video Editing (process_video_edit_data):

{
    "source_video_path": "relative/path/to/source_video.mp4",
    "instruction": "The editing instruction.",
    "target_video_path": "relative/path/to/target_video.mp4"
}

Image Editing (process_image_edit_data):

{
    "source_image_path": "relative/path/to/source_image.jpg",
    "instruction": "The editing instruction.",
    "target_image_path": "relative/path/to/target_image.jpg"
}

Multi-Image-to-Video (process_t2v_data_withref):

{
    "instruction": "A prompt describing the video to generate with reference images.",
    "reference_image_paths": [
        "relative/path/to/ref1.jpg",
        "relative/path/to/ref2.jpg"
    ],
    "target_video_path": "relative/path/to/target_video.mp4"
}

Reference-Guided Video Editing (process_video_edit_data_withref):

{
    "source_video_path": "relative/path/to/source_video.mp4",
    "reference_image_paths": [
        "relative/path/to/ref1.jpg"
    ],
    "instruction": "The editing instruction with reference guidance.",
    "target_video_path": "relative/path/to/target_video.mp4"
}

💡 All paths in JSON files are relative to the data_root specified in the dataset config.

Custom Process Functions (Optional)

You may also organize your JSON files in any format you prefer, as long as you implement a corresponding process_* function. We provide several reference implementations in src/dataset/processors.py. Each process function takes (dataset_info, data_info) and returns a list of segments describing the data flow. See the existing functions for examples.

Dataset Config

Create a YAML config file to register your datasets. See configs/dataset/train_demo.yaml as a reference. The config is organized into train, val, and eval sections, each containing dataset entries with the following arguments:

Argument	Description
task_weight	Controls the sampling probability of this task group relative to others during training.
process_func_name	Name of the processing function in src/dataset/processors.py that parses each JSON sample.
data_root	Base directory for resolving relative paths in JSON files.
data_json_dir	Directory containing the JSON files (0.json, 1.json, ...).
num_samples	Total number of samples in the directory.
sample_weight	Sampling weight of this dataset within its task group.

🏋️ Training

Training Config

The training behavior is fully controlled by a YAML config file (e.g., configs/train/stage3.yaml).

Key arguments:

Argument	Description
log_dir	Directory for saving logs, checkpoints, and generated samples.
dataset_config_path	Path to the dataset config YAML file.
train_steps	Total number of training iterations.
checkpointing_interval	Save a checkpoint every N steps.
validation_interval	Run validation every N steps.
evaluation_interval	Run evaluation benchmarks every N steps.

Model settings:

Argument	Description
model.trainable_modules.gen_model	Which modules to train. "all" trains the full generation model.
model.gradient_checkpointing	Enable gradient checkpointing to reduce GPU memory usage.
model.und.pretrained_model_path	Path to the pretrained understanding backbone.
model.gen.pretrained_model_path	Path to the pretrained generation backbone.
model.pretrained_ckpt_path	(Optional) Load weights from a previous training stage for continued training.

Data settings:

Argument	Description
data.train.resolution_buckets	List of resolution buckets for dynamic batching.
data.train.num_frames	Number of frames per training sample.
data.train.fps	Video FPS for frame sampling.
data.train.all_dropout_rate	Probability of dropping all conditions (for unconditional training).
data.train.text_dropout_rate	Probability of dropping text condition (for classifier-free guidance).

Launch Training

Once the data and configs are ready, you can simply start training with:

NUM_GPUS=8

accelerate launch --num_processes=${NUM_GPUS} \
    -m scripts.train.train \
    --config_path path/to/your/config.yaml

💡 All training outputs — including checkpoints, EMA weights, logs, and generated samples — are saved under the log_dir directory specified in the config.

📊 Evaluation

Environment Setup

Step 1: Prepare Benchmark Data

We evaluate on the following benchmarks. Download each dataset and organize it into the same single JSON format used for training data (see Data Preparation):

Benchmark	Category	Samples
GenEval	Image Generation	553
ImgEdit-Bench	Image Editing	737
VBench	Video Generation	165
OpenVE-Bench	Video Editing	431
RefVIE-Bench	Reference Video Editing	120
Intelligent-VBench-MI2V	Multi-Image-to-Video	320
Intelligent-VBench-TIV2V	Text-Image-Video-to-Video	210

💡 For Intelligent-VBench, we split the original benchmark into two subsets based on task type — MI2V and TIV2V. Their JSON files should be placed in separate directories.

After downloading, update the data_root and data_json_dir paths in configs/dataset/benchmarks.yaml to point to your local directories.

Step 2: Install Evaluation Dependencies

VBench:

mkdir -p libs && cd libs
git clone https://github.com/Vchitect/VBench.git

Add the following to libs/VBench/vbench/__init__.py:

import sys, os
local_lib_path = os.path.abspath("libs/VBench")
if local_lib_path not in sys.path:
    sys.path.append(local_lib_path)

If you encounter a NumPy 2.0 compatibility error (np.sctypes was removed), modify lines 45–47 of [YOUR_PYTHON_LIBS]/imgaug/imgaug.py:

# Replace:
# NP_FLOAT_TYPES = set(np.sctypes["float"])
# NP_INT_TYPES = set(np.sctypes["int"])
# NP_UINT_TYPES = set(np.sctypes["uint"])

# With:
NP_FLOAT_TYPES = {np.float16, np.float32, np.float64, np.longdouble}
NP_INT_TYPES = {np.int8, np.int16, np.int32, np.int64, np.longlong}
NP_UINT_TYPES = {np.uint8, np.uint16, np.uint32, np.uint64, np.ulonglong}

To save disk space, remove unnecessary files:

rm -rf libs/VBench/VBench-2.0 libs/VBench/.git libs/VBench/asset libs/VBench/vbench2_beta_trustworthiness

GenEval:

cd libs
git clone https://github.com/djghosh13/geneval.git
cd geneval
./evaluation/download_models.sh "../../checkpoints/"

cd ..
pip install mmcv-full
git clone https://github.com/open-mmlab/mmdetection.git
cd mmdetection && git checkout 2.x
pip install -v -e . --no-build-isolation

The GenEval model paths are configured in configs/evaluation/evaluation.yaml under model.evaluation.geneval:

model:
  evaluation:
    geneval:
      model_path: checkpoints/evaluation/mask2former_swin-s-p4-w7-224_lsj_8x2_50e_coco.pth
      model_config_path: libs/mmdetection/configs/mask2former/mask2former_swin-s-p4-w7-224_lsj_8x2_50e_coco.py
      clip_path: checkpoints/evaluation/ViT-L-14.pt

Step 3: Configure API Keys

Some benchmarks (OpenVE-Bench, RefVIE-Bench, ImgEdit-Bench, Intelligent-VBench) require LLM API calls for metric computation. Configure your API keys in configs/evaluation/evaluation.yaml under model.evaluation:

model:
  evaluation:
    # For OpenVE-Bench, RefVIE-Bench, Intelligent-VBench
    gemini:
      api_key: "YOUR_GEMINI_API_KEY"
      base_url: "YOUR_GEMINI_BASE_URL"
      model: "gemini-2.5-pro-06-17"
    # For ImgEdit-Bench
    openai:
      api_key: "YOUR_OPENAI_API_KEY"
      base_url: "YOUR_OPENAI_BASE_URL"
      model: "gpt-4.1"

Run Evaluation

Once the environment is set up, you can simply run evaluation with:

NUM_GPUS=8

accelerate launch --num_processes=${NUM_GPUS} \
    -m scripts.evaluation.evaluate \
    --config configs/evaluation/evaluation.yaml \
    --checkpoint_dir checkpoints/LoomVideo \
    --generation_configs configs/dataset/benchmarks.yaml \
    --output_dir results/evaluation \
    --calculate_metrics

📧 Contact

Jianzong Wu (吴健宗): jzwu@stu.pku.edu.cn

📄 Citation

If you find our work helpful, please consider giving us a ⭐ on this repo and citing our paper as follows:

@article{wu2026loomvideo,
  title={LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing},
  author={Wu, Jianzong and Lian, Hao and Yang, Jiongfan and Hao, Dachao and Tian, Ye and Tong, Yunhai and Zhu, Jingyuan and Chen, Biaolong and Qi, Qiaosong and Zhang, Aixi and He, Wanggui and Liu, Mushui and Huang, Pipei and Jiang, Hao},
  journal={arXiv preprint arXiv:2606.06042},
  year={2026}
}