MSAV-Script-100K

Paper | 🌐 Project Page | 🤗 Dataset

Summary

MSAV-Script-100K is a large-scale, high-quality dataset for multi-shot audio-visual generation, introduced in our paper Structured Stateful Relational Scripts with a High-Quality Dataset for Multi-Shot Audio-Visual Video Generation. It comprises 100,000 video clips with continuous source audio, selected for high visual and audio quality and strong audio-visual alignment.

Each video is paired with a Structured Stateful Relational Script (SSRS) that explicitly describes persistent entity and voice identities, shot-specific state changes, and audio-visual event relationships across shots. These annotations support training and fine-tuning models for coherent multi-shot audio-visual generation, with supervision for identity consistency, state evolution, and sound-source alignment.

SSRS Annotations

Each caption is a structured text string with seven top-level sections:

Section	Content
【Global Visual】	Shared scene, visual style, lighting, color, and atmosphere.
【Global Audio】	Overall soundscape, background score, and auditory atmosphere.
【Visual Entity Registry】	Reusable identities and appearance descriptions for subjects and objects.
【Audio Identity Registry】	Reusable voice identities and vocal characteristics.
【Audio Entity Registry】	Speech, sound effects, ambience, and music, with transcripts where available.
【Shot Sequence】	Shot timing, camera presentation, visual and auditory state changes, and audio-visual relations.
【Global Audiovisual Event Sequence】	Event progression and connections across shots.

Entity identifiers connect the registries to shot descriptions and the global event sequence. This lets a script represent, for example, the same voice continuing across a cut while the visible speaker changes. See Section 4 and Appendix H for the representation and examples. Descriptive text is in English; dialogue transcripts can retain their original language.

Directory

MSAV-Script-100K/
├── README.md
├── SSRS_captions.json
├── videos_00.tar
├── videos_01.tar
├── ...
└── videos_49.tar

The 100,000 videos are distributed across 50 tar archives, each approximately 19.5 GB. Extracting all archives creates videos/<first-two-characters>/<clip_id>.mp4, matching the relative paths in the captions.

SSRS_captions.json contains a top-level data array with one record per video:

Field	Type	Description
clip_id	string	Unique 64-character hexadecimal video identifier.
video_path	string	Relative path: videos/<first-two-characters>/<clip_id>.mp4.
caption	string	Complete SSRS annotation with section labels and line breaks.

Download

Download the release from Hugging Face using the Hub CLI:

pip install -U huggingface_hub
hf download shenlehan/MSAV-Script-100K --repo-type dataset --local-dir ./MSAV-Script-100K

Allow approximately 976 GB for the downloaded files, or approximately 1.95 TB if retaining both the tar archives and extracted videos. To download only the captions and documentation:

hf download shenlehan/MSAV-Script-100K SSRS_captions.json README.md --repo-type dataset --local-dir ./MSAV-Script-100K

Usage

Extract all 50 archives into the dataset directory:

cd MSAV-Script-100K
for archive in videos_*.tar; do
    tar -xf "$archive"
done
cd ..

Load the annotations with Python and resolve each video path against the dataset root:

import json
from pathlib import Path

root = Path("./MSAV-Script-100K")
with (root / "SSRS_captions.json").open(encoding="utf-8") as f:
    records = json.load(f)["data"]

sample = records[0]
video_path = root / sample["video_path"]
assert video_path.is_file(), video_path

print(sample["clip_id"])
print(video_path)
print(sample["caption"])

License

Our MSAV-Script-100K dataset is released under the CC-BY-4.0 license and is intended for research and non-commercial purposes. The video samples are collected from publicly available datasets. Users must follow the related licenses FineVideo, VCapAV, MiraData, VidMuse-V2M-Dataset to use these video samples.

Citation

If you use MSAV-Script-100K or SSRS in your research, please cite:

@article{zhao2026ssrs,
  title   = {Structured Stateful Relational Scripts with a High-Quality Dataset for Multi-Shot Audio-Visual Video Generation},
  author  = {Zhao, Chen and Shen, Lehan and Huang, Qilong and Li, Hongyu and Kang, Zhuoliang and Wei, Xiaoming and Yang, Jian and Tai, Ying},
  journal = {arXiv preprint arXiv:2610.15512},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.15512}
}