TransVLM Teaches Vision-Language Models to Find Full Video Transitions

HeyGen Research and the University of Melbourne have introduced TransVLM, a vision-language system designed to detect complete shot transitions in video. The Hugging Face Blog reports that the work was accepted at ECCV 2026 and published on August 27, 2026.

Instead of treating a transition as one boundary frame, TransVLM returns a segment with a start and end time. That makes it suitable for abrupt cuts as well as longer dissolves, fades, wipes, and other effects—cases where conventional shot-boundary tools often struggle.

Why the approach matters

Existing video pipelines typically split between specialist detectors that handle hard cuts and general-purpose vision-language models that can understand gradual changes but may miss instantaneous edits. TransVLM combines color frames with optical flow, giving the model an explicit signal for motion and pixel-level change. The two inputs are fused in the vision encoder, expanding its input channels from three to six without increasing the language model’s token count.

The system also uses overlapping sliding windows, allowing it to analyze videos far longer than the short clips used during training. Local predictions are merged into a global timeline, making the workflow practical for arbitrary-length footage.

According to the report, TransVLM reaches 78.3% segment F1 on public data and 89.5% on synthetic data, outperforming existing methods evaluated by the authors. For developers, more accurate transition spans could improve video search, captioning, action recognition, and the preparation of training data for text-to-video models—where incorrect scene boundaries can contribute to unwanted visual transitions.

Source: Hugging Face Blog