SenseTime previews NEO-unify, an encoder-free multimodal model

SenseTime and NTU have introduced NEO-unify (preview), a multimodal model designed to process and generate text and images without a separate vision encoder or visual VAE. The approach is detailed in the Hugging Face Blog, where the authors describe it as an end-to-end alternative to multimodal systems assembled from pretrained components.

One model for understanding and generation

NEO-unify uses a near-lossless visual interface, a native Mixture-of-Transformer backbone, and different learning objectives for each modality: autoregressive cross-entropy for text and pixel flow matching for images. The aim is to let the model develop a shared representation rather than continually translating between specialized subsystems.

The early results are notable, though they come from a preview. A 2-billion-parameter version reached 31.56 PSNR and 0.85 SSIM on MS COCO 2017 after 90,000 pretraining steps, compared with 32.65 PSNR and 0.91 SSIM for Flux VAE. With the understanding branch frozen, the same model scored 3.32 on ImgEdit after 60,000 mixed-training steps using public text-to-image and editing datasets.

For AI builders, the main appeal is efficiency and flexibility. The authors report that NEO-unify scales more efficiently with data than Bagel, achieving stronger results with fewer training tokens. They also say understanding stayed stable while image generation converged faster during joint training, suggesting limited conflict between the two capabilities.

The project points toward image-aware reasoning, editing loops, and broader multimodal or “world model” systems. However, SenseTime says the models are not yet fully released, so developers cannot evaluate the implementation directly. The next important test will be whether the promised open models match these early results in practical workflows.

Source: Hugging Face Blog