Inverse-LLaVA rethinks multimodal alignment by reversing the conventional mapping direction. Instead of projecting continuous visual features into the language model's text space, it maps intermediate text representations into their native visual feature space and combines both modalities through lightweight fusion modules inside the transformer.
The single-stage training pipeline uses zero alignment pre-training samples and 45% fewer total training samples than LLaVA-1.5. Across nine multimodal benchmarks, Inverse-LLaVA remains competitive while showing a distinct strength in reasoning: on MME, its overall cognition score improves by 27.2%, with gains of 69.2% in numerical calculation and 125% in text translation. The evaluation also identifies selective gaps in perception tasks that depend on explicit visual-text grounding, clarifying the trade-off created by removing paired alignment supervision.
The project website presents the architecture, benchmark analysis, and supporting material. The implementation and paper are linked above.