Inverse-LLaVA : Rethinking Multimodal Alignment via Text-to-Vision Mapping

Advisor: Tyler Derr

Text-to-vision fusion

Conventional LLaVA projects visual features into the language model's hidden space. Inverse-LLaVA reverses that mapping: separate query, key, and value branches project language states into CLIP feature space, combine them with frozen visual features, and add the resulting updates inside decoder attention. The released model uses Vicuna-7B-v1.5, CLIP ViT-L/14 at 336 pixels, and fusion at the first decoder layer. Fusion parameters and LoRA adapters are trained jointly.

The model uses one epoch of 665,298 visual-instruction mixture rows, with no separate alignment-pretraining stage. Compared with LLaVA-1.5's 558K alignment examples plus 665K instruction examples, this is 45.6% fewer task-training examples. Both approaches inherit pretrained backbones; this comparison excludes backbone pretraining and does not imply the same reduction in training time or FLOPs.

Results from the released checkpoint

The full evaluation covers nine primary benchmarks against the official LLaVA-1.5 LoRA and full-fine-tuning (FFT) checkpoints. The models share evaluation inputs and scoring protocols, while their training data and adaptation procedures differ. Selected results illustrate both the gains and remaining gaps:

Values are percentages except MME's official score. Bold marks the highest point estimate, not statistical significance. VizWiz improves by 2.40 percentage points over LoRA; its smaller difference from FFT and the ScienceQA differences have confidence intervals spanning zero. TextVQA remains lower. These results show that useful multimodal performance is possible without a separate alignment stage, with task-dependent differences rather than a uniform reasoning advantage. The project website gives all nine benchmarks and protocol details.

Reproducible research and release

The open-source implementation includes training, resumable checkpoints, pinned evaluation protocols, profiling, and representation analysis. The 7B release distributes learned fusion and LoRA weights with checksums and pinned Vicuna/CLIP revisions. Fresh-download inference checks validate loading; rescoring the exported answers reproduces all 33 locally scored comparison cells. External-server and hosted-judge results retain their separate procedures.

The research collection includes component and fusion-depth ablations, supervision interventions, training trajectories, and features from the same 100 image/prompt pairs. HD (wider visual features) and 13B (a larger language backbone) are 5%-data studies, separate from the full-data 7B model. Their checkpoint trajectories explore scaling under matched instruction exposure; they are not full-data scaling claims.

Start with the checkpoint guide for inference or the evaluation guide to reproduce the comparisons. The linked paper is an arXiv preprint; the results on this page correspond to the released checkpoint and evaluation records.