AI RESEARCHarXiv9d ago

A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss

Suryaansh Jain · Rahasya Barkur · Vishal G · Ryan Rossi · Franck Dernoncourt · Jack Wang · Koustava Goswami · Nedim Lipka · +3 more

arXiv:2609.00591v1Computer Vision

WHAT THIS RESEARCH IS ABOUT

Modern vision-language models easily generate fluent high-level captions but frequently overlook fine-grained details such as object counts, textures, materials, and spatial arrangements. While recent multi-stage pipelines can recover these missed details through iterative verification and rewriting, they suffer from substantially higher inference latency.

To solve this, the researchers introduce SimLoss, a reference-free embedding-space objective for single-pass detailed image captioning. SimLoss uses an InfoNCE contrastive loss to align the model projected hidden representations with a frozen image embedding prior to text decoding, eliminating the need for human-annotated detailed captions or synthetic multi-stage pipeline data. The framework is tested both as a fully differentiable fine-tuning model called SimLoss FFT and as a black-box reward approach called SimLoss GRPO.

The results show that embedding-space supervision can deliver the quality of multi-stage verification with the speed of single-pass generation. SimLoss FFT achieves the highest precision, nearly matches the F1 score of multi-stage methods, and runs roughly 20 times faster, while SimLoss GRPO delivers the best recall across evaluated baselines.

AI-assisted summary of the paper abstract.

KEY POINTS

01
Missing visual details
Standard vision-language models often omit specific attributes, counts, materials, and spatial relations in their captions, while multi-stage fix-up pipelines are slow.
02
Reference-free objective
SimLoss aligns a model hidden representation with a frozen image embedding using an InfoNCE contrastive loss before text decoding, avoiding the need for fine-grained reference captions.
03
Two training variants
The method is implemented as SimLoss FFT for differentiable fine-tuning and SimLoss GRPO for black-box reward-based optimization.
04
High speed and accuracy
SimLoss FFT achieves the highest precision and runs about 20 times faster than multi-stage systems, while SimLoss GRPO secures the strongest recall.

Abstract

An image may be worth a thousand words, but most captioning models describe it in only a few. Modern vision-language models produce fluent high-level captions, yet routinely miss the attributes, counts, textures, materials, and spatial relations that make an image visually specific. Recent multi-stage systems recover some of these details through generation, decomposition, verification, and rewriting, but they do so at the expense of substantially higher inference latency. We propose SimLoss, a reference-free embedding-space objective for single-pass fine-grained image captioning. SimLoss trains a vision-language model to align its projected hidden-state representation with a frozen image embedding through an InfoNCE contrastive loss, supplying a dense visual supervision signal before any text is decoded, and requiring neither human-written fine-grained captions nor pseudo-captions from a multi-stage pipeline. We instantiate it as SimLoss FFT, which backpropagates through a locally available embedding model, and SimLoss GRPO, which treats that model as a black-box reward. Compared with single-pass, multi-stage verification, reward-optimized, and perception-aware baselines, the fully differentiable fine-tuning variant, SimLoss FFT, achieves the highest precision while nearly matching the F1 score of the multi-stage method, all while retaining single-pass inference and running roughly 20 times faster than the multi-stage pipeline. The reward-based variant SimLoss GRPO attains the strongest recall. Together, these results show that embedding-space supervision can recover the quality of multi-stage verification at the latency of a single-pass captioner.

Discussion · 0

Original paper on arXiv