Extract LLM Logprobs to Run Single-Token Vision and Text Classification
Constraining multimodal language models to return a single token while capturing alternative log probabilities enables fast, non-hallucinating classification. Benchmarks show this technique achieves 1.0 FPS across three visual questions using Gemma 4 12B locally on an RTX 3090.
Impact: Medium
Why it matters
You can eliminate output generation latency and avoid JSON extraction failures by turning any vision or text LLM into a calibrated multi-choice classifier.
TL;DR
- 01Force vision and text LLMs to answer multiple-choice questions with a single character to eliminate output token decoding delay.
- 02Extract raw log probabilities from Chat Completions or Responses APIs to obtain true probability distributions.
- 03Reuse KV cache prefixes across multiple single-token questions evaluating the same image or document state.
Key facts
- Local throughput (RTX 3090)
- ~1.0 FPS (3 questions/frame)
- OpenAI gpt-6-luna throughput
- ~0.2 FPS
- Model weights size
- ~7 GB (Gemma 4 12B QAT) + 175 MB mmproj
- Output tokens per answer
- 1 token
Bypassing Text Generation Overhead
Traditional LLM classification pipelines force models to produce complete sentences or structured JSON schemas, exposing pipelines to formatting drift, syntax errors, and generation latency. By prompting the model to answer multiple-choice questions strictly with a single option letter (A, B, C, etc.) and inspecting token probabilities, developers can compute normalized confidence scores across all options in a single step.
# Requesting top logprobs for single-token selection
prompt = f"State:\n{state}\n\nQuestion: {question}\nOptions:\n{options_block}\nAnswer with the letter of the best option only."Local Gemma 4 12B Benchmarks
The pattern was tested on live webcam frames using a local setup with llama.cpp and Gemma 4 12B QAT (~7 GB model weights plus ~175 MB multimodal projector) on an Nvidia RTX 3090:
- Local Throughput: Achieved ~1.0 frames per second while evaluating three distinct queries per frame (detecting visible persons, scene setting, and relative brightness).
- Cloud Comparison: The same pipeline querying OpenAI
gpt-6-lunaran at ~0.2 FPS, largely bottlenecked by HTTP request round-trips per question. - Prefix Caching: Repeating queries over the same image benefits from KV-cache re-use, ensuring that processing input tokens dominates rather than output token generation.
API Differences and Normalization
Implementing this across providers requires handling subtle API differences. Local llama.cpp endpoints expose logprobs via the Chat Completions endpoint (top_logprobs), whereas OpenAI's API requires the Responses endpoint with include: ["message.output_text.logprobs"]. The returned log probabilities are normalized through a softmax-style exponentiation to calculate calibrated probabilities.
Try it in 2 minutes
# Fetch Gemma 4 12B QAT model and multimodal projector
curl -fL -o gemma-4-12b-it-qat-q4_0.gguf https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf/resolve/main/gemma-4-12b-it-qat-q4_0.gguf
curl -fL -o mmproj-gemma-4-12b-it-qat-q4_0.gguf https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf/resolve/main/mmproj-gemma-4-12b-it-qat-q4_0.gguf
llama-server -m gemma-4-12b-it-qat-q4_0.gguf --mmproj mmproj-gemma-4-12b-it-qat-q4_0.gguf --port 8060bash
✓ When to use
- Zero-shot image and document classification where categories change frequently in production.
- Edge or local deployments needing strict latency caps and zero hallucination risk on categorical flags.
✕ When NOT to use
- Open-ended conversational responses or complex multi-step reasoning generation.
- High-framerate video pipelines requiring >30 FPS, where lightweight dedicated CNNs/YOLO models are mandatory.
What to do today
- Download Gemma 4 12B QAT GGUF and run llama-server with logprobs enabled.
- Replace multi-line JSON classification prompts with single-letter multiple-choice questions.
- Implement a logprob normalization function using softmax exponentiation to calculate true confidence percentages.
What the community says
“providers could subsidize the cost of cached input tokens to virtually zero if they would allow for a more flexible API... Most of the cost is the infrastructure around keeping KV caches”
“I've created a Jev wrapper so that it can work via any OpenAI-compatible endpoints”
Sources