Hallucination in a vision-language model occurs when the model generates text that describes something not present in the image. A VLM might describe a cat sitting on the couch when the image shows only an empty couch, or claim there are three people when only two are visible. Unlike text-only LLM hallucinations (which fabricate facts), VLM hallucinations are grounded in a specific image, making them testable and measurable. This is one of the biggest obstacles to deploying VLMs in high-stakes applications.

VLM hallucinations stem from language priors overriding visual evidence. The language model has learned statistical patterns (couches often have cats, tables usually have chairs) and sometimes generates based on those priors rather than the actual image content. Evaluation benchmarks include POPE (polling-based object probing) and CHAIR (Caption Hallucination Assessment with Image Relevance). Mitigation strategies include chain-of-thought prompting (forcing the model to reason step-by-step before answering), RLHF with human feedback on visual accuracy, grounding mechanisms that tie generated text to specific image regions, and higher-resolution image encoding to preserve visual detail.

Hallucination is critical in medical imaging (a fabricated finding could affect diagnosis), autonomous driving (phantom objects cause false braking), quality inspection (reporting non-existent defects wastes investigation time), and legal document processing (incorrect OCR descriptions have compliance implications). Teams deploying VLMs in production must evaluate hallucination rates on their specific domain data and implement guardrails like confidence thresholds and human-in-the-loop verification.