SVHalluc is a CVPR 2026 benchmark for diagnosing semantic and temporal speech-vision grounding failures in audio-visual large language models. It contains 2,405 balanced, human-verified video-question pairs across six diagnostic tasks and reports that leading open AV-LLMs struggle to align spoken content with video, with cross-modal integration identified as a principal failure mode.
SVHalluc evaluates speech-vision hallucination in audio-visual large language models (AV-LLMs): cases in which a model produces an interpretation that does not correctly ground spoken content in the visual evidence of a video. The benchmark focuses on whether systems can connect what is said to what is depicted, rather than treating audio understanding and video understanding as separable competencies.
The CVPR 2026 paper, SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models, frames the target problem along two axes:
The arXiv record states that SVHalluc finds leading open audio-visual LLMs struggle to align spoken content with video. The CVPR paper further identifies cross-modal integration as a principal failure mode. This is a more specific diagnostic claim than a generic observation that video-language models make mistakes: the benchmark is intended to expose failures in the conjunction of modalities, including errors that may be hidden by apparently competent speech processing, visual recognition, or fluent language generation in isolation.
The official project page reports that SVHalluc comprises 2,405 balanced, human-verified video-question pairs distributed across six diagnostic tasks. The supplied sources do not enumerate the six task names, define their answer formats, or provide their individual construction protocols. They therefore support characterizing SVHalluc as a multi-task diagnostic benchmark, but not reconstructing each task's exact input-output specification.
SVHalluc operationalizes hallucination as a failure of speech-vision grounding. Its central premise is that a model can generate an answer that is plausible from speech alone or visually plausible from frames alone, yet unsupported when the audio and visual streams must be interpreted together.
The benchmark's semantic and temporal dimensions imply two corresponding error classes:
The paper's emphasis on cross-modal integration distinguishes SVHalluc from evaluations limited to automatic speech recognition, visual object recognition, captioning, or standard video question answering. A conventional aggregate video-QA score may not isolate whether a wrong answer arose from poor auditory perception, weak visual perception, deficient temporal representation, language priors, or an inability to integrate the modalities. SVHalluc is positioned as a diagnostic suite for the final category.
The supplied sources do not specify the model-side fusion architectures used by evaluated AV-LLMs. In particular, they do not establish whether the systems use modality-specific encoders with cross-attention, early or late fusion, transcript-based audio inputs, raw waveform representations, sampled frames, video tokens, shared multimodal sequence representations, or a particular temporal-tokenization strategy. Parameter counts, context windows, video-frame sampling rates, audio preprocessing, prompt templates, inference settings, and decoding configurations are likewise not specified in the supplied sources.
The official SVHalluc project page describes 2,405 balanced human-verified video-question pairs and six diagnostic tasks. Human verification indicates that the benchmark examples were validated by people, while the description of the set as balanced indicates that its creators controlled some aspect of the evaluation distribution.
However, the supplied material does not state what was balanced. It is not specified whether balance refers to labels, answer options, task types, positive versus negative examples, modality conditions, video domains, or other attributes. The sources also do not provide the number of source videos, video durations, languages, source-video domains, annotation instructions, annotator demographics, inter-annotator agreement, or quality-control criteria.
The available evidence supports an evaluation framing in which models are tested on targeted speech-vision grounding conditions rather than a single monolithic task. This matters because semantic and temporal alignment can fail independently. A model may identify relevant objects or actors but bind dialogue to the wrong visual event; conversely, it may localize an event in time but make an unsupported semantic association between speech and imagery.
The arXiv excerpt refers to “leading open audio-visual LLMs,” but neither it nor the project-page excerpt names the evaluated models. Model-specific baselines, rankings, closed-model coverage, hardware, and implementation details are not specified in the supplied sources. Likewise, the scoring metric is not specified. It would therefore be unsupported to claim that the benchmark uses accuracy, exact match, F1, hallucination rate, temporal localization IoU, calibration, or any particular multiple-choice protocol.
The principal reported finding is qualitative: the authors report that leading open AV-LLMs struggle to align spoken content with video. The CVPR paper identifies cross-modal integration as a principal failure mode, indicating that model limitations cannot be adequately inferred from performance on audio or visual understanding tasks alone.
No numerical results are provided in the supplied source excerpts. There are no reported aggregate scores, per-task results, confidence intervals, error counts, significance tests, leaderboard positions, or model-by-model comparisons available here. The best-performing and lowest-performing systems are not specified, and neither are margins between systems.
Accordingly, the supported conclusion is bounded: on SVHalluc's six-task, 2,405-pair evaluation, the authors find material limitations in open AV-LLMs' ability to ground spoken content in video semantically and temporally. The supplied materials do not support a precise comparative ranking of AV-LLMs or a quantified estimate of the performance gap to human performance, if such a comparison exists.
The official project page links to the paper, evaluation code, and benchmark data. This indicates that the authors provide resources beyond the paper itself and that the benchmark is intended to be evaluated using released materials.
The direct repository URL, software license, dataset license, release version, environment specification, installation instructions, inference scripts, model adapters, and checksums are not specified in the supplied sources. It is also not specified whether source videos are redistributed directly, referenced through external locations, or subject to separate access conditions.
A rigorous replication should follow the canonical links on the official project page and verify implementation details against the CVPR paper and released artifacts. Critical protocol details not available in the supplied excerpts include audio and video sampling, whether models receive audio waveforms or transcripts, prompting format, temporal-evidence presentation, output normalization, and the metric implementation used for final scoring.
SVHalluc is narrowly targeted at speech-vision hallucination. Its findings should not automatically be generalized to image-only hallucination, text-only factuality, long-horizon video reasoning, multilingual speech understanding, speaker identification, audio-event recognition, or real-time streaming settings. Coverage of languages, accents, recording conditions, video genres, editing styles, and adversarial audio-visual mismatches is not specified in the supplied sources.
The benchmark size of 2,405 video-question pairs is accompanied by human verification, but source-domain diversity and source-video composition are not described in the available excerpts. Without those details, transferability across deployment domains cannot be assessed from the supplied materials alone.
The sources also do not state whether controlled modality ablations were conducted. Such tests would be valuable for separating errors caused by audio perception, visual perception, language reasoning, temporal representation, and multimodal fusion. While the paper identifies cross-modal integration as a principal failure mode, the exact experimental procedure used to isolate that cause is not described in the provided source summaries.
Finally, the absence of numerical scores means the supplied materials cannot establish the severity of failure for a particular model, the comparative effectiveness of a mitigation technique, or the magnitude of improvement required for reliable deployment.
SVHalluc highlights a deployment risk that polished AV-LLM demonstrations can obscure: a system may generate a coherent explanation while incorrectly binding dialogue to the wrong person, action, event, or time interval. Such failures matter in video search, meeting review, accessibility tools, instructional-video assistants, media analysis, and content-moderation support, where an unsupported multimodal interpretation may be more harmful than an explicit abstention.
For evaluation design, the benchmark argues against using strong speech transcription, frame-level recognition, or fluent language generation as proxies for grounded audio-visual reasoning. Builders should test whether an answer is supported by the conjunction of auditory and visual evidence. Semantic alignment and temporal alignment should be inspected separately because a model can identify the correct entities while still assigning an utterance to the wrong event or segment.
SVHalluc also provides a benchmark-design pattern: use human-verified video-question pairs and targeted diagnostic tasks to expose multimodal integration failures. The released materials are necessary to determine the exact task formats and scoring protocol, but the paper's central engineering lesson is clear: AV-LLM validation should explicitly test whether speech and video evidence agree, rather than rewarding answers that can be generated from a plausible unimodal prior.
Published May 31, 2026, 12:00 AM
By Kainotomic Team
Published May 31, 2026, 12:00 AM
