When voice AI is involved in a real conversation that lasts for hours and spans dozens of rounds, can it remember "who said this sentence, with what tone, and what sounds are in the background"? A joint team from the University of Melbourne and the University of New South Wales believes that current evaluation methods cannot answer this question. So they introduced a benchmark specifically targeting this weakness: VoxMem.

They first pointed out two major flaws in the old evaluation methods: first, they only focus on the transcribed text, discarding the identity information, emotional cues, and ambient sounds hidden in the audio; second, they cannot isolate the variable "how long the history is," making it impossible to determine whether a model's poor performance is due to poor memory or being overwhelmed by length. VoxMem's approach is to create a two-dimensional classification method called "acoustic evidence × memory operation," breaking down the evaluation dimensions orthogonally, covering 15 combinations. All tasks remain unchanged across different historical lengths from 8K to 64K—effectively adjusting only the "conversation length" knob while keeping everything else fixed, allowing memory performance changes to be cleanly attributed to length alone.

This benchmark is substantial: it includes 3,196 evaluation instances, 34,743 audio conversations, totaling about 177 hours of audio, enough to simulate a long, multi-turn conversation. The actual test results on 15 mainstream audio large models are quite revealing. At a context length of around 32K, none of the models managed to exceed a 40% overall accuracy rate. More interestingly, there is a clear bias: models perform significantly better at remembering "what was said" than at remembering speaker identity, paralanguage cues (tone, emotion), and environmental sounds—three types of native audio information. In particular, the accuracy of tracking tone fluctuations and background sound changes is extremely low, basically leaving the models clueless. Moreover, as the history length increases, the accuracy of remembering all types of information declines.