Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model

Abstract

Recent advancements in audio generation have been significantly propelled by the capabilities of Large Language Models (LLMs). The existing research on audio LLM has primarily focused on enhancing the architecture and scale of audio language models, as well as leveraging larger datasets, and generally, acoustic codecs, such as EnCodec, are used for audio tokenization. However, these codecs were originally designed for audio compression, which may lead to suboptimal performance in the context of audio LLM. Our research aims to address the shortcomings of current audio LLM codecs, particularly their challenges in maintaining semantic integrity in generated audio. For instance, existing methods like VALL-E, which condition acoustic token generation on text transcriptions, often suffer from content inaccuracies and elevated word error rates (WER) due to semantic misinterpretations of acoustic tokens, resulting in word skipping and errors. To overcome these issues, we propose a straightforward yet effective approach called X-Codec. X-Codec incorporates semantic features from a pre-trained semantic encoder before the Residual Vector Quantization (RVQ) stage and introduces a semantic reconstruction loss after RVQ. By enhancing the semantic ability of the codec, X-Codec significantly reduces WER in speech synthesis tasks and extends these benefits to non-speech applications, including music and sound generation. Our experiments in text-to-speech, music continuation, and text-to-sound tasks demonstrate that integrating semantic information substantially improves the overall performance of language models in audio generation.

Text-to-Speech

We used the LibriTTS dataset as the training set and extracted each codec to get audio tokens for training a VALL-E model. We test our model on librispeech-test-clean.

Below are the inference results on continuation zero-shot speech synthesis of AR stage (1-token) and AR+NAR stage (8-token) .

↔️ Scroll horizontally to view the full table.

Text	Ground Truth	Prompt(First 3s)	X-Codec(Ours)	X-Codec(Ours)	Baseline Acoustic Codec	Baseline Acoustic Codec	Encodec(24khz)	Encodec(24khz)	SpeechTokenizer	SpeechTokenizer	DAC(16khz)	DAC(16khz)
Mode	/	/	AR	AR+NAR	AR	AR+NAR	AR	AR+NAR	AR	AR+NAR	AR	AR+NAR
You must look at him in the face, fight him, conquer him, with what scathe you may. You need not think to keep out of the way of him.
You'll easily judge why when you hear, because the thing had been such a scare, he continued to fix me.
One day when the boy was sent by his grandfather with a message to a relation, he passed along a street in which there was a great concourse of horsemen.
You hear what Sir FERDINANDO Brown has said, replied Captain Battleax.
It has occupied Mother a long time to find at the shops the exact shade for her new bonnet.
This was a formidable array of advantages. Slavery was playing with loaded dice.
In strict accuracy, nothing should be included under the head of conspicuous waste, but such expenditure as is incurred on the ground of an invidious pecuniary comparison.
The music came nearer and he recalled the words. The words of Shelley's fragment upon the moon wandering companionless, pale for weariness.
I must come another day and see your husband. I want to have a consultation with him about horses.
She pushed him toward the big chair by the fire and sat down on a stool at the opposite side of the hearth. Her knees drawn up to her chin, laughing like a happy little girl.