trelisresearch/audio-bits

audio-bits is a research project that measures how efficiently different methods of representing speech can be learned by a small language model. The central question is: when you convert spoken audio into a sequence of tokens, the discrete units that language models process, how much real information do those tokens actually carry per second of speech? The project compares two broad approaches. The first is text BPE (Byte Pair Encoding), the standard technique used to tokenize written text for language models like GPT-2. You transcribe speech to text, then tokenize the text the usual way. The second is audio codecs, programs that compress speech directly into discrete tokens without going through text first. Several codec methods are tested: NeuCodec, Mimi (in two configurations), and SNAC. Each representation is fed into the same small transformer model (called TinyGPT, with about 19 million parameters) trained for exactly one epoch on matched amounts of speech data. The model is then measured on how many bits of information it successfully extracted per second of audio. The headline finding is a three-level hierarchy: text BPE tops out around 22 bits per second, a semantic-only codec reaches about 62 bits per second, and full acoustic codec representations reach around 650 bits per second, though that ceiling mostly reflects reconstruction overhead in the codec rather than content. The code is written in Python and runs experiments on cloud GPU hardware via Modal. Training metrics are logged with Weights and Biases. Published datasets from the experiments are available on Hugging Face. Code is released under the MIT license.