Dynamic Code Assignment

Unlike static token vocabularies, discrete codes adapt to input statistics. The Gumbel-Softmax trick enables differentiable sampling during training:

where g_k ∼ Gumbel(0,1) and τ is a temperature parameter. As τ → 0, this approaches hard assignment while maintaining gradient flow.

Hierarchical Decoding

Multi-scale codebooks improve expressiveness. A two-level hierarchy first predicts coarse codes z^c then detail codes z^d:

This mirrors wavelet transforms in signal processing, where low-frequency components are encoded first. The computational cost scales as O(K_c + K_d) instead of O(K_cK_d).

Error Correction

Discrete codes introduce quantization errors. Residual vector quantization (RVQ) mitigates this by iteratively encoding residuals:

Each stage adds log₂K bits of precision. In practice, 2-4 stages achieve near-continuous reconstruction with manageable overhead.

Applications in Efficient Inference

Code-based representations enable:

• Subword skipping: Low-entropy regions can be represented with fewer codes
• Early exit: Decoding terminates when residual energy falls below threshold
• Adaptive compression: Dynamic bit allocation based on content complexity

For example, speech recognition systems using this approach achieve 3× faster inference than conventional transformers at equal WER.

Discrete Code Representation

Token-free models replace traditional subword tokenization with learned discrete codes, enabling variable-length input segmentation. The core idea involves mapping raw text to a sequence of fixed-length codes via a neural encoder, typically a convolutional or transformer-based architecture. Given an input sequence x = (x₁, ..., x_n), the encoder produces a code sequence z = (z₁, ..., z_m), where each z_i ∈ {1, ..., K} and K is the codebook size. The compression ratio n/m is dynamically adjusted based on input complexity.

Here, E is the encoder, C is the codebook matrix, and w is the local context window. The argmax operation is relaxed during training using Gumbel-Softmax.

Hierarchical Codebook Architecture

Effective models employ multi-level codebooks to capture linguistic structure:

• Phonetic-level codes for low-level acoustic/orthographic patterns
• Morpheme-level codes for morphological units
• Syntax-level codes for grammatical constructions

The hierarchical decomposition allows the model to learn composition rules similar to formal grammars. For example, the probability of a code sequence factors as:

Discrete Code Embedding Layer

The discrete code embedding layer replaces traditional token embeddings by mapping input sequences to a learned set of discrete codes. Given an input sequence x, the embedding layer applies a quantization function Q to project continuous representations into a discrete latent space:

where C is a fixed-size codebook of learned embeddings and E is a trainable encoder. This layer enables variable-length input handling without explicit token boundaries.

Hierarchical Convolutional Encoder

A stack of dilated convolutional layers processes the embedded discrete codes at multiple temporal resolutions. The hierarchical structure captures local and global dependencies through increasing receptive fields. For layer l, the convolution operation with dilation rate d_l is:

where W_l are learnable filters and K is the kernel size. The dilation rates typically follow an exponential progression (1, 2, 4, 8,...) to efficiently expand the context window.

Adaptive Self-Attention Mechanism

The model employs a modified self-attention layer that operates directly on discrete code sequences. Unlike standard attention, it incorporates relative position biases and code-specific attention gates:

where B_ij encodes relative positions between codes i and j, and G is a gating function conditioned on the discrete code values.

Dynamic Code Prediction Head

The output layer predicts probabilities over the codebook space using a mixture of experts approach. For each position, the model selects among M expert networks based on the current context:

Discrete Codebook Optimization

The training of token-free language models relies on learning a discrete codebook C ∈ ℝ^K×d, where K is the number of codes and d is the embedding dimension. The model learns to map input sequences to variable-length code sequences through a quantization process. Given an input sequence x_1:T, the encoder E produces continuous representations z_t = E(x_t), which are then quantized to the nearest codebook entry:

The training objective combines reconstruction loss with commitment loss to ensure the encoder commits to codebook entries:

where D is the decoder, sg[·] denotes the stop-gradient operation, and β controls the commitment strength (typically β = 0.25).

Variable-Length Sequence Modeling

Unlike token-based models that process fixed-length tokens, discrete code models must handle variable-length code sequences. This is achieved through:

• Dynamic chunking: Inputs are segmented into variable-length chunks using learned boundary predictors
• Hierarchical pooling: Multi-scale representations are created by pooling codes at different temporal resolutions
• Relative position embeddings: Allow the model to process codes regardless of absolute positions

The probability of a code sequence c_1:N given input x is factorized autoregressively:

Efficiency in Cross-Lingual Representation

Token-free models leverage discrete codes to represent subword units, bypassing the need for explicit tokenization. In machine translation (MT), this approach eliminates language-specific tokenizers, enabling a unified representation space. For example, a single sequence of discrete codes can encode both English and Mandarin subword structures, reducing preprocessing overhead. The model learns a shared embedding space where semantically similar units across languages map to proximate code vectors, improving alignment during translation.

Handling Morphologically Rich Languages

Languages with complex morphology (e.g., Finnish, Turkish) benefit from token-free models due to their ability to dynamically segment words into compositional codes. Traditional tokenizers struggle with agglutinative structures, often fragmenting words into impractical subword units. Discrete codes, however, adaptively represent morphemes as combinations of latent symbols. For a Finnish word like "taloissani" ("in my houses"), the model might decompose it into codes for "talo" (house), "i" (plural), "ssa" (inessive case), and "ni" (possessive), preserving grammatical meaning without predefined splits.

where w is a word, c_i are discrete codes, and 𝒱_code is the code vocabulary.

Low-Resource Translation

Token-free models excel in low-resource settings by leveraging transfer learning across languages. Discrete codes act as a pivot representation: a model pretrained on high-resource language pairs (e.g., English-French) can generalize to unseen pairs (e.g., Swahili-Tamil) by aligning their code distributions. This is formalized through cross-entropy minimization over shared code embeddings:

where x and y are source and target sentences, and θ denotes model parameters.

Case Study: Byte-Level Neural MT

Byte-based models (e.g., ByT5) demonstrate token-free translation by treating raw UTF-8 bytes as discrete codes. A byte sequence for "你好" (nǐ hǎo) might be represented as [0xE4, 0xBD, 0xA0, 0xE5, 0xA5, 0xBD], bypassing CJK tokenization entirely. Benchmarks show competitive BLEU scores with 2–3× faster inference than subword models, as byte sequences eliminate vocabulary lookup steps.

Latency and Memory Trade-offs

While token-free models reduce preprocessing latency, they may increase sequence lengths. For a 10-word English sentence, a subword model might emit 12 tokens, whereas a byte model emits ~40 codes. Transformer self-attention's O(n²) complexity thus becomes a bottleneck. Solutions include:

• Hierarchical pooling: Aggregating codes into chunks (e.g., 4-byte blocks) before attention.
• Hybrid approaches: Using discrete codes for training but dynamic tokenization for inference.

Traditional language models rely on tokenization, where text is split into subword units or characters before processing. Token-free approaches bypass this step by operating directly on raw bytes or learned discrete codes, enabling more flexible and efficient generation. The core mechanism involves mapping input sequences to a fixed set of codes, which are then processed autoregressively.

Discrete Code Representation

Instead of tokens, the input text x is transformed into a sequence of discrete codes c₁, c₂, ..., c_n using a learned codebook. Each code c_i is an integer indexing a high-dimensional embedding vector. The codebook is trained jointly with the model via vector quantization (VQ), minimizing the reconstruction error:

where E is the encoder, sg denotes the stop-gradient operation, and e_c is the codebook embedding. The first term ensures accurate reconstruction, while the latter two terms align the encoder outputs with the codebook.

Autoregressive Generation

Given a sequence of codes, generation proceeds autoregressively by predicting the next code c_t conditioned on previous codes c_<t. The model computes a distribution over the codebook:

where h_t is the hidden state from a transformer or RNN, and W, b are learnable parameters. Sampling strategies like top-k or nucleus filtering can be applied to improve output quality.

Advantages Over Token-Based Models

• No Out-of-Vocabulary Issues: Since codes are learned from data, there is no fixed vocabulary limit.
• Multilingual Support: The same codebook can represent multiple languages without explicit tokenization rules.
• Efficiency: Operating on raw bytes or codes reduces preprocessing overhead and memory usage.

Practical Implementation

Implementing a token-free model requires:

• A pretrained codebook (e.g., using VQ-VAE or similar methods).
• An autoregressive architecture (e.g., Transformer) modified to predict discrete codes.
• Efficient nearest-neighbor search for code assignment during inference.

For example, a transformer-based model can be trained with a cross-entropy loss over the codebook indices, treating code prediction as a classification task. During inference, the generated codes are mapped back to text via the decoder.

Case Study: ByT5

ByT5 demonstrates the effectiveness of token-free approaches by operating directly on UTF-8 bytes. It achieves competitive performance on multilingual tasks without language-specific tokenizers, highlighting the scalability of byte-level modeling.

Comparative Analysis Against Token-Based Models

Token-free architectures demonstrate competitive performance across standard NLP benchmarks while offering computational advantages. On the GLUE benchmark suite, discrete code models achieve within 2-3% of BERT's accuracy while reducing sequence length by 40-60%. The compression ratio r between input text and discrete codes follows:

where L represents sequence length. This compression directly translates to faster attention computation, with theoretical FLOPs scaling as O(rL²) instead of O(L²).

Memory Efficiency Metrics

Discrete code models reduce memory consumption in three key aspects:

• Embedding tables: 256-1024 codebook entries vs. 30k+ token vocabularies
• Attention key-value caches: 35-60% smaller for equivalent context windows
• Gradient buffers: 28% reduction in backpropagation memory

Quantitative measurements on the Pile dataset show memory-per-token ratios:

Latency and Throughput

Autoregressive generation speed improves non-linearly with sequence length due to reduced memory bandwidth pressure. Benchmarks on A100 GPUs show:

Downstream Task Adaptation

Fine-tuning efficiency improves due to compressed representations. On the SuperGLUE benchmark, discrete code models reach 90% of peak performance with 40% fewer training steps compared to token-based equivalents. The adaptation dynamics follow:

where d_code is the code dimension, typically 64-256 vs. 768-1024 for token embeddings.

Long-Context Performance

The compressed nature of discrete codes enables more effective long-context processing. On the SCROLLS benchmark, code-based models maintain 83% accuracy at 32k context length where token models degrade to 61%, demonstrating superior attention entropy:

with code models showing 18-22% lower entropy at each layer, indicating more focused attention patterns.

Token-free language models based on discrete codes face unique scalability challenges due to their reliance on learned codebooks and variable-length encoding schemes. Unlike token-based models that operate on fixed vocabularies, these systems must dynamically manage codebook growth while maintaining computational efficiency during training and inference.

Memory and Computational Complexity

The memory requirements for discrete code models scale with both the codebook size K and the sequence length L. The attention mechanism's complexity becomes:

where d represents the embedding dimension. This quadratic dependence on sequence length combined with linear codebook scaling creates bottlenecks when processing long documents. Recent work by Tay et al. (2022) shows that hybrid approaches using locality-sensitive hashing can reduce this to O(L log L), but at the cost of increased memory fragmentation.

Codebook Growth and Compression Tradeoffs

As model capacity increases, the optimal codebook size grows sublinearly with training data:

This relationship emerges from the information-theoretic bounds on code efficiency. Practical implementations must balance:

• Codebook lookup latency: Larger K increases GPU memory bandwidth pressure
• Compression ratio: More codes enable better sequence compression but require longer training
• Generalization: Overly large codebooks lead to code underutilization (Zipfian distribution)

Distributed Training Challenges

Data parallelism becomes inefficient when codebook parameters exceed GPU memory capacity. Model parallelism introduces synchronization overhead during code assignment:

where p is the number of devices and B the interconnect bandwidth. Current solutions employ:

• Hierarchical codebooks with device-local caching
• Dynamic code routing using learned importance scores
• Quantized gradient communication for code updates

Long-Range Dependency Modeling

Discrete codes introduce additional challenges in capturing long-range dependencies compared to token-based models. The effective context window shrinks due to:

where C is the average compression ratio and Δ_overlap represents the necessary overlap between encoded segments. Recent architectures address this through:

• Cross-segment attention with memory banks
• Recurrent state propagation between code sequences
• Learned positional interpolation schemes

The computational overhead of these solutions typically scales as O(L_eff^1.5), creating diminishing returns for very long sequences.

Token-free language models, which operate on discrete codes rather than traditional subword tokens, face unique challenges when processing rare or out-of-distribution (OOD) inputs. Unlike token-based models that can fall back on subword decomposition, code-based models must rely on their learned discrete representations, making robustness to unseen patterns critical.

Codebook Expansion for Rare Inputs

When encountering rare sequences, the model can dynamically expand its codebook by introducing new codes. Given an input sequence x with low probability under the current codebook C, the model computes:

where τ is a rarity threshold. If this condition holds, the model generates a new code c_new by:

and updates the codebook C' = C ∪ {c_new}. This approach maintains the model's ability to represent novel inputs without retraining.

Uncertainty Quantification for OOD Detection

For OOD detection, the model can leverage the code assignment probabilities. Let q(c|x) be the probability of assigning code c to input x. The uncertainty score U(x) is computed as:

High values of U(x) indicate inputs that are poorly represented by existing codes. In practice, models can use this score to trigger fallback mechanisms or human review.

Fallback Strategies

When rare or OOD inputs are detected, several fallback strategies can be employed:

• Nearest-code projection: Map the input to the most similar existing code using a distance metric in the embedding space.
• Hierarchical decomposition: Break down the input into constituent parts that may have existing codes.
• External memory lookup: Query an external database of rare patterns and their associated codes.

Case Study: Robustness in Multilingual Settings

In multilingual models using discrete codes, rare scripts or characters can be handled by:

where C_base represents a set of fundamental, script-agnostic codes. This allows the model to maintain functionality even for unseen writing systems while gradually learning new script-specific codes during deployment.

Token-free language models based on discrete codes introduce unique challenges and opportunities when integrated into conventional NLP pipelines. Unlike token-based approaches that rely on predefined vocabularies, these models operate on learned discrete representations that may not align directly with standard preprocessing steps.

Preprocessing Alignment

Traditional NLP pipelines typically include tokenization, stemming, and stopword removal - operations designed for word or subword tokens. For discrete code models, raw text must first be converted into the model's native code sequence format. This requires:

• A trained encoder to map character sequences to discrete codes
• Special handling of out-of-vocabulary characters through fallback mechanisms
• Optional code sequence normalization to match training distribution

where E is the encoding function mapping input text 𝒳 to variable-length code sequences 𝒞*.

Pipeline Modifications

Standard NLP components require adaptation to handle code sequences:

• Embedding layers must be replaced with code-specific lookup tables
• Attention mechanisms need adjustment for code-based positional encoding
• Batch processing requires padding/truncation at the code level rather than token level

The dimensional mismatch between code-based representations and traditional word embeddings necessitates careful projection layers when connecting to existing components:

Hybrid Approaches

Some pipelines benefit from hybrid token-code architectures:

• Using codes for early processing stages where sequence length matters
• Converting to tokens for downstream tasks requiring interpretability
• Joint training with both representations through shared latent spaces

The interface between code-based and token-based components can be implemented through:

where H_code is the code representation matrix and Pool is an adaptive pooling operation.

Performance Considerations

Integration impacts several practical aspects:

• Memory usage typically decreases due to shorter code sequences compared to tokens
• Computational efficiency varies by operation - matrix multiplies benefit while certain string operations may suffer
• Latency profiles change as encoding/decoding overhead replaces tokenization

Empirical measurements show the tradeoff curve between sequence length reduction and computational overhead follows:

where L_code is the code sequence length and α is the per-code processing time.

• ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models — Abstract. Most widely used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free models that operate directly on raw text (bytes or characters) have many benefits: They can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text ...
• CL M-TTS: IMPROVING NEURAL CODEC LANGUAGE MODELING FOR Z ... - OpenReview — discrete codes. We interchangeably use c t,1:D with c t. Subsequently, a neural language model p θ(c 1:T|x) is employed, aiming to predict c 1:T from the text transcript x. During the inference phase, the language model generates c 1:Tfor a given text x, which is subsequently transformed into speech through the VAE decoder and a pre-trained ...
• PDF LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive ... — Recent research has demonstrated that zero-shot TTS can be accomplished by harnessing the capabilities of language models on discrete tokens obtained from neural audio codecs [1, 5]. However, due to the high bandwidth nature of audio, a single audio frame is usually represented by multiple codes, which may also be sequentially dependent [6 ...
• PDF ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models — They can process text in any language out of thebox,theyaremorerobusttonoise,andthey minimize technical debt by removing complex and error-prone text preprocessing pipelines. Because byte or character sequences are longer than token sequences, past work on token-free models has often introduced new model ar-chitectures designed to amortize the ...
• PDF ByGPT5 : End-to-End Style-conditioned Poetry Generation with Token-free ... — style by a special token which we add to the model vocabulary. We use ByT5 (Xue et al.,2022), a token-free pre-trained encoder-decoder model, as a baseline. For comparison with subword-level approaches, we ne-tune mT5 (Xue et al.,2021b). Decoder-only As the input for encoder-decoder models is a relatively short sequence of styles, this
• Token Assorted: Mixing Latent and Text Tokens for Improved Language ... — map CoT tokens Cinto discrete latent tokens Z. To enhance abstraction performance, our VQ-VAE is trained on the whole input sequence X, but only applied to Cin the next stage. FollowingJiang et al.(2022;2023), we split Xinto chunks of length Land encode each chunk into L r latent codes, where ris a preset compression rate. More precisely,
• Token Assorted: Mixing Latent and Text Tokens for - arXiv.org — Reasoning capabilities are increasingly recognized as a critical component of Artificial General Intelligence (AGI) systems. Recent research has demonstrated that Large Language Models (LLMs) can exhibit sophisticated reasoning and planning abilities using chain-of-thought (CoT) methodologies, including prompting LLMs with examples where complex problems are broken down into explicit reasoning ...
• PDF Undetectable Watermarks for Language Models - IACR — language models. That is, watermarks can be detected only with the knowledge of a secret key; without the secret key, it is computationally intractable to distinguish watermarked outputs from those of the original model. In particular, it is impossible for a user to observe any degradation in the quality of the text.
• Can Discrete Information Extraction Prompts Generalize Across Language ... — model should meaningfully transfer to another model. Discretizing soft prompt tokens to their near-est vocabulary neighbours in order to overcome these issues does not help either. Khashabi et al. (2021) demonstrated that it is possible to find well-performing soft prompts whose nearest neighbor projections are arbitrarily fixed discrete tokens.
• ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free ... — In particular, we successfully pre-train and release ByGPT5, a new token-free decoder-only language model, and fine-tune it on a large custom corpus of English and German quatrains annotated with ...

• Matthew Dahl Abstract - arXiv.org — sources (e.g., law review articles, books, dic-tionaries); and signals. Each case law task consists of information from ... 6(2). Colleen V. Chien and Miriam Kim. 2025.Generative AI and Legal Aid: Results from a Field Study and ... How Language Models Use Long Contexts. Transactions of the Asso-ciation for Computational Linguistics, 12:157 ...
• ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models — Abstract. Most widely used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free models that operate directly on raw text (bytes or characters) have many benefits: They can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text ...
• Dissertation - eldorado.tu-dortmund.de — commercial books about language: The language instinct [Pin94]. In linguistics, researchers investigate and study language in all its aspects. Understanding language is the key to understand the history and the development of mankind. From the beginning of language studies, major directions in the field of linguistics have developed.
• PDF Evaluating Text-to-Speech Synthesis from a Large Discrete Token-based ... — 6466. modelEnCodec(Défossezetal.,2023),whichcom-pressesspeechaudiointo8separatecodebooks eachofsize1024at75Hz.Semantic-to-coarse ...
• Large language models in medical and healthcare fields: applications ... — Large language models (LLMs) are increasingly recognized for their advanced language capabilities, offering significant assistance in diverse areas like medical communication, patient data optimization, and surgical planning. Our survey meticulously searched for papers with keywords such as "medical," "clinical," "healthcare," and "LLMs" across various databases, including ACM ...
• PDF ZKPDL: A Language-Based System for Efficient Zero-Knowledge ... - KU — to use these protocols in practice, despite a wealth of po-tential applications. This paper introduces a simple description language, ZKPDL, and an interpreter for this language. ZKPDL implements non-interactive zero-knowledge proofs of knowledge, a primitive which has received much atten-tion in recent years. Using our language, a single pro-
• Crypto Tokens and Token Systems | Information Systems Frontiers - Springer — Cyptographic tokens are one of the cornerstones of the new blockchain world but the knowledge about these digital objects is still limited. In this research, we argue that crypto tokens, cryptographically secured digital tokens connected to DLT systems, form socio-technical systems through their reciprocal relationship with their foundational DLT systems. We also argue that today's crypto ...
• Classifiers Guided Controllable Text Generation for Discrete Diffusion ... — Figure 1 illustrates the generation process of the discrete diffusion language model. Starting with a fully masked sequence \(X_T\), the model proceeds through multiple iterations to produce the final generated text \(X_0\).Each iteration comprises two operation: prediction and masking. In the prediction step, the model using the partially masked sequence from the previous iteration to predict ...
• 402 results in SearchWorks catalog — all catalog, articles, website, & more in one search catalog books, media & more in the Stanford Libraries' collections articles+ journal articles & other e-resources
• A Retrieval Augmented Approach to Improving Accuracy of Biomedical Term ... — a retrieval augmented approach to improving accuracy of biomedical term normalization by large language models by thanh son do a master's thesis

• ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models — Abstract. Most widely used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free models that operate directly on raw text (bytes or characters) have many benefits: They can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text ...
• T-FREE: Tokenizer-Free Generative LLMs via Sparse Representations for ... — Figure 1: Method comparison of classic Tokenization (left) and T-Free (right) for text encoding (top) and decoding (bottom). Classic tokenizers learn a single-label vocabulary, i.e. a token is bijectively mapped into a single entry of the vocabulary. Instead, T-Free uses a bijective multi-label mapping over multiple activations of hashed character trigrams.
• 9.3. Language Models — Dive into Deep Learning 1.0.3 documentation - D2L — where statistical tools in Section 9.1 can be applied.. Language models are incredibly useful. For instance, an ideal language model should generate natural text on its own, simply by drawing one token at a time \(x_t \sim P(x_t \mid x_{t-1}, \ldots, x_1)\).Quite unlike the monkey using a typewriter, all text emerging from such a model would pass as natural language, e.g., English text.
• PDF ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models — token-based model and perform the minimal set of modifications to make it token-free, thereby limiting experimental confounds. We base ByT5 on the recent mT5 model (Xue et al., 2021), which 2Previous work has also developed token-free approaches for specific tasks: Gillick et al. (2016) for span labeling, Li
• Google AI Gemma open models - Google for Developers — Gemma 3's 128K-token context window lets your applications process and understand vast amounts of information enabling more sophisticated AI capabilities. ... Practical examples & tutorials for Google's open models. Explore notebooks. Hugging Face. ... MATH evaluates a language model's ability to solve complex mathematical word problems ...
• ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free ... — In particular, we successfully pre-train and release ByGPT5, a new token-free decoder-only language model, and fine-tune it on a large custom corpus of English and German quatrains annotated with ...
• NLTK :: Natural Language Toolkit — Natural Language Toolkit¶. NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries, and ...
• [2406.19223] T-FREE: Tokenizer-Free Generative LLMs via Sparse ... — As such, the benchmark results compare a Unigram model with 1.07 B 1.07 𝐵 1.07B parameter against a T-Free model with 0.84 B 0.84 𝐵 0.84B parameters (for v = 8 k 𝑣 8 𝑘 v=8k). Consequently, we demonstrate that an LLM using T-Free instead of Unigram performs better, despite having over 20% fewer parameters.
• PDF LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive ... — Audio Generation via Discrete Tokens The compression rate and reconstruction quality of RVQ codes inspire a number of works to formulate audio generation as a language model-ing task; however predicting codes sequentially poses a chal-lenge of high inference time. MusicGen [3] proposes reduc-ing the context size by predicting Q codes together ...
• GitHub - eugeneyan/open-llms: A list of open LLMs available for ... — OpenRAIL-M v1: Allows royalty-free access and flexible downstream use and sharing of the model and modifications of it, and comes with a set of use restrictions (see Attachment A) BSD-3-Clause : This version allows unlimited redistribution for any purpose as long as its copyright notices and the license's disclaimers of warranty are maintained.