React Native binding of llama.cpp - LLM inference in C/C++
Key Features:
- GPU/NPU Acceleration: Metal (iOS), Hexagon NPU (Android, Experimental) for on-device inference
- Multimodal Support: Support vision/audio understanding models through mmproj projector integration
- Text-to-Speech (Experimental): On-device neural TTS via codec.cpp — OuteTTS, Soprano, NeuTTS, CSM, Qwen3-TTS, MOSS, Chatterbox (incl. Turbo), BlueMagpie, CosyVoice / BreezyVoice — with reference-audio voice cloning
- Parallel Decoding: Slot-based concurrent request processing with automatic queue management
- Tool Calling: Universal function calling support via Jinja templates
- Grammar Sampling: GBNF and JSON schema support for structured, constrained output generation
Important
Starting with v0.10, llama.rn requires React Native's New Architecture.
For Old Architecture support or documentation for v0.9.x, please refer to the v0.9 branch.
npm install llama.rnllama.rn downloads the pre-built ios/rnllama.xcframework and android/src/main/jniLibs from the matching GitHub release during postinstall. Existing downloads are reused, and each archive is verified with SHA-256 before extraction.
Bun does not run dependency lifecycle scripts unless the package is trusted. If you install llama.rn with Bun and want the native downloads to run automatically, add it to trustedDependencies and re-run bun install:
{
"trustedDependencies": ["llama.rn"]
}If you prefer not to trust dependency lifecycle scripts, run the downloader manually before npx pod-install or your Android build:
node ./node_modules/llama.rn/install/download-native-artifacts.jsPlease re-run npx pod-install again.
By default, llama.rn will use pre-built rnllama.xcframework for iOS. If you want to build from source, please set RNLLAMA_BUILD_FROM_SOURCE to 1 in your Podfile.
Add proguard rule if it's enabled in project (android/app/proguard-rules.pro):
# llama.rn
-keep class com.rnllama.** { *; }
By default, llama.rn will use pre-built libraries for Android. If you want to build from source, please set rnllamaBuildFromSource to true in android/gradle.properties.
- Confirm the target device exposes an OpenCL-capable GPU (Qualcomm Adreno 700+ devices are currently supported & tested).
- Add
<uses-native-library android:name="libOpenCL.so" android:required="false" />to your app manifest so the loader can be loaded at runtime. - Configure
n_gpu_layers(> 0) when callinginitLlamato offload layers to the GPU. The native result exposesgpu,reasonNoGPU,devices, so you can confirm runtime behaviour.
- Confirm the target device has HTP (Hexagon Tensor Processor), Qualcomm SM8450+ (8 gen 1 or newer) devices are currently supported & tested).
- Add
<uses-native-library android:name="libcdsprpc.so" android:required="false" />to your app manifest so the loader can be loaded at runtime. - Add param
devices: ['HTP0'](orHTP*for all HTP sessions) to use HTP devices. - Configure
n_gpu_layers(> 0) when callinginitLlamato offload layers to the GPU. The native result exposesgpu,reasonNoGPU,devices, so you can confirm runtime behaviour.
For use with the Expo framework and CNG builds, you will need expo-build-properties to utilize iOS and OpenCL features. Simply add the following to your app.json/app.config.js file:
module.exports = {
expo: {
// ...
plugins: [
// ...
[
'llama.rn',
// optional fields, below are the default values
{
enableEntitlements: true,
entitlementsProfile: 'production',
forceCxx20: true,
enableOpenCL: true,
},
],
],
},
}You can search HuggingFace for available models (Keyword: GGUF).
For get a GGUF model or quantize manually, see quantize documentation in llama.cpp.
💡 You can find complete examples in the example project.
Load model info only:
import { loadLlamaModelInfo } from 'llama.rn'
const modelPath = 'file://<path to gguf model>'
console.log('Model Info:', await loadLlamaModelInfo(modelPath))Initialize a Llama context & do completion:
import { initLlama } from 'llama.rn'
// Initial a Llama context with the model (may take a while)
const context = await initLlama({
model: modelPath,
use_mlock: true,
n_ctx: 2048,
n_gpu_layers: 99, // number of layers to store in GPU memory (Metal/OpenCL)
// embedding: true, // use embedding
})
const stopWords = ['</s>', '<|end|>', '<|eot_id|>', '<|end_of_text|>', '<|im_end|>', '<|EOT|>', '<|END_OF_TURN_TOKEN|>', '<|end_of_turn|>', '<|endoftext|>']
// Do chat completion
const msgResult = await context.completion(
{
messages: [
{
role: 'system',
content: 'This is a conversation between user and assistant, a friendly chatbot.',
},
{
role: 'user',
content: 'Hello!',
},
],
n_predict: 100,
stop: stopWords,
// ...other params
},
(data) => {
// This is a partial completion callback
const { token } = data
},
)
console.log('Result:', msgResult.text)
console.log('Timings:', msgResult.timings)
// Or do text completion
const textResult = await context.completion(
{
prompt: 'This is a conversation between user and llama, a friendly chatbot. respond in simple markdown.\n\nUser: Hello!\nLlama:',
n_predict: 100,
stop: [...stopWords, 'Llama:', 'User:'],
// ...other params
},
(data) => {
// This is a partial completion callback
const { token } = data
},
)
console.log('Result:', textResult.text)
console.log('Timings:', textResult.timings)The binding's design inspired by server.cpp example in llama.cpp:
/completionand/chat/completions:context.completion(params, partialCompletionCallback)/tokenize:context.tokenize(content)/detokenize:context.detokenize(tokens)/embedding:context.embedding(content)/rerank:context.rerank(query, documents, params)- ... Other methods
Please visit the Documentation for more details.
You can also visit the example to see how to use it.
MTP speculative decoding can be enabled for GGUF models that contain MTP/NextN layers:
const context = await initLlama({
model: modelPath,
n_ctx: 4096,
n_batch: 1024,
n_ubatch: 512,
n_gpu_layers: 99,
flash_attn_type: 'auto',
cache_type_k: 'q8_0',
cache_type_v: 'q8_0',
speculative: {
type: 'draft-mtp',
n_max: 3,
},
})
const result = await context.completion({
messages: [
{
role: 'user',
content:
'Write a concise TypeScript function that groups an array of objects by a key.',
},
],
chat_template_kwargs: {
preserve_thinking: true,
},
n_predict: 128,
temperature: 0.6,
top_k: 20,
top_p: 0.95,
speculative: {
type: 'draft-mtp',
n_max: 3,
},
})
console.log(result.text)
console.log(result.draft_tokens, result.draft_tokens_accepted)Use speculative: false on a completion call to disable MTP for that request. For recurrent or hybrid models, enable MTP at initLlama time with a positive spec_draft_n_max or speculative.draft.n_max so llama.cpp can allocate rollback state. Current MTP support is text-only, including queued parallel completions.
Besides embedded MTP layers, a classic pair of a small draft model and a large target model is
supported by setting the speculative type to draft-simple (both models must share the same
vocabulary):
const context = await initLlama({
model: modelPath,
n_ctx: 4096,
n_batch: 1024,
n_ubatch: 512,
speculative: {
type: 'draft-simple',
n_max: 3, // also accepted as spec_draft_n_max
draft: {
model: draftModelPath,
n_gpu_layers: 0,
},
},
})The draft model must use the same vocabulary as the target; a mismatch throws
draft model vocab type must match target model to use speculation instead of running without
speculation. Like MTP, draft-simple is text-only for now, including queued parallel completions,
and can be disabled per request with speculative: false. Acceptance statistics are reported as
draft_tokens / draft_tokens_accepted on the completion result. Recurrent or hybrid models
(as target or draft) are not supported with draft-simple yet.
llama.rn supports multimodal capabilities including vision (images) and audio processing. This allows you to interact with models that can understand both text and media content.
Images (Vision):
- JPEG, PNG, BMP, GIF, TGA, HDR, PIC, PNM
- Base64 encoded images (data URLs)
- Local file paths
- * Not supported HTTP URLs yet
Audio:
- WAV, MP3 formats
- Base64 encoded audio (data URLs)
- Local file paths
- * Not supported HTTP URLs yet
First, you need a multimodal model and its corresponding multimodal projector (mmproj) file, see how to obtain mmproj for more details.
import { initLlama } from 'llama.rn'
// First initialize the model context
const context = await initLlama({
model: 'path/to/your/multimodal-model.gguf',
n_ctx: 4096,
n_gpu_layers: 99, // Recommended for multimodal models
// Important: Disable context shifting for multimodal
ctx_shift: false,
})
// Initialize multimodal support with mmproj file
const success = await context.initMultimodal({
path: 'path/to/your/mmproj-model.gguf',
use_gpu: true, // Recommended for better performance
})
// Check if multimodal is enabled
console.log('Multimodal enabled:', await context.isMultimodalEnabled())
if (success) {
console.log('Multimodal support initialized!')
// Check what modalities are supported
const support = await context.getMultimodalSupport()
console.log('Vision support:', support.vision)
console.log('Audio support:', support.audio)
} else {
console.log('Failed to initialize multimodal support')
}
// Release multimodal context
await context.releaseMultimodal()const result = await context.completion({
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: 'What do you see in this image?',
},
{
type: 'image_url',
image_url: {
url: 'file:///path/to/image.jpg',
// or base64: 'data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD...'
},
},
],
},
],
n_predict: 100,
temperature: 0.1,
})
console.log('AI Response:', result.text)// Method 1: Using structured message content (Recommended)
const result = await context.completion({
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: 'Transcribe or describe this audio:',
},
{
type: 'input_audio',
input_audio: {
data: 'data:audio/wav;base64,UklGRiQAAABXQVZFZm10...',
// or url: 'file:///path/to/audio.wav',
format: 'wav', // or 'mp3'
},
},
],
},
],
n_predict: 200,
})
console.log('Transcription:', result.text)A video_url part works with any vision model. The video is decoded by the platform (AVFoundation on iOS, MediaCodec on Android), so no ffmpeg is needed. Models with video support of their own (Qwen3-VL, LFM3-VL) get the frames, timestamps and frame size they were trained with; other models get frames sampled at 4 fps with a timestamp every 5 seconds, so keep the clip short or the context large.
const result = await context.completion({
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: 'What happens in this video?',
},
{
type: 'video_url',
video_url: {
url: 'file:///path/to/video.mp4',
// or base64: 'data:video/mp4;base64,AAAAIGZ0eXBpc29t...' (Android 9+)
},
},
],
},
],
n_predict: 200,
})
console.log('AI Response:', result.text)// Tokenize text with media
const tokenizeResult = await context.tokenize(
'Describe this image: <__media__>',
{
media_paths: ['file:///path/to/image.jpg']
}
)
console.log('Tokens:', tokenizeResult.tokens)
console.log('Has media:', tokenizeResult.has_media)
console.log('Media positions:', tokenizeResult.chunk_pos_media)- Context Shifting: Multimodal models require
ctx_shift: falseto maintain media token positioning - Memory: Multimodal models require more memory; use adequate
n_ctxand consider GPU offloading - Media Markers: The system automatically handles
<__media__>markers in prompts. When using structured message content, media items are automatically replaced with this marker - Model Compatibility: Ensure your model supports the media type you're trying to process
llama.rn has universal tool call support by using minja (as Jinja template parser) and chat.cpp in llama.cpp.
Example:
import { initLlama } from 'llama.rn'
const context = await initLlama({
// ...params
})
const { text, tool_calls } = await context.completion({
// ...params
tool_choice: 'auto',
tools: [
{
type: 'function',
function: {
name: 'ipython',
description:
'Runs code in an ipython interpreter and returns the result of the execution after 60 seconds.',
parameters: {
type: 'object',
properties: {
code: {
type: 'string',
description: 'The code to run in the ipython interpreter.',
},
},
required: ['code'],
},
},
},
],
messages: [
{
role: 'system',
content: 'You are a helpful assistant that can answer questions and help with tasks.',
},
{
role: 'user',
content: 'Test',
},
],
})
console.log('Result:', text)
// If tool_calls is not empty, it means the model has called the tool
if (tool_calls) console.log('Tool Calls:', tool_calls)You can check chat.cpp for models has native tool calling support, or it will fallback to GENERIC type tool call.
The generic tool call will be always JSON object as output, the output will be like {"response": "..."} when it not decided to use tool call.
GBNF (GGML BNF) is a format for defining formal grammars to constrain model outputs in llama.cpp. For example, you can use it to force the model to generate valid JSON, or speak only in emojis.
You can see GBNF Guide for more details.
llama.rn provided a built-in function to convert JSON Schema to GBNF:
Example gbnf grammar:
root ::= object
value ::= object | array | string | number | ("true" | "false" | "null") ws
object ::=
"{" ws (
string ":" ws value
("," ws string ":" ws value)*
)? "}" ws
array ::=
"[" ws (
value
("," ws value)*
)? "]" ws
string ::=
"\"" (
[^"\\\x7F\x00-\x1F] |
"\\" (["\\bfnrt] | "u" [0-9a-fA-F]{4}) # escapes
)* "\"" ws
number ::= ("-"? ([0-9] | [1-9] [0-9]{0,15})) ("." [0-9]+)? ([eE] [-+]? [0-9] [1-9]{0,15})? ws
# Optional space: by convention, applied in this grammar after literal chars when allowed
ws ::= | " " | "\n" [ \t]{0,20}
import { initLlama } from 'llama.rn'
const gbnf = '...'
const context = await initLlama({
// ...params
grammar: gbnf,
})
const { text } = await context.completion({
// ...params
messages: [
{
role: 'system',
content: 'You are a helpful assistant that can answer questions and help with tasks.',
},
{
role: 'user',
content: 'Test',
},
],
})
console.log('Result:', text)Also, this is how json_schema works in response_format during completion, it converts the json_schema to gbnf grammar.
llama.rn supports slot-based parallel request processing for concurrent completion requests, enabling multiple prompts to be processed simultaneously with automatic queue management. It is similar to the llama.cpp server.
import { initLlama } from 'llama.rn'
const context = await initLlama({
model: modelPath,
n_ctx: 8192,
n_gpu_layers: 99,
n_parallel: 4, // Max number of parallel slots supported
})
// Enable parallel mode with 4 slots
await context.parallel.enable({
n_parallel: 4, // new_n_ctx (2048) = n_ctx / n_parallel
n_batch: 512,
})
// Queue multiple completion requests
const request1 = await context.parallel.completion(
{
messages: [{ role: 'user', content: 'What is AI?' }],
n_predict: 100,
},
(requestId, data) => {
console.log(`Request ${requestId}:`, data.token)
}
)
const request2 = await context.parallel.completion(
{
messages: [{ role: 'user', content: 'Explain quantum computing' }],
n_predict: 100,
},
(requestId, data) => {
console.log(`Request ${requestId}:`, data.token)
}
)
// Cancel a request if needed
await request1.stop()
// Wait for completion
const result = await request2.promise
console.log('Result:', result.text)
// Disable parallel mode when done
await context.parallel.disable()context.parallel.enable(config?):
config.n_parallel(number): Number of concurrent slots (default: 2)config.n_batch(number): Batch size for processing (default: 512)- Returns:
Promise<boolean>
context.parallel.disable():
- Disables parallel mode
- Returns:
Promise<boolean>
context.parallel.configure(config):
- Reconfigures parallel mode (enables if not already enabled)
config.n_parallel(number): Number of concurrent slotsconfig.n_batch(number): Batch size for processing- Returns:
Promise<boolean>
context.parallel.completion(params, onToken?):
params: Same completion parameters ascompletion()onToken: Optional callback(requestId, data) => voidfor token streamingrequestId: Unique request identifierdata: Token data withtoken,content,reasoning_content,tool_calls,accumulated_text
- Returns:
Promise<{ requestId, promise, stop }>requestId: Unique request identifierpromise: Resolves toNativeCompletionResultwhen completestop: Function to cancel this request
context.parallel.embedding(text, params?):
text: Text content to get embedding forparams: Optional embedding parameters- Returns:
Promise<{ requestId, promise }>requestId: Unique request identifierpromise: Resolves to embedding result when complete
context.parallel.rerank(query, documents, params?):
query: Query string for rankingdocuments: Array of document strings to rankparams: Optional rerank parameters (e.g.,normalize)- Returns:
Promise<{ requestId, promise }>requestId: Unique request identifierpromise: Resolves to rerank results when complete
- Parallel mode uses slot-based architecture where each request occupies an available slot
- Slots share the same KV cache for efficient memory usage
- Request processing runs in a background loop that manages slot states automatically
- All standard completion parameters (temperature, top_k, etc.) work per-request
- The context must be initialized with sufficient
n_parallel(default: 8) to support desired slot count - State load/save are not fully supported on Android with OpenCL backend, but you can set
kv_unified: trueandflash_attn_type: 'off'context parameter to enable it.
The session file is a binary file that contains the state of the context, it can saves time of prompt processing.
const context = await initLlama({ ...params })
// After prompt processing or completion ...
// Save the session
await context.saveSession('<path to save session>')
// Load the session
await context.loadSession('<path to load session>')- * Session is currently not supported save state from multimodal context, so it only stores the text chunk before the first media chunk.
The embedding API is used to get the embedding of a text.
const context = await initLlama({
...params,
embedding: true,
})
const { embedding } = await context.embedding('Hello, world!')- You can use model like nomic-ai/nomic-embed-text-v1.5-GGUF for better embedding quality.
- You can use DB like op-sqlite with sqlite-vec support to store and search embeddings.
The rerank API is used to rank documents based on their relevance to a query. This is particularly useful for improving search results and implementing retrieval-augmented generation (RAG) systems.
const context = await initLlama({
...params,
embedding: true, // Required for reranking
pooling_type: 'rank', // Use rank pooling for rerank models
})
// Rerank documents based on relevance to query
const results = await context.rerank(
'What is artificial intelligence?', // query
[
'AI is a branch of computer science.',
'The weather is nice today.',
'Machine learning is a subset of AI.',
'I like pizza.',
], // documents to rank
{
normalize: 1, // Optional: normalize scores (default: from model config)
}
)
// Results are automatically sorted by score (highest first)
results.forEach((result, index) => {
console.log(`Rank ${index + 1}:`, {
score: result.score,
document: result.document,
originalIndex: result.index,
})
})- Model Requirements: Reranking requires models with
RANKpooling type (e.g., reranker models) - Embedding Enabled: The context must have
embedding: trueto use rerank functionality - Automatic Sorting: Results are returned sorted by relevance score in descending order
- Document Access: Each result includes the original document text and its index in the input array
- Score Interpretation: Higher scores indicate higher relevance to the query
- jinaai - jina-reranker-v2-base-multilingual-GGUF
- BAAI - bge-reranker-v2-m3-GGUF
- Other models with "rerank" or "reranker" in their name and GGUF format
Warning
Experimental. The TTS / codec integration is not production-ready. The API surface may change without a major version bump, and several model families do not yet produce correct speech on all backends — see Tested models before picking one.
llama.rn runs on-device neural text-to-speech through codec.cpp as the audio-codec / vocoder backend. Load a TTS backbone as usual, attach its codec (vocoder) GGUF, then drive the synthesis flow the library detects for the model.
The model family is auto-detected natively; getTTSCapabilities() reports it and getFormattedAudioCompletion() returns the right synthesis flow.
| Family | Variants |
|---|---|
| OuteTTS | v0.1 · v0.2 · v0.3 · v1.0 |
| Soprano | 1.1 (80M) |
| NeuTTS | Nano · Air (needs a phonemizer) |
| CSM | 1B |
| Qwen3-TTS | 0.6B · CustomVoice 0.6B / 1.7B |
| MOSS-TTSD | v0.5 |
| MOSS-TTS-Realtime | streaming interleave |
| Chatterbox (T3) | English · 23-language multilingual · Turbo |
| BlueMagpie-TTS | Barbet backbone + AudioVAE |
| CosyVoice | BreezyVoice · CosyVoice-300M · CosyVoice2 · CosyVoice3 |
"Supported" means the family is detected and wired end-to-end. It does not mean every family is verified on every backend — see Tested models.
codec.cpp's codec_audio_lm describes each model: its prompt, its sampling, when it stops and how its output turns into audio. The flow is getFormattedAudioCompletion() → completion() → decodeAudioTokens() / decodeAudioLatents():
getFormattedAudioCompletion()is likegetFormattedChat(): it keeps no state and returns the request'scompletionParams— the complete textprompt(the model's template, the instruct, the normalized text, a voice written as text such as OuteTTS / NeuTTS references, and MTMD-style<__voice__>/<__codec:NAME__>markers where the model needs embedding rows), the grammar (the tokens the model may emit and its minimum length) and alogit_biastable (Float32Array, one bias per token) for the tokens it may emit; a model can use either or both, andembeddingwhen the model reads hidden states. The prompt is plain text: inspect or edit it.completion()is the regular completion. On a context with a vocoder, codec.cpp builds its prompt the way MTMD builds media: text is tokenized, the markers become embedding rows, and thespeakergiven tocompletion()(aLlamaSpeakeror a builtin voice name; the model's default without one) fills<__voice__>. codec.cpp then takes part in every step: it says what the backbone reads next (a token or an embedding row), which steps are audio (never emitted as text; other tokens come out as text, for models that interleave both) and when the speech ends. The sampling is the completion's own, for the backbone and codec.cpp's codebook heads alike: the backbone GGUF's recommendedgeneral.sampling.*unless you set values; itsseedalso seeds codec.cpp's own noise.n_predictis yours to cap it (by default it runs until the model stops or the context is full). The result carries what it generated, the decoder input:audio_tokens(codec codes) oraudio_latentsfor latent models (Soprano, BlueMagpie, Pocket-TTS).embeddingskeeps its usual meaning, the backbone's hidden states, and is not the decoder input.- The decoder input decodes to PCM at any later time, in any order.
One request speaks one text. Long text is the app's to split (by sentence, say), one request and one completion per piece.
import { initLlama } from 'llama.rn'
const ctx = await initLlama({ model: backbonePath, n_gpu_layers: 99 })
// Attach the codec / vocoder GGUF
await ctx.initVocoder({ path: codecGgufPath }) // + optional use_gpu, kv_overrides
const caps = await ctx.getTTSCapabilities()
// { type, family, promptKind, requiresPhonemes, defaultLanguage, builtinVoices, canSpeakerFromAudio,
// canTakeInstruct, needsVoice }
// Arm the request; completionParams are the params to run it with
const { completionParams } = await ctx.getFormattedAudioCompletion({
prompt: 'Hello world',
speaker: 'ryan', // a builtin voice (caps.builtinVoices), a LlamaSpeaker (voice clone), or omit for the default
instruct: caps.canTakeInstruct ? 'Speak cheerfully.' : undefined, // how to speak, for models that take it
// supply a phonemizer only for models that need it (caps.requiresPhonemes)
phonemizer: caps.requiresPhonemes ? (text, lang) => toIPA(text, { language: lang }) : undefined,
})
// Run it with the same speaker and your sampling / length (the GGUF's recommended sampling applies
// to what you leave out), then decode its result to PCM float samples (now or later)
const result = await ctx.completion({ ...completionParams, speaker: 'ryan', seed: 42, n_predict: 2048 })
const sampleRate = await ctx.getAudioSampleRate()
const pcm = new Float32Array(
result.audio_latents?.length
? await ctx.decodeAudioLatents(result.audio_latents)
: await ctx.decodeAudioTokens(result.audio_tokens ?? []),
)
await ctx.releaseVocoder() // free the codec graphs when doneCreate a native speaker handle from a reference clip and pass it as speaker; codec.cpp derives the model's conditioning from it when the request runs. Models without a voice of their own (caps.needsVoice, e.g. Qwen3-TTS Base) require one:
const spk = await ctx.createSpeaker({
refAudio, // Float32Array (mono reference audio)
refAudioSampleRate: 24000,
refText: 'reference transcript',
})
const { completionParams } = await ctx.getFormattedAudioCompletion({ prompt: 'Hello world', speaker: spk })
const result = await ctx.completion({ ...completionParams, speaker: spk })
await spk.release() // free the native speaker when donecreateSpeaker clones the voice once (caps.canSpeakerFromAudio): it computes everything the model conditions on, the backbone prompt (x-vector, reference codes, transcript), the decoder, or both, and every request with that speaker reuses it. Decoding uses the voice of the latest text-to-speech completion, so decode a result before generating with another voice when the decoder depends on it (Chatterbox, CosyVoice, Qwen3-TTS Base with refText). Decoding is deterministic. Codec GGUFs converted before codec.cpp's speaker format carry one builtin voice and cannot clone; re-convert them for that.
A speaker can be saved to a speaker file of the same model and reloaded without the reference audio; the reloaded voice is identical:
await ctx.saveSpeaker(spk, `${dir}/my-voice.gguf`)
const speakers = await ctx.listSpeakers(`${dir}/my-voice.gguf`) // TTSSpeakerInfo[]
const saved = await ctx.loadSpeaker(`${dir}/my-voice.gguf`) // a LlamaSpeaker; { name } picks one in the fileinitVocoder({ kv_overrides }) overrides codec GGUF metadata at load, as llama.cpp-style KEY=TYPE:VALUE strings (TYPE is int, float, bool or str). Model graph settings live here: Chatterbox's guidance weight (codec.lm.chatterbox.sampling.cfg_weight), BlueMagpie's CFM steps / guidance (codec.lm.cfm_n_timesteps, codec.lm.cfm_cfg_value). The flow-matching decoders (Chatterbox S3G, CosyVoice) run n_timesteps × 2 (CFG) U-Net passes, which dominates decode time on a phone; fewer steps trade a little speaker fidelity for speed:
await ctx.initVocoder({
path: codecGgufPath,
kv_overrides: ['cosyvoice.flow.n_timesteps=int:6'], // or chatterbox_s3g.flow.n_timesteps; cfg_rate=float:0 drops CFG
})- OuteTTS / NeuTTS: the codec GGUFs in the
BricksDisplay/OuteTTS-*-GGUF/NeuTTS-*-GGUFrepos carry the builtin voices as the reference prompt (caps.builtinVoices); OuteTTS codecs without them are not supported. Use a codec GGUF only with its own repo's backbone: OuteTTS voices are stored as that backbone's token ids. OuteTTS 0.2 / 0.3 decode with OuteAI's WavTokenizer (the codec in those repos), notwavtokenizer-large-speech-75tokens. NeuTTS voice transcripts are espeak phonemes with stress marks, so phonemize the input the same way. - Qwen3-TTS CustomVoice: the codec GGUFs in the
BricksDisplay/Qwen3-TTS-12Hz-*-CustomVoice-GGUFrepos carry the builtin voices (caps.builtinVoices, defaultvivian); pick one withgetFormattedAudioCompletion({ prompt, speaker: 'ryan' }). ALlamaSpeakerwins over the builtin voice. The 1.7B takes aninstruct(how to speak, e.g.'Speak slowly in a sad, quiet whisper.');caps.canTakeInstructsays whether a model does. - Soprano:
Soprano-1.1-80M-soprano-tok.*.ggufdeclares thesopranopre-tokenizer, which llama.rn patches into llama.cpp and which matches the upstream tokenizer; the plainSoprano-1.1-80M.*.gguffiles keep thegpt-2pre-tokenizer for older llama.rn releases. - CSM: codec GGUFs downloaded from
BricksDisplay/CSM-1B-GGUFbefore 2026-10-20 lack the audio-end metadata and run to the frame cap; download them again, or passkv_overrides: ['codec.lm.eos_code_c0=int:0', 'codec.lm.eos_min_step=int:1']toinitVocoder. - Chatterbox (T3) decodes a guidance pair: create the context with
n_parallel>= 2, or turn the guidance off at load withinitVocoder({ kv_overrides: ['codec.lm.chatterbox.sampling.cfg_weight=float:0'] }); otherwisegetFormattedAudioCompletionthrows. Turbo needs one sequence. The Multilingual backbones inBricksDisplay/Chatterbox-Multilingual-TTS-GGUFsince 2026-10-21 (chatterbox-mtl-<q>.ggufwithcodec-<q>.gguf) carry T3's text tokenizer as their vocab (tokenizer.ggml.pre = chatterbox), so llama.rn tokenizes the prompt; the older vocab-less files still work through codec.cpp's own tokenizer.[lang]tags and other language preparation stay the caller's. - CosyVoice / BreezyVoice always needs a decoder speaker: a builtin voice (
caps.builtinVoices, e.g.zh_female) or aLlamaSpeaker. ALlamaSpeakerwithrefText(the clip's transcript) runs zero-shot cloning; without it, cross-lingual. Zero-shot puts the reference's speech tokens in the backbone context ahead of the text (BreezyVoice: 50 per second, so a 10 s clip is about 500 positions and a 30 s clip about 1500), son_ctxhas to cover the reference plus the output. Sampling is the completion's (the backbone GGUF'sgeneral.sampling.*unless set); upstream's repetition-aware sampling is approximated by a repetition penalty over the last 10 tokens. - BreezyVoice reads Mandarin with 注音 (bopomofo) annotations that upstream adds with g2pw.
caps.requiresPhonemesis true for it, and thephonemizerhook is where to add them; plain text works but polyphones may be misread. - CosyVoice-300M / BreezyVoice use a llama.cpp arch (
cosyvoice) whose relative positions are a KQ bias, which flash attention does not take. Use an F16 KV cache with them; a quantized V cache forces flash attention.
Every family in the catalog was run end-to-end (load → synthesize → decode) on an Apple M1 Max (Metal and CPU) and a Galaxy S25 Ultra / Snapdragon 8 Elite (Hexagon HTP and CPU), then the generated audio was transcribed with Whisper large-v3-turbo to check it actually says the requested text.
Legend: ✅ pass ·
| Family | Metal | macOS CPU | Hexagon HTP | Android CPU | Speech matches input |
|---|---|---|---|---|---|
| CSM 1B | ✅ | ✅ | ✅ | ✅ | ✅ Consistently correct |
| Chatterbox multilingual | ✅ | ✅ | ✅ | ✅ | ✅ Consistently correct |
| BlueMagpie-TTS Barbet-1B | ✅ | ✅ | ❌ cDSP crash on init | ✅ | ✅ Consistently correct |
| MOSS-TTS-Realtime | ✅ | ✅ | ❌ OOM | ❌ OOM | ✅ Correct where it runs |
| NeuTTS Nano · Air | ✅ | ✅ | ✅ | ✅ | |
| MOSS-TTSD v0.5 | ❌ grammar error | ✅ | ❌ grammar error | ✅ | |
| OuteTTS v0.3 | ✅ | ✅ | ✅ | ✅ | llama-tts shows the same; use top_k 4) |
| OuteTTS v1.0 | ✅ | ✅ | ✅ | ✅ | ✅ Correct (Metal/macOS verified after speaker + grammar fixes) |
| Soprano 1.1 | ✅ | ✅ | ✅ | ✅ | ❌ Does not match |
| Qwen3-TTS 0.6B | ✅ | ✅ | ✅ | ✅ | ✅ Correct on latest run |
| OuteTTS v0.1 · v0.2 | — | — | — | — | Not covered by this run |
Caveats:
- The last column is a Whisper content check, not an audio-quality (MOSS/MOS) score. Short clips can produce false negatives, so treat
⚠️ as "needs a manual listen", not as a confirmed bug. - A ✅ in a backend column only means the pipeline produced structurally valid, finite audio — read it together with the last column.
- MOSS-TTS-Realtime needs more RAM than a 12 GB Android device can give it; Android's low-memory killer terminates the app during load on both HTP and CPU.
- On Apple Silicon, CPU-only inference (
n_gpu_layers: 0) additionally hits aSIGILLin the q4_K repack kernel inside the iOS runtime, so the Metal path is the only supported one there for now. - One-pass functional runs from Debug builds, not a statistically sampled benchmark.
isVocoderEnabled()reports whether a codec is attached;releaseVocoder()frees the codec graphs when you're done.- The example app's Text-to-Speech screen wires each family end-to-end (download → init → synthesize → play).
- This is audio output. Audio input (understanding) is a separate feature — see Multimodal (Vision & Audio).
We have provided a mock version of llama.rn for testing purpose you can use on Jest:
jest.mock('llama.rn', () => require('llama.rn/jest/mock'))iOS:
- The Extended Virtual Addressing and Increased Memory Limit capabilities are recommended to enable on iOS project.
- Metal:
- We have tested to know some devices is not able to use Metal (GPU) due to llama.cpp used SIMD-scoped operation, you can check if your device is supported in Metal feature set tables, Apple7 GPU will be the minimum requirement.
- It's also not supported in iOS simulator due to this limitation, we used constant buffers more than 14.
Android:
- Currently only supported arm64-v8a / x86_64 platform, this means you can't initialize a context on another platforms. The 64-bit platform are recommended because it can allocate more memory for the model.
- The OpenCL backend is supported, but currently it limited to Qualcomm Adreno GPU and Q4_0 / Q6_K data types. Please check OpenCL backend for more details.
See the contributing guide to learn how to contribute to the repository and the development workflow.
- BRICKS: Our product for building interactive signage in simple way. We provide LLM functions as Generator LLM/Assistant.
- ChatterUI: Simple frontend for LLMs built in react-native.
- PocketPal AI: An app that brings language models directly to your phone.
- InferrLM: On-device AI for iOS & Android.
- Airgap: Open-source React Native framework for shipping on-device, offline-first customer support chatbots. Runs Gemma 4 E2B locally via
llama.rn. Seven industry templates ship in the repo (telco, retail, healthcare, banking, education, insurance, airlines). - Quo: Private AI assistant that runs open models offline on Android, desktop and in the browser. The Android app runs on
llama.rn, with a small model built in.
- llama.node: An another Node.js binding of
llama.cppbut made API same asllama.rn.
MIT
Made with create-react-native-library
Built and maintained by BRICKS.