--- library_name: transformers license: other license_name: lfm1.0 license_link: LICENSE pipeline_tag: any-to-any tags: - liquid - lfm3 - lfm3-omni - omni - audio - vision - speech-to-speech - edge - conversational - custom_code language: - en - ar - zh - fr - de - it - ja - ko - pt - es - vi - th - id - hi - ru - pl - da - el - fi - he - nl - sv - tr ---

API Try Docs Blog

# LFM3-Omni-2B LFM3-Omni-2B is [Liquid AI](https://www.liquid.ai/)'s first **omni-modal** model in the LFM3 family: a single end-to-end model that **understands text, images, video and audio**, and **answers in text and speech**. It is built for **on-device** use: one model replaces a separate ASR model, vision-language model, LLM and TTS model. - **Any input, text or speech output:** ask about a photo by voice, transcribe or translate speech, chat out loud, or ask questions about a video and its soundtrack together. - **Watches and listens together:** for videos with sound, the model sees and hears them at the same time, so it can answer questions that need both. - **Small enough for devices:** at 2.2B parameters, it is designed to run efficiently on phones, laptops and other edge hardware. Find more information about the LFM3 family in [our release blog post](https://www.liquid.ai/blog/lfm3). ## Model details

The [LFM3 family](https://huggingface.co/collections/LiquidAI/lfm3) includes base, text, vision, audio, omni, and decision models. LFM3-Omni architecture: a spoken question passes through the LFM3 audio encoder and audio connector, an image through the LFM3 vision encoder and vision connector, and a text prompt through the tokenizer and embedding, all into the LFM3-1.7B Liquid Foundation Model. Its output stream of text and audio tokens drives the LM head for text and the depthformer and Liquid Audio Codec for 24 kHz speech.

LFM3-Omni architecture: a spoken question passes through the LFM3 audio encoder and audio connector, an image through the LFM3 vision encoder and vision connector, and a text prompt through the tokenizer and embedding, all into the LFM3-1.7B Liquid Foundation Model. Its output stream of text and audio tokens drives the LM head for text and the depthformer and Liquid Audio Codec for 24 kHz speech.

Property	Specification
Checkpoint	Post-trained omni-modal model
Parameters	2.17B total
Architecture	LFM3 backbone, LFM3 vision encoder, LFM3 causal audio encoder, LiquidCodec audio tokenizer.
Inputs	Text, images, video with or without its soundtrack, and 16 kHz audio. Images and video are processed at native resolution; each 2×2 block of patches becomes one language token. Video is sampled at 2 fps.
Outputs	Text, 24 kHz speech, or interleaved text and speech
Context length	32K tokens (input + output)
Vocabulary	128,000 text tokens
Text languages	English, Arabic, Chinese, French, German, Italian, Japanese, Korean, Portuguese, Spanish, Vietnamese, Thai, Indonesian, Hindi, Russian, Polish
Speech languages and locales	Arabic, Danish, German, Greek, British English, American English, Finnish, French, Hebrew, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Spanish, Swedish, Turkish, Chinese
License	LFM Open License v1.0

This is a **non-reasoning** model. `enable_thinking=True` is not supported. ## Inference

Choose a checkpoint variant for your inference framework and hardware.

Checkpoint	Framework	Description
BF16	SGLang, Transformers	BF16 weights for GPU inference. SGLang serves text output.
GGUF	llama.cpp	Local inference on CPUs and GPUs, including a realtime speech server.

### Sampling parameters - **Text output:** greedy decoding works well. - **Speech or interleaved output:** use `audio_temperature=0.4`, `audio_repetition_penalty=1.2` ### Generation modes (optional) To force a generation mode, pass the optional `generation_mode` argument to `model.generate`.

Mode	`generation_mode` (optional)	Opening token
Text	`"text"`	none
Speech	`"audio"`	`<\|audio_start\|>`
Text and speech, interleaved	`"interleaved"`	`<\|mixed_start\|>`

By default, the model selects the output modality based on the system prompt and input modality, typically matching the input modality. To encourage a specific output modality, append the corresponding instructions to the end of your system prompt.

Mode	System Prompt
Text	`Respond in text.`
Text and speech, interleaved	`Respond in interleaved text and audio.`

### Transformers quick start Install the dependencies, then load the model and processor with the checkpoint's custom code. The codec implementation is included in this repository, and the processor loads its weights from `audio_tokenizer/`. ```bash pip install "transformers>=5.20.2" torch torchvision torchcodec accelerate pillow librosa soundfile ``` ```python import torch import soundfile as sf from transformers import AutoModelForMultimodalLM, AutoProcessor model_id = "LiquidAI/LFM3-Omni-2B" device = "cuda" if torch.cuda.is_available() else "cpu" dtype = torch.bfloat16 if device == "cuda" else torch.float32 model = AutoModelForMultimodalLM.from_pretrained( model_id, dtype=dtype, device_map=device, trust_remote_code=True ).eval() processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True) codec = processor.audio_tokenizer.to(device=device).eval() ```

Chat template

### Usage examples The examples below reuse the model, codec, and processor loaded above. Replace `audio.wav`, `question.wav`, `recording.wav`, `photo.jpg`, or `clip.mp4` with your own files. Audio files are resampled to 16 kHz mono on load.

Speech recognition (ASR)

To use the model for transcription, use the system prompt `Perform ASR in .`, where `` is the English name of the language. Below, we transcribe a `.wav` file containing English speech. ```python conversation = [ {"role": "system", "content": "Perform ASR in English."}, {"role": "user", "content": [{"type": "audio", "audio": "audio.wav"}]}, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text") text = processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]], skip_special_tokens=True, ) print(text) ```

Speech synthesis (TTS)

For speech synthesis, use the system prompt `Perform TTS in using a voice.`. Use one of the speech languages listed in [Model details](#model-details) and a `female` or `male` voice. For English, say `American English` or `British English`. Pass `generation_mode="audio"` so the answer opens with `<|audio_start|>`. ```python conversation = [ {"role": "system", "content": "Perform TTS in American English using a female voice."}, {"role": "user", "content": "The quick brown fox jumps over the lazy dog"}, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=256, generation_mode="audio", audio_temperature=0.4, audio_repetition_penalty=1.2, ) waveform = codec.decode( outputs.audio_codes[:1, :, :outputs.audio_lengths[0]] ).audio_values[0] # 24 kHz sf.write("tts.wav", waveform.squeeze(0).float().cpu().numpy(), 24_000) ```

Voice chat: speech and text responses

In this mode, the model will respond to the user in both speech and text. The recommended system prompt is `Respond with interleaved text and audio.`. User input can be either text or audio, and we pass `generation_mode="interleaved"` so the answer opens with `<|mixed_start|>`. The assistant responds in a conversational style with interleaved text and speech. This example saves the complete response. ```python conversation = [ {"role": "system", "content": "Respond with interleaved text and audio."}, {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]}, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=1024, generation_mode="interleaved", audio_temperature=0.4, audio_repetition_penalty=1.2, ) text = processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]], skip_special_tokens=True, ) waveform = codec.decode( outputs.audio_codes[:1, :, :outputs.audio_lengths[0]] ).audio_values[0] # 24 kHz print(text) sf.write("reply.wav", waveform.squeeze(0).float().cpu().numpy(), 24_000) ```

Speech-to-text chat

For a text-only answer, no specific system prompt is needed. `Respond with text.` makes it explicit. Pass `generation_mode="text"`. LFM3-Omni-2B is a non-reasoning model: every answer opens with an empty ``. ```python conversation = [ {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]}, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text") text = processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]], skip_special_tokens=True, ) print(text) ```

Audio understanding

Include the recording and a question in the user turn. Questions can address speech, music, background sounds, or events. No specific system prompt is required. To ground answers in the recording, use `Answer questions about the provided audio clip. Base every answer only on what can be heard.`. ```python conversation = [ { "role": "system", "content": "Answer questions about the provided audio clip. Base every answer only on what can be heard.", }, { "role": "user", "content": [ {"type": "audio", "audio": "recording.wav"}, {"type": "text", "text": "What is happening in this recording?"}, ], }, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text") print(processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]], skip_special_tokens=True, )) ``` For multiple-choice questions, include the choices in the text part and end it with `Please choose the answer from the options.`

Translation

Use a system prompt that specifies the source language, target language, and output format: - **Speech to text:** `Please translate the following spoken {source} into {target} text.` Replace `{source}` and `{target}` with the English names of the source and target languages. For text input, set the user message's `content` to the text to translate. Translation to or from English generally works best.

Image understanding

Include the image and a question in the user turn. To ground answers in the image, use the system prompt `Answer questions about the provided image. Base every answer only on what can be seen.`. For captions, use `Describe the provided image.`. Images are processed at their native resolution, up to 2,048 tokens per image. ```python conversation = [ { "role": "system", "content": "Answer questions about the provided image. Base every answer only on what can be seen.", }, { "role": "user", "content": [ {"type": "image", "image": "photo.jpg"}, {"type": "text", "text": "What is written on the sign?"}, ], }, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text") print(processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]], skip_special_tokens=True, )) ```

Spoken questions about an image

The question can be asked by voice, with the recording in place of the text. Add `Respond with text.` to the system prompt for a text answer, or follow the voice chat example with `Respond with interleaved text and audio.` for a spoken one. ```python conversation = [ { "role": "system", "content": "Answer questions about the provided image. Base every answer only on what can be seen. Respond with text.", }, { "role": "user", "content": [ {"type": "image", "image": "photo.jpg"}, {"type": "audio", "audio": "question.wav"}, ], }, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text") print(processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]], skip_special_tokens=True, )) ```

Video understanding

Include the video and a question in the user turn. No specific system prompt is required. Video is sampled at 2 frames per second (4 to 384 frames), and each pair of frames is prefixed with its timestamp. ```python conversation = [ { "role": "user", "content": [ {"type": "video", "video": "clip.mp4"}, {"type": "text", "text": "What happens in this video?"}, ], }, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text") print(processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]], skip_special_tokens=True, )) ```

Video with its soundtrack

To let the model see and hear a video, use the system prompt `Answer questions about the video using what you see and hear.` and pass two flags: - `load_audio_from_video=True` reads the audio track out of the video file. - `use_audio_in_video=True` interleaves that audio with the frames, so each pair of frames is followed by the sound that goes with it. Without them, the model gets the frames only. ```python conversation = [ {"role": "system", "content": "Answer questions about the video using what you see and hear."}, { "role": "user", "content": [ {"type": "video", "video": "clip.mp4"}, {"type": "text", "text": "What happens here, and what can you hear?"}, ], }, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", load_audio_from_video=True, processor_kwargs={"use_audio_in_video": True}, ).to(device) with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text") print(processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]], skip_special_tokens=True, )) ```

Tool calling from spoken requests

Pass function schemas through `tools=...` to `apply_chat_template()`. The chat template places them in the system prompt between `` and ``. Requests may be spoken or written. A tool call looks like this: ``` 30 ``` Values other than strings are written as JSON. Your application parses and runs the calls, then appends the assistant's `tool_calls` (with `arguments` as a dict) and a `tool` message with the results to the conversation. Generate again with the same `tools`: the model either makes more calls or answers. ```python import json import re tools = [{ "type": "function", "function": { "name": "set_volume", "description": "Set the device speaker volume.", "parameters": { "type": "object", "properties": { "volume": {"type": "integer", "description": "Volume from 0 to 100."} }, "required": ["volume"], }, }, }] functions = {"set_volume": lambda volume: {"volume": volume}} # your implementations CALL_RE = re.compile(r"\s*\n]+)>(.*?)\s*", re.DOTALL) PARAM_RE = re.compile(r"\n]+)>\n?(.*?)\n?", re.DOTALL) def parse_tool_calls(text): calls = [] for name, body in CALL_RE.findall(text): arguments = {} for key, value in PARAM_RE.findall(body): try: arguments[key] = json.loads(value) except json.JSONDecodeError: arguments[key] = value # plain string calls.append({"type": "function", "function": {"name": name, "arguments": arguments}}) return calls def respond(conversation): inputs = processor.apply_chat_template( conversation, tools=tools, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=1024, generation_mode="text") return processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:outputs.sequence_lengths[0]], skip_special_tokens=True, ) conversation = [ {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]}, ] text = respond(conversation) while calls := parse_tool_calls(text): results = [functions[c["function"]["name"]](**c["function"]["arguments"]) for c in calls] conversation += [ {"role": "assistant", "tool_calls": calls}, {"role": "tool", "content": json.dumps(results)}, ] text = respond(conversation) print(text) ``` See the [Tool Use documentation](https://docs.liquid.ai/lfm/key-concepts/tool-use) for details.

## Fine-tuning

LFM3-Omni-2B is designed for **easy customization** for a given domain, task, or application with Hugging Face Transformers.

Training	Use case	Notebook
SFT	Fine-tune on your own omni data (text, audio, images) with Transformers
SFT	Visual question answering

## Performance

### Omni

Benchmark	LFM3-Omni 2B	Qwen2.5-Omni 3B	Qwen2.5-Omni 7B	Gemma 4 E2B	Gemma 4 E4B	Phi-4 multimodal
Total parameters	2.2B	6B	10.7B	5B	8B	5.6B
Video + audio understanding
Daily-Omni	70.09	58.81	61.32	50.63	54.14	48.87
WorldSense	47.10	44.99	45.37	38.11	43.51	42.37
AV-SpeakerBench	46.79	40.44	43.24	32.32	36.36	37.95
Image + audio understanding
OmniBench (overall)	46.32	47.11	49.65	16.81	20.67	38.97
OmniBench (speech)	48.25	45.65	47.99	16.08	21.27	38.91
Spoken visual QA
RealWorldQA	64.68	61.05	69.93	39.35	49.02	64.31
MMStar	51.79	50.80	57.27	38.87	44.60	54.80
POPE	83.78	85.34	85.37	74.20	77.82	83.00
ERQA	39.43	37.34	39.16	20.63	32.12	30.81
EmbSpatial	67.59	53.40	59.51	44.08	54.01	42.60

### Vision

Benchmark	LFM3-Omni 2B	Qwen2.5-Omni 3B	Gemma 4 E2B	Gemma 4 E4B	Phi-4 multimodal	Cosmos3 Edge
Total parameters	2.2B	6B	5B	8B	5.6B	4B
General
MMStar	55.67	55.67	46.47	52.00	59.33	56.07
MMMU (val)	42.44	48.00	37.81	43.14	54.76	37.14
MMMU-Pro	27.28	24.80	24.34	28.67	27.63	13.76
MMBench	73.68	79.30	67.61	73.54	82.73	76.63
SEEDBench (image)	75.07	75.27	72.53	74.78	72.24	74.73
RealWorldQA	66.54	63.14	48.63	53.07	64.84	72.81
MathVista (mini)	62.50	58.80	51.00	54.30	65.40	53.90
POPE	89.34	84.62	78.78	81.24	83.34	83.52
Documents and OCR
DocVQA	92.47	92.71	73.49	77.91	92.60	84.43
InfoVQA	64.08	74.55	38.72	48.30	65.98	60.94
OCRBench	806	863	731	761	795	751
Grounding
RefCOCO	73.55	78.76	63.18	63.36	—^*	74.53
Video
MVBench	58.03	66.43	42.05	47.10	52.60	71.65
Video-MME (no subtitles)	57.70	62.59	47.70	52.44	53.44	55.89
Spatial and embodied
ERQA	38.50	40.25	35.25	34.50	35.75	41.00
EmbSpatial	70.55	65.58	47.64	62.28	57.83	76.24
RoboSpatial	49.14	52.00	27.71	38.29	41.14	50.57

### Audio

Benchmark	LFM3-Omni 2B	Qwen2.5-Omni 3B	Gemma 4 E2B	Gemma 4 E4B	Phi-4 multimodal
Total parameters	2.2B	6B	5B	8B	5.6B
English ASR WER ↓	6.36	6.30	8.26	7.82	5.14
Multilingual ASR WER / CER ↓	6.72	7.41	8.08	6.54	—^†
English TTS UTMOSv2 ↑	3.80	3.42	—	—	—
VoiceBench ↑	64.73	66.37	63.28	72.90	64.97
Tool calling internal eval ↑	52.20	44.30	55.60	72.40	—
Audio understanding MMAR ↑	51.40	54.41	44.49	49.10	44.69

Evaluation details

All scores were measured by Liquid AI. Every model uses its own chat template, prompt format, and image and video preprocessing, with greedy decoding for text output. LFM3-Omni-2B speech output uses audio_temperature=0.4 and audio_repetition_penalty=1.2.

Scale: all scores are on a 0–100 scale except OCRBench, which is out of 1,000. DocVQA and InfoVQA use ANLS.
Daily-Omni, WorldSense, and AV-SpeakerBench: video with its soundtrack, multiple-choice accuracy.
OmniBench: an image and an audio clip with each question, scored with the official prompt and answer parser on all 1,142 questions. OmniBench (speech) is the subset whose audio is speech.
Spoken visual QA: the image benchmarks with each question given only as speech.
POPE: F1 over all examples in the adversarial, popular, and random subsets.
MMMU-Pro: vision setting, with the question and options shown in the image.
RefCOCO: boxes are read in each model's native coordinate format. ^*Phi-4-multimodal does not return bounding boxes in a parseable format (72% of answers fail to parse), so its score is not reported.
MVBench and Video-MME: video frames without audio, sampled with each model's recommended settings. Video-MME is without subtitles.
RoboSpatial: RoboSpatial-Home, scored with the official RoboSpatial-Eval rule, with points read in each model's native coordinate format.
English ASR: average word error rate (WER) in percent on the ESB / Open ASR benchmark suite, lower is better.
Multilingual ASR: mean error rate in percent on FLEURS across English, Spanish, Portuguese, German, French, Italian, Chinese, Japanese, and Korean, lower is better. WER is used for all languages except Chinese, Japanese, and Korean, which use character error rate (CER). ^†Phi-4-multimodal does not support Korean speech, so its score is not reported.
English TTS: UTMOSv2 speech quality score on Seed-TTS-eval English, on a 1–5 scale, higher is better.
VoiceBench: audio input, text output, mean of the 9 subsets on a 0–100 scale (the three 1–5 judge scores are multiplied by 20), higher is better.
Tool calling: internal evaluation with audio input, percentage score, higher is better.
Audio understanding: MMAR, percentage score, higher is better.

## Citation

```bibtex @article{liquidAI2026LFM3, author = {Liquid AI}, title = {Introducing LFM3: Every Modality for Specialized Intelligence}, journal = {Liquid AI Blog}, year = {2026}, note = {www.liquid.ai/blog/lfm3}, } ``` For questions and feedback, join [Discord](https://discord.com/invite/liquid-ai). For custom deployment and fine-tuning support, [contact Liquid AI](https://www.liquid.ai/connect?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface).