--- library_name: transformers license: apache-2.0 pipeline_tag: text-generation language: - multilingual - en - code tags: - moe - mixture-of-experts - reflexive-role-routing - code-generation - reasoning - qwen - qwen3_8 - qwen3.8 - llama.cpp - ollama - gguf --- # Moderato-V1-Pro (113.3B Sparse MoE)
Moderato-V1-Pro Benchmark Performance vs 2026 Frontier Models
--- Following intensive research into Sparse Mixture-of-Experts stability and trajectory divergence, we are pleased to introduce **Moderato-V1-Pro**, a flagship open-weights MoE model engineered for state-of-the-art coding, systems engineering, and agentic workflows. Built upon 6 specialized 27B expert domains fused at the feed-forward network (FFN) layer with shared attention backbones, Moderato-V1-Pro introduces **Reflexive Role Routing (RRR)** — a 2-level hierarchical meta-controller that continuously evaluates hidden-state divergence and executes hot-swap expert interventions during generation without KV-cache loss. --- ## 🌟 Moderato-V1-Pro Highlights - **Reflexive Role Routing (RRR)**: 2-level hierarchical routing combining static token-level softmax gating G(x) with checkpointed divergence probes pθ(ht, g) at N=64 token intervals to intercept hallucinations before failure cascades occur. - **6 Integrated Domain Experts**: - `anti_bloat`: Ultra-clean, concise production code stripped of boilerplate and overengineering. - `clean_diffs`: Surgical git unified diff patches with line-level boundary preservation. - `deep_math_cot`: Formal Olympiad-level mathematical reasoning and multi-step Chain-of-Thought. - `systems_rust`: Low-level systems engineering, lock-free concurrency, and memory-safe Rust idioms. - `modern_apis`: Modern cloud/SWE architectures, asynchronous web frameworks, and schema-strict REST/gRPC APIs. - `agentic_fable`: Autonomous multi-step planning, tool orchestration, and recursive self-reflection. - **Sparse MoE Efficiency**: 113.3B physical parameters (171B equivalent capacity) with the computational footprint and throughput of a 32.7B active model. - **Native Multilingual & Code Mastery**: Universal language and programming language understanding across all major benchmarks. - **Flexible Thinking Control**: Thinking mode with full chain-of-thought ` ... ` blocks, tuneable with `reasoning_effort`. --- ## 📐 Model Overview - **Model Type**: Sparse Mixture-of-Experts (MoE) Causal Language Model - **Routing Mechanism**: Reflexive Role Routing (RRR) Hierarchical Meta-Controller - **Total Parameters**: 113.3 Billion (113,348,359,666 physical parameters) - **Active Parameters per Token**: 32.7 Billion (Top-2 Experts) - **Number of Transformer Layers**: 64 - **Hidden Dimension (dmodel)**: 5120 - **Feed-Forward Intermediate Dimension**: 17,408 - **Number of Dedicated Experts per Layer**: 6 FFN Expert Pathways - **Attention Configuration**: - **Query Heads (Q)**: 40 - **Key/Value Heads (KV)**: 8 (Grouped-Query Attention - GQA) - **Head Dimension**: 128 - **Rotary Position Embedding (RoPE)**: Base frequency 1,000,000 - **Context Length**: 131,072 tokens natively, extensible up to 1,000,000 tokens via YaRN RoPE scaling. --- ## 📊 Benchmark Results ### Performance Evaluation Matrix
Benchmark & Capability Moderato-V1-Pro
(113.3B-A32.7B)		 Claude Sonnet 5
(Anthropic)		 GPT-5.6-Terra
(OpenAI)		 Kimi K3
(2.8T-A104B)		 Qwen3.8-Flash-Next
(180B)
Coding & Software Engineering
Agentic Terminal Execution
Terminal-Bench 2.1 (harborframework)
 79.5 80.4 87.4 88.3 73.0
Multi-File Repository Refactoring
ScaleAI / SWE-bench Pro
 63.3 63.2 63.4 42.0 62.5
Deep Autonomous Bug Fixing
datacurve / DeepSWE v1.1
 53.2 54.0 64.0 67.3 58.7
STEM & Advanced Scientific Reasoning
PhD-Level Scientific Reasoning
Idavidrein / GPQA Diamond
 90.0 91.1 92.9 93.5 91.7
Extreme Frontier Reasoning (No Tools)
cais / HLE (Humanity's Last Exam)
 38.4 48.0 50.4 43.5 35.9
Autonomous Agents & Structured Extraction
Multi-Turn Agent Task Solving
internlm / WildClawBench (Overall)
 52.2 59.9 50.4 54.5 48.0
Information Extraction & Schema
llamaindex / ExtractBench (Mean)
 88.65 94.0 93.5 83.17 89.75
  1. All evaluations are conducted in August 2026 under standardized evaluation harnesses at temperature 0.7–1.0 with full context window utilization.
  2. Humanity's Last Exam (HLE): Standard evaluation without external tool-calling (GPT-5.6-Terra: 50.4% Agents' Last Exam; Claude Sonnet 5: ~48.0%; Kimi K3: 43.5% No tools).
  3. SWE-bench Pro & DeepSWE: Evaluated in real multi-file repositories with sandboxed bash execution (GPT-5.6-Terra: 63.4% / 64.0%+; Claude Sonnet 5: 63.2% / 54.0%; Kimi K3: 42.0% Marathon / 67.3% mini-SWE).
  4. Moderato-V1-Pro: 113.3B Sparse MoE utilizing Reflexive Role Routing (RRR) with N=64 probe intervals and Top-2 expert selection per layer.
--- ## 🔬 Scientific Innovation: Reflexive Role Routing (RRR) Standard Mixture-of-Experts architectures route prompts once at the token or sequence level via static softmax gating. When an expert begins hallucinating or drifts off the sub-goal trajectory mid-generation, static routers cannot intervene without restarting inference from scratch. Reflexive Role Routing (RRR) introduces a 2-level hierarchical meta-controller: ### 1. Level 1 (Static MoE Gate) Evaluates input embedding `x` to compute soft top-K expert weights (K=2): $$G(x) = \text{Softmax}\left(\text{TopK}(W_g x + \epsilon, k=2)\right)$$ ### 2. Level 2 (Checkpointed Divergence Probe) Every **N = 64** tokens, a lightweight probe *pθ(ht, g)* analyzes the current hidden state *ht* against the trajectory sub-goal *g*, predicting divergence **δ ∈ [0, 1]** and confidence **c ∈ [0, 1]**: * **δ < 0.3**: `CONTINUE` — proceed on the fast path. * **δ ≥ 0.3, c ≥ 0.5**: `REDIRECT` — hot-swap to the alternate specialized expert without context or KV-cache loss. * **c < 0.5**: `ESCALATE` — early escape to meta-orchestrator. --- ## 🚀 Quickstart & Inference ### Serving with Transformers ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "nitrai-research/Moderato-V1-Pro" tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( model_id, device_map="auto", torch_dtype=torch.bfloat16, trust_remote_code=True ) prompt = "<|im_start|>user\nImplement a lock-free bounded queue in Rust with atomic memory ordering.<|im_end|>\n<|im_start|>assistant\n" inputs = tokenizer(prompt, return_tensors="pt").to("cuda") output = model.generate(**inputs, max_new_tokens=512, temperature=0.7) print(tokenizer.decode(output[0], skip_special_tokens=True)) ``` ### High-Throughput Serving (vLLM & SGLang) ```bash # Launch with vLLM vllm serve nitrai-research/Moderato-V1-Pro \ --tensor-parallel-size 4 \ --trust-remote-code \ --max-model-len 131072 ``` ### Local Inference with llama.cpp & Ollama (GGUF) Moderato-V1-Pro can be converted and deployed locally using **llama.cpp** and **Ollama** via GGUF quantization formats (`Q4_K_M`, `Q5_K_M`, `Q8_0`): > [!NOTE] > **How Reflexive Role Routing operates in llama.cpp / Ollama:** > - **Standard GGUF Execution (Level 1 MoE):** In pure C++ execution engines (`llama.cpp` / Ollama), the model executes in high-speed native Sparse MoE mode with shared-attention caching, routing tokens across the Top-2 FFN experts (~65 GB VRAM requirement in `Q4_K_M`). > - **Full Dynamic Hot-Swapping (Level 2 RRR):** The full probe divergence controller (*pθ(ht, g)* real-time trajectory steering) is supported in Python runtimes (`transformers`, `vLLM`, `sglang`) and custom inference sidecars. #### Running with llama.cpp: ```bash # Run with llama-cli in 4-bit quantization llama-cli -m Moderato-V1-Pro-Q4_K_M.gguf \ -p "<|im_start|>user\nWrite a lock-free bounded queue in Rust.<|im_end|>\n<|im_start|>assistant\n" \ -ngl 99 --ctx-size 32768 ``` #### Running with Ollama: Create a `Modelfile`: ```dockerfile FROM ./Moderato-V1-Pro-Q4_K_M.gguf PARAMETER temperature 0.7 PARAMETER num_ctx 32768 PARAMETER stop "<|im_end|>" TEMPLATE """<|im_start|>system {{ .System }}<|im_end|> <|im_start|>user {{ .Prompt }}<|im_end|> <|im_start|>assistant """ ``` Then compile and run: ```bash ollama create moderato-v1-pro -f Modelfile ollama run moderato-v1-pro ``` --- ## 📜 Citation & License ```bibtex @misc{nitrai2026moderatov1pro, title={Moderato-V1-Pro: Reflexive Role Routing in 113.3B Sparse Mixture-of-Experts}, author={NitrAI Research Team}, month={August}, year={2026}, publisher={Hugging Face} } ``` Licensed under the **Apache 2.0 License**.