--- library_name: transformers language: - en model-index: - name: Mellum2 Base results: - task: type: text-generation name: Text Generation dataset: type: humaneval name: HumanEval metrics: - name: pass@1 type: pass@1 value: 41.46 verified: false - task: type: text-generation name: Text Generation dataset: type: humaneval_plus name: HumanEval+ metrics: - name: pass@1 type: pass@1 value: 37.20 verified: false - task: type: text-generation name: Text Generation dataset: type: mbpp name: MBPP metrics: - name: pass@1 type: pass@1 value: 62.40 verified: false - task: type: text-generation name: Text Generation dataset: type: mbpp_plus name: MBPP+ metrics: - name: pass@1 type: pass@1 value: 78.31 verified: false - task: type: text-generation name: Text Generation dataset: type: multipl-e name: MultiPL-E HumanEval, 7 languages metrics: - name: pass@1 type: pass@1 value: 20.97 verified: false - task: type: text-generation name: Text Generation dataset: type: cruxeval name: CRUXEval-I metrics: - name: pass@1 type: pass@1 value: 45.38 verified: false - task: type: text-generation name: Text Generation dataset: type: cruxeval name: CRUXEval-O metrics: - name: pass@1 type: pass@1 value: 43.88 verified: false - task: type: text-generation name: Text Generation dataset: type: cais/mmlu name: MMLU metrics: - name: accuracy type: acc value: 70.87 verified: false - task: type: text-generation name: Text Generation dataset: type: mmlu-pro name: MMLU-Pro metrics: - name: exact match type: exact_match value: 59.31 verified: false - task: type: text-generation name: Text Generation dataset: type: bbh name: BBH metrics: - name: exact match type: exact_match value: 74.90 verified: false - task: type: text-generation name: Text Generation dataset: type: ai2_arc name: ARC-Challenge metrics: - name: normalized accuracy type: acc_norm value: 53.50 verified: false - task: type: text-generation name: Text Generation dataset: type: hellaswag name: HellaSwag metrics: - name: normalized accuracy type: acc_norm value: 73.72 verified: false - task: type: text-generation name: Text Generation dataset: type: winogrande name: WinoGrande metrics: - name: accuracy type: acc value: 65.51 verified: false - task: type: text-generation name: Text Generation dataset: type: truthful_qa name: TruthfulQA MC2 metrics: - name: MC2 type: mc2 value: 44.51 verified: false - task: type: text-generation name: Text Generation dataset: type: gsm8k name: GSM8K metrics: - name: exact match type: exact_match value: 81.73 verified: false - task: type: text-generation name: Text Generation dataset: type: hendrycks_math name: MATH metrics: - name: exact match type: exact_match value: 9.96 verified: false - task: type: text-generation name: Text Generation dataset: type: gpqa name: GPQA Diamond metrics: - name: accuracy type: acc value: 31.31 verified: false - task: type: text-generation name: Text Generation dataset: type: gpqa name: GPQA Main metrics: - name: accuracy type: acc value: 35.04 verified: false license: apache-2.0 --- Mellum # Mellum2 Base Pretrain > [!Note] > Use this checkpoint as a starting point for research on long-context extension or for 8K-context continued pretraining and fine-tuning. For downstream applications use [Base](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Base), [Instruct](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct), or [Thinking](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking) instead. ## Mellum2 Base Highlights Mellum2 Base is a pretrained causal language model trained by JetBrains. The model uses a Mixture-of-Experts architecture with 64 experts and activates 8 experts per token. It uses a combination of sliding-window and full attention layers, with a context length of 8,192 tokens. This is a checkpoint before long-context extension. ## Mellum2 Model Family This repository contains one checkpoint from the Mellum2 family. | Checkpoint | Description | |---|---| | Base Pretrain | Base checkpoint before long-context extension | | [Base](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Base) | Final base model | | [Instruct SFT](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct-SFT) | Supervised instruction-tuned checkpoint | | [Thinking SFT](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking-SFT) | Supervised thinking checkpoint | | [Instruct](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct) | RL-tuned instruction model | | [Thinking](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking) | RL-tuned thinking model | ## Model Overview **Mellum2 Base** has the following features: - Number of Layers: 28 - Hidden Size: 2304 - Intermediate Size: 7168 - MoE Intermediate Size: 896 - Number of Experts: 64 - Number of Activated Experts: 8 - Number of Attention Heads (GQA): 32 for Q and 4 for KV - Context Length: 8,192 - Sliding Window: 1,024 - Vocabulary Size: 98,304 - Precision: bfloat16 - License: Apache 2.0 ## Serving with vLLM This checkpoint has an 8K context length (long-context extension is applied in [Base](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Base)). ```sh vllm serve JetBrains/Mellum2-12B-A2.5B-Base-Pretrain --max-model-len 8192 ``` ## Quickstart Text-Only Input ```python from openai import OpenAI # Configured by environment variables client = OpenAI() messages = [ {"role": "user", "content": "Write a Python function to reverse a string."}, ] chat_response = client.chat.completions.create( model="JetBrains/Mellum2-12B-A2.5B-Base-Pretrain", messages=messages, max_tokens=8192, temperature=0.6, top_p=0.95, extra_body={ "top_k": 20, }, ) print("Chat response:", chat_response) ``` ## Evaluation Evaluation results are available in the model card. All values are self-reported by JetBrains. For more details, see the [Mellum2 Technical Report](https://arxiv.org/abs/2605.31268). ## License Released under the Apache 2.0 license.