---
library_name: transformers
pipeline_tag: text-generation
language:
- en
- zh
tags:
- mxfp4
- fp8
- vllm
- llm-compressor
- compressed-tensors
base_model: zai-org/GLM-5.2
---

# GLM-5.2-MXFP4xFP8_BLOCK

## Model Overview
- **Model Architecture:** GlmMoeDsaForCausalLM
  - **Input:** Text
  - **Output:** Text
- **Model Optimizations:**
  - **Weight quantization:** Mixed (MXFP4 / FP8_BLOCK)
  - **Activation quantization:** FP8
- **Version:** 1.0
- **Model Developers:** RedHatAI

This model is a quantized version of [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2).

### Model Optimizations

This model was obtained by applying mixed-precision quantization to [zai-org/GLM-5.2](https://huggingface.co/zai-org/GLM-5.2), quantizing the MoE expert linears to 4-bit MXFP4 and the remaining quantized linears (attention, dense, and shared-expert layers) to 8-bit block-scaled FP8 with dynamic FP8 activation scaling, ready for inference with vLLM.

This optimization reduces the number of bits per parameter from 16 to an average of ~4.4 bits, shrinking the on-disk model size from ~1.47 TB (BF16) to ~411 GB and reducing disk and GPU memory requirements by approximately 72%.

Only the linear operators within transformer blocks are quantized using [LLM Compressor](https://github.com/vllm-project/llm-compressor).

Note: this quantized checkpoint is experimentally supported in vLLM.

## Deployment

### Use with vLLM

```
vllm serve RedHatAI/GLM-5.2-MXFP4xFP8_BLOCK \
  --tensor-parallel-size 8 \
  --kv-cache-dtype fp8_e4m3 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 5}' \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice
```

## Creation

This model was created by applying [LLM Compressor](https://github.com/vllm-project/llm-compressor) with the mixed-precision scheme described above (RTN, no calibration data), exported in compressed-tensors format.