# Evaluating Beam

- Beam is Reflection's open-weight text model for coding, reasoning and tool use, with a `1M`-token context window.
- Self-hosted weights are available in `BF16`, `FP8` and `NVFP4`. Compare against the [reference scores](https://huggingface.co/reflection/Beam-1.0-501B-A23B#benchmark-results) for the precision you test.

## Connect to Beam

- Create a Reflection API key at [platform.reflection.ai](https://platform.reflection.ai). Set your Chat Completions base URL to `https://api.reflection.ai/openai/v1` ([API quickstart](https://developers.reflection.ai/quickstart)).
- To self-host Beam, follow the [deployment guide](https://developers.reflection.ai/deployment-overview) in the developer docs.
- Check that your outgoing request contains `"model": "Beam-1.0-501B-A23B"`.
- Before a formal eval, ask your Reflection contact to increase your organization's quota and rate limits.
  - Include your registered email and organization, eval workload, expected token usage, request rate, duration and any zero data retention requirement.
  - Check your limits in [Billing](https://platform.reflection.ai/billing) and confirm the increase before starting. Ask your contact to confirm daily allowance, concurrency and the per-request output cap.

## Set up the model

- Use `max` for `reasoning_effort`. Beam reaches comparable solution quality to leading open models using approximately `3x` fewer reasoning tokens.
- Start with these model settings:

```json
{
  "model": "Beam-1.0-501B-A23B",
  "reasoning_effort": "max",
  "temperature": 0.7,
  "top_p": 0.99,
  "max_completion_tokens": 32768
}
```

- Suggested output budgets per API call, including reasoning:
  - `32768`: start here for general evals.
  - `65536`: start here for SWE-bench. Raise to `131072` if `finish_reason` is `length`.
  - `16384`: use for short tasks only after checking that answers finish without being cut off. Keep `reasoning_effort` at `max`.
  - Stay within Beam's `131072`-token output limit. Input plus output budget must fit the context window; `null` uses the configured default.
- Save the model ID, prompts, settings and test software version. If hosting Beam yourself, also save the weights, tokenizer, chat template and server settings.

## Check your endpoint with Beam Verifier

- Use it for self-hosted or third-party Beam endpoints to catch deployment issues before the full eval.
- `beam-verifier verify` checks API behavior, including tool calls, streaming and multi-turn conversations.
- `beam-verifier eval` runs GPQA-Diamond and AIME. Use its benchmark presets when comparing with reference scores.
- [Setup and usage](https://developers.reflection.ai/beam-verifier) · [GitHub](https://github.com/reflection-oss/beam-verifier)

## Agent and conversation setup

- For tool-based evals, start with [Mirror](https://developers.reflection.ai/mirror-overview). Beam has been validated with Mirror, Pi, OpenCode and Hermes.
- Your agent or API client must preserve `reasoning_content` and any `tool_calls` unchanged in prior assistant messages. It must return one `tool` message per call, with `tool_call_id` matching the call's `id`.

## Errors and debugging

- Log request IDs when debugging failures. If `finish_reason` is `length`, the output was cut off; track service failures separately from wrong answers.
- Follow rate limits and wait as directed by `Retry-After`. Do not retry when `x-should-retry` is `false`.
- For debugging, share the failing request, settings and request ID with Reflection.

## Example: [SWE-bench Verified](https://www.swebench.com/SWE-bench/guides/datasets/)

- Run [Mirror](https://developers.reflection.ai/mirror-overview) on each issue from its original repository commit, using the `bash`-only eval setup. Set `max_completion_tokens` to `65536`. Keep reference fixes and grading tests hidden.
- Save the generated patches in `predictions.jsonl` using the [official prediction format](https://www.swebench.com/SWE-bench/guides/evaluation/).
- On the evaluation machine, start [Docker Engine](https://www.swebench.com/SWE-bench/guides/docker_setup/) (Linux) or [Docker Desktop](https://www.swebench.com/SWE-bench/guides/docker_setup/) (macOS/Windows). With `swebench` installed, run the command below; the harness creates the task containers.

```bash
python -m swebench.harness.run_evaluation \
  --dataset_name SWE-bench/SWE-bench_Verified \
  --predictions_path predictions.jsonl \
  --max_workers 4 \
  --run_id beam-verified-001
```

- Report the percentage of issues fixed with one attempt per issue. Include failed or missing runs.
- Use a new `run_id` whenever the patches change.

## Developer references

- [API quickstart](https://developers.reflection.ai/quickstart) and [models](https://developers.reflection.ai/models)
- [Generation settings](https://developers.reflection.ai/text-generation) and [reasoning](https://developers.reflection.ai/reasoning)
- [Tool calling](https://developers.reflection.ai/tool-calling)
- [Rate limits](https://developers.reflection.ai/rate-limits) and [errors](https://developers.reflection.ai/errors)