Evals quickstart

1. Install

npm install @resslai/eval

Requires Node 20+. The example below uses TypeScript. Install tsx if your project does not already have a TypeScript runner:

npm install --save-dev tsx

2. Create a worker

Create a file that runs your agent for each evaluation task. Pass task.ticket as the prompt and add task.mcp to your agent’s MCP servers. This example uses the Claude Agent SDK. The same pattern works with any agent that accepts an MCP server configuration.

agent.ts

import { EvalWorker } from "@resslai/eval";
import { query } from "@anthropic-ai/claude-agent-sdk";

const MODEL = "your-model-id";

await new EvalWorker({
  baseUrl: process.env.RESSL_EVAL_URL!,
  apiKey: process.env.RESSL_KEY!,
  runId: process.env.RESSL_RUN_ID!,
  model: MODEL,
  concurrency: 4,
  runAgent: async (task) => {
    for await (const _ of query({
      prompt: task.ticket,
      options: {
        model: MODEL,
        mcpServers: { evalmock: task.mcp },
        allowedTools: ["mcp__evalmock__*"],
      },
    })) {
      // Consume the stream until the agent finishes.
    }
  },
}).start();

Replace your-model-id with the model used by your agent. The model option is a label for grouping comparable runs, so it must match the value passed to your agent. Add a start script so CI can run the worker:

package.json

{
  "scripts": {
    "start": "tsx agent.ts"
  }
}

runAgent does not return a score. Ressl grades the final state of the mock world after the function finishes.

3. Copy the dataset slug

Open the Datasets tab in the console, pick a dataset, and click the copy icon next to its name. Use the copied slug in RESSL_DATASET. A display name and slug may be different.

4. Add the workflow

The SDK package includes a GitHub Actions workflow. Copy it into your repository:

mkdir -p .github/workflows
cp node_modules/@resslai/eval/ci/ressl-eval.yml .github/workflows/

On each push to main, the workflow:

Starts an evaluation for the current commit.
Runs your worker with npm start.
Waits for the worker to finish the task queue.
Posts the result as a commit comment and a GitHub check.

Add these values under Settings → Secrets and variables → Actions:

Name	Kind	Value
`RESSL_API_KEY`	secret	`rsk_…` from the console
`ANTHROPIC_API_KEY`	secret	API key used by the example agent
`RESSL_EVAL_URL`	variable	`https://simulation.ressl.ai`
`RESSL_DATASET`	variable	Dataset slug from step 3
`RESSL_TASKS`	variable	Optional comma-separated task IDs

The included workflow is configured for the Claude example above. If your agent uses another provider, update the workflow to pass the required credentials.

Set RESSL_TASKS to a few task IDs for quick checks on each push. Leave it empty to run the full dataset.

5. Run an evaluation locally

Start an evaluation before running the worker. If the queue is empty, the worker exits.

curl -sS -X POST https://simulation.ressl.ai/api/v1/evals \
  -H "Authorization: Bearer rsk_..." \
  -H "content-type: application/json" \
  -d '{"commit":"'"$(git rev-parse HEAD)"'","dataset":"itbench","tasks":["task-c2"]}'

{ "runId": "…", "status": "running", "taskCount": 1 }

Then drain it with your worker:

export RESSL_EVAL_URL="https://simulation.ressl.ai"
export RESSL_KEY="rsk_..."
npm start

You can follow the run from the Runs tab in the console.

Get Started

Providers

Continuous evals

Evals quickstart

1. Install

2. Create a worker

3. Copy the dataset slug

4. Add the workflow

5. Run an evaluation locally

​1. Install

​2. Create a worker

​3. Copy the dataset slug

​4. Add the workflow

​5. Run an evaluation locally

1. Install

2. Create a worker

3. Copy the dataset slug

4. Add the workflow

5. Run an evaluation locally