SDK reference

npm install @resslai/eval

import { EvalWorker, type EvalTask, type McpConfig } from "@resslai/eval";

EvalWorker

await new EvalWorker(options).start();

start() resolves once this worker’s run has no tasks left. Call stop() to finish the current work and stop pulling tasks.

Option	Default	Description
`baseUrl`	required	Ressl API URL, such as `https://simulation.ressl.ai`
`apiKey`	required	Organization API key beginning with `rsk_`
`runId`	required	The run to drain — the `runId` returned when the run was triggered
`model`	required	Model identifier used by your agent
`runAgent`	required	Async function that runs your agent for one task
`concurrency`	`1`	Maximum number of tasks to run at the same time
`pollIntervalMs`	`1000`	Delay between empty-queue polls, in milliseconds
`onEvent`	none	Callback for worker events

The model value must match the model used inside runAgent. Ressl uses this value to compare runs that use the same dataset and model. The constructor throws if model or runId is empty.

EvalTask

Ressl passes this object to runAgent:

interface EvalTask {
  runId: string;
  taskResultId: string;
  taskId: string;
  ticket: string;
  mcp: McpConfig;
}

interface McpConfig {
  type: "http";
  url: string;
  headers?: Record<string, string>;
}

Field	Description
`runId`	Evaluation run that contains the task
`taskResultId`	Identifier used by the worker to report completion
`taskId`	Task identifier in the dataset
`ticket`	Prompt to give your agent
`mcp`	MCP server for the task’s isolated mock world

Add task.mcp to the MCP configuration used for that agent call:

mcpServers: {
  evalmock: task.mcp,
}

Tasks use separate mock worlds, so they can run in parallel. If your organization reaches its world limit, the worker waits and retries.

Events

Use onEvent to log or monitor worker activity:

Event	When
`task`	The worker pulled a task
`reported`	The worker reported task completion
`idle`	The queue was empty on this poll
`drained`	All worker loops stopped and `start()` is resolving
`error`	A request failed and may be retried

onEvent: (event, data) => {
  console.log(event, data);
}

The task event includes { runId, taskResultId, taskId }. The reported event includes { taskResultId, error? }. Error data includes the failed operation and either an error message or HTTP status.

Failure behavior

The worker retries temporary request failures with exponential backoff, up to 30 seconds between attempts.
A 401 or 403 response stops the worker.
The worker tries up to three times to report task completion.
If runAgent throws, the worker reports the error. Ressl still grades the state left in the mock world.

Trigger API

The included GitHub Actions workflow uses this endpoint to start a run.

POST https://simulation.ressl.ai/api/v1/evals
Authorization: Bearer rsk_...

{
  "commit": "1234567",
  "dataset": "itbench",
  "ref": "refs/heads/main",
  "tasks": ["task-c2"]
}

Field	Required	Meaning
`commit`	yes	Git SHA to evaluate
`dataset`	yes	Dataset slug
`ref`	no	Git ref to display with the run
`tasks`	no	Task IDs to run; omit this field to run the full dataset

{ "runId": "…", "status": "running", "taskCount": 89 }

The API key determines the organization. The endpoint returns 404 if the dataset is unavailable or the task filter has no matches.

Run status API

Check the run until status is succeeded or failed:

GET https://simulation.ressl.ai/api/v1/evals/{runId}
Authorization: Bearer rsk_...

{
  "runId": "…",
  "commit": "1234567",
  "dataset": "itbench",
  "model": "claude-sonnet-5",
  "status": "succeeded",
  "resolved": 71,
  "total": 89,
  "assertions_passed": 402,
  "assertions_counted": 431,
  "conclusion": "success",
  "regression": { "baselineCommit": "7ab12c3" },
  "comment_markdown": "## Ressl eval...",
  "tasks": [
    {
      "task": "task-c2",
      "status": "graded",
      "reward": 1,
      "assertions_passed": 6,
      "assertions_counted": 6,
      "error": null
    }
  ]
}

Field	Meaning
`status`	`running`, `succeeded`, or `failed`
`conclusion`	`success`, `failure`, or `neutral` when no baseline exists
`resolved` / `total`	Fully resolved tasks and total tasks
`assertions_passed` / `assertions_counted`	Passed and total assertions
`regression`	Comparison with the baseline run, or `null`
`comment_markdown`	Markdown summary used by the CI workflow
`tasks`	Per-task status, reward, assertions, and error

Get Started

Providers

Continuous evals

SDK reference

EvalWorker

EvalTask

Events

Failure behavior

Trigger API

Run status API

​EvalWorker

​EvalTask

​Events

​Failure behavior

​Trigger API

​Run status API

EvalWorker

EvalTask

Events

Failure behavior

Trigger API

Run status API