---
name: ressl
description: >
  Create and call hosted mock SaaS APIs (any granted provider) via the Ressl
  platform API, and run continuous evals that score an agent against those mock
  worlds on every commit. Use when an agent or eval needs a live HTTPS mock of a
  SaaS product without a real tenant, or when wiring an agent harness to Ressl’s
  eval CI.
license: MIT
compatibility: Requires an org API key (rsk_…) from https://simulation.ressl.ai Settings.
metadata:
  author: ressl
  version: "1.1"
  control_plane: "https://simulation.ressl.ai"
  mock_host_pattern: "https://{snapshotId}.{slug}.mock.ressl.cc"
---

# Ressl

Two surfaces, one API key:

| Surface | Use when |
|---------|----------|
| **Snapshots** | You need a hosted mock of a SaaS HTTP API to point an agent at |
| **Continuous evals** | You want an agent scored against mock worlds on every commit, with regressions failing the build |

## Do not confuse hosts

| Purpose | Base |
|---------|------|
| Control API (list, create, evals) | `https://simulation.ressl.ai` |
| Mock traffic | `https://{snapshotId}.{slug}.mock.ressl.cc` |

Never invent mock hosts under `ressl.ai`. Mocks live on `*.mock.ressl.cc`.

## Auth

1. Human creates an API key in the console: Settings → API keys (`rsk_…`).
2. Send on every control-plane request:

```http
Authorization: Bearer rsk_...
```

Also accepted: `X-API-Key: rsk_...`.

Keys are org-scoped. Invalid/revoked → `401`. The organization is always taken
from the key, never from a request body.

---

# Snapshots

## When to use

- You need a **hosted mock** of a SaaS HTTP API for an agent, harness, or eval.
- The org already has access to that provider on Ressl.
- You want a normal HTTPS base URL that expires after a TTL.

## Workflow

### 1. List providers

```http
GET https://simulation.ressl.ai/providers/list
Authorization: Bearer rsk_...
```

Response:

```json
{ "providers": ["jira", "salesforce", "slack"] }
```

Use only slugs from this list.

### 2. Create a snapshot

```http
POST https://simulation.ressl.ai/providers/{slug}/create-snapshot
Authorization: Bearer rsk_...
Content-Type: application/json
```

Body (all optional):

```json
{
  "ttl": "1h",
  "seed": { "<slug>": { } },
  "config": {}
}
```

- `ttl`: `15m` / `1h` / `1d` style; default `1h`; max `7d`.
- `seed`: initial data; prefer `{ "<slug>": { … } }`. Bare objects are auto-wrapped. Empty/default is `{ "<slug>": {} }`.
- `config`: provider overrides. For **salesforce**, describe JSON that **fully replaces** baked metadata; omit to keep baked. Other slugs reject `config` today. Details: docs site `/providers/salesforce`.

Provisioning may take 1–2 minutes.

Success includes:

- `snapshotId`, `slug`, `url`, `expiresAt`, `ttl`

Errors: `400` bad input, `403` no grant, `502` provision failure.

### 3. Call the mock

Use `url` as the HTTP base. Paths match that provider’s real API. Snapshot URLs are public but unguessable until `expiresAt`.

## Constraints

- One provider per snapshot (`EMULATOR_TOOLS` = single slug).
- Org must be granted the slug (`403` otherwise).
- No list/delete snapshot HTTP APIs in v1 — TTL tears the sandbox down.
- Salesforce `config` replaces describes for that snapshot only; see `/providers/salesforce` on the docs site for seed/config templates.

## Agent tips

- Always call `/providers/list` before create if the allowed set is unknown.
- Persist `url` + `expiresAt` for the run; do not hardcode hosts.
- On `502`, retry create once after a short wait; surface `detail` if present.

---

# Continuous evals

## When to use

- A customer harness should be scored on every push against a **dataset** of
  graded tasks, with regressions failing the commit check.
- You are wiring `@resslai/eval` into an existing agent, or debugging a run.

Do NOT create snapshots for this. Eval tasks come with their own mock world; the
worker receives it per task.

## Model

- A **dataset** is a set of graded tasks the org owns, referenced by its
  **slug** (from the console’s Datasets tab — the slug is not the display name,
  e.g. *ITSMBench* is `itbench`).
- A **run** is one commit evaluated against one dataset.
- Each task is dispatched to a worker with its own MCP mock world. Grading reads
  the world’s final state; nothing the worker reports is trusted as a score.
- Metrics: **resolution** (tasks scored 1, out of the tasks in the run) and
  **assertions** (assertion pass fraction). Each run is compared to the previous
  run on the same **dataset + model**; either dropping is a failure.

## Workflow

### 1. Write the worker

```ts
import { EvalWorker } from "@resslai/eval";   // npm install @resslai/eval (Node 20+)

new EvalWorker({
  baseUrl: process.env.RESSL_EVAL_URL,   // https://simulation.ressl.ai
  apiKey: process.env.RESSL_KEY,         // rsk_…
  runId: process.env.RESSL_RUN_ID,       // the run being drained — from the trigger response
  model: MODEL,                          // the SAME value the agent runs — labels the run
  concurrency: 4,                        // tasks are isolated; parallel is safe
  runAgent: async (task) => {
    // The caller's own agent, unchanged, plus the injected per-task world:
    //   mcpServers: { evalmock: task.mcp }
    // Drain the agent to completion. Return nothing.
  },
}).start();                              // resolves when THIS run is drained
```

`task` is `{ runId, taskResultId, taskId, ticket, mcp }`. Give `ticket` to the
agent as the prompt; bind `mcp` as an MCP server inside the process.

`model` is required and validated — an empty value throws. It must be the same
variable the harness hands its agent, or runs are mislabeled and baselines
compare the wrong things.

### 2. Trigger a run

```http
POST https://simulation.ressl.ai/api/v1/evals
Authorization: Bearer rsk_...
Content-Type: application/json
```

```json
{ "commit": "1234567", "dataset": "itbench", "ref": "refs/heads/main", "tasks": ["task-c2"] }
```

- `commit` (required): the git SHA being evaluated (7–64 hex chars).
- `dataset` (required): the dataset slug.
- `ref` (optional): git ref, shown on the run.
- `tasks` (optional): task ids for a smoke subset; omit to run the whole dataset.

Returns `{ runId, status, taskCount }`. A dataset the org does not own → `404`.

### 3. Start the worker, then read the verdict

Order matters: trigger first, **then** start the worker — it needs the `runId`
from the trigger response, and a worker whose run has no tasks exits immediately.

Once it has drained:

```http
GET https://simulation.ressl.ai/api/v1/evals/{runId}
Authorization: Bearer rsk_...
```

Returns `status` (`running` | `succeeded` | `failed`), `conclusion`
(`success` = no regression, `failure` = a metric dropped, `neutral` = baseline,
no prior run), and `comment_markdown` — the report to post on the commit.

## Constraints

- The worker dials out only; nothing inbound is exposed and model keys stay with
  the caller.
- CI never specifies the model — the worker’s `model` labels the run.
- `runAgent` returns nothing; scoring reads the world, and tool calls are logged
  server-side at the MCP gateway.
- If `runAgent` throws, the error is recorded and the task is **still graded**
  against whatever state the agent left.
- `401`/`403` is fatal to the worker; transient errors back off to 30s.

## Agent tips

- `@resslai/eval` ships the CI workflow: copy `node_modules/@resslai/eval/ci/ressl-eval.yml`
  into `.github/workflows/`, then set `RESSL_API_KEY` + `ANTHROPIC_API_KEY`
  (secrets) and `RESSL_EVAL_URL`, `RESSL_DATASET`, optional `RESSL_TASKS` (vars).
- A full dataset takes hours; use `tasks`/`RESSL_TASKS` for per-push smoke runs
  and the whole dataset nightly.
- Never guess a dataset slug — read it from the console’s Datasets tab.