# E3 Rollouts — integration guide

## Overview

E3 Rollouts is a shared home for experiment percentages and JSON configuration. Save a value in the [dashboard](https://rollouts.e3g.ai/), read it in your service, and apply it to new calls or requests. You can make the same edits through [Cosmos MCP](https://cosmos.e3g.ai/mcp).

**Saving configuration does not route traffic by itself.** Your service owns refreshing the value, selecting a variant, and keeping a fallback.

## Quickstart

Read the existing Ender configuration. This request does not change anything.

```sh
curl --fail --silent --show-error \
  https://rollouts.e3g.ai/v1/entries/ender.stt.production
```

The response includes the configuration and its revision. These are example values; read the endpoint for the current revision.

```json
{
  "key": "ender.stt.production",
  "kind": "rollout",
  "value": {"flux": 85, "qwen": 15},
  "revision": 4,
  "updated_at": "2026-09-24T06:43:00+00:00",
  "note": "Snapshot; not connected to live traffic"
}
```

Use `value` as your configuration and `revision` to identify which version your service used. Continue with [code examples](#code-examples) or the [consumer contract](#consumer-contract).

> **Canonical domain:** use `https://rollouts.e3g.ai` for the dashboard, API, and direct MCP. Python's default User-Agent is blocked by the current Cloudflare rule; the Python example identifies itself as `E3Rollouts/1.0`, verified to work. See [operations and limits](#operations-and-limits) if a client cannot set headers.

## Code examples

### Python

Uses only the standard library. Run this from startup or a background refresh, outside the audio processing path.

```python
import json
from urllib.request import Request, urlopen

url = (
    'https://rollouts.e3g.ai'
    '/v1/entries/ender.stt.production'
)
request = Request(url, headers={'User-Agent': 'E3Rollouts/1.0'})
with urlopen(request, timeout=1) as response:
    record = json.load(response)
assert record['kind'] == 'rollout'
weights = record['value']
revision = record['revision']
```

This minimal snippet raises on failure. The [reference implementation](#reference-implementation) includes caching, validation, timeout, and fallback behavior.

### JavaScript · Node.js

```javascript
const response = await fetch(
  'https://rollouts.e3g.ai/v1/entries/ender.stt.production',
  { signal: AbortSignal.timeout(1000) }
);
if (!response.ok) throw new Error(`Rollouts HTTP ${response.status}`);
const record = await response.json();
console.log(record.value, record.revision);
```

## Create and update

Writes persist immediately. Only run this when a configuration change is intended. This creates an illustrative key, not an active negotiation experiment:

```sh
curl --fail-with-body -X PUT \
  https://rollouts.e3g.ai/v1/entries/example.my-experiment \
  -H 'Content-Type: application/json' \
  --data '{"kind":"rollout","value":{"control":85,"experiment":15},"expected_revision":0,"note":"Initial example"}'
```

Use expected_revision 0 only for a new key. For an existing key, GET it first and use its exact revision in the PUT body. A successful write increments revision and records history atomically. HTTP 409 means another edit won: reread and review instead of blindly retrying. HTTP 422 means invalid key/value/shape; 413 means request too large; 404 means key missing; 503 means storage unavailable. Revert by saving a historical value against the current revision. No delete endpoint exists.

## API reference

Base URL: [rollouts.e3g.ai](https://rollouts.e3g.ai/). Reads and writes currently require no service key. Keep secrets and caller data out of this store.

An entry has `key`, `kind` (`json` or `rollout`), `value`, `revision`, `updated_at`, and `note`.

| Method | Path | Purpose |
| --- | --- | --- |
| GET | [/healthz](/healthz) | Database readiness |
| GET | [/v1/entries](/v1/entries) | List entries |
| GET | [/v1/entries/{key}](/v1/entries/ender.stt.production) | Read value/revision; link opens the Ender example |
| PUT | `/v1/entries/{key}` | [Create/update](#create-and-update) |
| GET | [/v1/entries/{key}/history](/v1/entries/ender.stt.production/history) | Last 100 revisions; link opens the Ender example |
| GET | [/docs](/docs) | This guide |
| GET | [/docs.md](/docs.md) | Markdown guide |
| POST | `/mcp` | [Streamable HTTP setup](#cosmos-and-mcp) |

### Validation and errors

Keys and variant names allow letters, digits, dots, underscores and dashes, begin with a letter or digit, and are at most 128 characters. Rollout values contain 1–32 integer percentages from 0 to 100 totaling exactly 100. Generic JSON must be finite and at most 48KB; request bodies are limited to 64KB.

| Status | Meaning | Action |
| --- | --- | --- |
| 404 | Missing key | Check the key; use your fallback |
| 409 | Stale revision | Reread and review before retrying |
| 413 | Body too large | Keep the request below 64KB |
| 422 | Invalid value | Check kind, key, and percentage totals |
| 503 | Storage unavailable | Keep last-known-good configuration |

## Cosmos and MCP

Connect through [Cosmos](https://cosmos.e3g.ai/mcp), using your normal Cosmos authentication. Start with `documentation`, locate E3 Rollouts, and call `rollouts__get_docs` for this guide.

| Cosmos tool | Arguments |
| --- | --- |
| rollouts__get_docs | {} |
| rollouts__list_entries | {} |
| rollouts__read_entry | {"key":"ender.stt.production"} |
| rollouts__read_history | {"key":"ender.stt.production"} |
| rollouts__update_entry | {"key":"example.my-experiment","kind":"rollout","value":{"control":85,"experiment":15},"expected_revision":0,"note":"Initial example"} |

The update example is a persistent mutation, not a read-only test.

### Direct connection

Use the [direct MCP endpoint](https://rollouts.e3g.ai/mcp) with a **Streamable HTTP MCP client**. It is a protocol endpoint, not a webpage. Direct tools use the same names without `rollouts__`; no service key is needed. Refresh your client's tool list after connecting.

```text
Transport: Streamable HTTP
URL: https://rollouts.e3g.ai/mcp
Authentication: none
Read tool: read_entry
Arguments: {"key": "ender.stt.production"}
```

## Consumer contract

Fetch outside the audio/turn critical path. Validate the expected key, kind, variants and revision. Cache in each service; refresh periodically with a bounded timeout. Keep last-known-good on failure and use the existing environment default on a cold start. Freeze a decision for the lifetime of a call/request and record the key/revision/source with telemetry. A process-local cache is lost on restart; durable cache and fleet propagation policies require explicit consumer implementation.

Percentage weights alone do not assign traffic. Consumers own stable hashing, subject IDs, variant ordering, fallback and experiment isolation. Changing these can reshuffle assignments. Verify actual traffic independently of stored weights.

## Reference implementation

> **Don't touch — example only.** [Negotiation draft PR #1059](https://github.com/e3-solutions/negotiation/pull/1059) is a reference. Do not merge, deploy, or enable it.

The draft includes a [cached Python consumer](https://github.com/e3-solutions/negotiation/blob/codex/cor-4421-rollouts-example/voice-agent/examples/rollouts_consumer.py), [tests](https://github.com/e3-solutions/negotiation/blob/codex/cor-4421-rollouts-example/voice-agent/examples/test_rollouts_consumer.py), and [integration notes](https://github.com/e3-solutions/negotiation/blob/codex/cor-4421-rollouts-example/voice-agent/examples/README.md). It is not wired into negotiation's runtime and does not change Railway variables or call routing. GitHub access to the private repository is required.

### Negotiation versus Ender

Negotiation's `VOICE_AGENT_STT_PROVIDER` supports `deepgram` or `proxy`. Ender independently owns upstream transcriber/turn-detector selection behind that proxy. `ender.stt.production` (`flux`/`qwen`) is a snapshot and must not be interpreted as negotiation's provider selector. The draft proposes a separate JSON key `negotiation.stt.provider.production` with `{"provider":"proxy"}`; that key is not created or activated. Keep proxy secrets, endpoints and timeouts in the deployment secret store.

## Scheduled ramps

Select an existing key in the [dashboard](https://rollouts.e3g.ai/). Its allocation and **Schedule** appear together. Edit **Every**, the service and direction, **Points / step**, and **Stop when**. The paired service automatically gets the opposite change; the preview shows both services now, after the next step, and at the final limit. With three or more services, the preview lists the shares that stay unchanged. A target cannot exceed the selected pair's combined share.

**One coordinated rule per key** is intentional for migrating traffic between an incumbent and a new service. For example: Qwen +5 points each hour, Flux −5, stopping at Qwen 25%. It moves 15/85 → 20/80 → 25/75, then stops. Review the canary results, raise the limit to 50% or 100%, and save again. To drain the new service instead, choose Qwen −5 until 0%; Flux returns to 100%. Other variants under the key stay unchanged.

To change an existing schedule, select its key, edit the loaded fields, and choose **Save schedule changes**. Saving resets the next interval from now. **Pause** preserves the current allocation; **Save & resume** starts from the current percentages. Completed schedules can be extended by raising the limit and saving again.

Allocation percentages are directly editable beside the pie chart. With exactly two services, editing either percentage balances the other to 100%. With more services, set the percentages to total 100%. The chart previews unsaved values. **Save changes** at the top right saves the allocation and pauses any active schedule; it is disabled until the configuration differs from the saved version and is valid. Restoring the original values disables it again. **Discard changes** restores the saved configuration. Save or discard allocation edits before configuring a schedule. Key metadata and the change note are under **Key settings & change note**.

- Common intervals: **5, 15, 30 minutes; 1 hour; 24 hours**. Enter the amount and choose minutes or hours.
- Custom intervals use **minutes or whole hours**, with a minimum of 5 minutes and increments of 5 minutes. Maximum: 365 days.
- The first step happens after the interval, on a five-minute scheduler check. Railway may run late. This is elapsed-time scheduling, not a local-time daily appointment.
- Upward and downward ramps both stop exactly at the target. Other variants are unchanged; the sum remains 100%.
- **Pause** keeps the current percentages. **Save & resume** starts a new interval from now.
- Manual configuration edits automatically pause an active schedule. Refresh before replacing a schedule that another person or cron has changed.
- Schedules persist across deployments. After downtime, at most one step runs; missed steps are not applied in a burst.
- Each executed step writes a new configuration revision and history note in the same transaction. Duplicate cron requests cannot apply the same due step twice.

Each due step also requires a fresh good [feedback heartbeat](#feedback-heartbeats). The service does not calculate success rates or latency itself; callers decide whether their observed health is good or failed. Consumer refresh latency still applies.

### REST and MCP

| Action | HTTP | Cosmos tool |
| --- | --- | --- |
| List and check scheduler heartbeat | `GET /v1/schedules` | `rollouts__get_schedules` |
| Read a schedule | `GET /v1/entries/{key}/schedule` | `rollouts__get_schedule` |
| Start, replace, or resume | `PUT /v1/entries/{key}/schedule` | `rollouts__set_schedule` |
| Pause | `POST /v1/entries/{key}/schedule/pause` | `rollouts__stop_schedule` |

Read the entry and schedule first. `expected_revision` is the schedule revision (0 if none); `expected_entry_revision` is the current configuration revision. Start/replace body:

```json
{
  "variant": "qwen",
  "counterpart": "flux",
  "step": 5,
  "target": 50,
  "interval_minutes": 15,
  "heartbeat_window_hours": 24,
  "expected_revision": 0,
  "expected_entry_revision": 4
}
```

These revisions are illustrative. Saving this body starts an automatic ramp: use fresh revisions and an explicitly intended target. MCP takes the same fields plus `key`. Pause takes `expected_revision` (and `key` for MCP). Invalid intervals return 422; stale revisions return 409. No existing production ramp is enabled by deployment alone.

## Feedback heartbeats

Every schedule owns a unique identifier and a generic endpoint: `POST https://rollouts.e3g.ai/heartbeat/{identifier}`. Save a schedule, then copy its URL from **Heartbeat endpoint**, or read `heartbeat_id` from `GET /v1/entries/{key}/schedule`. Identifiers stay unchanged when editing or resuming that schedule; different keys get different identifiers. Unknown identifiers return 404 instead of silently creating a disconnected signal.

Send `{"status":"good"}` after a successful health assessment, or `{"status":"fail"}` when the service is unstable. Use one authoritative reporter per identifier in this first version; receipt order determines the latest status, so independent reporters can overwrite each other. A process merely being alive is not evidence of good service performance.

```bash
# Replace IDENTIFIER with heartbeat_id from the saved schedule.
curl --fail-with-body -X POST https://rollouts.e3g.ai/heartbeat/IDENTIFIER \
  -H 'Content-Type: application/json' \
  -d '{"status":"good"}'

curl --fail-with-body -X POST https://rollouts.e3g.ai/heartbeat/IDENTIFIER \
  -H 'Content-Type: application/json' \
  -d '{"status":"fail"}'
```

```python
import json
from urllib.request import Request, urlopen

def send_rollout_health(identifier: str, healthy: bool):
    request = Request(
        f"https://rollouts.e3g.ai/heartbeat/{identifier}",
        data=json.dumps({"status": "good" if healthy else "fail"}).encode(),
        headers={"Content-Type": "application/json", "User-Agent": "E3Rollouts/1.0"},
        method="POST",
    )
    with urlopen(request, timeout=5) as response:
        return json.load(response)

# Pass your actual health assessment; do not send unconditional good signals.
# send_rollout_health(identifier, healthy=measured_health_is_good)
```

```javascript
async function sendRolloutHealth(identifier, healthy) {
  const response = await fetch(
    `https://rollouts.e3g.ai/heartbeat/${encodeURIComponent(identifier)}`,
    { method: 'POST', headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ status: healthy ? 'good' : 'fail' }),
      signal: AbortSignal.timeout(5000) });
  if (!response.ok) throw new Error(`Heartbeat failed: ${response.status}`);
  return response.json();
}
```

**Rule:** at a due cron step, the latest signal must be `good` and its server receipt time must be less than `heartbeat_window_hours` old. Default: 24 hours; configurable from 1 to 8760 whole hours. A newer fail overrides an earlier good. Missing, failed, expired, or future-dated server records hold the allocation. Only server timestamps are accepted.

Holding does not reverse traffic, pause the schedule, or write a configuration revision. The scheduler records the hold reason and advances to the next regular interval. A later good signal allows the next due step; it does not immediately change traffic or replay missed steps. One good signal may authorize several steps until it expires. This is a rolling freshness window, not a requirement for a new signal after every step. Choose a reporting cadence comfortably shorter than your window.

Existing schedules acquire identifiers and a 24-hour window on deployment; they hold until their first good signal. A failure after a completed step does not undo that step. A completed schedule stays completed. The first version holds on instability; automatic reverse steps and richer health policies are not implemented.

`GET /heartbeat/{identifier}` returns `stable`, `state` (`good`, `fail`, `missing`, or `stale`), `received_at` (Unix seconds), and the window. Schedule reads include the same feedback state. Only the latest signal is retained; this is not a health-event archive. The endpoint has the same open access as the configuration API: identifiers are routing identifiers, not credentials.

MCP exposes `get_heartbeat(identifier)` and `send_heartbeat(identifier, status)`; Cosmos uses the `rollouts__` prefix. `set_schedule` accepts `heartbeat_window_hours`. Tools should report measured health only, never invent a good signal to advance a rollout.

## Operations and limits

One Railway instance uses SQLite on /data. Values survive redeploys; history shares the same database. Brief redeploy downtime is possible. Backups are not configured, and history is not an independent backup. A separate Railway cron worker checks due schedules every five minutes; the UI shows its last check-in. The private tick endpoint requires a machine token, while the dashboard and configuration API remain open. No per-user audit attribution, routing engine, automatic consumer integration, SDK or automatic fallback exists in the server. Production adoption must explicitly address the currently public write access.

## Links and resources

| Resource | Link |
| --- | --- |
| Source repository | [e3-solutions/rollouts](https://github.com/e3-solutions/rollouts) · E3 organization access |
| Rollout editor | [Open dashboard](https://rollouts.e3g.ai/) |
| Shareable documentation | [Integration guide](https://rollouts.e3g.ai/docs) |
| Source for agents | [Markdown guide](https://rollouts.e3g.ai/docs.md) |
| Cosmos MCP | [Cosmos endpoint](https://cosmos.e3g.ai/mcp) |
| Direct MCP | [Rollouts endpoint](https://rollouts.e3g.ai/mcp) |
| Example integration | [Draft PR #1059 — Don't touch](https://github.com/e3-solutions/negotiation/pull/1059) |
| Implementation tracking | [COR-4421](https://linear.app/coreedgesolution/issue/COR-4421) |
| E3 design language | [Doctrine](https://doctrine.e3g.ai/) |

### Client troubleshooting

The Railway hostname `https://rollouts-production.up.railway.app` remains a fallback for clients whose User-Agent is blocked by Cloudflare and cannot be configured. Cosmos currently uses that transport internally. All public links and examples use the canonical `rollouts.e3g.ai` domain. MCP endpoints require an MCP client, not a browser GET.