Synthetic monitoring for MCP dependencies

Trust your MCP dependencies.

MCP hides tool failures inside HTTP 200s. Orchestra calls the servers you depend on, on a schedule, and goes red the moment the answer, shape, speed, or size changes.

$ claude mcp add orchestra -- uvx --from "orchestra-mcp[agent]" orchestra mcp

> Set up drift watches for the MCP servers this project depends on.

Download for macOS Apple Silicon · free

Read the docs

version: 1
name: context7-watch
server:
  transport: http
  url: https://mcp.context7.com/mcp
defaults:
  fail_on: breaking
steps:
  - id: contract
    type: snapshot          # tools still there, same shape
    baseline: baselines/context7.surface.json
    tools: ["query-docs"]
  - id: query_docs
    type: tool_call         # same answer, recorded live
    tool: "query-docs"
    input: { libraryId: "/vercel/next.js" }
    expect: { under_ms: 2700, under_bytes: 50000 }
    response_baseline:
      path: baselines/context7.query-docs.response.json
      ignore: ["$.meta.request_id"]

Written by your agent from real calls. Existence, shape, behavior, latency, and size, one file per dependency, red the day the vendor moves.

Works with any MCP server

stdio HTTP SSE OAuth 2.1

Compatible with Atlassian, Notion, GitHub, and internal servers.

0.2s engine cold start

13 assertion operators, value-level

3 transports: stdio · HTTP · SSE

100% local: tokens never leave your machine

Who this is for

You ship MCP servers, or you depend on them

Same engine, same files, two postures. One gates a pull request. The other watches a vendor you do not control.

Producers

You ship an MCP server. orchestra init records a behavior suite and a CI workflow in minutes, so a pull request that changes the contract goes red with the exact list of what moved.

Consumers

You depend on servers you do not control. When one of them changes, nothing in your stack errors. That is the sharper problem, and it is the one below.

For MCP consumers

Nothing errors. The output just gets worse.

Clients re-read tools/list every session, so a renamed tool or a dropped field is absorbed at runtime. No exception, no 404. The agent starts flailing and quality degrades until a customer notices.

The protocol is built that way: a failed tool returns isError: true inside a normal result. An expired credential, an outage, and a rate limit all arrive as HTTP 200, and your uptime check passes.

Orchestra replays the calls you actually depend on and goes red when one moves, naming the tool, the path, and both values.

Response mismatch vs baselines/search.response.json (1 breaking):
  breaking  value changed: $.content[0].text: "42" -> "41"

Not "something changed". The tool, the path, the old value, the new one.

version: 1
name: vendor-watch
server:
  transport: http
  url: https://mcp.vendor.com/mcp
  auth:
    type: bearer
    token: "{{env.VENDOR_KEY}}"
defaults:
  fail_on: breaking       # reworded docs stay quiet
steps:
  - id: surface
    type: snapshot
    baseline: baselines/contract.surface.json
    tools: [search, get_doc]  # only what you call
  - id: search
    type: tool_call
    tool: "search"
    input: { query: "orchestra mcp testing" }
    response_baseline:
      path: baselines/search.response.json
      ignore:                # masked on both sides
        - "$.meta.request_id"
        - "$.results[*].fetched_at"

name: MCP contract

on:
  schedule:
    - cron: "0 8 * * *"
  workflow_dispatch: {}

jobs:
  mcp:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          uvx --from orchestra-mcp \
            orchestra run tests/mcp \
            --output junit --report-dir reports
        env:
          VENDOR_KEY: ${{ secrets.VENDOR_KEY }}

Both files are generated. No daemon: the CI scheduler is the process that runs forever, and a red run is the alert.

Set one up

$ uvx --from orchestra-mcp orchestra init --vendor https://mcp.vendor.com/mcp

It connects, asks which tools you actually call, records real responses for them, and writes the suite plus the cron workflow. Files are named after the vendor, so several vendors share one suite directory. Use --scope search,get_doc to pin the subset up front, or --name to choose the filename stem yourself.

How it works

Ask once. Watched from then on.

You never write the file. Your agent calls the tools you depend on, keeps what came back, and commits a workflow that replays it every morning.

your terminal

$ claude mcp add orchestra -- uvx --from "orchestra-mcp[agent]" orchestra mcp

✓ Added orchestra · 7 tools

Set up drift watches for the MCP servers this project depends on.

Reading .mcp.json and pipeline code…

Found 2 dependencies: context7, stripe

These 3 tools are called in src/rag/retrieve.ts. Record them?

✓ approved

Live calls, made for real

query-docs 412ms · 18 KB

{ "structuredContent": {
    "library": "/vercel/next.js",
    "total": 42 } }

pinned $.total = 42

baseline recorded, 1 volatile field masked

tests/mcp/context7.yaml

- id: query_docs
  type: tool_call
  tool: "query-docs"
  input: { libraryId: "/vercel/next.js" }
  expect: { under_ms: 2700, under_bytes: 54000 }
  response_baseline:
    path: baselines/context7.query-docs.json
    ignore: ["$.meta.request_id"]
- id: query_docs_check_0
  type: assert
  check: { op: jsonpath_eq, path: "$.total", value: 42 }

Actions mcp-context7.yml · cron 0 8 * * *

✓ Aug 14 · 08:00 UTC 4 steps · 2.1s

✓ Aug 15 · 08:00 UTC 4 steps · 1.9s

✗ Aug 16 · 08:00 UTC 1 breaking

Response mismatch vs baselines/context7.query-docs.json

  breaking  $.total
    baseline: 42
    current:  0

  info  server version: 1.4.2 -> 1.5.0

Nothing errored. The vendor just started returning nothing.

How it works

Four jobs, ordered by time to value

Not by how clever they are. The first two take seconds and need no setup.

01 · seconds

Inspect

Every tool, resource, and prompt with full input schemas. No client restart, no config file, no credentials for local servers.

$ orchestra inspect ./server.yaml

02 · seconds

Call

Invoke one tool with real arguments and print the result. Never prompts, never opens a browser, so it is safe to script and safe for agents.

$ orchestra call pg add -i a:=19 -i b:=23

03 · minutes

Record a suite

Init walks the tools, calls each with input you approve, shows you the response, and asks which fields to pin. Writes the collection, the baselines, and the workflow.

Pin values (y/N each):
$.tempo = 128 [y] pinned
$.elapsed_ms = 42 [n] shape only

04 · ongoing

Know when it changes

First run records, later runs diff. Every difference is labelled, one threshold decides what fails, and accepting a change is deliberate.

$ orchestra run tests/mcp --update-baselines

Two layers, one lifecycle

Behavior is the primary layer: real calls, real inputs, pinned values, expected-error tests, and response baselines that record a tool's whole response and diff every later run. Shape is the supporting layer: the tool-surface snapshot you get with zero input, covering every tool on the server rather than only the ones you recorded.

Both layers behave identically. First run records the baseline. Later runs diff against it. Differences land on one severity ladder, one fail_on threshold decides what fails the build, and --update-baselines is how you accept a change on purpose.

Severity	Behavior layer (response baseline)	Shape layer (surface snapshot)
breaking	a recorded value changed, a field disappeared, a type changed, a list shortened	tool removed, property removed, property now required, type narrowed
additive	a new field appeared, a list grew	tool added, new optional property, property no longer required
cosmetic	not used at this layer	description changed, other schema detail moved
info	only masked paths changed	server name or version changed

fail_on: breaking fails only on real breaks and reports the rest as context. That knob is the answer to "we change descriptions constantly, this will be noisy".

The failure mode that kills value snapshots is nondeterminism: a monitor that flaps gets deleted. Every path in a step's ignore list is masked on both sides before the compare and before the write, so timestamps and request ids never land in git and never wake anyone. The recorder proposes that list for you: it calls the tool twice back to back, and anything that moved between the two calls is volatile by construction.

Agent mode

Your coding agent writes the tests

Orchestra is itself an MCP server. Run orchestra mcp, point a coding agent at it, and the agent can inspect a server, call a tool, and author behavior tests without shelling out.

This is the strongest setup path there is, because the agent reads your pipeline code and finds the real inputs your production system already sends. Say this once:

> Use Orchestra to set up drift watches for the MCP servers this project depends on.

Longer recipe in Watch your MCP dependencies.

orchestra_list_servers     # configured connections
orchestra_inspect          # the tool surface
orchestra_call             # one call, one result
orchestra_record_behavior  # call → a test
orchestra_write_watch      # tests → the watch file
orchestra_check_contract   # diff vs baselines
orchestra_run_collection   # run the suite

{
  "server_ref": "vendor",
  "tool": "search",
  "input": { "query": "orchestra mcp testing" },
  "pin_paths": ["$.structuredContent.total"],
  "response_baseline": {
    "path": "baselines/search.response.json",
    "collection_dir": "tests/mcp"
  }
}

Checks derived from the live response, then written to the two files it names. Nothing else on disk is touched.

Pins what you name

Each path in pin_paths becomes a jsonpath_eq check carrying the live value. Paths it cannot pin come back with a reason instead of failing the call, and it lists the other candidates it saw.

Finds the volatile fields

Ask for a response baseline and it calls the tool a second time back to back. Whatever differs between the two becomes the ignore list, so the baseline does not flap on its first scheduled run.

Says what it will do

It calls the tool for real, twice when a baseline is requested. A tool that does not declare readOnlyHint: true may change state, and the tool description says so before the agent fires it.

For MCP producers

From a recorded call to a red pull request

A test written in the same commit as a change agrees with the change by construction. To report that a change is breaking you need an artifact captured before it. That committed artifact is the whole idea: plain YAML and JSON in your repo, diffed and reviewed like any other file.

{
  "version": 1,
  "tool": "search",
  "response": {
    "content": [
      { "type": "text", "text": "42" }
    ],
    "structuredContent": { "total": 42 },
    "meta": { "request_id": "__ignored__" },
    "isError": false
  }
}

name: MCP contract

on: [pull_request]

jobs:
  mcp:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - run: |
          uvx --from orchestra-mcp \
            orchestra run tests/mcp \
            --output junit --report-dir reports
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: mcp-reports
          path: reports/

ci · orchestra run

$ orchestra run tests/mcp --output junit

📄 Loading collection: tests/mcp/contract.yaml

✓ snapshot surface · no drift

✓ call search · 214ms

✓ response baseline search · no drift

✓ assert $.structuredContent.total == 42

🧪 JUnit report: reports/junit.xml

Suite: 1/1 collections passed · exit 0

JUnit XML for your CI provider, plus a summary JSON with severity counts per collection, so a scheduled multi-server run can route the alert to whoever owns the failing server.

Demo

From live response to regression test

Connect to Linear's MCP server, call a tool, pin the live response as an assertion, and run the collection. No setup, no YAML written by hand.

Features

Built for MCP, not adapted from REST

Every feature maps to a real gap in how teams test Model Context Protocol servers today.

Value-level assertions

Thirteen operators, including equality, numeric comparison, substring, regex, and length checks against any JSONPath in the response.

Response baselines

Record a tool's whole response, diff every later run, mask the fields that move. The behavior twin of the surface snapshot, same lifecycle.

Expected-error tests

A tool that returns an error can be recorded as an expected-error test, so "this still rejects bad input" is part of the contract too.

Git-native workspaces

Collections, baselines, and connections are plain YAML and JSON you commit, diff, and review. Nothing lives in a cloud account.

Auth for real servers

Bearer tokens, API-key headers, and basic auth read from environment variables, never written into files. OAuth 2.1 sign-in lives behind orchestra auth login.

Enterprise networks

Corporate proxies, internal CAs, and custom gateway headers supported out of the box.

The landscape

How Orchestra stacks up

Great tools exist for poking at an MCP server. Orchestra is where the poke becomes a test that keeps running.

Capability	MCP Inspector	Postman	MCP CLIs	Orchestra
Browse & call tools in a UI	✓	✓	◐	✓
Scriptable CLI calls	✓	◐	✓	✓
Save a call as a rerunnable test	◐	◐	✕	✓
Assertions on responses	◐	✕	◐	✓
CI runs with JUnit output	◐	✕	◐	✓
Contract baseline, severity-labeled diff	✕	✕	◐	✓
Watch mode for the dev loop	✕	✕	✕	✓
OAuth 2.1 sign-in	✓	✓	✓	✓

✓ supported · ◐ partial · ✕ not available. Verified against each tool's own documentation, August 2026. "MCP CLIs" is the best of mcpc and mcp-cli combined.

Why Orchestra

Built the way platform teams expect

Local-first

Tokens and server data stay on your machine. No cloud account required to test.

One command to start

No install step. uvx --from orchestra-mcp orchestra init scaffolds a suite and a workflow in the repo you are standing in.

Open workflow

Plain YAML files, no lock-in. Your collections live in git alongside your server code.

Find out before your customers do.

Record what your tools return today, and get told the morning it changes.

Download for macOS Apple Silicon · free

Quickstart Contract testing Contact

Contact

Get in touch

Questions, feedback, or interested in Orchestra for your team? Send a note and I'll get back to you.