From b5924fd6b78eddee25a74cc405ee6a4ed1767c88 Mon Sep 17 00:00:00 2001 From: Paul Carleton Date: Fri, 23 Jan 2026 11:30:15 +0000 Subject: [PATCH 1/2] docs: add SDK integration guide and fix spec references - Add SDK_INTEGRATION.md with guide for SDK maintainers on integrating conformance tests - Link to guide from README.md - Update spec references to use 2025-11-25 instead of draft URLs --- README.md | 2 + SDK_INTEGRATION.md | 206 +++++++++++++++++++ src/scenarios/client/auth/spec-references.ts | 16 +- 3 files changed, 216 insertions(+), 8 deletions(-) create mode 100644 SDK_INTEGRATION.md diff --git a/README.md b/README.md index bbc3c8c..da649d5 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,8 @@ A framework for testing MCP (Model Context Protocol) client and server implement > [!WARNING] > This repository is a work in progress and is unstable. Join the conversation in the #conformance-testing-wg in the MCP Contributors discord. +**For SDK maintainers:** See [SDK Integration Guide](./SDK_INTEGRATION.md) for a streamlined guide on integrating conformance tests into your SDK repository. + ## Quick Start ### Testing Clients diff --git a/SDK_INTEGRATION.md b/SDK_INTEGRATION.md new file mode 100644 index 0000000..7df2e6e --- /dev/null +++ b/SDK_INTEGRATION.md @@ -0,0 +1,206 @@ +# Using MCP Conformance Tests in SDK Repositories + +This guide explains how to integrate the MCP conformance test suite into your language SDK repository. The conformance framework tests your MCP implementation against the protocol specification to ensure compatibility. + +## Quick Start + +Install and run conformance tests: + +```bash +# Client testing (framework starts a test server, runs your client against it) +npx @modelcontextprotocol/conformance client --command "your-client-command" --scenario initialize + +# Server testing (your server must already be running) +npx @modelcontextprotocol/conformance server --url http://localhost:3000/mcp --scenario server-initialize +``` + +## Two Testing Modes + +### Client Testing + +The framework **starts a test server** and spawns your client against it. Your client receives the server URL as its final command-line argument. + +```bash +# Run a single scenario +npx @modelcontextprotocol/conformance client \ + --command "python tests/conformance/client.py" \ + --scenario initialize + +# Run a suite of tests +npx @modelcontextprotocol/conformance client \ + --command "python tests/conformance/client.py" \ + --suite auth +``` + +**Available client suites:** `all`, `core`, `extensions`, `auth`, `metadata`, `sep-835` + +Your client should: +1. Accept the server URL as its last argument +2. Read `MCP_CONFORMANCE_SCENARIO` env var to determine which scenario is being tested +3. Read `MCP_CONFORMANCE_CONTEXT` env var for scenario-specific data (e.g., OAuth credentials) + +### Server Testing + +Your server must be **running before** invoking the conformance tool. The framework connects to it as an MCP client. + +```bash +# Start your server first +your-server --port 3001 & + +# Then run conformance tests +npx @modelcontextprotocol/conformance server \ + --url http://localhost:3001/mcp \ + --suite active +``` + +**Available server suites:** `active` (default), `all`, `pending` + +**Note:** Server testing requires you to manage server lifecycle (start, health-check, cleanup) yourself. + +--- + +## Expected Failures (Baseline) File + +The expected-failures feature lets your CI pass while you work on fixing known issues. It catches regressions by failing when: +- A previously passing test starts failing (regression) +- A previously failing test starts passing (stale baseline - remove the entry) + +### File Format + +Create a YAML file (e.g., `conformance-baseline.yml`): + +```yaml +server: + - tools-call-with-progress + - resources-subscribe +client: + - auth/client-credentials-jwt +``` + +### Usage + +```bash +npx @modelcontextprotocol/conformance server \ + --url http://localhost:3000/mcp \ + --expected-failures ./conformance-baseline.yml +``` + +### Exit Code Behavior + +| Scenario Result | In Baseline? | Exit Code | Meaning | +|-----------------|--------------|-----------|---------| +| Fails | Yes | 0 | Expected failure | +| Fails | No | 1 | Unexpected regression | +| Passes | Yes | 1 | Stale baseline - remove entry | +| Passes | No | 0 | Normal pass | + +--- + +## GitHub Action + +The conformance repo provides a reusable GitHub Action that handles Node.js setup and conformance execution. + +### Client Testing Example + +```yaml +name: Conformance Tests +on: [push, pull_request] + +jobs: + conformance: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up your SDK + run: | + # Your SDK setup (pip install, npm install, etc.) + pip install -e . + + - uses: modelcontextprotocol/conformance@v0.1.11 + with: + mode: client + command: 'python tests/conformance/client.py' + suite: auth + expected-failures: ./conformance-baseline.yml +``` + +### Server Testing Example + +```yaml +name: Conformance Tests +on: [push, pull_request] + +jobs: + conformance: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up and start server + run: | + pip install -e . + python -m myserver --port 3001 & + # Wait for server to be ready + timeout 15 bash -c 'until curl -s http://localhost:3001/mcp; do sleep 0.5; done' + + - uses: modelcontextprotocol/conformance@v0.1.11 + with: + mode: server + url: http://localhost:3001/mcp + suite: active + expected-failures: ./conformance-baseline.yml +``` + +### Action Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| `mode` | Yes | `server` or `client` | +| `url` | Server mode | URL of the server to test | +| `command` | Client mode | Command to run the client | +| `expected-failures` | No | Path to YAML baseline file | +| `suite` | No | Test suite to run | +| `scenario` | No | Run a single scenario by name | +| `timeout` | No | Timeout in ms for client tests (default: 30000) | +| `verbose` | No | Show verbose output (default: false) | +| `node-version` | No | Node.js version (default: 20) | + +--- + +## Writing Conformance Clients/Servers + +### Example Client Pattern + +See [`src/conformance/everything-client.ts`](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/src/conformance/everything-client.ts) in the TypeScript SDK for a reference implementation. The recommended pattern is a single client that routes behavior based on the scenario: + +```python +import os +import sys +import json + +def main(): + server_url = sys.argv[-1] # URL passed as last argument + scenario = os.environ.get("MCP_CONFORMANCE_SCENARIO", "") + context = json.loads(os.environ.get("MCP_CONFORMANCE_CONTEXT", "{}")) + + if scenario.startswith("auth/"): + run_auth_scenario(server_url, scenario, context) + else: + run_default_scenario(server_url) + +if __name__ == "__main__": + main() +``` + +### Example Server Pattern + +See [`src/conformance/everything-server.ts`](https://github.com/modelcontextprotocol/typescript-sdk/blob/main/src/conformance/everything-server.ts) in the TypeScript SDK for a reference implementation that handles all server scenarios. + +--- + +## Additional Resources + +- [Conformance README](./README.md) +- [Design documentation](./src/runner/DESIGN.md) +- [TypeScript SDK conformance examples](https://github.com/modelcontextprotocol/typescript-sdk/tree/main/src/conformance) diff --git a/src/scenarios/client/auth/spec-references.ts b/src/scenarios/client/auth/spec-references.ts index 52a08ca..0bc3a9c 100644 --- a/src/scenarios/client/auth/spec-references.ts +++ b/src/scenarios/client/auth/spec-references.ts @@ -19,15 +19,15 @@ export const SpecReferences: { [key: string]: SpecReference } = { }, MCP_PRM_DISCOVERY: { id: 'MCP-2025-06-18-PRM-discovery', - url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#protected-resource-metadata-discovery-requirements' + url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#protected-resource-metadata-discovery-requirements' }, MCP_AUTH_DISCOVERY: { id: 'MCP-Authorization-metadata-discovery', - url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#authorization-server-metadata-discovery' + url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#authorization-server-metadata-discovery' }, MCP_DCR: { id: 'MCP-Dynamic-client-registration', - url: 'https://modelcontextprotocol.io/specification/draft/basic/client#dynamic-client-registration' + url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/client#dynamic-client-registration' }, OAUTH_2_1_AUTHORIZATION_ENDPOINT: { id: 'OAUTH-2.1-authorization-endpoint', @@ -39,23 +39,23 @@ export const SpecReferences: { [key: string]: SpecReference } = { }, MCP_ACCESS_TOKEN_USAGE: { id: 'MCP-Access-token-usage', - url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#access-token-usage' + url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#access-token-usage' }, MCP_SCOPE_SELECTION_STRATEGY: { id: 'MCP-Scope-selection-strategy', - url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#scope-selection-strategy' + url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#scope-selection-strategy' }, MCP_SCOPE_CHALLENGE_HANDLING: { id: 'MCP-Scope-challenge-handling', - url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#scope-challenge-handling' + url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#scope-challenge-handling' }, MCP_AUTH_ERROR_HANDLING: { id: 'MCP-Auth-error-handling', - url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#error-handling' + url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#error-handling' }, MCP_CLIENT_ID_METADATA_DOCUMENTS: { id: 'MCP-Client-ID-Metadata-Documents', - url: 'https://modelcontextprotocol.io/specification/draft/basic/authorization#client-id-metadata-documents' + url: 'https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization#client-id-metadata-documents' }, IETF_CIMD: { id: 'IETF-OAuth-Client-ID-Metadata-Document', From 9122a079371faa45274ff13f4d4b773afe964ce7 Mon Sep 17 00:00:00 2001 From: Paul Carleton Date: Fri, 23 Jan 2026 11:37:15 +0000 Subject: [PATCH 2/2] fix: prettier formatting and correct action version to v0.1.10 --- SDK_INTEGRATION.md | 40 +++++++++++++++++++++------------------- 1 file changed, 21 insertions(+), 19 deletions(-) diff --git a/SDK_INTEGRATION.md b/SDK_INTEGRATION.md index 7df2e6e..a092115 100644 --- a/SDK_INTEGRATION.md +++ b/SDK_INTEGRATION.md @@ -35,6 +35,7 @@ npx @modelcontextprotocol/conformance client \ **Available client suites:** `all`, `core`, `extensions`, `auth`, `metadata`, `sep-835` Your client should: + 1. Accept the server URL as its last argument 2. Read `MCP_CONFORMANCE_SCENARIO` env var to determine which scenario is being tested 3. Read `MCP_CONFORMANCE_CONTEXT` env var for scenario-specific data (e.g., OAuth credentials) @@ -62,6 +63,7 @@ npx @modelcontextprotocol/conformance server \ ## Expected Failures (Baseline) File The expected-failures feature lets your CI pass while you work on fixing known issues. It catches regressions by failing when: + - A previously passing test starts failing (regression) - A previously failing test starts passing (stale baseline - remove the entry) @@ -87,12 +89,12 @@ npx @modelcontextprotocol/conformance server \ ### Exit Code Behavior -| Scenario Result | In Baseline? | Exit Code | Meaning | -|-----------------|--------------|-----------|---------| -| Fails | Yes | 0 | Expected failure | -| Fails | No | 1 | Unexpected regression | -| Passes | Yes | 1 | Stale baseline - remove entry | -| Passes | No | 0 | Normal pass | +| Scenario Result | In Baseline? | Exit Code | Meaning | +| --------------- | ------------ | --------- | ----------------------------- | +| Fails | Yes | 0 | Expected failure | +| Fails | No | 1 | Unexpected regression | +| Passes | Yes | 1 | Stale baseline - remove entry | +| Passes | No | 0 | Normal pass | --- @@ -117,7 +119,7 @@ jobs: # Your SDK setup (pip install, npm install, etc.) pip install -e . - - uses: modelcontextprotocol/conformance@v0.1.11 + - uses: modelcontextprotocol/conformance@v0.1.10 with: mode: client command: 'python tests/conformance/client.py' @@ -144,7 +146,7 @@ jobs: # Wait for server to be ready timeout 15 bash -c 'until curl -s http://localhost:3001/mcp; do sleep 0.5; done' - - uses: modelcontextprotocol/conformance@v0.1.11 + - uses: modelcontextprotocol/conformance@v0.1.10 with: mode: server url: http://localhost:3001/mcp @@ -154,17 +156,17 @@ jobs: ### Action Inputs -| Input | Required | Description | -|-------|----------|-------------| -| `mode` | Yes | `server` or `client` | -| `url` | Server mode | URL of the server to test | -| `command` | Client mode | Command to run the client | -| `expected-failures` | No | Path to YAML baseline file | -| `suite` | No | Test suite to run | -| `scenario` | No | Run a single scenario by name | -| `timeout` | No | Timeout in ms for client tests (default: 30000) | -| `verbose` | No | Show verbose output (default: false) | -| `node-version` | No | Node.js version (default: 20) | +| Input | Required | Description | +| ------------------- | ----------- | ----------------------------------------------- | +| `mode` | Yes | `server` or `client` | +| `url` | Server mode | URL of the server to test | +| `command` | Client mode | Command to run the client | +| `expected-failures` | No | Path to YAML baseline file | +| `suite` | No | Test suite to run | +| `scenario` | No | Run a single scenario by name | +| `timeout` | No | Timeout in ms for client tests (default: 30000) | +| `verbose` | No | Show verbose output (default: false) | +| `node-version` | No | Node.js version (default: 20) | ---