Skip to content

Run every referee scenario in the server conformance leg - #652

Merged
koic merged 1 commit into
modelcontextprotocol:mainfrom
koic:run_every_referee_scenario_in_the_server_conformance_leg
Nov 3, 2026
Merged

koic merged 1 commit into
modelcontextprotocol:mainfrom
koic:run_every_referee_scenario_in_the_server_conformance_leg

Conversation

@koic

@koic koic commented Nov 2, 2026

Copy link
Copy Markdown
Member

Motivation and Context

The server conformance leg ran the referee's default active suite, which leaves out the extension-tagged scenarios, while the client leg has selected every scenario with --suite all. The Skills server implementation therefore never met the referee's sep-2640-skills-* server scenarios, although the conformance server already serves the conformance/skills directory for them, and the all suite's other additions (stateless server, SSE polling, caching, standard header validation, input-required results) were not run either.

The server leg now selects --suite all the way the client leg does, with a requirement set or a single scenario still replacing the selection. Of the 34 scenarios this adds, 23 pass. The rest reach three areas the SDK does not implement and are listed in expected_failures.yml the way the TypeScript and Python SDKs list theirs: the Tasks extension (SEP-2663); the server side of SEP-2243 custom headers, where the server does not bind Mcp-Param-* headers to x-mcp-header tool parameters and the conformance server exposes no such tool, so the referee reports the scenario's checks as untestable; and SEP-1699 resumability, where a POST SSE stream carries no priming event or retry field, which the referee reports as warnings that its baseline counts as failures. tasks-status-notifications is left out of the baseline because the referee runs no check in it, and the baseline check would flag an entry for a passing scenario as stale.

How Has This Been Tested?

test/conformance/server_runner_test.rb, and bundle exec rake conformance:test: the server leg runs 65 scenarios, the 11 baselined failures reported as expected and the three sep-2640-skills-* scenarios passing, where it ran 31 before the change. REQUIREMENTS=2026-07-28 still runs the frozen set without a suite.

Breaking Changes

None.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

## Motivation and Context

The server conformance leg ran the referee's default `active` suite, which leaves out the extension-tagged scenarios,
while the client leg has selected every scenario with `--suite all`. The Skills server implementation
therefore never met the referee's `sep-2640-skills-*` server scenarios, although the conformance server already serves
the `conformance/skills` directory for them, and the `all` suite's other additions (stateless server, SSE polling, caching,
standard header validation, `input-required` results) were not run either.

The server leg now selects `--suite all` the way the client leg does, with a requirement set or a single scenario
still replacing the selection. Of the 34 scenarios this adds, 23 pass. The rest reach three areas the SDK does not implement
and are listed in `expected_failures.yml` the way the TypeScript and Python SDKs list theirs: the Tasks extension (SEP-2663);
the server side of SEP-2243 custom headers, where the server does not bind `Mcp-Param-*` headers to `x-mcp-header` tool parameters
and the conformance server exposes no such tool, so the referee reports the scenario's checks as untestable; and SEP-1699 resumability,
where a POST SSE stream carries no priming event or `retry` field, which the referee reports as warnings that its baseline counts as failures.
`tasks-status-notifications` is left out of the baseline because the referee runs no check in it, and the baseline check would flag
an entry for a passing scenario as stale.

## How Has This Been Tested?

test/conformance/server_runner_test.rb, and `bundle exec rake conformance:test`: the server leg runs 65 scenarios,
the 11 baselined failures reported as expected and the three `sep-2640-skills-*` scenarios passing, where it ran 31 before the change.
`REQUIREMENTS=2026-07-28` still runs the frozen set without a suite.

## Breaking Changes

None.
@koic
koic force-pushed the run_every_referee_scenario_in_the_server_conformance_leg branch from 430ff37 to 1ab5242 Compare November 2, 2026 21:52
@koic
koic merged commit 69c08b6 into modelcontextprotocol:main Nov 3, 2026
13 checks passed
@koic
koic deleted the run_every_referee_scenario_in_the_server_conformance_leg branch November 3, 2026 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants