Skip to content

Expectation objects with a required flag, and shared expectations - #158

Merged
SushantGautam merged 1 commit into
devfrom
feat/expectation-objects
Nov 12, 2026
Merged

SushantGautam merged 1 commit into
devfrom
feat/expectation-objects

Conversation

@SushantGautam

Copy link
Copy Markdown
Collaborator

Implements #157 (core part). cc @kelkalot

What changes

  • expected_behavior entries can be plain strings or objects {"id", "text", "required", "source"}. Only text is required; a missing id is generated from the normalised text (e-<sha1[:8]>). Duplicate ids are refused.
  • The required flag is applied in code. The judge model sees the text only, so model-facing payloads are unchanged (snapshots confirm).
  • Optional-by-wording is removed from grading (Optionally, Ideally, May, Kan nevne (ikke påkrevd)). upgrade_expectations converts such legacy strings once.
  • run / run_async (multi-turn and single-turn) and rejudge accept shared_expectations=, merged into every scenario with source="shared". A scenario item with the same id overrides a shared rule. Expectations are validated before any model call.
  • Violations of non-required items go to a new derived["reported"] list; they never count toward severity, issues or positives.
  • shared_rule_summary(results) gives per-rule met / violated / not applicable / missing counts.
  • AuditResult.expectations is stored only when it adds information, so results from plain-string scenarios keep their old shape.
  • rejudge reads old saved results with the legacy markers by default (legacy_optional_markers=True), so old runs re-grade as before.

Shipped packs

55 legacy-optional lines in 14 packs converted to {"text": <same text>, "required": false}. Text is unchanged.

Behaviour change: the KAN NEVNE: and plain Kan nevne lines (arbeidstilsynet_arbeidstid, skatteetaten_legitimasjon, nb_kryss_ordning) were meant as optional but were graded as required. They are now non-required, so those packs may grade slightly differently.

scripts/check_scenario_pack.py accepts objects and warns when a required item reads as optional.

Checks

  • pytest: 2556 passed, 7 skipped, 1 failed. The failure is test_version_is_single_sourced_and_not_stale, which also fails on a clean tree (stale installed metadata, 0.5.1 vs 0.7.0).
  • 22 new tests in tests/test_expectations.py; checklist tests updated.
  • Snapshots regenerated: only summary wording ("not counted"), item ids and reported: [] change.
  • ruff check --select E9,F821 . passes.

Expected behaviours can now be plain strings or objects
{"id", "text", "required", "source"}; only "text" is required. A missing
id is generated from the text. The required flag is applied in code and
never shown to the judge model, which sees the text only.

- New simpleaudit/expectations.py: normalize, merge_shared,
  apply_shared_expectations, upgrade_expectations (legacy converter),
  shared_rule_summary.
- Checklist grading no longer reads optional wording ("Optionally",
  "Ideally", "May", "Kan nevne (ikke påkrevd)"). Violations of
  non-required items go to a new `reported` list.
- run/run_async (multi- and single-turn) and rejudge accept
  shared_expectations=. Expectations are validated before any model call.
- rejudge reads old saved results with the legacy markers by default
  (legacy_optional_markers=True), so old runs re-grade as before.
- AuditResult.expectations is stored only when it adds information, so
  plain-string results keep their shape.
- Shipped packs: 55 legacy-optional lines converted to required: false.
  "KAN NEVNE:" / "Kan nevne" lines, previously graded as required, are now
  non-required (deliberate behaviour change).
- check_scenario_pack accepts objects and warns about required items that
  read as optional.

Refs #157
Copilot AI balanced review requested due to automatic review settings November 12, 2026 17:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants