Repository navigation
Expectation objects with a required flag, and shared expectations - #158
Merged
Merged
Conversation
Expected behaviours can now be plain strings or objects
{"id", "text", "required", "source"}; only "text" is required. A missing
id is generated from the text. The required flag is applied in code and
never shown to the judge model, which sees the text only.
- New simpleaudit/expectations.py: normalize, merge_shared,
apply_shared_expectations, upgrade_expectations (legacy converter),
shared_rule_summary.
- Checklist grading no longer reads optional wording ("Optionally",
"Ideally", "May", "Kan nevne (ikke påkrevd)"). Violations of
non-required items go to a new `reported` list.
- run/run_async (multi- and single-turn) and rejudge accept
shared_expectations=. Expectations are validated before any model call.
- rejudge reads old saved results with the legacy markers by default
(legacy_optional_markers=True), so old runs re-grade as before.
- AuditResult.expectations is stored only when it adds information, so
plain-string results keep their shape.
- Shipped packs: 55 legacy-optional lines converted to required: false.
"KAN NEVNE:" / "Kan nevne" lines, previously graded as required, are now
non-required (deliberate behaviour change).
- check_scenario_pack accepts objects and warns about required items that
read as optional.
Refs #157
Contributor
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
This was referenced Nov 12, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #157 (core part). cc @kelkalot
What changes
expected_behaviorentries can be plain strings or objects{"id", "text", "required", "source"}. Onlytextis required; a missingidis generated from the normalised text (e-<sha1[:8]>). Duplicate ids are refused.requiredflag is applied in code. The judge model sees the text only, so model-facing payloads are unchanged (snapshots confirm).Optionally,Ideally,May,Kan nevne (ikke påkrevd)).upgrade_expectationsconverts such legacy strings once.run/run_async(multi-turn and single-turn) andrejudgeacceptshared_expectations=, merged into every scenario withsource="shared". A scenario item with the same id overrides a shared rule. Expectations are validated before any model call.derived["reported"]list; they never count toward severity, issues or positives.shared_rule_summary(results)gives per-rule met / violated / not applicable / missing counts.AuditResult.expectationsis stored only when it adds information, so results from plain-string scenarios keep their old shape.rejudgereads old saved results with the legacy markers by default (legacy_optional_markers=True), so old runs re-grade as before.Shipped packs
55 legacy-optional lines in 14 packs converted to
{"text": <same text>, "required": false}. Text is unchanged.Behaviour change: the
KAN NEVNE:and plainKan nevnelines (arbeidstilsynet_arbeidstid, skatteetaten_legitimasjon, nb_kryss_ordning) were meant as optional but were graded as required. They are now non-required, so those packs may grade slightly differently.scripts/check_scenario_pack.pyaccepts objects and warns when a required item reads as optional.Checks
pytest: 2556 passed, 7 skipped, 1 failed. The failure istest_version_is_single_sourced_and_not_stale, which also fails on a clean tree (stale installed metadata, 0.5.1 vs 0.7.0).tests/test_expectations.py; checklist tests updated.reported: []change.ruff check --select E9,F821 .passes.