Skip to content

Release 0.8.0: expectation objects and shared expectations - #160

Merged
SushantGautam merged 4 commits into
mainfrom
dev
Nov 12, 2026
Merged

SushantGautam merged 4 commits into
mainfrom
dev

Conversation

@SushantGautam

Copy link
Copy Markdown
Collaborator

Merges dev into main for the 0.8.0 release (#158, #159).

Expected behaviours can now be plain strings or objects
{"id", "text", "required", "source"}; only "text" is required. A missing
id is generated from the text. The required flag is applied in code and
never shown to the judge model, which sees the text only.

- New simpleaudit/expectations.py: normalize, merge_shared,
  apply_shared_expectations, upgrade_expectations (legacy converter),
  shared_rule_summary.
- Checklist grading no longer reads optional wording ("Optionally",
  "Ideally", "May", "Kan nevne (ikke påkrevd)"). Violations of
  non-required items go to a new `reported` list.
- run/run_async (multi- and single-turn) and rejudge accept
  shared_expectations=. Expectations are validated before any model call.
- rejudge reads old saved results with the legacy markers by default
  (legacy_optional_markers=True), so old runs re-grade as before.
- AuditResult.expectations is stored only when it adds information, so
  plain-string results keep their shape.
- Shipped packs: 55 legacy-optional lines converted to required: false.
  "KAN NEVNE:" / "Kan nevne" lines, previously graded as required, are now
  non-required (deliberate behaviour change).
- check_scenario_pack accepts objects and warns about required items that
  read as optional.

Refs #157
Expectation objects with a required flag, and shared expectations
Copilot AI balanced review requested due to automatic review settings November 12, 2026 23:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@SushantGautam
SushantGautam merged commit b23ef73 into main Nov 12, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants