Skip to content

fix(decision judges): honour required flags and shared rules - #161

Merged
SushantGautam merged 1 commit into
devfrom
fix/decision-expectation-flags
Nov 13, 2026
Merged

SushantGautam merged 1 commit into
devfrom
fix/decision-expectation-flags

Conversation

@SushantGautam

Copy link
Copy Markdown
Collaborator

Decision judges that check expectations (decision_safety, decision_expectations) did not use expectation objects from 0.8.0:

  • a violated required: false item still raised the severity;
  • items had no id/source, so shared rules could not be counted per rule (shared_rule_summary only read checklist judgments);
  • a conditional shared rule ("when the user …") could only be met or violated.

Changes:

  • Items carry id, required and source from scenario_meta["expectations"]; with plain texts (or objects that do not line up) every item is a required scenario item, as before.
  • Violated non-required items go to reported and never change the severity.
  • Each shared rule gets an applies_N yes/no question; clearly not applying (p ≤ 0.3) gives not_applicable. Scenario items get no extra question.
  • shared_rule_summary reads a decision judgment's expectations when there is no checklist.

Tests: 6 new tests in tests/test_decision_judge.py; full suite 2563 passed.

🤖 Generated with Claude Code

decision_safety and decision_expectations ignored expectation objects: a
violated item with required: false still raised the severity, items had no
id or source so shared rules could not be counted per rule, and a
conditional shared rule could only be met or violated.

- Items now carry id, required and source from scenario_meta["expectations"]
  (falling back to required scenario items when they do not line up).
- A violated non-required item is listed under "reported" and never
  changes the severity, as with the checklist judge.
- A shared rule gets an extra applies_N question; clearly not applying
  gives status not_applicable.
- shared_rule_summary also reads a decision judgment's "expectations".

Co-Authored-By: Claude <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 13, 2026 09:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@SushantGautam
SushantGautam merged commit f313a1f into dev Nov 13, 2026
8 checks passed
@SushantGautam
SushantGautam deleted the fix/decision-expectation-flags branch November 13, 2026 09:45
This was referenced Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants