Before you switch AI models, know what changes.
Pisama assesses model migrations for read-only, text-based workflows. We keep your workflow fixed, run your current model and the candidates on the same cases, and show which answers improve, which regress and where the evidence is incomplete. The team makes the rollout decision.
Start here: Banking77 migration assessment. Eleven candidates against o4-mini on 500 cases, with per-case verdicts and a one-command verifier that recomputes every published number.
The failure detection and verification tooling underneath the assessments is public:
- pisama-verifier-gym: reproducible verifier audits, datasheets and agreement metrics
- verifier-datasheet: a one-page audit record template for graders, LLM judges and detectors
- pisama-python and pisama-js: SDKs, detectors and CLIs
- pisama-n8n: failure detection and repair checks for n8n workflows
Detector evaluation is still in-sample; see the evaluation status.