Reference Desk answers questions about continuous delivery practice from published sources: DORA, MinimumCD, Beyond MinimumCD and the Twelve-Factor App. Every answer quotes its sources word for word, and every quote is checked against the passage it came from.
Reference Desk holds published practice material only, no customer or team data.
Each script runs from the repo root, in this order:
scripts/normalise.pycleans each source incorpus/raw/into Markdown with one front matter schema, and writescorpus/units.jsonlandSOURCES.md.scripts/chunk.pycuts each page into chunks at headings and sentence ends, each within 480 tokens, and writescorpus/chunks.jsonl.scripts/embed.pyembeds the chunks with bge-small-en-v1.5 and writescorpus/chunks.npyandcorpus/manifest.json.scripts/search.py "question"searches with BM25 and embeddings, merged by reciprocal rank fusion, with at most 2 passages from any one page.scripts/answer.py "question"answers from the top 8 passages in exact quotes, using the model set in.env.scripts/verify.pychecks every quote in a saved answer against its passage.scripts/eval.pyscores retrieval against the questions ineval.jsonl.
The decisions behind each step, with the research they rest on, are in docs/adr/.
uv sync
git clone https://github.com/dora-team/dora.dev corpus/raw/dora.dev
git clone https://github.com/Minimum-CD/cd-manifesto corpus/raw/cd-manifesto
git clone https://github.com/bdfinst/cd-migration corpus/raw/cd-migration
git clone https://github.com/twelve-factor/twelve-factor corpus/raw/twelve-factor
cp .env.example .env
./scripts/normalise.py && ./scripts/chunk.py && ./scripts/embed.pyAdd your OpenRouter key to .env before running answer.py; .env.example
explains each setting. The first chunk.py and embed.py runs download the
tokeniser and model from Hugging Face, and later runs use the cached copies.
SOURCES.md lists the commit of each source that the current corpus was built
from.
./scripts/eval.pyThis scores BM25, embeddings and the fused search on the dev questions, overall
and by question type, and saves the run to corpus/evals/. The holdout
questions are scored too, but listed one by one only with --show-holdout:
tune on dev and look at the holdout last.
To try a change to retrieval without changing the defaults, score a variant from
scripts/experiments.py on the dev questions and compare it, question by question,
with a saved run:
./scripts/eval.py --variant NAME --compare corpus/evals/<run>.jsonA change wins if it fixes at least 3 questions and breaks at most 1, or breaks none
and raises the fused recall@8. scripts/expand.py writes questions and context for
each chunk with a local model under Ollama, for the expansion variants. That text is
only matched against: it is never shown, quoted or cited.
The tests need neither the corpus nor the network:
uv run python -m unittest discover -s testsThe code is MIT licensed: see LICENSE.md. The corpus is not included, and
each source keeps its own licence: see SOURCES.md.