All pull requests

Pull requests

• Fix or remove tasks whose Hub subsets don't exist (storycloze, xcopa:en, wikifact, mlmm_hellaswag is/no)

  #1500·Zephyr4772 opened Nov 13, 2026·

  ·

• Fix Czech word tokenizer: spaCy language code is cs, not cz

  #1499·Zephyr4772 opened Nov 12, 2026·

  ·

• Depend on langcodes[data]: multilingual tasks fail with ModuleNotFoundError: language_data

  #1498·Zephyr4772 opened Nov 12, 2026·

  ·

• Fix Windows crash and lost details: percent-encode task names in cache and details paths

  #1497·Zephyr4772 opened Nov 12, 2026·

  ·

• Add a file backend to score stored responses

  #1496·Rutvij97 opened Nov 12, 2026·

  ·
  1

• Read user config, task and fixture files as UTF-8 (non-ASCII system prompts crash on Windows)

  #1495·Zephyr4772 opened Nov 12, 2026·

  ·

• Fix timeout decorator on Windows (math metrics scored every answer as wrong)

  #1494·Zephyr4772 opened Nov 12, 2026·

  ·
  1

• Keep the evaluated doc out of its own few-shot examples

  #1492·David-Wu1119 opened Nov 11, 2026·

  ·
  1

• Fix HLE judge metric crashing when metrics are computed

  #1491·David-Wu1119 opened Nov 11, 2026·

  ·
  1

• Bump the actions group across 1 directory with 5 updates

  dependenciesgithub_actions
  #1490·dependabot[bot] opened Nov 11, 2026Bot·

  ·
  1

• Allow selecting few-shot examples by explicit ids

  #1489·amalsaidov opened Nov 10, 2026·

  ·
  1

• fix(ci): surface failures in the vLLM nightly workflow

  #1488·voropaevv opened Nov 9, 2026·

  ·
  1

• fix: don't expand an exact task name into its own subsets

  #1487·KhalidKhan541 opened Nov 8, 2026·

  ·
  1

• Add HAE-RAE Bench (five original subsets)

  #1485·jhwanseok opened Nov 6, 2026·

  ·
  1

  2

• fix(metrics): stop counting missing samples as failures in pass@k

  #1484·aidless opened Nov 6, 2026·

  ·
  1

• fix: compute multiclass F1 averages over all classes

  #1482·aidless opened Nov 6, 2026·

  ·

• Add a Count column with the evaluated sample count to the markdown results summary

  #1480·amkkoussay opened Nov 5, 2026·

  ·
  1

• Read numbers grouped with no-break, narrow no-break or thin spaces

  #1478·ChrystianSchutz opened Nov 5, 2026·

  ·
  1

• Do not let an extracted expression end with an operator (bold answers)

  #1477·ChrystianSchutz opened Nov 5, 2026·

  ·
  1

• fix(cache): persist regenerated responses after invalid cache entries

  #1474·Cookie0103 opened Nov 3, 2026·

  ·

• Fill in the translation literals for Latvian, Lithuanian, Maltese and Slovenian

  #1473·inchang-ing opened Nov 3, 2026·

  ·
  1

• fix(litellm): log and back off on non-filter BadRequestError

  #1472·pcbeingused333 opened Nov 3, 2026·

  ·

• fix(hle): scale confidence to 0-1 before computing calibration error

  #1470·damienen opened Nov 2, 2026·

  ·
  1

• Combine subtask stderrs in quadrature for the _average and all rows

  #1468·Andreas-Hum opened Nov 1, 2026·

  ·
  1

• Fix double BOS when using chat templates (vllm, transformers, sglang backends)

  #1467·EvanZhuang opened Nov 1, 2026·

  ·
  1