Skip to content

Speed up escaping strings and keys when writing TOML - #709

Merged
frostming merged 1 commit into
python-poetry:masterfrom
deadstrobe5:perf/escape-sets
Nov 10, 2026
Merged

frostming merged 1 commit into
python-poetry:masterfrom
deadstrobe5:perf/escape-sets

Conversation

@deadstrobe5

@deadstrobe5 deadstrobe5 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

escape_string() walked the value one character at a time and sliced s[i:] at every position as soon as the value held anything to escape, so a value with a quote cost time quadratic in its length. It now builds one regex from the sequences actually present (longest first, so """ still matches whole) and replaces them in a single re.sub. A value with nothing to escape returns early. StringType.escaped_sequences and invalid_sequences now return sets built once at import instead of rebuilding them on every call, and SingleKey checks bare keys against a module frozenset instead of concatenating the allowed characters for every character of the key.

Every string value and every new key goes through this path when a document is built from Python data, which is how Poetry writes poetry.lock and how most tools call tomlkit.dumps(dict). Same-process A/B on CPython 3.14, 7 alternating rounds, median:

Workload Before After Speedup
Build + dumps() of a 760 KB poetry.lock (255 packages) 1651 ms 991 ms 1.67x
Build + dumps() of a 198 KB Cargo.lock 616 ms 317 ms 1.94x
Build + dumps() of a small pyproject.toml 6.93 ms 5.05 ms 1.37x
tomlkit.string('a"b') 29.3 us 10.6 us 2.77x
dumps() of one 20 KB value with quotes 44.4 ms 0.73 ms 61x

Output is unchanged. A differential fuzz (about 170k cases: random strings with quotes, backslashes, CR/LF, control characters, """ and ''', for all four string types, keys, String inputs, concatenation, pickling and dumps()) gave identical results. The function always returns a plain str, also for str subclasses such as String, as before. The new test covers repeated quotes and every escape kind in one value.

Duplicate Check

  • I searched the open PRs and found none that already fixes this issue.
  • Related open PRs (if any, and why this PR is still needed): Re-escape strings concatenated across quote styles #653 also touches String.__add__ around escape_string(), but only its call site, not the escaping itself.

Agent Drafting Metadata

  • Agent: Claude Code
  • Model: Claude Opus 5.5
  • Notes: the agent profiled the write path, wrote the change and the test, and ran the differential fuzz and the benchmarks above. Human attestation is pending.

@human-attestation

Copy link
Copy Markdown

🛡️ Human Attestation required. A human needs to confirm this PR once before it can merge.

Attest this PR — sign in with GitHub, pass the challenge, and confirm one of the statements.

@frostming
frostming merged commit 73078ea into python-poetry:master Nov 10, 2026
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants