Skip to content

Exact byte position in the stream reader; remove the System and Legacy readers (#639) - #756

Merged
Hirogen merged 3 commits into
Developmentfrom
fix/639-exact-byte-position
Nov 20, 2026
Merged

Hirogen merged 3 commits into
Developmentfrom
fix/639-exact-byte-position

Conversation

@Hirogen

@Hirogen Hirogen commented Nov 20, 2026

Copy link
Copy Markdown
Member

Fixes #639

Problem

A log that grows while it contains a binary block jumped to the top on every append. The stream readers advanced their byte Position by re-encoding the decoded text. Each invalid byte decodes to U+FFFD, which re-encodes as three UTF-8 bytes, so Position ran ahead of the file: 100,055 bytes on the reporter's sample. LogfileReader.FileSize comes from that position. On the next poll newSize < FileSize, so the file was treated as recreated, reloaded, and the view reset.

Changes

  • Direct reader finds lines in the raw bytes (8dadc374). It searches for \r/\n as code units of the encoding (one byte, or two/four for UTF-16/UTF-32) and decodes each line into the pooled char block itself. Position advances by the bytes a line really occupies, so it is exact for any content and any line ending.

    • Pure-ASCII lines are widened with Ascii.ToUtf16.
    • UTF-16 LE lines without surrogates are copied.
    • Every other line goes through the encoding.
    • U+FFFD display is unchanged.
  • System and Legacy readers removed (b85d1a5f). Both read through StreamReader, which never reports the bytes it consumed, so neither could be made exact without becoming a copy of Direct. They were only kept as a one-release safety net. Also removed:

    • ReaderType and Preferences.ReaderType
    • the dead UseLegacyReader
    • the readerType parameter of LogfileReader
    • the Settings combobox and its tooltip resource

    Settings saved with "ReaderType" still load: Newtonsoft ignores the unknown member, and a test pins this for every stored value.

  • Review follow-ups (65608a3d): comment rules from CODING_STANDARDS.md, a shared test oracle, and non-ASCII cases in the allocation test.

Tests

Tests were written to fail on the old reader. They use the raw bytes as the oracle: a line ends at the next terminator unit, and its text is that line's bytes decoded on their own.

  • Invalid UTF-8: stray continuation bytes, truncated sequences before \n, \r\n and EOF, NUL runs, overlong and encoded-surrogate forms, and a genuine EF BF BD, which must still count as 3 bytes.
  • Multi-byte, invalid and \r\n sequences straddling buffer boundaries (16 K to 128 K).
  • Seeking back to recorded line starts after a binary block.
  • UTF-16 LE/BE (lone surrogate, surrogate pairs, odd trailing byte) and GB2312 with invalid lead bytes.
  • 50 seeded random byte streams.
  • LogfileReader tail test: a file with a binary block grows by appended lines and takes the growth path. No new-file report, LineCount increases, FileSize equals the size on disk.
  • A deterministic allocation test (bytes allocated on the reading thread) for ASCII and non-ASCII UTF-8 and UTF-16, so per-line allocation can't creep back in.

Full suite: all green except TrySetDataObject_ClipboardAvailable_PlacesDataAndReturnsTrue, which also fails on Development on my machine.

Performance

DirectReaderBenchmarks measures the reader the way LogfileReader drives it: TryReadLine, with blocks detached and returned every 500 lines. Its workloads: ASCII medium and large, Unicode, mixed line endings, long lines, UTF-16, and invalid bytes. It replaces StreamReaderBenchmarks, which only covered System/Legacy.

Old vs. new Direct reader, interleaved (old and new alternate per iteration to cancel machine drift), median of 40 rounds:

Workload Old (µs) New (µs) Δ pass 1 Δ pass 2
AsciiMedium 1,730 1,485 −9.7% −14.1%
AsciiLarge 14,499 10,244 −32.8% −29.4%
Unicode 5,211 4,284 −15.5% −17.8%
MixedLineEndings 1,800 1,413 −24.5% −21.5%
LongLines 12,173 4,119 −66.0% −66.2%
Utf16 6,911 4,845 −31.6% −29.9%
InvalidBytes 5,088 3,880 −15.8% −23.7%

(µs columns from pass 2.)

Allocations (BenchmarkDotNet MemoryDiagnoser, per read):

Workload Old New
AsciiMedium 9.25 KB 9.23 KB
AsciiLarge 42.45 KB 42.20 KB
Unicode 21.43 KB 20.32 KB
MixedLineEndings 21.48 KB 20.32 KB
LongLines 9,890 KB 13.91 KB
Utf16 21.69 KB 20.90 KB
InvalidBytes 23.41 KB 22.13 KB

Note: sequential BenchmarkDotNet runs on the dev laptop drifted 10–20% between the old and new blocks. One of three runs showed MixedLineEndings 33% slower for the new reader, another showed it 7% faster. The interleaved runs above are consistent across all passes.

Out of scope / follow-ups

  • XML mode still reads characters through the base StreamReader (ReadChar) and counts re-encoded chars.
  • A binary blob with no newline at all grows the reader's buffers to the blob size (the previous Direct reader did the same).

BRUNER Patrick added 3 commits November 20, 2026 14:42
A log with a binary block in the middle jumped to the top on every append.
The Direct reader advanced its byte Position by re-encoding the decoded text;
each invalid byte decodes to U+FFFD, which re-encodes as three UTF-8 bytes, so
Position ran ahead of the file (by 100,055 bytes on the reporter's sample).
LogfileReader takes FileSize from it, saw newSize < FileSize on the next poll,
treated the file as recreated and reloaded it.

The reader now finds line terminators in the raw bytes (as code units of the
encoding: one byte, or two/four for UTF-16/UTF-32) and decodes each line into
the pooled char block itself, so Position advances by the bytes a line really
occupies. Pure-ASCII lines are widened with Ascii.ToUtf16 and native-order
UTF-16 lines without surrogates are copied; other lines go through the
encoding.

Tests use the raw bytes as oracle: invalid UTF-8 cases, sequences straddling
buffer boundaries, seeking back to recorded line starts, UTF-16 and GB2312
edge cases, seeded random input, and a LogfileReader tail test that grows a
file with a binary block. A deterministic test guards zero allocations per
line. DirectReaderBenchmarks measures the reader as LogfileReader drives it;
against the previous reader it is faster on every workload (interleaved A/B).
With Direct exact for every byte sequence and line ending, the System and
Legacy readers no longer cover anything Direct does not. Both read through
StreamReader, which never reports the bytes it consumed, so neither could be
made exact without becoming a copy of Direct. They were kept only as a
one-release safety net (ADR 0006 update), and System already guessed one
newline length per file.

Removes PositionAwareStreamReaderSystem, PositionAwareStreamReaderLegacy,
the ReaderType enum, Preferences.ReaderType (and the dead UseLegacyReader),
the readerType parameter of LogfileReader, and the Settings combobox with its
tooltip resource. Settings saved with "ReaderType" still load: Newtonsoft
ignores the unknown member, which a new LoadOrCreateNew test pins for each
stored value.

Tests that used the System reader as oracle now check against the raw bytes;
reader tests that ran per reader run against Direct, and duplicates are gone.
The ReaderTest case that pinned System's overcount at EOF now expects the
exact position. StreamReaderBenchmarks (System/Legacy only) is replaced by
DirectReaderBenchmarks; the two ReadThroughput cases that only differed in
reader type are dropped.
…on case

- Drop XML doc comments from private members and tests
  (docs/CODING_STANDARDS.md) and shorten the remaining comments.
- Move the raw-byte oracle into RawByteLineOracle instead of one fixture
  calling into another.
- Cover valid non-ASCII UTF-8 and UTF-16 in the per-line allocation test,
  which goes through the encoding's decoder instead of the ASCII fast path.
- Rename _charFill to _usedChars and drop the big-endian UTF-16 branch
  (Windows only runs little-endian).
@Hirogen
Hirogen merged commit ff4b398 into Development Nov 20, 2026
3 checks passed
@Hirogen
Hirogen deleted the fix/639-exact-byte-position branch November 20, 2026 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scrolling to the top of the file

1 participant