Repository navigation
Exact byte position in the stream reader; remove the System and Legacy readers (#639) - #756
Merged
Merged
Conversation
added 3 commits
November 20, 2026 14:42
A log with a binary block in the middle jumped to the top on every append. The Direct reader advanced its byte Position by re-encoding the decoded text; each invalid byte decodes to U+FFFD, which re-encodes as three UTF-8 bytes, so Position ran ahead of the file (by 100,055 bytes on the reporter's sample). LogfileReader takes FileSize from it, saw newSize < FileSize on the next poll, treated the file as recreated and reloaded it. The reader now finds line terminators in the raw bytes (as code units of the encoding: one byte, or two/four for UTF-16/UTF-32) and decodes each line into the pooled char block itself, so Position advances by the bytes a line really occupies. Pure-ASCII lines are widened with Ascii.ToUtf16 and native-order UTF-16 lines without surrogates are copied; other lines go through the encoding. Tests use the raw bytes as oracle: invalid UTF-8 cases, sequences straddling buffer boundaries, seeking back to recorded line starts, UTF-16 and GB2312 edge cases, seeded random input, and a LogfileReader tail test that grows a file with a binary block. A deterministic test guards zero allocations per line. DirectReaderBenchmarks measures the reader as LogfileReader drives it; against the previous reader it is faster on every workload (interleaved A/B).
With Direct exact for every byte sequence and line ending, the System and Legacy readers no longer cover anything Direct does not. Both read through StreamReader, which never reports the bytes it consumed, so neither could be made exact without becoming a copy of Direct. They were kept only as a one-release safety net (ADR 0006 update), and System already guessed one newline length per file. Removes PositionAwareStreamReaderSystem, PositionAwareStreamReaderLegacy, the ReaderType enum, Preferences.ReaderType (and the dead UseLegacyReader), the readerType parameter of LogfileReader, and the Settings combobox with its tooltip resource. Settings saved with "ReaderType" still load: Newtonsoft ignores the unknown member, which a new LoadOrCreateNew test pins for each stored value. Tests that used the System reader as oracle now check against the raw bytes; reader tests that ran per reader run against Direct, and duplicates are gone. The ReaderTest case that pinned System's overcount at EOF now expects the exact position. StreamReaderBenchmarks (System/Legacy only) is replaced by DirectReaderBenchmarks; the two ReadThroughput cases that only differed in reader type are dropped.
…on case - Drop XML doc comments from private members and tests (docs/CODING_STANDARDS.md) and shorten the remaining comments. - Move the raw-byte oracle into RawByteLineOracle instead of one fixture calling into another. - Cover valid non-ASCII UTF-8 and UTF-16 in the per-line allocation test, which goes through the encoding's decoder instead of the ASCII fast path. - Rename _charFill to _usedChars and drop the big-endian UTF-16 branch (Windows only runs little-endian).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #639
Problem
A log that grows while it contains a binary block jumped to the top on every append. The stream readers advanced their byte
Positionby re-encoding the decoded text. Each invalid byte decodes to U+FFFD, which re-encodes as three UTF-8 bytes, soPositionran ahead of the file: 100,055 bytes on the reporter's sample.LogfileReader.FileSizecomes from that position. On the next pollnewSize < FileSize, so the file was treated as recreated, reloaded, and the view reset.Changes
Direct reader finds lines in the raw bytes (
8dadc374). It searches for\r/\nas code units of the encoding (one byte, or two/four for UTF-16/UTF-32) and decodes each line into the pooled char block itself.Positionadvances by the bytes a line really occupies, so it is exact for any content and any line ending.Ascii.ToUtf16.SystemandLegacyreaders removed (b85d1a5f). Both read throughStreamReader, which never reports the bytes it consumed, so neither could be made exact without becoming a copy of Direct. They were only kept as a one-release safety net. Also removed:ReaderTypeandPreferences.ReaderTypeUseLegacyReaderreaderTypeparameter ofLogfileReaderSettings saved with
"ReaderType"still load: Newtonsoft ignores the unknown member, and a test pins this for every stored value.Review follow-ups (
65608a3d): comment rules fromCODING_STANDARDS.md, a shared test oracle, and non-ASCII cases in the allocation test.Tests
Tests were written to fail on the old reader. They use the raw bytes as the oracle: a line ends at the next terminator unit, and its text is that line's bytes decoded on their own.
\n,\r\nand EOF, NUL runs, overlong and encoded-surrogate forms, and a genuineEF BF BD, which must still count as 3 bytes.\r\nsequences straddling buffer boundaries (16 K to 128 K).LogfileReadertail test: a file with a binary block grows by appended lines and takes the growth path. No new-file report,LineCountincreases,FileSizeequals the size on disk.Full suite: all green except
TrySetDataObject_ClipboardAvailable_PlacesDataAndReturnsTrue, which also fails onDevelopmenton my machine.Performance
DirectReaderBenchmarksmeasures the reader the wayLogfileReaderdrives it:TryReadLine, with blocks detached and returned every 500 lines. Its workloads: ASCII medium and large, Unicode, mixed line endings, long lines, UTF-16, and invalid bytes. It replacesStreamReaderBenchmarks, which only coveredSystem/Legacy.Old vs. new Direct reader, interleaved (old and new alternate per iteration to cancel machine drift), median of 40 rounds:
(µs columns from pass 2.)
Allocations (BenchmarkDotNet
MemoryDiagnoser, per read):Note: sequential BenchmarkDotNet runs on the dev laptop drifted 10–20% between the old and new blocks. One of three runs showed MixedLineEndings 33% slower for the new reader, another showed it 7% faster. The interleaved runs above are consistent across all passes.
Out of scope / follow-ups
StreamReader(ReadChar) and counts re-encoded chars.