Skip to content

Fix fsspec data loss after flush and reduce stream buffering costs - #407

Open
shcheklein wants to merge 1 commit into
mainfrom
fix/fsspec-stream-io
Open

shcheklein wants to merge 1 commit into
mainfrom
fix/fsspec-stream-io

Conversation

@shcheklein

Copy link
Copy Markdown
Member

On main, calling flush() on a Drive writer uploads the bytes written so far and marks the stream closed. A later binary write() still accepts bytes, but close() skips the final upload. For example, writing b"first", flushing, writing b"last", and closing leaves b"first" on Drive.

This change makes flush() flush the local buffer and close() upload once. Binary writes return their byte count. Small files stay in memory and larger files spill to a seekable temporary file. Upload errors propagate on close and the spool is released. Callers that need the file to be visible on Drive should close the writer or exit its context manager.

Small reads previously copied the entire unread tail of the current Drive response. The reader now keeps a view of that tail. A positive block_size passed to fs.open() controls the size of Drive download requests.

Local measurements

On macOS with CPython 3.11.18, comparing this branch with 4798a5c. The read result is the median of seven runs. The write measurement uses tracemalloc around the writer, reuses one input block, and excludes network traffic.

Workload main This PR
Read one 8 MiB response in 8 KiB pieces 193.85 ms 0.71 ms
Stage 64 MiB in 1 MiB writes, peak Python allocation 64.13 MiB 5.01 MiB

Verification

The offline suite passed locally (134 passed, 2 skipped). The filesystem unit tests passed on fsspec 2021.7.0 and 2026.9.0. Coverage includes flush followed by another write in text and binary mode, disk rollover, cleanup after a failed upload, byte-range request sizes, and mixed partial reads and peeks. The existing live Drive round-trip test now flushes between two writes and checks the downloaded bytes. Ruff, mypy, and the Sphinx build passed.

Based on main. Git verified a clean merge with the separate protocol-registration PR #406.

monkeypatch.setattr(fs, "upload_fobj", upload)
with fs.open("root/file", mode) as writer:
value = b"abc" if mode == "wb" else "abc"
assert writer.write(value) == 3
assert not writer.closed
assert not uploads
backing_file.assert_not_called()
assert writer.write(value) == 3

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the Drive fsspec write-handling and buffering changes.

No open issues 🎉

What changed in this PR

This PR fixes data loss when writing after flush() through the Drive fsspec integration and reduces stream buffering overhead.

Changes:

  • Keep writes local until close, using temporary-file spooling for larger uploads and propagating upload errors.
  • Avoid copying unread response tails and honor positive download block_size values.
  • Add regression coverage and document flush, close, and buffering behavior.
File Description
pydrive2/​test/​test_fs.py Adds a flush between writes in the live Drive round-trip test.
pydrive2/​test/​test_fs_unit.py Tests partial reads, peeks, download sizes, writer rollover, and upload-failure cleanup.
pydrive2/​fs/​utils.py Uses memory views for unread data and preserves buffered bytes after peek failures.
pydrive2/​fs/​spec.py Adds spooled writes, upload-on-close behavior, and configurable download request sizes.
docs/​fsspec.rst Documents download sizing and local buffering versus upload behavior.

Review effort: Balanced

@shcheklein shcheklein left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-generated review

Overall this is a clean, well-scoped fix with strong test coverage and impressive memory/latency improvements. The flush()/close() lifecycle and IterStream memoryview slicing are much cleaner than main.

Three edge-case bugs found (see inline comments):

  1. Temp-file leak on write() after close() in GDriveBufferedWriter.write() (pydrive2/fs/spec.py).
  2. TypeError on block_size="default" in _gdrive_open_file() (pydrive2/fs/spec.py), which is a standard fsspec sentinel.
  3. memoryview export left open on exception in IterStream.readinto() (pydrive2/fs/utils.py), preventing callers from resizing/clearing a bytearray inside an except block.

Comment thread pydrive2/fs/spec.py
Comment on lines +718 to +722
def write(self, data):
# A single large write would otherwise fill BytesIO before rollover.
if memoryview(data).nbytes > self.fs.blocksize:
self.buffer.rollover()
return self.buffer.write(data)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-generated review comment

Bug: calling write() with len(data) > fs.blocksize after close() leaks an open temporary file.

If the writer was closed while still in memory (_rolled is False), calling write() with a payload larger than self.fs.blocksize invokes self.buffer.rollover() before any closed check. In CPython's tempfile.SpooledTemporaryFile.rollover(), a new TemporaryFile is opened and assigned to self.buffer._file before file.getvalue() raises ValueError: I/O operation on closed file. That leaves self.buffer.closed == False (an unclosed temp file on disk), while subsequent writer.close() calls return early because self.closed is already True.

Guard write() with self._checkClosed() first:

def write(self, data):
    self._checkClosed()
    if memoryview(data).nbytes > self.fs.blocksize:
        self.buffer.rollover()
    return self.buffer.write(data)

(Minor: if you also want to prevent BytesIO from temporarily growing up to ~2× blocksize across two consecutive writes just under fs.blocksize, you can track buffered bytes until rollover without incurring tell() syscalls after rollover.)

Comment thread pydrive2/fs/spec.py
# it does not create a file on the remote
gdrive_file = self.client.CreateFile(param)
options = {}
if block_size is not None and block_size > 0:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-generated review comment

Edge case: block_size="default" raises TypeError.

In fsspec, "default" is an accepted sentinel for block_size (used by AbstractBufferedFile and several fsspec implementations). On main, _open() ignored block_size, so fs.open(path, "rb", block_size="default") worked; here block_size > 0 raises TypeError: '>' not supported between instances of 'str' and 'int'.

Suggested change
if block_size is not None and block_size > 0:
if block_size not in (None, "default") and block_size > 0:

Comment thread pydrive2/fs/utils.py
Comment on lines 37 to +43
def readinto(self, b):
try:
n = len(b) # We're supposed to return at most this much
chunk = self.leftover or next(self.iterator)
output, self.leftover = chunk[:n], chunk[n:]

n_out = len(output)
b[:n_out] = output
return n_out
except StopIteration:
return 0 # indicate EOF
dest = memoryview(b).cast("B")
if dest.readonly:
raise TypeError("readinto() requires a writable buffer")
chunk = self._take(len(dest))
dest[: len(chunk)] = chunk
return len(chunk)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-generated review comment

Bug: dest memoryview is not released when _take() raises, locking caller buffers.

If self._take(len(dest)) raises (e.g., an ApiRequestError/OSError mid-download or ValueError on a closed stream), the exception traceback retains this stack frame and its local dest view. While the exception is alive in an except block, resizing or clearing the caller's bytearray (e.g., buf.clear()) fails with BufferError: Existing exports of data: object cannot be re-sized.

Wrapping dest in a with block releases the export immediately on both normal return and error (and checking self._checkClosed() first also ensures a closed stream raises ValueError even for read-only buffers):

def readinto(self, b):
    self._checkClosed()
    with memoryview(b).cast("B") as dest:
        if dest.readonly:
            raise TypeError("readinto() requires a writable buffer")
        chunk = self._take(len(dest))
        dest[: len(chunk)] = chunk
        return len(chunk)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants