Repository navigation
Fix fsspec data loss after flush and reduce stream buffering costs - #407
shcheklein wants to merge 1 commit into
Conversation
| monkeypatch.setattr(fs, "upload_fobj", upload) | ||
| with fs.open("root/file", mode) as writer: | ||
| value = b"abc" if mode == "wb" else "abc" | ||
| assert writer.write(value) == 3 |
| assert not writer.closed | ||
| assert not uploads | ||
| backing_file.assert_not_called() | ||
| assert writer.write(value) == 3 |
There was a problem hiding this comment.
🟢 Approval recommended
There are no active review issues for the Drive fsspec write-handling and buffering changes.
No open issues 🎉
What changed in this PR
This PR fixes data loss when writing after flush() through the Drive fsspec integration and reduces stream buffering overhead.
Changes:
- Keep writes local until close, using temporary-file spooling for larger uploads and propagating upload errors.
- Avoid copying unread response tails and honor positive download
block_sizevalues. - Add regression coverage and document flush, close, and buffering behavior.
| File | Description |
|---|---|
pydrive2/test/test_fs.py |
Adds a flush between writes in the live Drive round-trip test. |
pydrive2/test/test_fs_unit.py |
Tests partial reads, peeks, download sizes, writer rollover, and upload-failure cleanup. |
pydrive2/fs/utils.py |
Uses memory views for unread data and preserves buffered bytes after peek failures. |
pydrive2/fs/spec.py |
Adds spooled writes, upload-on-close behavior, and configurable download request sizes. |
docs/fsspec.rst |
Documents download sizing and local buffering versus upload behavior. |
Review effort: Balanced
shcheklein
left a comment
There was a problem hiding this comment.
🤖 AI-generated review
Overall this is a clean, well-scoped fix with strong test coverage and impressive memory/latency improvements. The flush()/close() lifecycle and IterStream memoryview slicing are much cleaner than main.
Three edge-case bugs found (see inline comments):
- Temp-file leak on
write()afterclose()inGDriveBufferedWriter.write()(pydrive2/fs/spec.py). TypeErroronblock_size="default"in_gdrive_open_file()(pydrive2/fs/spec.py), which is a standardfsspecsentinel.memoryviewexport left open on exception inIterStream.readinto()(pydrive2/fs/utils.py), preventing callers from resizing/clearing abytearrayinside anexceptblock.
| def write(self, data): | ||
| # A single large write would otherwise fill BytesIO before rollover. | ||
| if memoryview(data).nbytes > self.fs.blocksize: | ||
| self.buffer.rollover() | ||
| return self.buffer.write(data) |
There was a problem hiding this comment.
🤖 AI-generated review comment
Bug: calling write() with len(data) > fs.blocksize after close() leaks an open temporary file.
If the writer was closed while still in memory (_rolled is False), calling write() with a payload larger than self.fs.blocksize invokes self.buffer.rollover() before any closed check. In CPython's tempfile.SpooledTemporaryFile.rollover(), a new TemporaryFile is opened and assigned to self.buffer._file before file.getvalue() raises ValueError: I/O operation on closed file. That leaves self.buffer.closed == False (an unclosed temp file on disk), while subsequent writer.close() calls return early because self.closed is already True.
Guard write() with self._checkClosed() first:
def write(self, data):
self._checkClosed()
if memoryview(data).nbytes > self.fs.blocksize:
self.buffer.rollover()
return self.buffer.write(data)(Minor: if you also want to prevent BytesIO from temporarily growing up to ~2× blocksize across two consecutive writes just under fs.blocksize, you can track buffered bytes until rollover without incurring tell() syscalls after rollover.)
| # it does not create a file on the remote | ||
| gdrive_file = self.client.CreateFile(param) | ||
| options = {} | ||
| if block_size is not None and block_size > 0: |
There was a problem hiding this comment.
🤖 AI-generated review comment
Edge case: block_size="default" raises TypeError.
In fsspec, "default" is an accepted sentinel for block_size (used by AbstractBufferedFile and several fsspec implementations). On main, _open() ignored block_size, so fs.open(path, "rb", block_size="default") worked; here block_size > 0 raises TypeError: '>' not supported between instances of 'str' and 'int'.
| if block_size is not None and block_size > 0: | |
| if block_size not in (None, "default") and block_size > 0: |
| def readinto(self, b): | ||
| try: | ||
| n = len(b) # We're supposed to return at most this much | ||
| chunk = self.leftover or next(self.iterator) | ||
| output, self.leftover = chunk[:n], chunk[n:] | ||
|
|
||
| n_out = len(output) | ||
| b[:n_out] = output | ||
| return n_out | ||
| except StopIteration: | ||
| return 0 # indicate EOF | ||
| dest = memoryview(b).cast("B") | ||
| if dest.readonly: | ||
| raise TypeError("readinto() requires a writable buffer") | ||
| chunk = self._take(len(dest)) | ||
| dest[: len(chunk)] = chunk | ||
| return len(chunk) |
There was a problem hiding this comment.
🤖 AI-generated review comment
Bug: dest memoryview is not released when _take() raises, locking caller buffers.
If self._take(len(dest)) raises (e.g., an ApiRequestError/OSError mid-download or ValueError on a closed stream), the exception traceback retains this stack frame and its local dest view. While the exception is alive in an except block, resizing or clearing the caller's bytearray (e.g., buf.clear()) fails with BufferError: Existing exports of data: object cannot be re-sized.
Wrapping dest in a with block releases the export immediately on both normal return and error (and checking self._checkClosed() first also ensures a closed stream raises ValueError even for read-only buffers):
def readinto(self, b):
self._checkClosed()
with memoryview(b).cast("B") as dest:
if dest.readonly:
raise TypeError("readinto() requires a writable buffer")
chunk = self._take(len(dest))
dest[: len(chunk)] = chunk
return len(chunk)
On
main, callingflush()on a Drive writer uploads the bytes written so far and marks the stream closed. A later binarywrite()still accepts bytes, butclose()skips the final upload. For example, writingb"first", flushing, writingb"last", and closing leavesb"first"on Drive.This change makes
flush()flush the local buffer andclose()upload once. Binary writes return their byte count. Small files stay in memory and larger files spill to a seekable temporary file. Upload errors propagate on close and the spool is released. Callers that need the file to be visible on Drive should close the writer or exit its context manager.Small reads previously copied the entire unread tail of the current Drive response. The reader now keeps a view of that tail. A positive
block_sizepassed tofs.open()controls the size of Drive download requests.Local measurements
On macOS with CPython 3.11.18, comparing this branch with
4798a5c. The read result is the median of seven runs. The write measurement usestracemallocaround the writer, reuses one input block, and excludes network traffic.Verification
The offline suite passed locally (134 passed, 2 skipped). The filesystem unit tests passed on fsspec 2021.7.0 and 2026.9.0. Coverage includes flush followed by another write in text and binary mode, disk rollover, cleanup after a failed upload, byte-range request sizes, and mixed partial reads and peeks. The existing live Drive round-trip test now flushes between two writes and checks the downloaded bytes. Ruff, mypy, and the Sphinx build passed.
Based on
main. Git verified a clean merge with the separate protocol-registration PR #406.