Repository navigation
Copy files on Drive and reduce metadata and export requests - #405
Conversation
# Conflicts: # pydrive2/test/test_drive_v3.py
There was a problem hiding this comment.
🟡 Changes recommended
The server-side copy path bypasses existing retries for transient Drive errors, and MIME types supplied through CreateFile are missed, causing unnecessary failing media requests before export.
2 open issues
What changed in this PR
This PR builds on the Drive API v3 migration to reduce memory use and API requests in PyDrive2. Server-side copying addresses the in-memory copy problem described in #138 while preserving native Workspace formats.
Changes:
- Replace download-and-upload copies with Drive’s server-side copy, creating destination directories when needed.
- Narrow metadata requests and derive download progress totals from media responses.
- Share download/export selection logic and add request-level and live integration coverage.
| File | Description |
|---|---|
pydrive2/test/test_fs.py |
Add a live copy test covering new destination directories and source preservation. |
pydrive2/test/test_drive_v3.py |
Test copy requests, export selection, resource keys, BOM removal, permission errors, and progress. |
pydrive2/fs/spec.py |
Use server-side copying, narrower metadata fields, and media-based progress totals. |
pydrive2/files.py |
Share disk and stream download selection and directly export known Workspace types. |
Review effort: Balanced
shcheklein
left a comment
There was a problem hiding this comment.
AI generated (Copilot) — 4-pass review of Drive v3 performance improvements (#405).
Extracting _DownloadContent, narrowing fsspec field masks, server-side files.copy in cp_file(), and deriving download total size from media responses are all high-value wins. Below are the bugs, design refinements, simplifications, and test-quality issues found across the four passes (also left inline on the relevant lines).
Pass 1 — Functionality, bugs & gaps
- Regression:
GDriveFileSystem.cp_file()lost@_gdrive_retry(pydrive2/fs/spec.py:560-569): Previouslycp_filewent through_gdrive_open_fileand_gdrive_upload_fobj, both decorated with@_gdrive_retry. Nowsource.Copy(...)runs with zero retry handling — any transient 5xx or 403 rate-limit error fails immediately instead of retrying. Extract@_gdrive_retry def _gdrive_copy_file(...)(matching_gdrive_upload_fobj,_gdrive_delete_file, etc.). _DownloadContent()only checksself.metadata.get("mimeType"), ignoringself.get("mimeType")(pydrive2/files.py:418): For an un-uploaded handle created with a known MIME type (drive.CreateFile({"id": doc_id, "mimeType": "application/vnd.google-apps.document"})),self.metadatais{}whileself["mimeType"]holds the MIME type, so it still pays for the wastedget_media403 round-trip. Usingself.metadata.get("mimeType") or self.get("mimeType")(matchingfile_idright above) fixes this._DownloadContent()indexeserror.error["code"]directly (pydrive2/files.py:428): If a non-standard/empty error payload omits"code", this raisesKeyError: 'code'instead of re-raisingApiRequestError. Useerror.error.get("code") != 403(asspec.pyalready does).- Inconsistent
remove_bombehavior betweenGetContentFileandGetContentIOBuffer(pydrive2/files.py:365vspydrive2/files.py:400):GetContentFilechecks(export_type or mimetype) == "text/plain"(stripping BOM on non-export downloads whenmimetype="text/plain"), whileGetContentIOBuffer(and thereforeFetchContent/GetContentString) checksexport_type == "text/plain"(only stripping on Workspace exports). GetContentFileopens/truncatesfilenametwice on export fallback (pydrive2/files.py:353): On ID-only Workspace docs,open(filename, "w+b")runs for the failingget_mediacall and again forexport_media(and still leaves a 0-byte file on disk on unrecoverable errors like 404/403).
Pass 2 — Design, reusability, unification & organization
- Follow
spec.py's_gdrive_*pattern for copy: Extracting@_gdrive_retry def _gdrive_copy_file(self, source_id, parent_id, name)keepscp_fileconsistent with_gdrive_upload_fobj,_gdrive_get_file,_gdrive_open_file, and_gdrive_delete_file, and ensures retries don't re-run_get_item_id. - Centralize fsspec field-mask constants:
_gdrive_list(nextPageToken,files(id,name,mimeType,parents,size,md5Checksum)) andinfo(mimeType,size,md5Checksum) inline field masks; defining module-level constants next toFOLDER_MIME_TYPEkeeps the contract withls/find/infoexplicit and easy to maintain. - Unify BOM prefix resolution across
GetContentFileandGetContentIOBufferso both apply identical semantics instead of diverging on(export_type or mimetype)vsexport_type. - Test location: Like
#404, addingfiles.pyandfs/spec.pyunit tests intotest_drive_v3.pyturns it into a grab-bag file. Place download/export tests with file unit tests andcp_file/get_file/field-mask tests with fs unit tests.
Pass 3 — Simplification & reducing complexity
_DownloadContentis a great simplification over the duplicated try/except in#404.- Extract an
error_response(code, reason)helper in tests (pydrive2/test/test_drive_v3.py:180-193,215-227): Both export and permission-error tests repeat the same 12-line nested JSON error structure inline. - Avoid branching inside
test_workspace_export_requests: Splittingif stream: ... else: ...into focused stream vs file tests (or a parametrized reader helper) keeps each test straight-line and simpler to read.
Pass 4 — Test quality, naming, fixtures, mocks & idiomatic pytest
- Over-mocking private internals (
_get_item_id) (pydrive2/test/test_drive_v3.py:238-246,260):- Both
test_filesystem_copy_uses_drive_and_creates_parentsandtest_filesystem_download_progress_uses_media_responsepatch privatefs._get_item_idand assert its internalcall_args_list. - Because
HttpMockSequenceis already in use here, driving the real HTTP flow instead of patching_get_item_idavoids coupling tests to private internals and actually verifies the narrowed_gdrive_listfieldsparameter (which currently has zero offline test coverage!).
- Both
- Use real
fsspec.Callback()instead ofmocker.Mock()intest_filesystem_download_progress_uses_media_response, and test a multi-chunk (206 Partial Content+Content-Range) download to actually exercise thetotal != reported_sizeguard (plus a chunked response withoutContent-Lengthwheretotal is None). - Missing unit test coverage for PR
#405changes:- Narrowed
fieldson_gdrive_listandinfo(). cp_file()retry behavior on transient 5xx / 403 rate-limit errors (once@_gdrive_retryis restored).- Un-uploaded
drive.CreateFile({"id": "doc", "mimeType": "application/vnd.google-apps.document"})skippingget_media.
- Narrowed
- Good: Standalone pytest functions (no
Test*classes) and a live integration test (test_copy_file_to_new_directoryintest_fs.py) verifying real Drive behavior.
There was a problem hiding this comment.
🟡 Changes recommended
Disk downloads retain the previous chunk while allocating the next response, adding roughly 100 MiB of peak memory at the default chunk size and risking memory exhaustion in constrained processes.
1 open issue
2 resolved since last review
Review effort: Balanced
shcheklein
left a comment
There was a problem hiding this comment.
AI generated — Copilot | Four-pass review of current head 8cd79f3
I reviewed the performance branch against the exact #404 head and checked server-side copy, destination-directory creation, retries, resource keys, regular and Workspace downloads, BOM/encoding behavior, progress callbacks, fsspec field masks, memory transfer behavior, and the live/offline split. Findings are inline; the four-pass summary follows.
Pass 1 — functionality and performance
- Performance gap: the known-Workspace fast path is used by disk and stream downloads, but
GetContentString/FetchContentstill force a metadata request first when the caller suppliedmimeType. A verified handle madeGET /files/docbeforeGET /files/doc/export. - Server-side
cp_fileis otherwise correct: it resolves/creates destination parents, sends onefiles.copy, preserves the source format, and retries only the copy request through the existingApiRequestErrorpolicy. - Download selection, fallback-on-
fileNotDownloadable, malformed-error handling, resource-key headers, multi-chunk BOM removal, incremental UTF-8 decoding, and media-derived progress sizes are correct. - The new destination-preservation test covers pre-buffer errors only; its name currently implies a stronger whole-download guarantee.
Pass 2 — reuse, placement, and ownership
- Good consolidation:
GetContentFilenow reusesGetContentIOBuffer,_INFO_FIELDScomposes into_LIST_FIELDS, disk/stream tests share readers, and copy/retry helpers match the existingfs/spec.pypattern. MediaIoReadableis now the right owner of streaming concerns (BOM prefix, incremental decoding, progress). The previousIoBufferencoding responsibility is now dead and can be removed rather than retained asIoBuffer(None).- The test additions are in the correct file-module and fs-module unit files; the one live copy test is appropriately in the Drive integration suite.
Pass 3 — simplification
- Remove the now-production-dead
_RemovePrefix; its callers moved to streaming prefix removal._InsertPrefixalso has no production caller, so theFilePrefixTestclass is testing private legacy machinery rather than observable behavior. The public BOM tests already cover the retained contract. - Simplify
IoBufferto a bytes-only one-chunk sink and keep all decoding/prefix logic inMediaIoReadable. - A shared "resolve download type / known MIME" helper would let direct disk, stream, and string downloads make the same request decision without duplicating decorators or metadata rules.
Pass 4 — tests, fixtures, naming, and mocks
- Good: all newly added tests are function-based, use the shared
mock_drivetransport plus real v3 discovery, and use a real fsspecCallback. The copy test drives path lookup, directory creation, and request bodies end-to-end instead of mocking private methods. - Extend the direct-export test across all four members of
WORKSPACE_EXPORT_TYPES; only the document MIME is currently exercised. - Clarify/extend the destination-error test as noted inline.
- No class-based tests, private mocks, or implementation-only assertions were added.
Validation
python -m pytest -m "not manual and not drive": 113 passed, 2 skipped.ruff check .andruff format --check .: passed.python -m mypy: passed.sphinx-build docs … -b dirhtml -a -W: passed.- Live Drive tests were not run locally because credentials are not available in this review environment.


Performance on Drive v3
Stacked on #404. The base of this PR is
drive-api-v3. Closes #138 when the stack reachesmain.GDriveFileSystem.cp_file()resolves the source and destination folder, then uses Drive’s server-side copy. It can create missing destination directories and preserves native Workspace formats.fileNotDownloadableresponse. Handles with a supplied Workspace MIME type use one export request for disk, stream, and string reads.Validation
Compared both branches with a simulated 8 MiB file, instrumenting the actual HTTP transport after path resolution. The migration branch downloaded 8 MiB and uploaded 8 MiB across three requests. This branch made one
files.copyrequest and transferred zero file-content bytes.For a simulated 32 MiB disk download with 8 MiB chunks, peak Python allocations measured by
tracemallocdropped from 16.54 MiB to 8.53 MiB. A regression test checks that the previous response body is released before the next HTTP request begins.The complete macOS / Python 3.11 offline suite passed (126 passed, two Windows-specific tests skipped), including frozen-app packaging. Ruff, formatting, and mypy passed. Tests cover copy request bodies, known and unknown MIME types, resource keys, BOM removal, permission errors, retry behavior, and progress reporting with multiple chunks or an unknown download size. A live Drive test covers copying into a new directory and reading the source and result.