Skip to content

Copy files on Drive and reduce metadata and export requests - #405

Merged
shcheklein merged 7 commits into
drive-api-v3from
drive-v3-performance
Nov 21, 2026
Merged

shcheklein merged 7 commits into
drive-api-v3from
drive-v3-performance

Conversation

@shcheklein

@shcheklein shcheklein commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Performance on Drive v3

Stacked on #404. The base of this PR is drive-api-v3. Closes #138 when the stack reaches main.

  • GDriveFileSystem.cp_file() resolves the source and destination folder, then uses Drive’s server-side copy. It can create missing destination directories and preserves native Workspace formats.
  • fsspec listings and file information request only the fields they consume. Download progress gets the total size from media responses, removing the separate metadata request.
  • Downloads of known Docs, Sheets, Slides, and Drawings go straight to export. ID-only handles still try a media download first and fall back on Drive’s specific fileNotDownloadable response. Handles with a supplied Workspace MIME type use one export request for disk, stream, and string reads.

Validation

Compared both branches with a simulated 8 MiB file, instrumenting the actual HTTP transport after path resolution. The migration branch downloaded 8 MiB and uploaded 8 MiB across three requests. This branch made one files.copy request and transferred zero file-content bytes.

For a simulated 32 MiB disk download with 8 MiB chunks, peak Python allocations measured by tracemalloc dropped from 16.54 MiB to 8.53 MiB. A regression test checks that the previous response body is released before the next HTTP request begins.

The complete macOS / Python 3.11 offline suite passed (126 passed, two Windows-specific tests skipped), including frozen-app packaging. Ruff, formatting, and mypy passed. Tests cover copy request bodies, known and unknown MIME types, resource keys, BOM removal, permission errors, retry behavior, and progress reporting with multiple chunks or an unknown download size. A live Drive test covers copying into a new directory and reading the source and result.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The server-side copy path bypasses existing retries for transient Drive errors, and MIME types supplied through CreateFile are missed, causing unnecessary failing media requests before export.

2 open issues
What changed in this PR

This PR builds on the Drive API v3 migration to reduce memory use and API requests in PyDrive2. Server-side copying addresses the in-memory copy problem described in #138 while preserving native Workspace formats.

Changes:

  • Replace download-and-upload copies with Drive’s server-side copy, creating destination directories when needed.
  • Narrow metadata requests and derive download progress totals from media responses.
  • Share download/export selection logic and add request-level and live integration coverage.
File Description
pydrive2/​test/​test_fs.py Add a live copy test covering new destination directories and source preservation.
pydrive2/​test/​test_drive_v3.py Test copy requests, export selection, resource keys, BOM removal, permission errors, and progress.
pydrive2/​fs/​spec.py Use server-side copying, narrower metadata fields, and media-based progress totals.
pydrive2/​files.py Share disk and stream download selection and directly export known Workspace types.

Review effort: Balanced

Comment thread pydrive2/fs/spec.py Resolved
Comment thread pydrive2/files.py Resolved

@shcheklein shcheklein left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI generated (Copilot) — 4-pass review of Drive v3 performance improvements (#405).

Extracting _DownloadContent, narrowing fsspec field masks, server-side files.copy in cp_file(), and deriving download total size from media responses are all high-value wins. Below are the bugs, design refinements, simplifications, and test-quality issues found across the four passes (also left inline on the relevant lines).


Pass 1 — Functionality, bugs & gaps

  1. Regression: GDriveFileSystem.cp_file() lost @_gdrive_retry (pydrive2/fs/spec.py:560-569): Previously cp_file went through _gdrive_open_file and _gdrive_upload_fobj, both decorated with @_gdrive_retry. Now source.Copy(...) runs with zero retry handling — any transient 5xx or 403 rate-limit error fails immediately instead of retrying. Extract @_gdrive_retry def _gdrive_copy_file(...) (matching _gdrive_upload_fobj, _gdrive_delete_file, etc.).
  2. _DownloadContent() only checks self.metadata.get("mimeType"), ignoring self.get("mimeType") (pydrive2/files.py:418): For an un-uploaded handle created with a known MIME type (drive.CreateFile({"id": doc_id, "mimeType": "application/vnd.google-apps.document"})), self.metadata is {} while self["mimeType"] holds the MIME type, so it still pays for the wasted get_media 403 round-trip. Using self.metadata.get("mimeType") or self.get("mimeType") (matching file_id right above) fixes this.
  3. _DownloadContent() indexes error.error["code"] directly (pydrive2/files.py:428): If a non-standard/empty error payload omits "code", this raises KeyError: 'code' instead of re-raising ApiRequestError. Use error.error.get("code") != 403 (as spec.py already does).
  4. Inconsistent remove_bom behavior between GetContentFile and GetContentIOBuffer (pydrive2/files.py:365 vs pydrive2/files.py:400): GetContentFile checks (export_type or mimetype) == "text/plain" (stripping BOM on non-export downloads when mimetype="text/plain"), while GetContentIOBuffer (and therefore FetchContent / GetContentString) checks export_type == "text/plain" (only stripping on Workspace exports).
  5. GetContentFile opens/truncates filename twice on export fallback (pydrive2/files.py:353): On ID-only Workspace docs, open(filename, "w+b") runs for the failing get_media call and again for export_media (and still leaves a 0-byte file on disk on unrecoverable errors like 404/403).

Pass 2 — Design, reusability, unification & organization

  1. Follow spec.py's _gdrive_* pattern for copy: Extracting @_gdrive_retry def _gdrive_copy_file(self, source_id, parent_id, name) keeps cp_file consistent with _gdrive_upload_fobj, _gdrive_get_file, _gdrive_open_file, and _gdrive_delete_file, and ensures retries don't re-run _get_item_id.
  2. Centralize fsspec field-mask constants: _gdrive_list (nextPageToken,files(id,name,mimeType,parents,size,md5Checksum)) and info (mimeType,size,md5Checksum) inline field masks; defining module-level constants next to FOLDER_MIME_TYPE keeps the contract with ls/find/info explicit and easy to maintain.
  3. Unify BOM prefix resolution across GetContentFile and GetContentIOBuffer so both apply identical semantics instead of diverging on (export_type or mimetype) vs export_type.
  4. Test location: Like #404, adding files.py and fs/spec.py unit tests into test_drive_v3.py turns it into a grab-bag file. Place download/export tests with file unit tests and cp_file/get_file/field-mask tests with fs unit tests.

Pass 3 — Simplification & reducing complexity

  1. _DownloadContent is a great simplification over the duplicated try/except in #404.
  2. Extract an error_response(code, reason) helper in tests (pydrive2/test/test_drive_v3.py:180-193,215-227): Both export and permission-error tests repeat the same 12-line nested JSON error structure inline.
  3. Avoid branching inside test_workspace_export_requests: Splitting if stream: ... else: ... into focused stream vs file tests (or a parametrized reader helper) keeps each test straight-line and simpler to read.

Pass 4 — Test quality, naming, fixtures, mocks & idiomatic pytest

  1. Over-mocking private internals (_get_item_id) (pydrive2/test/test_drive_v3.py:238-246,260):
    • Both test_filesystem_copy_uses_drive_and_creates_parents and test_filesystem_download_progress_uses_media_response patch private fs._get_item_id and assert its internal call_args_list.
    • Because HttpMockSequence is already in use here, driving the real HTTP flow instead of patching _get_item_id avoids coupling tests to private internals and actually verifies the narrowed _gdrive_list fields parameter (which currently has zero offline test coverage!).
  2. Use real fsspec.Callback() instead of mocker.Mock() in test_filesystem_download_progress_uses_media_response, and test a multi-chunk (206 Partial Content + Content-Range) download to actually exercise the total != reported_size guard (plus a chunked response without Content-Length where total is None).
  3. Missing unit test coverage for PR #405 changes:
    • Narrowed fields on _gdrive_list and info().
    • cp_file() retry behavior on transient 5xx / 403 rate-limit errors (once @_gdrive_retry is restored).
    • Un-uploaded drive.CreateFile({"id": "doc", "mimeType": "application/vnd.google-apps.document"}) skipping get_media.
  4. Good: Standalone pytest functions (no Test* classes) and a live integration test (test_copy_file_to_new_directory in test_fs.py) verifying real Drive behavior.

Comment thread pydrive2/files.py Resolved
Comment thread pydrive2/files.py Resolved
Comment thread pydrive2/files.py Resolved
Comment thread pydrive2/fs/spec.py Resolved
Comment thread pydrive2/fs/spec.py Resolved
Comment thread pydrive2/fs/spec.py Resolved
Comment thread pydrive2/test/test_drive_v3.py Resolved
Comment thread pydrive2/test/test_drive_v3.py Resolved
Comment thread pydrive2/test/test_drive_v3.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Disk downloads retain the previous chunk while allocating the next response, adding roughly 100 MiB of peak memory at the default chunk size and risking memory exhaustion in constrained processes.

1 open issue
2 resolved since last review

Review effort: Balanced

Comment thread pydrive2/files.py Resolved

@shcheklein shcheklein left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI generated — Copilot | Four-pass review of current head 8cd79f3

I reviewed the performance branch against the exact #404 head and checked server-side copy, destination-directory creation, retries, resource keys, regular and Workspace downloads, BOM/encoding behavior, progress callbacks, fsspec field masks, memory transfer behavior, and the live/offline split. Findings are inline; the four-pass summary follows.

Pass 1 — functionality and performance

  • Performance gap: the known-Workspace fast path is used by disk and stream downloads, but GetContentString / FetchContent still force a metadata request first when the caller supplied mimeType. A verified handle made GET /files/doc before GET /files/doc/export.
  • Server-side cp_file is otherwise correct: it resolves/creates destination parents, sends one files.copy, preserves the source format, and retries only the copy request through the existing ApiRequestError policy.
  • Download selection, fallback-on-fileNotDownloadable, malformed-error handling, resource-key headers, multi-chunk BOM removal, incremental UTF-8 decoding, and media-derived progress sizes are correct.
  • The new destination-preservation test covers pre-buffer errors only; its name currently implies a stronger whole-download guarantee.

Pass 2 — reuse, placement, and ownership

  • Good consolidation: GetContentFile now reuses GetContentIOBuffer, _INFO_FIELDS composes into _LIST_FIELDS, disk/stream tests share readers, and copy/retry helpers match the existing fs/spec.py pattern.
  • MediaIoReadable is now the right owner of streaming concerns (BOM prefix, incremental decoding, progress). The previous IoBuffer encoding responsibility is now dead and can be removed rather than retained as IoBuffer(None).
  • The test additions are in the correct file-module and fs-module unit files; the one live copy test is appropriately in the Drive integration suite.

Pass 3 — simplification

  • Remove the now-production-dead _RemovePrefix; its callers moved to streaming prefix removal. _InsertPrefix also has no production caller, so the FilePrefixTest class is testing private legacy machinery rather than observable behavior. The public BOM tests already cover the retained contract.
  • Simplify IoBuffer to a bytes-only one-chunk sink and keep all decoding/prefix logic in MediaIoReadable.
  • A shared "resolve download type / known MIME" helper would let direct disk, stream, and string downloads make the same request decision without duplicating decorators or metadata rules.

Pass 4 — tests, fixtures, naming, and mocks

  • Good: all newly added tests are function-based, use the shared mock_drive transport plus real v3 discovery, and use a real fsspec Callback. The copy test drives path lookup, directory creation, and request bodies end-to-end instead of mocking private methods.
  • Extend the direct-export test across all four members of WORKSPACE_EXPORT_TYPES; only the document MIME is currently exercised.
  • Clarify/extend the destination-error test as noted inline.
  • No class-based tests, private mocks, or implementation-only assertions were added.

Validation

  • python -m pytest -m "not manual and not drive": 113 passed, 2 skipped.
  • ruff check . and ruff format --check .: passed.
  • python -m mypy: passed.
  • sphinx-build docs … -b dirhtml -a -W: passed.
  • Live Drive tests were not run locally because credentials are not available in this review environment.

Comment thread pydrive2/files.py Resolved
Comment thread pydrive2/files.py Resolved
Comment thread pydrive2/test/test_files_unit.py Resolved
Comment thread pydrive2/test/test_files_unit.py Resolved
@shcheklein
shcheklein merged commit 983e23a into drive-api-v3 Nov 21, 2026
21 checks passed
@shcheklein
shcheklein deleted the drive-v3-performance branch November 21, 2026 22:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants