Skip to content

Migrate PyDrive2 to Google Drive API v3 - #404

Open
shcheklein wants to merge 8 commits into
mainfrom
drive-api-v3
Open

shcheklein wants to merge 8 commits into
mainfrom
drive-api-v3

Conversation

@shcheklein

@shcheklein shcheklein commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Drive API v3

Closes #19.
Closes #42.
Closes #138.

PyDrive2 now builds the Drive v3 service for the public file API and the fsspec backend. Existing file and folder IDs continue to work. Applications passing raw Drive metadata, queries, or request parameters need to switch to v3 names. The file management guide includes an upgrade section and a Workspace import example.

  • Use name, string parent IDs, pageSize, and files(...) projections. File lists retain their pagination token when callers project only file fields.
  • Create and update files through v3, move files with addParents and removeParents, and trash or restore them with metadata updates. Media uploads can specify a source MIME type separately from the destination Workspace format.
  • Download binary content and export Workspace documents through Drive media endpoints. Fetch permissions and revisions across pages, including shared-drive permission lists.
  • Update the fsspec adapter, examples, live tests, and frozen-app discovery document check.
  • The performance changes from Copy files on Drive and reduce metadata and export requests #405 use Drive’s server-side copy, export known Workspace files directly, and release consumed disk-download chunks before requesting the next one.

Validation

The full offline suite on macOS with Python 3.11 passed (128 passed, two Windows-specific tests skipped). Ruff, mypy, and the Sphinx build with warnings treated as errors passed. Request-level tests use real v3 discovery and a simulated HTTP transport to check pagination, moves, trash, copy, imports, permissions, revisions, folder downloads, root aliases, and files returned from multiple parent-query batches. The tests also exercise caller-supplied HTTP transports and nested field selectors on create and update. Regression tests cover updating a listed file with its ID and rejecting an update before any write when the listing omitted the ID. The existing GitHub Actions suite exercises live Drive credentials.

@shcheklein shcheklein left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI generated (Copilot) — 4-pass review of Drive API v3 migration (#404).

Overall this is a clean, well-scoped migration to Drive API v3 — real discovery + HttpMockSequence offline tests, supportsAllDrives=True on media/permission endpoints, spaces="appDataFolder", and fixing the old != bug in _DeletePermission are all strong improvements. Below are the concrete findings across the four passes (also left inline on the relevant lines).


Pass 1 — Functionality, bugs & gaps

  1. _ListChildren (GetPermissions / GetRevisions) does not wrap errors.HttpError into ApiRequestError (pydrive2/files.py:704): Unlike every other GoogleDriveFile operation, transient 403/5xx errors or calling GetRevisions() on a folder/Workspace doc leak raw googleapiclient.errors.HttpError and bypass pydrive_retry.
  2. Field-mask parsing bugs in Upload() (and _GetList()) (pydrive2/files.py:543-548):
    • Naive fields.replace(" ", "").split(",") treats a nested subfield like permissions(role,id,type) as if top-level id were requested, dropping file id from files.create responses and causing subsequent Upload() calls to create duplicates.
    • Upload() omits .strip() on fields != "*" (inconsistent with _GetList) and crashes with AttributeError if a caller passes param={"fields": None} because setdefault preserves None.
  3. Upload() checks self.get("mimeType") before self.UpdateMetadata(metadata) when clearing self.content (pydrive2/files.py:556-563): Updating an existing Workspace doc from an ID-only handle (drive.CreateFile({"id": doc_id})) leaves stale local bytes in self.content because self.get("mimeType") is still None before UpdateMetadata(metadata) runs.
  4. InsertPermission() omits fields in v3 (pydrive2/files.py:664): Drive v3 permissions.create returns only minimal fields (kind, id, type, role) by default, dropping emailAddress, domain, allowFileDiscovery, etc. (inconsistent with GetPermissions(), which requests permissions(*)). Defaulting param.setdefault("fields", "*") restores parity.
  5. FetchContent() only maps folder and shortcut to FileNotDownloadableError (pydrive2/files.py:500-506): Non-exportable Workspace types (Google Forms, Sites, Maps) or unsupported export MIME types now raise ApiRequestError instead of FileNotDownloadableError.
  6. GDriveFileSystem.find() root-alias fallback can misattribute multi-parent files (pydrive2/fs/spec.py:505-513): Checking only item["parents"][0] before falling back to self.root misattributes a multi-parent file whose matching directory ID is at index >= 1.
  7. Shared-drive delete/trash error detection in GDriveFileSystem._gdrive_delete_file(): Untouched line exc.GetField("location") == "file.permissions" should be verified on Drive v3 now that Trash() uses files.update(body={"trashed": True}) (v3 403s commonly report reason == "insufficientFilePermissions" without location).
  8. Minor gaps: _FilesDelete still mutates caller's param dict in place; GetAbout(fields=None) will 400 on v3 (fields=fields or "*" is safer); docs/filemanagement.rst upgrade guide doesn't document the new curated DEFAULT_FILE_FIELDS default vs v2's full-resource default; examples/.../upload.py calls GetAbout(fields="importFormats") even when updating an existing file.

Pass 2 — Design, reusability, unification & organization

  1. Extract _ensure_field(fields, required) in pydrive2/files.py to unify the duplicated field-mask checks in GoogleDriveFileList._GetList and GoogleDriveFile.Upload (with proper top-level parenthesis awareness).
  2. Unify request execution via a _Execute(self, request) helper on GoogleDriveFile: methods currently mix _AddResourceKeyHeaders(req).execute(...), _WrapRequest(req).execute(...), and inconsistent try/except HttpError wrapping (some with from error, some without, _ListChildren missing it altogether).
  3. Unify file_id lookup via a small _FileId() helper (currently split across self.metadata.get("id") or self.get("id"), self.metadata.get("id") or self["id"], self["id"], and self.get("id")).
  4. Clean up leftover internal title names in pydrive2/fs/spec.py (_get_remote_item_ids, _gdrive_create_dir, _create_dir, _gdrive_upload_fobj) now that they manipulate v3 name.
  5. Test module placement: pydrive2/test/test_drive_v3.py is named after the migration PR rather than the units under test, and mixes GoogleDrive, GoogleDriveFileList, GoogleDriveFile, and GDriveFileSystem tests in one file. Putting offline tests in module-oriented files (e.g. test_files_unit.py / test_fs_unit.py, or colocated with per-test @pytest.mark.drive markers) and sharing a mock_drive fixture in pydrive2/test/conftest.py (reusable by test_file.py's auth_with_resource_key_mock too) will age much better.

Pass 3 — Simplification & reducing complexity

  1. GoogleDriveFile.GetContentString (pydrive2/files.py:302-312): not isinstance(self.content, io.BytesIO) already subsumes self.content is None, and not self.dirty["content"] is repeated twice. Factoring out stale cuts the condition in half.
  2. GoogleDriveFile.Upload (pydrive2/files.py:516-563): Using _Execute and _ensure_field simplifies Upload substantially.
  3. GDriveFileSystem file-entry dict construction (info, ls, find in pydrive2/fs/spec.py): All three repeat the same {type, name, size, checksum} mapping with slight drift (int(size) if size is not None else None vs else size); a small _file_entry helper unifies them.
  4. Split multi-scenario tests in test_drive_v3.py: test_v3_create_move_trash_and_copy, test_v3_list_and_about, and test_v3_permissions_and_revisions bundle unrelated behaviors and rely on brittle positional requests[i] indexing.

Pass 4 — Test quality, naming, fixtures & idiomatic pytest

  • Good: All new tests are standalone pytest functions (no unittest.TestCase classes) and exercise real v3 discovery via HttpMockSequence rather than mocking out googleapiclient.
  • Improve:
    • Promote mock_drive / response to pytest fixtures in pydrive2/test/conftest.py.
    • Rename/split umbrella test_v3_* tests into focused, behavior-named tests.
    • Add missing unit coverage for: _MoveParents on an ID-only file (the files.get(fields="parents") path used by GDriveFileSystem.mv), _MoveParents raising ValueError for len(parents) > 1, Upload(param={"fields": ...}) retaining id (including nested subfields), ListFile({"fields": "*"}) not prepending nextPageToken, and GetPermissions/GetRevisions error wrapping.
    • In test_file.py::test_GFile_Conversion_Lossless_String, prefer self.getTempFile() over a hardcoded relative "_tmp_downloaded_file_name.txt" in cwd.

Comment thread pydrive2/drive.py Resolved
Comment thread pydrive2/files.py Resolved
Comment thread pydrive2/files.py Resolved
Comment thread pydrive2/files.py Resolved
Comment thread pydrive2/files.py Resolved
Comment thread pydrive2/fs/spec.py Resolved
Comment thread docs/filemanagement.rst Resolved
Comment thread pydrive2/test/test_drive_v3.py Resolved
Comment thread pydrive2/test/test_drive_v3.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved regressions can break subsequent parent moves and custom-transport upload, trash, and untrash operations, while Workspace updates may return cached source content instead of the converted document.

4 open issues
What changed in this PR

Migrates PyDrive2’s public file API and fsspec backend to Google Drive API v3, including field selection and pagination support for the workflows described in #19 and #42.

Changes:

  • Replaces v2 metadata and request formats across file operations and the filesystem adapter.
  • Adds Workspace import support and paginated permission and revision retrieval.
  • Updates documentation, examples, live tests, and offline request-level coverage.
File Description
README.rst Updates API references and usage examples to v3.
pydrive2/​test/​test_util.py Uses pageSize in listing helpers.
pydrive2/​test/​test_filelist.py Migrates listing tests to v3 fields and queries.
pydrive2/​test/​test_file.py Updates file-operation, permission, and conversion tests.
pydrive2/​test/​test_drive_v3.py Adds simulated HTTP coverage for v3 requests.
pydrive2/​test/​README.rst Updates the Drive API documentation link.
pydrive2/​fs/​spec.py Migrates filesystem metadata, queries, and parent handling.
pydrive2/​files.py Implements v3 file operations, imports, and pagination.
pydrive2/​drive.py Adds field selection to account information requests.
pydrive2/​auth.py Builds the Drive v3 service.
pydrive2/​apiattr.py Uses v3 pagination and removes legacy URL handling.
pydrive2/​__pyinstaller/​test_hook-googleapiclient.py Checks for the bundled v3 discovery document.
examples/​using_folders.py Uses v3 folder names and parent IDs.
examples/​Upload-and-autoconvert-to-Google-Drive-Format-Example/​upload.py Adapts conversion uploads and filename queries to v3.
examples/​strip_bom_example.py Uses explicit Workspace import MIME types.
docs/​quickstart.rst Migrates introductory examples and fixes name lookup.
docs/​filemanagement.rst Adds migration guidance and a Workspace import example.
docs/​filelist.rst Documents v3 queries, field masks, and pagination.

Review effort: Balanced

Comment thread pydrive2/files.py Resolved
Comment thread pydrive2/files.py Resolved
Comment thread pydrive2/files.py Resolved
Comment thread pydrive2/files.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

FetchContent() unnecessarily sends a failing binary-media request before exporting Workspace documents with cached MIME types, adding an avoidable round trip to each uncached content read.

1 open issue
4 resolved since last review

Review effort: Balanced

Comment thread pydrive2/files.py Resolved

@shcheklein shcheklein left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI generated — Copilot | Four-pass review of current head 9aa2564

I reviewed the v3 migration against the v2 behavior and checked auth/discovery, partial fields and pagination, metadata/list projections, uploads/imports, parent moves, trash/delete, copy, permissions/revisions, fsspec listing/info/move/delete, shared drives, root aliases, examples, PyInstaller discovery, and the offline/live-test split. Findings are inline; below is the compact pass-by-pass result.

Pass 1 — functionality and compatibility

  • Functional regression: parent-query batching can return the same legacy multi-parent file more than once; this changes ls/find results relative to the old single OR query.
  • Edge-case inconsistency: the one-parent guard is applied to updates but not creates. The v3 request path should have one rule for both.
  • No other correctness break found in the migrated request shapes. In particular, projected pagination, import media_mime_type, canonical root moves, resource-key headers, permissions/revisions pagination, trash/update, and shared-drive permission errors are wired through v3 correctly.

Pass 2 — design, reuse, and placement

  • Good reuse: _Execute, _ensure_field, _file_entry, the shared mock_drive fixture, and error_response/query helpers remove duplicated behavior.
  • The new offline tests are in module-oriented test_files_unit.py / test_fs_unit.py locations rather than a migration-named grab bag.
  • The migration guide is useful but should call out at least the common raw permission parameter rename noted inline, since callers can pass raw request params.

Pass 3 — simplification

  • Upload is doing the right job, but parent validation should be centralized before the create/update branch instead of being hidden only in _MoveParents.
  • The batched listing needs one shared seen-ID filter rather than downstream ad-hoc deduplication.

Pass 4 — tests and pytest style

  • Good: all newly added tests are function-based, use real v3 discovery with HttpMockSequence, exercise request paths/bodies, and use real pagination behavior rather than mocking the implementation.
  • Missing regression coverage is the same-ID-across-batches case; the existing batching matrix otherwise covers counts, laziness, duplicate parent names, and 251-directory fan-out.
  • The legacy live conversion helper still creates fixed files in the test working directory and uses manual cleanup. Reusing getTempFile/addCleanup (or function-scoped tmp_path fixtures if those live tests are converted) would remove cross-run/cwd coupling. The remaining unittest.TestCase classes are pre-existing, not introduced by this PR; a separate class-to-function cleanup would avoid inflating this migration.

Validation

  • python -m pytest -m "not manual and not drive": 97 passed, 2 skipped.
  • ruff check . and ruff format --check .: passed.
  • python -m mypy: passed.
  • sphinx-build docs … -b dirhtml -a -W: passed.
  • Live Drive tests were not run locally because credentials are not available in this review environment.

Comment thread pydrive2/fs/spec.py Resolved
Comment thread pydrive2/files.py Resolved
Comment thread docs/filemanagement.rst Resolved
Comment thread pydrive2/test/test_file.py Fixed
* Copy files on Drive and reduce metadata and export requests

* Release consumed download chunks and export known Workspace files directly

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Listed files marked as uploaded but lacking an ID now use files.create on Upload(), silently creating duplicates instead of raising an error for a missing existing-file ID.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread pydrive2/files.py Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fs.cp_file: out of memory error (don't store file in RAM) docs: How to get specific fields when listing files. migrate to API v3

2 participants