Repository navigation
Migrate PyDrive2 to Google Drive API v3 - #404
shcheklein wants to merge 8 commits into
Conversation
shcheklein
left a comment
There was a problem hiding this comment.
AI generated (Copilot) — 4-pass review of Drive API v3 migration (#404).
Overall this is a clean, well-scoped migration to Drive API v3 — real discovery + HttpMockSequence offline tests, supportsAllDrives=True on media/permission endpoints, spaces="appDataFolder", and fixing the old != bug in _DeletePermission are all strong improvements. Below are the concrete findings across the four passes (also left inline on the relevant lines).
Pass 1 — Functionality, bugs & gaps
_ListChildren(GetPermissions/GetRevisions) does not wraperrors.HttpErrorintoApiRequestError(pydrive2/files.py:704): Unlike every otherGoogleDriveFileoperation, transient 403/5xx errors or callingGetRevisions()on a folder/Workspace doc leak rawgoogleapiclient.errors.HttpErrorand bypasspydrive_retry.- Field-mask parsing bugs in
Upload()(and_GetList()) (pydrive2/files.py:543-548):- Naive
fields.replace(" ", "").split(",")treats a nested subfield likepermissions(role,id,type)as if top-levelidwere requested, dropping fileidfromfiles.createresponses and causing subsequentUpload()calls to create duplicates. Upload()omits.strip()onfields != "*"(inconsistent with_GetList) and crashes withAttributeErrorif a caller passesparam={"fields": None}becausesetdefaultpreservesNone.
- Naive
Upload()checksself.get("mimeType")beforeself.UpdateMetadata(metadata)when clearingself.content(pydrive2/files.py:556-563): Updating an existing Workspace doc from an ID-only handle (drive.CreateFile({"id": doc_id})) leaves stale local bytes inself.contentbecauseself.get("mimeType")is stillNonebeforeUpdateMetadata(metadata)runs.InsertPermission()omitsfieldsin v3 (pydrive2/files.py:664): Drive v3permissions.createreturns only minimal fields (kind,id,type,role) by default, droppingemailAddress,domain,allowFileDiscovery, etc. (inconsistent withGetPermissions(), which requestspermissions(*)). Defaultingparam.setdefault("fields", "*")restores parity.FetchContent()only mapsfolderandshortcuttoFileNotDownloadableError(pydrive2/files.py:500-506): Non-exportable Workspace types (Google Forms, Sites, Maps) or unsupported export MIME types now raiseApiRequestErrorinstead ofFileNotDownloadableError.GDriveFileSystem.find()root-alias fallback can misattribute multi-parent files (pydrive2/fs/spec.py:505-513): Checking onlyitem["parents"][0]before falling back toself.rootmisattributes a multi-parent file whose matching directory ID is at index>= 1.- Shared-drive delete/trash error detection in
GDriveFileSystem._gdrive_delete_file(): Untouched lineexc.GetField("location") == "file.permissions"should be verified on Drive v3 now thatTrash()usesfiles.update(body={"trashed": True})(v3 403s commonly reportreason == "insufficientFilePermissions"withoutlocation). - Minor gaps:
_FilesDeletestill mutates caller'sparamdict in place;GetAbout(fields=None)will 400 on v3 (fields=fields or "*"is safer);docs/filemanagement.rstupgrade guide doesn't document the new curatedDEFAULT_FILE_FIELDSdefault vs v2's full-resource default;examples/.../upload.pycallsGetAbout(fields="importFormats")even when updating an existing file.
Pass 2 — Design, reusability, unification & organization
- Extract
_ensure_field(fields, required)inpydrive2/files.pyto unify the duplicated field-mask checks inGoogleDriveFileList._GetListandGoogleDriveFile.Upload(with proper top-level parenthesis awareness). - Unify request execution via a
_Execute(self, request)helper onGoogleDriveFile: methods currently mix_AddResourceKeyHeaders(req).execute(...),_WrapRequest(req).execute(...), and inconsistenttry/except HttpErrorwrapping (some withfrom error, some without,_ListChildrenmissing it altogether). - Unify
file_idlookup via a small_FileId()helper (currently split acrossself.metadata.get("id") or self.get("id"),self.metadata.get("id") or self["id"],self["id"], andself.get("id")). - Clean up leftover internal
titlenames inpydrive2/fs/spec.py(_get_remote_item_ids,_gdrive_create_dir,_create_dir,_gdrive_upload_fobj) now that they manipulate v3name. - Test module placement:
pydrive2/test/test_drive_v3.pyis named after the migration PR rather than the units under test, and mixesGoogleDrive,GoogleDriveFileList,GoogleDriveFile, andGDriveFileSystemtests in one file. Putting offline tests in module-oriented files (e.g.test_files_unit.py/test_fs_unit.py, or colocated with per-test@pytest.mark.drivemarkers) and sharing amock_drivefixture inpydrive2/test/conftest.py(reusable bytest_file.py'sauth_with_resource_key_mocktoo) will age much better.
Pass 3 — Simplification & reducing complexity
GoogleDriveFile.GetContentString(pydrive2/files.py:302-312):not isinstance(self.content, io.BytesIO)already subsumesself.content is None, andnot self.dirty["content"]is repeated twice. Factoring outstalecuts the condition in half.GoogleDriveFile.Upload(pydrive2/files.py:516-563): Using_Executeand_ensure_fieldsimplifiesUploadsubstantially.GDriveFileSystemfile-entry dict construction (info,ls,findinpydrive2/fs/spec.py): All three repeat the same{type, name, size, checksum}mapping with slight drift (int(size) if size is not None else Nonevselse size); a small_file_entryhelper unifies them.- Split multi-scenario tests in
test_drive_v3.py:test_v3_create_move_trash_and_copy,test_v3_list_and_about, andtest_v3_permissions_and_revisionsbundle unrelated behaviors and rely on brittle positionalrequests[i]indexing.
Pass 4 — Test quality, naming, fixtures & idiomatic pytest
- Good: All new tests are standalone pytest functions (no
unittest.TestCaseclasses) and exercise real v3 discovery viaHttpMockSequencerather than mocking outgoogleapiclient. - Improve:
- Promote
mock_drive/responseto pytest fixtures inpydrive2/test/conftest.py. - Rename/split umbrella
test_v3_*tests into focused, behavior-named tests. - Add missing unit coverage for:
_MoveParentson an ID-only file (thefiles.get(fields="parents")path used byGDriveFileSystem.mv),_MoveParentsraisingValueErrorforlen(parents) > 1,Upload(param={"fields": ...})retainingid(including nested subfields),ListFile({"fields": "*"})not prependingnextPageToken, andGetPermissions/GetRevisionserror wrapping. - In
test_file.py::test_GFile_Conversion_Lossless_String, preferself.getTempFile()over a hardcoded relative"_tmp_downloaded_file_name.txt"in cwd.
- Promote
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved regressions can break subsequent parent moves and custom-transport upload, trash, and untrash operations, while Workspace updates may return cached source content instead of the converted document.
4 open issues
What changed in this PR
Migrates PyDrive2’s public file API and fsspec backend to Google Drive API v3, including field selection and pagination support for the workflows described in #19 and #42.
Changes:
- Replaces v2 metadata and request formats across file operations and the filesystem adapter.
- Adds Workspace import support and paginated permission and revision retrieval.
- Updates documentation, examples, live tests, and offline request-level coverage.
| File | Description |
|---|---|
| README.rst | Updates API references and usage examples to v3. |
| pydrive2/test/test_util.py | Uses pageSize in listing helpers. |
| pydrive2/test/test_filelist.py | Migrates listing tests to v3 fields and queries. |
| pydrive2/test/test_file.py | Updates file-operation, permission, and conversion tests. |
| pydrive2/test/test_drive_v3.py | Adds simulated HTTP coverage for v3 requests. |
| pydrive2/test/README.rst | Updates the Drive API documentation link. |
| pydrive2/fs/spec.py | Migrates filesystem metadata, queries, and parent handling. |
| pydrive2/files.py | Implements v3 file operations, imports, and pagination. |
| pydrive2/drive.py | Adds field selection to account information requests. |
| pydrive2/auth.py | Builds the Drive v3 service. |
| pydrive2/apiattr.py | Uses v3 pagination and removes legacy URL handling. |
| pydrive2/__pyinstaller/test_hook-googleapiclient.py | Checks for the bundled v3 discovery document. |
| examples/using_folders.py | Uses v3 folder names and parent IDs. |
| examples/Upload-and-autoconvert-to-Google-Drive-Format-Example/upload.py | Adapts conversion uploads and filename queries to v3. |
| examples/strip_bom_example.py | Uses explicit Workspace import MIME types. |
| docs/quickstart.rst | Migrates introductory examples and fixes name lookup. |
| docs/filemanagement.rst | Adds migration guidance and a Workspace import example. |
| docs/filelist.rst | Documents v3 queries, field masks, and pagination. |
Review effort: Balanced
There was a problem hiding this comment.
🟡 Changes recommended
FetchContent() unnecessarily sends a failing binary-media request before exporting Workspace documents with cached MIME types, adding an avoidable round trip to each uncached content read.
1 open issue
4 resolved since last review
Review effort: Balanced
shcheklein
left a comment
There was a problem hiding this comment.
AI generated — Copilot | Four-pass review of current head 9aa2564
I reviewed the v3 migration against the v2 behavior and checked auth/discovery, partial fields and pagination, metadata/list projections, uploads/imports, parent moves, trash/delete, copy, permissions/revisions, fsspec listing/info/move/delete, shared drives, root aliases, examples, PyInstaller discovery, and the offline/live-test split. Findings are inline; below is the compact pass-by-pass result.
Pass 1 — functionality and compatibility
- Functional regression: parent-query batching can return the same legacy multi-parent file more than once; this changes
ls/findresults relative to the old singleORquery. - Edge-case inconsistency: the one-parent guard is applied to updates but not creates. The v3 request path should have one rule for both.
- No other correctness break found in the migrated request shapes. In particular, projected pagination, import
media_mime_type, canonical root moves, resource-key headers, permissions/revisions pagination, trash/update, and shared-drive permission errors are wired through v3 correctly.
Pass 2 — design, reuse, and placement
- Good reuse:
_Execute,_ensure_field,_file_entry, the sharedmock_drivefixture, anderror_response/queryhelpers remove duplicated behavior. - The new offline tests are in module-oriented
test_files_unit.py/test_fs_unit.pylocations rather than a migration-named grab bag. - The migration guide is useful but should call out at least the common raw permission parameter rename noted inline, since callers can pass raw request params.
Pass 3 — simplification
Uploadis doing the right job, but parent validation should be centralized before the create/update branch instead of being hidden only in_MoveParents.- The batched listing needs one shared seen-ID filter rather than downstream ad-hoc deduplication.
Pass 4 — tests and pytest style
- Good: all newly added tests are function-based, use real v3 discovery with
HttpMockSequence, exercise request paths/bodies, and use real pagination behavior rather than mocking the implementation. - Missing regression coverage is the same-ID-across-batches case; the existing batching matrix otherwise covers counts, laziness, duplicate parent names, and 251-directory fan-out.
- The legacy live conversion helper still creates fixed files in the test working directory and uses manual cleanup. Reusing
getTempFile/addCleanup(or function-scopedtmp_pathfixtures if those live tests are converted) would remove cross-run/cwd coupling. The remainingunittest.TestCaseclasses are pre-existing, not introduced by this PR; a separate class-to-function cleanup would avoid inflating this migration.
Validation
python -m pytest -m "not manual and not drive": 97 passed, 2 skipped.ruff check .andruff format --check .: passed.python -m mypy: passed.sphinx-build docs … -b dirhtml -a -W: passed.- Live Drive tests were not run locally because credentials are not available in this review environment.
* Copy files on Drive and reduce metadata and export requests * Release consumed download chunks and export known Workspace files directly
There was a problem hiding this comment.
🟡 Changes recommended
Listed files marked as uploaded but lacking an ID now use files.create on Upload(), silently creating duplicates instead of raising an error for a missing existing-file ID.
1 open issue
1 resolved since last review
Review effort: Balanced

Drive API v3
Closes #19.
Closes #42.
Closes #138.
PyDrive2 now builds the Drive v3 service for the public file API and the fsspec backend. Existing file and folder IDs continue to work. Applications passing raw Drive metadata, queries, or request parameters need to switch to v3 names. The file management guide includes an upgrade section and a Workspace import example.
name, string parent IDs,pageSize, andfiles(...)projections. File lists retain their pagination token when callers project only file fields.addParentsandremoveParents, and trash or restore them with metadata updates. Media uploads can specify a source MIME type separately from the destination Workspace format.Validation
The full offline suite on macOS with Python 3.11 passed (128 passed, two Windows-specific tests skipped). Ruff, mypy, and the Sphinx build with warnings treated as errors passed. Request-level tests use real v3 discovery and a simulated HTTP transport to check pagination, moves, trash, copy, imports, permissions, revisions, folder downloads, root aliases, and files returned from multiple parent-query batches. The tests also exercise caller-supplied HTTP transports and nested field selectors on create and update. Regression tests cover updating a listed file with its ID and rejecting an update before any write when the listing omitted the ID. The existing GitHub Actions suite exercises live Drive credentials.