Skip to content

Register pydrive2:// with fsspec - #406

Open
aryankalantri wants to merge 7 commits into
iterative:mainfrom
aryankalantri:fix/fsspec-register-gdrive-354
Open

aryankalantri wants to merge 7 commits into
iterative:mainfrom
aryankalantri:fix/fsspec-register-gdrive-354

Conversation

@aryankalantri

@aryankalantri aryankalantri commented Nov 21, 2026 •

Copy link
Copy Markdown

Fixes #354.

Installing PyDrive2[fsspec] registers pydrive2:// through the standard fsspec.specs entry-point group. Applications can open pydrive2://FILE_ID or pydrive2://FOLDER_ID/path/to/file with their existing credentials. The filesystem factory defaults to My Drive and accepts a folder ID through the path keyword.

The dedicated scheme preserves fsspec's existing gdrive provider. DVC's gdrive:// remotes continue to construct the PyDrive2 filesystem directly. fsspec registry · DVC adapter

Tests cover discovery in a fresh interpreter, case-sensitive file IDs, and sibling URLs sharing a filesystem and folder lookup. The live test writes a named file through fsspec and reads it back by ID.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The offline registration test raises AttributeError instead of exercising discovery, and the documentation example passes an unsupported positional argument that raises TypeError.

2 open issues
What changed in this PR

This PR addresses #354 by registering PyDrive2’s existing filesystem with fsspec, enabling discovery through the gdrive protocol without importing PyDrive2 directly.

Changes:

  • Adds an fsspec entry point and declares the filesystem’s gdrive protocol.
  • Defaults the constructor’s path to "root" so fsspec can instantiate it without a positional path.
  • Adds usage documentation and offline tests for registration and missing-credential behavior.
File Description
pyproject.toml Registers the gdrive fsspec entry point.
pydrive2/​test/​test_fs_registration.py Adds offline checks for discovery, protocol declaration, and credential errors.
pydrive2/​fs/​spec.py Declares the protocol and defaults path to "root".
docs/​fsspec.rst Documents fsspec factory and URL usage, including credential requirements.

Review effort: Balanced

Comment thread pydrive2/test/test_fs_registration.py Resolved
Comment thread docs/fsspec.rst Resolved
@shcheklein shcheklein changed the title feat(fs): register gdrive protocol with fsspec (#354) Register pydrive2:// with fsspec and improve streamed file I/O Nov 22, 2026

@shcheklein shcheklein left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reproduced both inline findings and pushed fixes in 4615fa3. The original package metadata also passed its own test while fsspec continued selecting a different backend. The new test starts a fresh Python process and resolves PyDrive2 through the public fsspec API.

I used the same entry-point group as Hugging Face and the same URL-to-storage-options hook as sshfs. PyDrive2 has its own scheme, so installing it preserves fsspec's existing Google Drive provider. DVC constructs the class directly and can continue using its own gdrive:// remotes. Hugging Face registration · sshfs URL handling · DVC adapter

One binary file-ID read took one mocked media GET. Reading two sibling files reused the shared-drive and parent-folder lookups. In a local Python 3.11 benchmark, consuming one 8 MiB in-memory chunk in 8 KiB reads improved from 192 ms to 0.69 ms. Drive network time is outside that measurement.

The backend requires credentials and exposes sequential reads. Each filesystem is rooted at one ID, so batch sibling paths under the same folder. Drive allows duplicate names and a write creates a new item. Those semantics are documented for downstream callers.

@shcheklein

Copy link
Copy Markdown
Member

/test-drive

1 similar comment
@shcheklein

Copy link
Copy Markdown
Member

/test-drive

@shcheklein

Copy link
Copy Markdown
Member

/test-drive

Restore the existing stream reader, download chunking, and buffered writer. Small-read copying and write buffering are separate concerns from registering a filesystem protocol.
@shcheklein shcheklein changed the title Register pydrive2:// with fsspec and improve streamed file I/O Register pydrive2:// with fsspec Nov 22, 2026
@shcheklein

Copy link
Copy Markdown
Member

/test-drive

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Regsister fsspec interface with fsspec

3 participants