Skip to content

MnemoSync: Google Drive sync on a shared sync engine - #4

Open
ehitimum wants to merge 6 commits into
codex/mnemosync-backupfrom
codex/mnemosync-drive
Open

ehitimum wants to merge 6 commits into
codex/mnemosync-backupfrom
codex/mnemosync-drive

Conversation

@ehitimum

Copy link
Copy Markdown
Owner

Adds optional Google Drive sync for MnemoScript projects, built on a new transport-neutral sync engine. Stacked on #3 (codex/mnemosync-backup); retarget after #2 and #3 merge.

Why the engine is here too. A review of #2 and #3 against the MnemoSync plan found them sound, but only Phase 0 is done (strategy, CI/undo fixes, backup/restore). The local-sync parts the Drive design depends on (sync state, revisions, merge, transport interface) did not exist yet, so this PR builds them first. LAN sync can use the same Transport trait later.

How it works

  • Sync state. Projects are split into entities: project details, folders, chapter details, chapter text, images.
    • Each change is an immutable revision. Its id is a hash of its content and its parents, so it is the same on every device and doesn't depend on the clock.
    • State lives in SQLite (<data dir>/sync/mnemosync.db), outside project folders. Project JSON files stay the source of truth.
    • Each sync round compares the files on disk with the last known state, so a crash after a save loses nothing.
  • Merging.
    • Metadata merges field by field, deterministically.
    • Chapter text edited on two devices keeps both versions as a conflict.
    • An edit beats a concurrent delete.
    • The folder tree is repaired after merging (no chapters in deleted folders, no cycles).
    • The open chapter's text, or its deletion, is held back until the writer loads it. Remote renames and moves still apply without losing unsaved typing.
  • Drive.
    • Only the hidden appDataFolder is used (drive.appdata scope).
    • Content files are uploaded before the immutable batch files that reference them.
    • New batches are found through the changes feed, with a full listing as fallback.
    • Duplicate or forged files are ignored. Auth, quota, rate-limit and outage errors back off without losing queued work.
    • Image references sync as mnemo-asset:<file>; the disk format is unchanged.
  • Sign-in.
    • Desktop uses a loopback redirect with PKCE in the writer's own browser, with the refresh token in the OS keychain.
    • Android uses Google Play services' AuthorizationClient through an inline google-auth plugin; no refresh token is stored.
    • A different Google account is refused while projects are tied to the previous one.
  • UI. Built separately for the desktop and phone shells:
    • "Sync with Google Drive" settings card: connect, per-project switches, download from other devices, sync notes.
    • A status pill, shown only for synced projects.
    • A floating banner for the open chapter.
    • A side-by-side conflict review that keeps the other version as its own chapter by default.

Design, Google Cloud setup and open items: docs/MNEMOSYNC_DRIVE.md.

Bugs fixed along the way (each with a regression test)

  • Restore left fantasy-map "My Imports" paths pointing at the original project (MnemoSync Phase 0: verified project backups and safe restore #3).
  • Imported images were written non-atomically, so a crash or a sync scan could see half a file.
  • Forgotten projects kept syncing.
  • Two bugs in this PR's own first version, caught in self-review:
    • turning sync off and on reused batch numbers;
    • re-enabling under a new Google account published nothing.
  • Vitest's one-worker-per-core default made App-mounting suites and the streamed AI mock time out on a busy machine. Workers are now capped at 50% and the AI waits widened.

Testing

  • npm run check passes: lint, types, 145 tests, build. npm run test:regression: 56 tests.
  • cargo test passes: 90 tests (32 sync), 3 existing live-AI tests ignored. cargo clippy -- -D warnings is clean.
  • The engine tests simulate 2–3 devices with reordered and duplicated deliveries, offline rounds, late uploads, conflicts, deletes, folder cycles and forged data.
  • npx tauri android build --apk --debug --target aarch64 builds, including the Kotlin plugin and play-services-auth. It has not been run on a device.

Not verified

  • Live Google sign-in and real desktop ↔ Android sync. These need your own Desktop and Android OAuth clients in one Cloud project (app/src-tauri/.env.local); steps are in the doc.
  • Real quota and outage behaviour (only unit-tested).
  • Visual review: the local browser preview is blocked by a saved permission policy.

Not included

Smart Merge for projects created separately on two devices, LAN sync, web sync, history pruning.

🤖 Generated with Claude Code

ehitimum added a commit that referenced this pull request Nov 15, 2026
Data loss (each with an engine regression):
- An unreadable chapter or a missing documents folder no longer becomes a
  delete on every device; the last good state is kept with one note.
- Engine writes re-check the file under project::files_lock (shared with the
  save commands) and capture a newer save first, so it merges or conflicts.
- Conflict review and resolve scan first, so the latest typing is a version.
- Delete + two different edits keeps both edits as a conflict.
- Remote deletes and replaced damaged copies go to <project>/sync-trash/.

Sign-in and Drive client:
- A stray loopback request no longer ends the sign-in.
- Retrying Connect keeps a valid grant unless Google rejects it.
- Drive sync is off on Linux (no keyring backend) and says why.
- Read the newer error shape (SERVICE_DISABLED); shorter text in release.
- Expired change cursor detected by status; catalogue date kept current.

Smaller: id validation in sync commands, Android device model name, Drive
space used in settings, "Back up, then sync", Beta label, stricter live-test
cleanup. Docs and test matrix updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ehitimum

ehitimum commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner Author

Changes for the review (commit f21be89, rebased from 3311192)

Both reviews (at 50e4beb and the follow-up at 5444c1c) are addressed. Each fix has a regression test.

Data-loss findings

# Finding Change
1 Unreadable chapter becomes a delete everywhere read_disk records chapter files that exist but cannot be read (bad JSON, id ≠ file name, locked), and a missing documents/ folder. scan keeps their last synced state and adds one note instead of a tombstone. Remote deletes, and damaged copies replaced by a good version, move to <project>/sync-trash/ (30 days) instead of remove_file. Tests: a_chapter_damaged_on_one_device_is_never_deleted_elsewhere, a_missing_documents_folder_deletes_nothing, a_remote_delete_moves_the_chapter_to_the_sync_trash.
2 Remote update overwrites an unscanned save Every engine write goes through write_checked. It re-reads the entity and compares it with the scanned state under project::files_lock, which Document::save/delete and Project::save also take, so the check and the write are atomic. On a difference, it scans first, then decides again (merge or conflict). Engine-made differences (download skeleton, empty file before its text, trashed chapter) are recognised. Test: a_save_after_the_scan_is_never_overwritten_by_a_remote_update.
3 Resolve drops recent typing conflict_versions and resolve scan first. Choosing "this device" from an older screen keeps the latest text. If a save lands mid-resolve, it becomes a new conflict, not an overwrite. Test: resolving_a_conflict_keeps_typing_saved_since_the_last_round.
4 Delete + two different edits keeps one Tombstone heads are absorbed into the lowest live head (the same on every device), then the live heads stay a normal conflict. Test: a_delete_and_two_different_edits_keep_both_edits.

Other findings

  • 5 Linux keyring: Drive sync is off on Linux (auth::unavailable_reason), and the card shows why instead of offering Connect.
  • 6 Stray loopback request: a request without the right state gets a 400 page and the sign-in keeps waiting (Redirect::Foreign).
  • 7 Unvalidated ids: valid_ids is applied to sync_conflict, sync_resolve_conflict, sync_disable_project, sync_apply_staged and sync_set_active.
  • Smaller points: Android names itself by model (ro.product.model). The settings card shows the Drive space used (drive_usage). An expired change cursor is detected by status code (send_raw + cursor_expired). The catalogue date moves forward when a round publishes (at most hourly). Turning a project on offers "Back up, then sync", and the card has a Beta label.

Follow-up review (5444c1c)

  1. Retry: Connect keeps a valid grant and only signs in again on an Auth error. Other errors are returned and the grant is kept.
  2. SERVICE_DISABLED: classify also reads error.details[].reason, with a test body in the newer shape. Release builds show "isn't set up correctly in this build".
  3. Live test: cleanup failure fails the test. The project is named zz-live-test-… and its id is printed. The header says Windows/macOS only.
  • The test matrix now states the live run covers the engine happy path only. The machine-specific DEVLOG lines were removed.

Checks

npm run check (149 tests incl. regressions, lint, types, build) · cargo test 100 passed · cargo clippy --all-targets -- -D warnings clean.

Not verified: Android sign-in and real desktop ↔ Android sync. The new settings-card controls were not checked visually.

Still open:

  • A way to restore from sync-trash/ inside the app.
  • Whether to put sync behind an off-by-default experimental switch, rather than the Beta label.

🤖 Generated with Claude Code

ehitimum and others added 6 commits November 15, 2026 16:38
Add the local sync core used by every transport: per-entity immutable
revisions (project, folder, chapter metadata, chapter text, assets) with
content-derived ids, SQLite WAL state outside project folders, disk scan
change capture, causal fast-forward, deterministic field merges, preserved
text conflicts, edit-beats-delete, staged updates for the open chapter,
frozen batches for crash-safe publishing, and portable asset references.

Fix backup restore leaving fantasy-map "My Imports" paths pointing at the
original project (regression tests in backup/tests.rs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Desktop sign-in uses the loopback redirect with PKCE (S256) in the writer's
browser and only the drive.appdata scope; the refresh token lives in the OS
secret store, access tokens in memory. The OAuth client comes from the
environment or a git-ignored .env.local at build time.

The Drive transport publishes gzip blobs before immutable batch files in
appDataFolder, finds new batches through the changes feed with a listing
fallback, ignores duplicate uploads and malformed files, and classifies
401/403/429/5xx/quota errors without discarding queued work.

A background worker syncs enabled projects after saves (debounced) and on a
poll with backoff, refuses silent Google account switches, and exposes
status, download, conflict and staged-update commands.

Also write imported images atomically so a crash or a sync scan never sees
a half-written asset, and stop syncing a project that leaves the library.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lict review

Settings gains a Google Drive card (desktop and phone variants) with
sign-in, per-project sync toggles, downloads from other devices and sync
notes; a quiet status pill appears only for synced projects. The open
chapter is reported to the backend so its text is never replaced under the
editor: a banner offers to load the newer version (after saving), and
parallel edits open a side-by-side review that keeps the other version as
its own chapter by default. Remote renames and moves merge into the open
chapter without dropping unsaved typing.

Regression suites cover the settings flows, banner, review (inert previews)
and the App save boundary. Widen the AI suite's streamed-response waits and
cap Vitest at half the cores: App-mounting suites timed out on busy desktops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the inline google-auth Android plugin (Google Play services
AuthorizationClient for drive.appdata, no refresh token stored) with its
ACL permission, and silent re-authorization when the short-lived token
lapses.

Fix two sync bugs found in review: turning a project's sync off and on
reset its batch counter so new batches reused published numbers, and
re-enabling under another Google account published nothing (both covered
by engine regressions). Report needs-auth after signing out while projects
stay bound to the account. Mark desktop-only pairing helpers so Android
builds are warning-free.

Document the design as built, Google Cloud setup and open items in
MNEMOSYNC_DRIVE.md and update architecture, devlog, roadmap, testing,
threat model, test matrix and the MnemoSync handoff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Drive test

The first live setup showed a raw "403: accessNotConfigured" when the
Cloud project lacked the Drive API; it now says what to enable. A retry
after such a failure reuses the stored grant instead of repeating the
browser consent.

Add an ignored live test (cargo test live_drive -- --ignored) that runs two
engines against the real appDataFolder under a throwaway project and
cleans up. Passed against the writer's own Cloud project.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Data loss (each with an engine regression):
- An unreadable chapter or a missing documents folder no longer becomes a
  delete on every device; the last good state is kept with one note.
- Engine writes re-check the file under project::files_lock (shared with the
  save commands) and capture a newer save first, so it merges or conflicts.
- Conflict review and resolve scan first, so the latest typing is a version.
- Delete + two different edits keeps both edits as a conflict.
- Remote deletes and replaced damaged copies go to <project>/sync-trash/.

Sign-in and Drive client:
- A stray loopback request no longer ends the sign-in.
- Retrying Connect keeps a valid grant unless Google rejects it.
- Drive sync is off on Linux (no keyring backend) and says why.
- Read the newer error shape (SERVICE_DISABLED); shorter text in release.
- Expired change cursor detected by status; catalogue date kept current.

Smaller: id validation in sync commands, Android device model name, Drive
space used in settings, "Back up, then sync", Beta label, stricter live-test
cleanup. Docs and test matrix updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant