Skip to content

Resubmit collection backups that Solr has lost track of - #910

Open
HoustonPutman wants to merge 6 commits into
apache:mainfrom
HoustonPutman:backup-notfound-resubmit
Open

HoustonPutman wants to merge 6 commits into
apache:mainfrom
HoustonPutman:backup-notfound-resubmit

Conversation

@HoustonPutman

Copy link
Copy Markdown
Contributor

Fixes #824

This replaces #846. #846 treats a notfound async status as a terminal failure. This PR instead submits the backup again, so a recurring backup keeps going and still produces a backup, rather than recording a failed run. It builds on #907, which handles a backup's async ID being present in Solr before it is submitted.

The problem

A collection backup whose async status Solr reports as notfound stayed inProgress forever. Because the SolrBackup never finished, a recurring backup was never scheduled again and maxSaved pruning stopped. Nothing was reported.

Changes

  • Resubmit on notfound. Once a collection backup has been notfound for more than 5 seconds after it was submitted, it is submitted again, with the same name and async ID. Its start time is reset on every submission, so the 5 second grace also applies after a resubmission.
  • Delete the async status only after the result is saved. Solr's async status for a finished collection backup used to be deleted before the SolrBackup status recording the result was saved. If that save failed, the next reconcile found the backup still in progress, with Solr reporting it as notfound. That is one way into SolrBackup may never finish if REQUESTSTATUS returns notfound for an accepted backup #824.
    • Reconcile now fetches the SolrCloud and builds its auth context.
    • reconcileSolrCloudBackup returns the collections whose async requests finished.
    • Reconcile deletes their async statuses only once the status patch succeeds.
  • Don't finish without a collection that failed to start. If a collection's BACKUP call failed, its status was never recorded. Once the other collections finished, the SolrBackup was marked finished, and successful, without it. Its status is now recorded as not started, so the SolrBackup waits for it.
  • Error handling.
    • A failure for one collection no longer stops the other collections from being checked. The errors are returned together.
    • A failure to list the collections ends the reconcile.
    • Errors are only recorded as events once, as BackupError by Reconcile. The duplicate BackupCloudNotReady and BackupCollectionListError events are removed. Events were added in Emit Kubernetes events from the operator controllers #836, which has not been released.

Testing

  • New unit tests in controllers/solrbackup_controller_unit_test.go, against a fake Solr Collections API:
    • resubmitting a notfound backup and seeing it through to completion;
    • no resubmit within the grace period, including right after a resubmission;
    • the async status is deleted only after the status is saved, and kept when the save fails;
    • a collection that failed to start holds up the SolrBackup;
    • a failure to list collections ends the reconcile.
  • events_test.go: the cloud-not-ready test now goes through Reconcile, and checks that exactly one warning event is recorded.
  • Recover a SolrBackup whose InProgress status update was lost #907's tests in solrbackup_async_recovery_test.go are updated for the deletion now happening after the status save, and for the status entry now being recorded when the start fails.
  • Each new test fails without its fix. All plain controller tests pass, and go vet is clean, at every commit.
  • Not yet run: the envtest suite (make test) and the backup e2e tests (make e2e-tests TEST_FILES=backups_test.go).

A collection backup whose async status Solr reports as "notfound" stayed in
progress forever, which also stopped recurring backups from being scheduled
again (apache#824). Once a backup has been "notfound" for more than 5 seconds after
it started, it is now submitted again.

Solr's async status for a finished collection backup is now only deleted once
the SolrBackup status recording the result has been saved. Deleting it first,
and then failing to save the status, is one way a backup became "notfound".

The SolrCloud is now fetched, and its auth context built, in Reconcile, so the
deletes can use it after the status is saved. A failure for one collection no
longer stops the other collections from being checked, a failure to list the
collections ends the reconcile, and errors are only recorded as events once,
by Reconcile.
When a collection's backup could not be started, its status was never
recorded. Once the other collections finished, the SolrBackup was then marked
finished, and successful, without it. Record the collection's status even when
the backup cannot be started, so the SolrBackup waits for it.
The 5 second grace before resubmitting a "notfound" backup is measured from
the collection backup's start time, which was only set for the first
submission. After one resubmission, any later "notfound" resubmitted the
backup again straight away. Set the start time on every submission.
@HoustonPutman
HoustonPutman requested a review from janhoy November 19, 2026 20:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain to block approval of the backup-recovery changes.

No open issues 🎉

What changed in this PR

This PR addresses #824 by recovering collection backups that Solr no longer tracks, allowing recurring SolrBackups to progress rather than remain stuck.

Changes:

  • Resubmits backups reported as notfound after a five-second grace period, resetting the grace period on each submission.
  • Defers async-status deletion until results are saved and prevents failed-to-start collections from being omitted from completion checks.
  • Continues processing other collections after individual errors, consolidates warning events, and adds regression tests.
File Description
helm/​solr-operator/​Chart.yaml Documents backup recovery and reliability fixes in the chart changelog.
controllers/​solrbackup_controller.go Implements resubmission, deferred async cleanup, and revised error handling.
controllers/​solrbackup_controller_unit_test.go Adds regression tests for recovery, grace periods, status persistence, and collection failures.
controllers/​solrbackup_async_recovery_test.go Updates recovery assertions for deferred cleanup and failed-start status recording.
controllers/​events_test.go Verifies cloud-not-ready failures emit exactly one warning through Reconcile.

Review effort: Balanced

@janhoy janhoy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skimmed through and had Copilot check with not comments.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SolrBackup may never finish if REQUESTSTATUS returns notfound for an accepted backup

3 participants