Repository navigation
Resubmit collection backups that Solr has lost track of - #910
Open
HoustonPutman wants to merge 6 commits into
Open
HoustonPutman wants to merge 6 commits into
HoustonPutman wants to merge 6 commits into
Conversation
A collection backup whose async status Solr reports as "notfound" stayed in progress forever, which also stopped recurring backups from being scheduled again (apache#824). Once a backup has been "notfound" for more than 5 seconds after it started, it is now submitted again. Solr's async status for a finished collection backup is now only deleted once the SolrBackup status recording the result has been saved. Deleting it first, and then failing to save the status, is one way a backup became "notfound". The SolrCloud is now fetched, and its auth context built, in Reconcile, so the deletes can use it after the status is saved. A failure for one collection no longer stops the other collections from being checked, a failure to list the collections ends the reconcile, and errors are only recorded as events once, by Reconcile.
When a collection's backup could not be started, its status was never recorded. Once the other collections finished, the SolrBackup was then marked finished, and successful, without it. Record the collection's status even when the backup cannot be started, so the SolrBackup waits for it.
The 5 second grace before resubmitting a "notfound" backup is measured from the collection backup's start time, which was only set for the first submission. After one resubmission, any later "notfound" resubmitted the backup again straight away. Set the start time on every submission.
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain to block approval of the backup-recovery changes.
No open issues 🎉
What changed in this PR
This PR addresses #824 by recovering collection backups that Solr no longer tracks, allowing recurring SolrBackups to progress rather than remain stuck.
Changes:
- Resubmits backups reported as
notfoundafter a five-second grace period, resetting the grace period on each submission. - Defers async-status deletion until results are saved and prevents failed-to-start collections from being omitted from completion checks.
- Continues processing other collections after individual errors, consolidates warning events, and adds regression tests.
| File | Description |
|---|---|
helm/solr-operator/Chart.yaml |
Documents backup recovery and reliability fixes in the chart changelog. |
controllers/solrbackup_controller.go |
Implements resubmission, deferred async cleanup, and revised error handling. |
controllers/solrbackup_controller_unit_test.go |
Adds regression tests for recovery, grace periods, status persistence, and collection failures. |
controllers/solrbackup_async_recovery_test.go |
Updates recovery assertions for deferred cleanup and failed-start status recording. |
controllers/events_test.go |
Verifies cloud-not-ready failures emit exactly one warning through Reconcile. |
Review effort: Balanced
janhoy
approved these changes
Nov 20, 2026
janhoy
left a comment
Contributor
There was a problem hiding this comment.
Skimmed through and had Copilot check with not comments.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #824
This replaces #846. #846 treats a
notfoundasync status as a terminal failure. This PR instead submits the backup again, so a recurring backup keeps going and still produces a backup, rather than recording a failed run. It builds on #907, which handles a backup's async ID being present in Solr before it is submitted.The problem
A collection backup whose async status Solr reports as
notfoundstayedinProgressforever. Because the SolrBackup never finished, a recurring backup was never scheduled again andmaxSavedpruning stopped. Nothing was reported.Changes
notfound. Once a collection backup has beennotfoundfor more than 5 seconds after it was submitted, it is submitted again, with the same name and async ID. Its start time is reset on every submission, so the 5 second grace also applies after a resubmission.notfound. That is one way into SolrBackup may never finish if REQUESTSTATUS returns notfound for an accepted backup #824.Reconcilenow fetches the SolrCloud and builds its auth context.reconcileSolrCloudBackupreturns the collections whose async requests finished.Reconciledeletes their async statuses only once the status patch succeeds.BackupErrorbyReconcile. The duplicateBackupCloudNotReadyandBackupCollectionListErrorevents are removed. Events were added in Emit Kubernetes events from the operator controllers #836, which has not been released.Testing
controllers/solrbackup_controller_unit_test.go, against a fake Solr Collections API:notfoundbackup and seeing it through to completion;events_test.go: the cloud-not-ready test now goes throughReconcile, and checks that exactly one warning event is recorded.solrbackup_async_recovery_test.goare updated for the deletion now happening after the status save, and for the status entry now being recorded when the start fails.go vetis clean, at every commit.make test) and the backup e2e tests (make e2e-tests TEST_FILES=backups_test.go).