bug: sandbox create leaves stale resource on ImagePullFailed, blocking retry #2675

Activity

1. github-actions

   added

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Aug 10, 2026
2. franciscojavierarceo commented on Aug 12, 2026

   

   franciscojavierarceo

   on Aug 12, 2026

   More actions

   I’m taking this issue toward a PR. I traced the failure to the Docker driver retaining failed provisioning entries in its pending map after publishing ImagePullFailed. The fix will publish the failure, remove the failed pending resource, and emit the deletion event so the gateway record and sandbox name are released. I’ll cut a PR once the local Docker reproduction and tests are complete.
3. krishicks commented on Aug 13, 2026

   

   krishicks

   on Aug 13, 2026

   Collaborator

   More actions

   I worked through this with an agent and I think we could reframe this issue, maybe by creating a replacement issue. What follows is output from the agent. @rhuss what do you think?

   ---

   The issue is better framed as a deficient asynchronous provisioning lifecycle, rather than stale cleanup after ImagePullFailed.

   Provisioning can already represent “accepted but not yet satisfied,” including retries. The missing piece is an explicit distinction between:

   • provisioning is still being reconciled;
   • provisioning has terminally failed and reconciliation has stopped;
   • the user has cancelled provisioning and cleanup is underway;
   • backend absence has been confirmed and the name can be reused.

   Error currently conflates backend-specific conditions, transient failures, and terminal abandonment.

   Proposed user story

   As a user who has submitted a sandbox creation request, I want OpenShell to retain and accurately report the request while the compute driver attempts to satisfy it, including retryable failures, so that I can observe progress, wait for recovery, explicitly abandon the attempt, or replace it with a new request without manually reasoning about gateway and backend state.

   Reframed problem statement

   Sandbox creation is asynchronous. A successful CreateSandbox response means the gateway durably accepted the request and the compute driver accepted responsibility for reconciling it. It does not mean that the sandbox has been provisioned.

   After acceptance, the driver may encounter conditions such as image pull failures, insufficient capacity, scheduling failures, registry outages, or backend API errors. OpenShell does not currently model whether such a condition is:

   1. transient and still being retried;
   2. blocked pending an external change;
   3. terminal because the driver has abandoned reconciliation; or
   4. being cancelled and cleaned up at the user’s request.

   Instead, OpenShell infers terminal failure from backend-specific reason strings and may retain or delete records inconsistently across drivers. This makes it unsafe to determine when a sandbox name may be reused.

   Proposed state model

   The existing top-level phases could support the model if their semantics were tightened:

   Phase	Meaning
   Provisioning	The request was accepted and the compute driver is actively reconciling it. Transient or externally recoverable failures remain in this phase while the driver continues retrying.
   Ready	The backend resource is healthy and the sandbox is usable through the gateway.
   Error	The compute driver has stopped reconciling and declared the provisioning attempt terminal. Status conditions should preserve the reason and indicate whether backend resources remain.
   Deleting	OpenShell is cancelling reconciliation and removing partial or complete backend resources. The sandbox name remains reserved.
   Record absent	The compute driver has confirmed that backend resources are absent. The sandbox name may now be reused.

   More detail should be represented through structured conditions rather than additional phases:

   Ready=False
   Reconciling=True
   Reason=ImagePullBackOff
   Message=Registry unavailable; retrying
   NextRetryTime=...
   

   For terminal failure:

   Ready=False
   Reconciling=False
   Reason=ProvisioningFailed
   Message=Retry deadline exhausted
   

   The compute-driver protocol should explicitly communicate a disposition such as Retrying or Terminal, rather than requiring the gateway to infer it from strings like ImagePullBackOff.

   Abandoning an attempt

   The existing delete operation should also serve as cancellation:

   1. The user calls sandbox delete.
   2. OpenShell transitions the record to Deleting.
   3. The driver stops any reconciliation or retry task.
   4. The driver removes partial or complete backend resources.
   5. The driver confirms backend absence.
   6. OpenShell removes the durable record and releases the name.

   This operation should be idempotent and recoverable after gateway or driver restart. A timeout should leave the record in Deleting, not
   remove it optimistically.

   Replacing an attempt

   A --replace option could orchestrate cancellation followed by creation:

   1. Lock the workspace/name.
   2. Resolve the existing sandbox by its stable ID.
   3. Mark it Deleting.
   4. Cancel its reconciliation and delete its backend resource.
   5. Confirm that the old backend resource is absent.
   6. Remove the old record.
   7. Create a new record with a new sandbox ID and generation.
   8. Ask the driver to reconcile the new request.

   OpenShell must not implement replacement by deleting the gateway record first. That can orphan a Kubernetes CR or container and allows stale events from the old resource to interfere with the new request.

   Without --replace, same-name creation should continue returning AlreadyExists, but include the current phase and actionable guidance:

   sandbox 'example' already exists and is still provisioning
   hint: wait for it, or run `openshell sandbox delete example`
   

   For a terminal attempt:

   sandbox 'example' has failed
   hint: inspect it, delete it, or retry with `openshell sandbox create --replace ...`
   

   Suggested acceptance criteria

   • A successful create response means “accepted,” not “ready.”
   • Retryable driver failures remain in Provisioning.
   • Drivers explicitly distinguish active reconciliation from terminal failure.
   • OpenShell does not classify terminality from backend reason strings.
   • Users can delete a provisioning sandbox to cancel the attempt.
   • Cancellation stops driver retries and removes partial backend resources.
   • A name remains reserved until backend absence is confirmed.
   • Replacement deletes the old backend resource before creating a new generation.
   • Stale events from an old sandbox ID cannot mutate its replacement.
   • Docker, Podman, and Kubernetes satisfy the same lifecycle contract.
   • Tests cover transient recovery, terminal failure, cancellation during retry, replacement, gateway restart, and stale backend events.

   This framing makes stale records one symptom of the larger problem: OpenShell lacks a precise contract for who owns an accepted but unsatisfied creation request and how that ownership ends.
4. krishicks

   removed

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Aug 13, 2026
5. krishicks

   self-assigned this

   on Aug 13, 2026
6. krishicks

   added

   state:needs-infoAssessment needs specific evidence or reproduction detailsAssessment needs specific evidence or reproduction details

   on Aug 13, 2026
7. rhuss commented on Aug 14, 2026

   

   rhuss

   on Aug 14, 2026

   Last edited by rhuss

   ContributorAuthor

   More actions

   Yes, I think it makes sense to look at the bigger picture, too, at particularly the distinction between active reconciliation, terminal failure, and user cancellation. Those are real gaps in the current provisioning contract.

   That said, I think we should keep #2675 scoped to the immediate regression (it's a small consolidation IMO): "user can't retry a sandbox name after ImagePullFailed without manual delete." #2720 fixes this with a focused, two-layer approach (Docker driver cleanup + gateway-level auto-replace of Error-phase sandboxes on create). It's minimal and doesn't preclude the broader redesign.
   The only question whether replacing automatically an errored container is maybe too much magic.

   The lifecycle redesign (explicit cancellation, --replace, unified driver contract, gateway restart recovery) is worth pursuing as a separate issue or RFC. The acceptance criteria in the analysis above would make a strong starting point for that.

   I can wrap my brain around that.
8. github-actions commented on Aug 28, 2026

   

   github-actionsBot

   on Aug 28, 2026 – with GitHub Actions

   More actions

   This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.
9. github-actions

   added

   state:staleInactive item at risk of automatic closure.Inactive item at risk of automatic closure.

   on Aug 28, 2026
10. rhuss commented on Sep 5, 2026

    

    rhuss

    on Sep 5, 2026

    ContributorAuthor

    More actions

    Not stale, just need a bit more time :-)
11. github-actions

    removed

    state:staleInactive item at risk of automatic closure.Inactive item at risk of automatic closure.

    on Sep 6, 2026
12. github-actions commented on Sep 20, 2026

    

    github-actionsBot

    on Sep 20, 2026 – with GitHub Actions

    More actions

    This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.
13. github-actions

    added

    state:staleInactive item at risk of automatic closure.Inactive item at risk of automatic closure.

    on Sep 20, 2026
14. rhuss commented on Oct 12, 2026

    

    rhuss

    on Oct 12, 2026

    ContributorAuthor

    More actions

    A related observation with the Kubernetes driver (v0.1.3) that fits the "active reconciliation vs. terminal failure" framing:

    A sandbox whose image can't be pulled never reaches a terminal phase. It stays in Provisioning with the pod in ImagePullBackOff, and the CLI shows neither the cause nor that retrying is pointless:

    ✓ Sandbox allocated (2s)
    ✓ Image pulled (3s)
    ⠹ Pulling image... Error: ImagePullBackOff (15s)
    

    Two things make this harder to diagnose:

    1. "Image pulled" is checked off for the wrong image. attach_kube_progress_metadata in crates/openshell-driver-kubernetes/src/driver.rs treats every Pulled event as completing the pull step, without looking at involvedObject.fieldPath. Here it was the bootstrap init container's image (ghcr.io/nvidia/openshell/sandbox:0.1.3, already on the node). The later Pulling event for workspace-init then reactivates the step.
    2. The reason is dropped. Failed events aren't mapped, but Kubernetes already says what is wrong. Three cases I hit:
       • reading manifest latest in docker.io/library/<name>: requested access to the resource is denied (the reference had no namespace)
       • pinging container registry localhost: ... connection refused (the image existed only in a local engine)
       • toomanyrequests: You have reached your unauthenticated pull rate limit (Docker Hub, with no pull secret on the sandbox service account)

    In all three cases the sandbox stayed in Provisioning until I deleted it by hand. Treating persistent ErrImagePull / ImagePullBackOff with these reasons as a terminal ImagePullFailed, and showing the event message, would cover the Kubernetes side of this issue.
15. github-actions

    removed

    state:staleInactive item at risk of automatic closure.Inactive item at risk of automatic closure.

    on Oct 13, 2026
16. procoduck commented on Oct 29, 2026

    

    procoduck

    on Oct 29, 2026

    More actions

    I see the same thing on 0.1.6 in managed workspace mode: a sandbox whose image can't be pulled
    stays Provisioning with ConfigurationPending, and nothing in the CLI names the image. I've
    written up where the reason gets lost in #5028 : the Kubernetes driver and the gateway both
    rewrite Ready, and managed and operator modes don't forward pod events. It also covers the
    supervisor-image case.
17. rhuss commented on Nov 11, 2026

    

    rhuss

    on Nov 11, 2026 – with ChatGPT Codex Connector

    ContributorAuthor

    More actions

    Thanks for the detailed report and #5028. I went back through the related work, and I think we should separate diagnostics from recovery.

    Let’s use #5028 to track preserving image-pull diagnostics through the driver, gateway, and CLI, including supervisor images and event forwarding in managed/operator modes. That should also cover the misleading “Image pulled” progress originally reported in #4415.

    For #2675, retaining a failed sandbox is useful: it preserves diagnostics and avoids releasing the name while backend resources may still exist. #2720’s final revision took that direction, but closed without merging. We should carry forward its actionable duplicate-name errors and recovery guidance, updated for the stop/start recovery available today.

    We should avoid treating ImagePullBackOff alone as terminal: Kubernetes may still be retrying. Showing the blocker should not require abandoning the provisioning attempt.

    Related scheduling work is already underway in #4907 and #5344. I’d build on those, then assess the remaining lifecycle-contract gaps against the deadline and recovery improvements that have landed.