Skip to content

[release/2.1] Proper fixes for EPIC batch install script - #2211

Merged
climbfuji merged 4 commits into
JCSDA:release/2.1from
EdwardSnyder-NOAA:batch-install-fixes
Nov 16, 2026
Merged

climbfuji merged 4 commits into
JCSDA:release/2.1from
EdwardSnyder-NOAA:batch-install-fixes

Conversation

@EdwardSnyder-NOAA

Copy link
Copy Markdown
Contributor

Use of generative AI

Please refer to our temporary spack-stack AI policy for further information.

  • I confirm that this contribution was submitted by a real person. If AI tools or agents were used in its preparation, I have personally reviewed and take responsibility for all submitted content, code, and proposed changes. This contribution is not autonomously generated by AI.

Description

A few git commits from the dev branch to get the batch_install.sh script working with EPIC GitHub Actions workflow for the release/2.1 branch.

Dependencies

None.

Issues addressed

None.

Applications affected

None.

Systems affected

None.

Testing

  • CI: Note whether the automatic tests (GitHub actions tests that run automatically for every commit) pass or not
    • GitHub actions CI tests pass
    • GitHub actions CI tests do not pass (provide explanation)
    • GitHub actions CI tests skipped (provide explanation if necessary)
  • New tests added: List and describe any new tests added to GitHub actions
    • ...
  • Additional testing: Add information on any additional tests conducted
    • ...

Checklist

  • This PR addresses one issue/problem/enhancement or has a very good reason for not doing so.
  • These changes have been tested on the affected systems and applications.
  • All dependency PRs/issues have been resolved and this PR can be merged.
  • All necessary updates to the documentation (spack-stack wiki) will be made when this PR is merged

rickgrubin-noaa and others added 3 commits November 10, 2026 11:16
… EPIC Tier 1 hosts (JCSDA#2043)

Add a batch install script for EPIC Tier 1 hosts, and update mirrors.yaml files for those hosts to define source and binary cache locations.

File util/epic/intel_oneapi-2026.0.0_install.sh is added for potential future need / use.

Both scripts shamelessly borrowed from NRL host scripts.

(cherry picked from commit c17b15c)
…JCSDA#2115)

Add required module pathing and loading for MSU hosts to successfully build oneAPI envs. GCC envs are not effected.

Module pathing:
- specify module use pathing for the EPIC-installed oneAPI compilers
- specify module use pathing for the system-installed oneAPI MPI and MKL

Module loading:

load module git-lfs so that packages that require it are successfully fetched

This is accomplished by adding configs/sites/tier1/{hercules, orion}/setup.sh scripts and referencing them in the top-level spack setup.sh script. This allows for removing incomplete module pathing commands from util/epic/batch_install.sh and instead place them in site-specific setup scripts. Placing these updates in configs/sites/tier1/{hercules, orion}/setup.sh ensures that they're available to both manual and automated builds.

Note that a module pathing command for site derecho in util/epic/batch_install.sh was removed because it's already in configs/sites/tier1/derecho/setup.sh

(cherry picked from commit b2f2346)
@climbfuji climbfuji changed the title Proper fixes for EPIC batch install script [release/2.1] Proper fixes for EPIC batch install script Nov 10, 2026
@climbfuji climbfuji self-assigned this Nov 10, 2026

@RatkoVasic-NOAA RatkoVasic-NOAA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@natalie-perlin

Copy link
Copy Markdown
Collaborator

@EdwardSnyder-NOAA - thank you for making this change! How did you test these approaches?

@EdwardSnyder-NOAA

Copy link
Copy Markdown
Contributor Author

@EdwardSnyder-NOAA - thank you for making this change! How did you test these approaches?

For the past few weeks, I have been using the batch_install.sh script from the develop branch to build spack-stack-2.1.1 in my spack-stack GitHub Actions workflow. If you do a history of this file in the develop branch, you will see I'm bringing in those commits to the file here in the release/2.1 branch. I excluded your commit and Matt's CARGO_HOME rename commit since that was pertaining to the release/2.2 branch. Basically I'm trying to sync up the batch_install.sh file from the release/2.1 branch with the file in the develop branch.

@natalie-perlin

Copy link
Copy Markdown
Collaborator

@EdwardSnyder-NOAA - thank you for making this change! How did you test these approaches?

For the past few weeks, I have been using the batch_install.sh script from the develop branch to build spack-stack-2.1.1 in my spack-stack GitHub Actions workflow. If you do a history of this file in the develop branch, you will see I'm bringing in those commits to the file here in the release/2.1 branch. I excluded your commit and Matt's CARGO_HOME rename commit since that was pertaining to the release/2.2 branch. Basically I'm trying to sync up the batch_install.sh file from the release/2.1 branch with the file in the develop branch.

@EdwardSnyder-NOAA - do you mean the commit from #2210?
It would apply to all branches, not just release/2.2 (it is not there yet; still pending the update). Bootstrapping mirrors would likely fail IF 1) you already have prior mirrors, and you create a new spack-stack version as a separate directory (e.g., spack-stack-2.2.0), and 2) the bootstrap tag has changed in spack. I am not sure what the outcome would be from combining these factors... My attempt to bootstrap a mirror on Ursa for a new spack-stack version has resulted in an error, as the prior installation already populated the mirror metadata.

@EdwardSnyder-NOAA

Copy link
Copy Markdown
Contributor Author

@EdwardSnyder-NOAA - thank you for making this change! How did you test these approaches?

For the past few weeks, I have been using the batch_install.sh script from the develop branch to build spack-stack-2.1.1 in my spack-stack GitHub Actions workflow. If you do a history of this file in the develop branch, you will see I'm bringing in those commits to the file here in the release/2.1 branch. I excluded your commit and Matt's CARGO_HOME rename commit since that was pertaining to the release/2.2 branch. Basically I'm trying to sync up the batch_install.sh file from the release/2.1 branch with the file in the develop branch.

@EdwardSnyder-NOAA - do you mean the commit from #2210? It would apply to all branches, not just release/2.2 (it is not there yet; still pending the update). Bootstrapping mirrors would likely fail IF 1) you already have prior mirrors, and you create a new spack-stack version as a separate directory (e.g., spack-stack-2.2.0), and 2) the bootstrap tag has changed in spack. I am not sure what the outcome would be from combining these factors... My attempt to bootstrap a mirror on Ursa for a new spack-stack version has resulted in an error, as the prior installation already populated the mirror metadata.

@natalie-perlin - I was referring to commit #2210. Are you planning on cherry picking this commit to release/2.1 then? I'm fine having this commit to future release branches. I'd figure since we are versioning control the batch_install.sh script it is okay to leave it from the release/2.1 branch since that script was working before your commit. That is why I excluded it from this PR. If I add your commit, then I'll have to go back and rerun everything, which I guess isn't a big deal since everything is automated at this point.

@natalie-perlin

Copy link
Copy Markdown
Collaborator

@EdwardSnyder-NOAA - thank you for making this change! How did you test these approaches?

For the past few weeks, I have been using the batch_install.sh script from the develop branch to build spack-stack-2.1.1 in my spack-stack GitHub Actions workflow. If you do a history of this file in the develop branch, you will see I'm bringing in those commits to the file here in the release/2.1 branch. I excluded your commit and Matt's CARGO_HOME rename commit since that was pertaining to the release/2.2 branch. Basically I'm trying to sync up the batch_install.sh file from the release/2.1 branch with the file in the develop branch.

@EdwardSnyder-NOAA - do you mean the commit from #2210? It would apply to all branches, not just release/2.2 (it is not there yet; still pending the update). Bootstrapping mirrors would likely fail IF 1) you already have prior mirrors, and you create a new spack-stack version as a separate directory (e.g., spack-stack-2.2.0), and 2) the bootstrap tag has changed in spack. I am not sure what the outcome would be from combining these factors... My attempt to bootstrap a mirror on Ursa for a new spack-stack version has resulted in an error, as the prior installation already populated the mirror metadata.

@natalie-perlin - I was referring to commit #2210. Are you planning on cherry picking this commit to release/2.1 then? I'm fine having this commit to future release branches. I'd figure since we are versioning control the batch_install.sh script it is okay to leave it from the release/2.1 branch since that script was working before your commit. That is why I excluded it from this PR. If I add your commit, then I'll have to go back and rerun everything, which I guess isn't a big deal since everything is automated at this point.

@EdwardSnyder-NOAA - oh, sure, older branches such as release/2.1 may not need to be updated.
The main goal is to have this script tested in an actual environment, at least on one of the Tier 1 platforms.

…ors with a version tag (JCSDA#2210)

spack bootstrap mirror writes metadata/sources and metadata/binaries at fixed paths. Rerunning it for a newer Spack (e.g., when doing spack-stack v2.2.0 install or spack-stack-nvhpc-dev) and attempting to use mirrors overwrites the metadata (including clingo.json, gnupg.json, patchelf.json) that installs from older spack-stack versions rely on.

Change (util/epic/batch_install.sh):
- Build a metadata key from Spack's bootstrap release: -. The tag is from BINARY_TARBALL in spack/lib/spack/spack/cmd/bootstrap.py, e.g. v2.2; the hash is the first 8 characters of the sha256 of the three github-actions-v2/*.json files.
- Keep generating the mirror in a temp directory. Merge bootstrap_cache/ into the shared cache, which is safe because its files are stored by checksum and by name. Copy the metadata to metadata// instead of metadata/.
- Rewrite url: in the copied metadata.yaml files to the absolute bootstrap_cache path, since the relative ../../bootstrap_cache no longer resolves one level deeper.
- Register local-sources and local-binaries from metadata//{sources,binaries}.

(cherry picked from commit bc146a5)
@EdwardSnyder-NOAA

Copy link
Copy Markdown
Contributor Author

@natalie-perlin - I tested this PR by running both modes of the batch_install.sh script (i.e. building the local cache and installing the spack-stack unified environment) on Hercules at this location: /apps/contrib/spack-stack/gha/ss-pr-test/envs. Both modes passed successfully.

@climbfuji
climbfuji enabled auto-merge (squash) November 16, 2026 20:53
@climbfuji
climbfuji merged commit 05b8be6 into JCSDA:release/2.1 Nov 16, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants