[martinkilbinger]

What

• profiles/candide/config.yaml: bind /n23data1/mkilbing/shapepipe/tilestore to /local/scratch instead of node /tmp.
• Add n34 and n35 to the default and the four tile_shape exclude lists.
• Update the matching comments in workflow/config.yaml and workflow/README.md.

Why

• Node /tmp: it is a ~29 GB disk shared by every job on the node, and each tile store is ~5.6 GB. Stores left behind by killed jobs filled n34's /tmp in September.
• n34 and n35: ShapePipe jobs must not be sent to these nodes.

Trade-offs

• Speed: ngmix now reads the tile store over NFS. On nibi this was much slower than a local disk; it has not been measured on candide.
• Per-run override: tile_store_root still moves the store for one run.
• Per-job /scratch: /scratch/$USER/$SLURM_JOB_ID would be the proper place, but the profile cannot name it (literal paths only, and --cleanenv drops SLURM_JOB_ID). It needs a code change in tile_local(), which can trigger reruns of finished tiles, so it is left for a separate PR at a campaign boundary.
• Fixed path: the bind names one user's directory, as the PYTHONPATH pin already does. Other users must edit it, and the directory must exist.

🤖 Generated with Claude Code

[cailmdaley]

Approved. This only touches the candide profile; nothing changes for nibi, fir or production. One timed candide tile before the next campaign would show what NFS reads through n23 cost ngmix.

— Claude Opus on behalf of Cail

[cailmdaley]

@martinkilbinger why are n34 and n35 being excluded?

— Claude Opus on behalf of Cail

[martinkilbinger]

@martinkilbinger why are n34 and n35 being excluded?

— Claude Opus on behalf of Cail

These were nodes where something went wrong with mounting the apptainer's tmp dir, and thus /tmp was filled up against Henry's recommendation. This was a one-off incident, so maybe we can drop the exclusion.

[cailmdaley]

apptainer doesn't have its own writable /tmp in my understanding, it should be read-only and the user mounts the directories they need. maybe the simplest cross-host approach is to have shapepipe put intermediate products in the apptainer's /tmp, and bind different storage to apptainer /tmp on each machine. i suppose it doesn't matter whether we call it /local/scratch or /tmp inside the container though..

ah, in fact i hadn't noticed the other excluded nodes.. i looked in to this a few months ago and convinced myself that in fact all shapepipe runs on all the nodes (in fact we have a test for this in our suite), and there's no reason to exclude any. i worry that n09,n17,n23,n34,n35,n36 have been cargo-culted back in. can i open a PR that removes all of them and shows that we can do a small image sim run on each?

[martinkilbinger]

apptainer doesn't have its own writable /tmp in my understanding, it should be read-only and the user mounts the directories they need. maybe the simplest cross-host approach is to have shapepipe put intermediate products in the apptainer's /tmp, and bind different storage to apptainer /tmp on each machine. i suppose it doesn't matter whether we call it /local/scratch or /tmp inside the container though..

ah, in fact i hadn't noticed the other excluded nodes.. i looked in to this a few months ago and convinced myself that in fact all shapepipe runs on all the nodes (in fact we have a test for this in our suite), and there's no reason to exclude any. i worry that n09,n17,n23,n34,n35,n36 have been cargo-culted back in. can i open a PR that removes all of them and shows that we can do a small image sim run on each?

yes please