Propagate enable_profiling through ProcessedContainerSettings

[jhiemstrawisc]

ProcessedContainerSettings.from_container_settings() rebuilds the processed container settings from the raw ContainerSettings, but it never copied the enable_profiling field, so the processed settings always used the dataclass default of False. run_container_singularity() reads the flag off the processed settings (config.enable_profiling), so the profiling branch never ran: no peer cgroup was created, cgroup_wrapper.sh never executed, and no usage-profile.tsv was written, regardless of the user's containers.enable_profiling config value.

This regressed in #390, which moved enable_profiling from a top-level config field to the nested container settings and added the field to both ContainerSettings and ProcessedContainerSettings, but missed wiring it through the conversion in from_container_settings().

Add the missing assignment, plus a regression test asserting that enable_profiling survives the full config-parsing path.

[agitter]

The code changes look good. I have not tested it yet, which one of us may want to do before merging. I tried testing in CHTC but hit other problems described in #459 (review)

[ntalluri]

I updated my code on my current CHTC set up with these changes. I am currently waiting for my jobs to get scheduled to test if this fixed the bug.

[ntalluri]

I accidentally approved this. I still need to test this on CHTC, I realized that the SPRAS Docker image I was using wasn't including this code change so it still looked like a bug.

[jhiemstrawisc]

I'm testing this now, and I've found a few more subtle issues -- don't merge yet!

[jhiemstrawisc]

The most recent commits address two other issues I ran into while testing. The first fixes a runtime error that I think I introduced at some point related to the way I used subprocess to invoke apptainer on the EP. The second declares the profiling file as an output so it's transferred back to the AP. This second point was previously hidden by the fact that the executor used to transfer the EP's entire output director rather than individual files (which was itself a bug because it caused Snakemake to re-run completed jobs).

At this point I've tested profiling end to end and this works for me.

[ntalluri]

I tested this update and was able to get the profiling files. However, every value inside the files is N/A:

egfr_large-strwr-params-CYXCVH5/usage-profile.tsv 
peak_memory_bytes	cpu_usage_usec	cpu_user_usec	cpu_system_usec
N/A	N/A	N/A	N/A

egfr_large-strwr-params-KZG4KAJ/usage-profile.tsv 
peak_memory_bytes	cpu_usage_usec	cpu_user_usec	cpu_system_usec
N/A	N/A	N/A	N/A

[ntalluri]

Looking at one of the reconstruct stdout outputs I get:

Container image override (local .sif): apptainer_images/strwr.sif
singularity build --sandbox unpacked/strwr apptainer_images/strwr.sif
WARNING: 'nodev' mount option set on /srv, it could be a source of failure during build process
INFO:    Starting build...
INFO:    Verifying bootstrap image apptainer_images/strwr.sif
INFO:    Extracting local image...
INFO:    Creating sandbox directory...
INFO:    Build complete: unpacked/strwr
WARNING: 'nodev' mount option set on /srv, it could be a source of failure during build process
INFO:    Starting build...
INFO:    Verifying bootstrap image apptainer_images/strwr.sif
INFO:    Extracting local image...
INFO:    Creating sandbox directory...
INFO:    Build complete: unpacked/strwr
Resolved singularity image to sandbox: unpacked/strwr
Failed to create cgroup: [Errno 30] Read-only file system: '/sys/fs/cgroup/spras-peer-19'
Reading memory and CPU stats from cgroup
Failed to read memory usage from cgroup: [Errno 2] No such file or directory: '/sys/fs/cgroup/spras-peer-19/memory.peak'
Failed to read cpu.stat from cgroup: [Errno 2] No such file or directory: '/sys/fs/cgroup/spras-peer-19/cpu.stat'
    My cgroup path is: /sys/fs/cgroup/spras-peer-19
    /usr/local/lib/python3.11/site-packages/spras/cgroup_wrapper.sh: line 15: /sys/fs/cgroup/spras-peer-19/cgroup.procs: No such file or directory
    Executing command: apptainer exec --bind /srv/scratch/output/egfr/prepared/egfr_large-strwr-inputs:/spras/AOZVQUF --bind /srv/scratch/output/egfr/prepared/egfr_large-strwr-inputs:/spras/4IWLUPR --bind /srv/scratch/output/egfr/prepared/egfr_large-strwr-inputs:/spras/ZBD3RLG --bind /srv/scratch/output/egfr/egfr_large-strwr-params-CYXCVH5:/spras/GN4QBOE/egfr_large-strwr-params-CYXCVH5 --cleanenv --containall --pwd /spras --env SPRAS=True unpacked/strwr python /ST_RWR/ST_RWR.py --network /spras/ZBD3RLG/network.txt --sources /spras/AOZVQUF/sources.txt --targets /spras/4IWLUPR/targets.txt --output /spras/GN4QBOE/egfr_large-strwr-params-CYXCVH5/output.txt --alpha 0.05

the outputs for the profiling are all NA, seems like the cgroup files do not exist

[ntalluri]

Something is broken where the profiling files all have N/A values.

[agitter]

The error Failed to create cgroup: [Errno 30] Read-only file system: '/sys/fs/cgroup/spras-peer-19' seems to point to

spras/spras/profiling.py

Lines 23 to 32 in 11091cd

	mycgroup = os.path.join("/sys/fs/cgroup", cgroup_rel.lstrip("/"))
	peer_cgroup = os.path.join(os.path.dirname(mycgroup), f"spras-peer-{os.getpid()}")
	# Create the peer cgroup directory
	try:
	os.makedirs(peer_cgroup, exist_ok=True)
	except Exception as e:
	print(f"Failed to create cgroup: {e}")
	return peer_cgroup

Did that work previously and no longer does? Should we return None or fail differently in that block of code if we can't create the directory?