Skip to content

Remove volume when removing container#2409

Merged
Didayolo merged 1 commit into
developfrom
fix_remove_volumes
Jun 18, 2026
Merged

Remove volume when removing container#2409
Didayolo merged 1 commit into
developfrom
fix_remove_volumes

Conversation

@Didayolo

@Didayolo Didayolo commented Jun 15, 2026

Copy link
Copy Markdown
Member

Original PR

Description

This change resolves an issue, which occured with our codabench_worker using Podman. (Not sure if the same happens with Docker at some point)

The problem

Our worker stopped working after couple of weeks and it took some time to understand the root cause of the problem, since the worker got always stuck when creating the container. We only recognized after we investigated several options that podman volume ls | wc -l resulted in 2047 entries, which was too close to a "threshold". As it turns out, the create_container was failing as podman could not create additional volumes and therefore our process was always running into an execution timeout.

We could resolve the problem by removing all dangling volumes that have been created (podman volumes rm --all). After that "cleaning", the worker is now operational again.

The fix

Therefore, we propose the change to have remove_container also remove the volumes via the option v=True. In our tests, this helps to avoid the problem described above.

Checklist

  • Code review by reviewer
  • Hand tested by reviewer
  • CircleCi tests are passing
  • Ready to merge

@Didayolo Didayolo force-pushed the fix_remove_volumes branch from e507319 to 31f6ea0 Compare June 18, 2026 09:55
@Didayolo Didayolo merged commit 6cec6b5 into develop Jun 18, 2026
1 of 2 checks passed
@Didayolo Didayolo deleted the fix_remove_volumes branch June 18, 2026 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants