semble 0.5.2, Python 3.12 (via uv), macOS.
Every path semble indexes gets a cache entry in ~/Library/Caches/semble/, keyed by sha256 of the resolved path (which makes sense and is a huge improvement over earlier versions that just handled this in memory). But nothing ever removes an entry when its directory is deleted, so anything short-lived that gets indexed leaves a dead entry behind forever.
My tooling creates new worktrees constantly, 3 to 5 a day, and most live less than a week; there are 77 on disk right now across 19 repos. Measuring the cache 23 days after its last reset, a stretch that included two unusually quiet weeks: 2.47 GB across 17 entries, of which 4 point at paths that no longer exist, 0.69 GB or about 28% of the cache. At my normal pace the dead share piles up considerably faster than that.
Repro with a worktree, though any deleted directory behaves the same:
semble search "whatever" /path/to/worktree # builds index, creates sha-named dir in the cache
git worktree remove /path/to/worktree
ls ~/Library/Caches/semble/ # dir is still there
Expected: some way to drop entries whose source path no longer exists.
Actual: semble clear only offers all/index/savings, and clearing everything is a real cost with big repos. A cold index of my largest (~10k files, ~2.2M lines) takes about 106 seconds so reclaiming the dead 0.69 GB means minutes of rebuilding live indexes.
I've worked around it with an external script that reads root_path from each entry's metadata.json and drops entries whose path is gone, but this feels like something semble should handle itself.
Would you be open to a semble clear orphans (or maybe an automatic prune on startup) that drops entries whose root_path no longer exists?
Note: entries for git URLs store the temp clone dir as root_path, which is always gone by then, so a naive existence check would nuke all of those. They should be able to be easily excluded though because for local entries the dir name is the sha256 of root_path, while for URL entries it's the hash of the URL, so the check can skip entries where the hash doesn't match. I just don't use semble for remote repos much so not sure what the right expectation is there.
If this sounds reasonable I'm happy to take a stab at a PR, per the contributing guide. Wanted to check first whether you'd want it at all, and if so which shape you'd prefer.
semble 0.5.2, Python 3.12 (via uv), macOS.
Every path semble indexes gets a cache entry in
~/Library/Caches/semble/, keyed by sha256 of the resolved path (which makes sense and is a huge improvement over earlier versions that just handled this in memory). But nothing ever removes an entry when its directory is deleted, so anything short-lived that gets indexed leaves a dead entry behind forever.My tooling creates new worktrees constantly, 3 to 5 a day, and most live less than a week; there are 77 on disk right now across 19 repos. Measuring the cache 23 days after its last reset, a stretch that included two unusually quiet weeks: 2.47 GB across 17 entries, of which 4 point at paths that no longer exist, 0.69 GB or about 28% of the cache. At my normal pace the dead share piles up considerably faster than that.
Repro with a worktree, though any deleted directory behaves the same:
Expected: some way to drop entries whose source path no longer exists.
Actual:
semble clearonly offersall/index/savings, and clearing everything is a real cost with big repos. A cold index of my largest (~10k files, ~2.2M lines) takes about 106 seconds so reclaiming the dead 0.69 GB means minutes of rebuilding live indexes.I've worked around it with an external script that reads
root_pathfrom each entry's metadata.json and drops entries whose path is gone, but this feels like something semble should handle itself.Would you be open to a
semble clear orphans(or maybe an automatic prune on startup) that drops entries whoseroot_pathno longer exists?Note: entries for git URLs store the temp clone dir as
root_path, which is always gone by then, so a naive existence check would nuke all of those. They should be able to be easily excluded though because for local entries the dir name is the sha256 ofroot_path, while for URL entries it's the hash of the URL, so the check can skip entries where the hash doesn't match. I just don't use semble for remote repos much so not sure what the right expectation is there.If this sounds reasonable I'm happy to take a stab at a PR, per the contributing guide. Wanted to check first whether you'd want it at all, and if so which shape you'd prefer.