feat: extract lfs if the file contents has changed

[aclauer]

Contribution path

• Small, safe change that does not need a tracking issue
• Linked issue or discussion: DIM-XXX / #XXX / URL

Problem

Previously we would only extract lfs files only if there wasn't already an extracted copy. But if the data changed, this means we would skip the extraction and continue using old data.

Solution

Now we check if the compressed copy has changed since the last time we extracted. If we request data from this file and it has in fact changed, we reextract so we have the most up to date data.

Happens lazily, so we only reextract when we need the data.

How to Test

AI assistance

Checklist

• I have read and approved the CLA.

[codecov]

Codecov Report

❌ Patch coverage is 99.47917% with 1 line in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/utils/test_data.py	99.22%	1 Missing ⚠️

@@            Coverage Diff             @@
##             main    #3354      +/-   ##
==========================================
+ Coverage   75.80%   75.83%   +0.03%     
==========================================
  Files        1172     1172              
  Lines      113568   113752     +184     
  Branches    10258    10264       +6     
==========================================
+ Hits        86090    86265     +175     
- Misses      24492    24497       +5     
- Partials     2986     2990       +4     

Flag	Coverage Δ
OS-ubuntu-24.04-arm	69.97% <99.47%> (+0.04%)	⬆️
OS-ubuntu-latest	71.97% <99.47%> (+0.04%)	⬆️
Py-3.10	71.97% <99.47%> (+0.04%)	⬆️
Py-3.11	71.96% <98.43%> (+0.03%)	⬆️
Py-3.12	71.97% <99.47%> (+0.04%)	⬆️
Py-3.13	71.96% <99.47%> (+0.04%)	⬆️
Py-3.14	71.97% <99.47%> (+0.03%)	⬆️
Py-3.14t	71.97% <99.47%> (+0.04%)	⬆️
SelfHosted-Large	29.61% <38.54%> (+<0.01%)	⬆️
SelfHosted-Linux	35.75% <38.54%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/utils/data.py	76.27% <100.00%> (+8.31%)	⬆️
dimos/utils/test_data.py	96.97% <99.22%> (+1.43%)	⬆️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[greptile-apps]

Greptile Summary

The PR makes lazy LFS data resolution detect changed archives using per-checkout identity stamps and serialize extraction with file locks.

• Records archive size and modification time after successful extraction.
• Re-extracts stale or unstamped data through a temporary staging directory.
• Adds coverage for archive changes, legacy and corrupt stamps, malformed archives, single-file archives, and concurrent callers.

Confidence Score: 4/5

The PR is not yet safe to merge because a failed replacement can still destroy the prior usable extraction.

The reply from "" states that the prior issue was fixed, but the current implementation still removes the destination before calling os.replace; if that call raises, cleanup removes the staged copy as well, preserving the reported data-loss failure.

Files Needing Attention: dimos/utils/data.py, dimos/utils/test_data.py

Important Files Changed

Filename	Overview
dimos/utils/data.py	Adds archive identity tracking, locking, and staged replacement, but the previously reported failed-swap data-loss path remains reachable because the old extraction is deleted before replacement.
dimos/utils/test_data.py	Adds broad extraction-state coverage, although the prior failed-swap scenario is only tested for validation failure before deletion rather than failure of the replacement operation itself.

_{Reviews (4): Last reviewed commit: "Mypy, more like my pie!" | Re-trigger Greptile}

[TomCC7]

fyi we had a similar pr already: #2884

The conclusion is that we can just rely on if the lfs file is a pointer or not (if we don't pull manually) to determine if it's fresh

[aclauer]

fyi we had a similar pr already: #2884

The conclusion is that we can just rely on if the lfs file is a pointer or not (if we don't pull manually) to determine if it's fresh

oh lol i see, i can do that

[paul-nechifor]

@aclauer LFS is an extremely contentious issue. You might want to check this thread to see what you're getting into: https://discord.com/channels/1341146487186391173/1528884577412579520/1528884580394733709 😅