Data freshness metric

[vandanavk]

Is your feature request related to a problem? Please describe.
I did not find Data freshness metric in the list of Analyzers. We would need this to verify we have the latest data

Describe the solution you'd like
An analyzer in deequ and pydeequ that can check data for staleness by days, hours, minutes

Describe alternatives you've considered
Considered using Python code to determine freshness but its probably more optimized in deequ (Scala)

[github-actions]

Thanks for the feature request! Currently, neither Deequ nor PyDeequ has a built-in "data freshness" analyzer. However, you can achieve this today using the existing Maximum analyzer on a timestamp column combined with a check assertion:

from pydeequ.checks import Check, CheckLevel
from pydeequ.verification import VerificationSuite, VerificationResult
import time

current_epoch = time.time()
max_staleness_seconds = 86400  # 1 day

check = Check(spark, CheckLevel.Error, "Freshness Check") \
    .hasMax("timestamp_column", lambda x: (current_epoch - x) <= max_staleness_seconds)

result = VerificationSuite(spark) \
    .onData(df) \
    .addCheck(check) \
    .run()

This assumes your timestamp column stores epoch seconds. You'd adapt the lambda for hours/minutes by changing the threshold.

That said, a dedicated DataFreshness analyzer that handles timestamp formats and returns staleness as a metric would be a nice addition. This would need to be implemented upstream in the Deequ Scala library first, then wrapped here in PyDeequ. I'll leave this open as an enhancement request. Contributions are welcome if anyone wants to take this on!

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: cf3780e5) — may not be fully accurate. Reply if this doesn't help.