Skip to content

Fix residual NotNaN for null partition values#3689

Open
BharatDeva wants to merge 1 commit into
apache:mainfrom
BharatDeva:fix/residual-not-nan-null-3498
Open

Fix residual NotNaN for null partition values#3689
BharatDeva wants to merge 1 commit into
apache:mainfrom
BharatDeva:fix/residual-not-nan-null-3498

Conversation

@BharatDeva

Copy link
Copy Markdown

Rationale for this change

ResidualVisitor.visit_not_nan currently treats a partition value of None as AlwaysFalse(). That is inconsistent with the normal evaluator behavior and with Iceberg Java semantics: None is not NaN, so NotNaN(None) should evaluate to true.

This PR addresses only that residual-evaluator edge case from #3498. The stricter metrics-evaluator items discussed in the issue are already covered by other active work.

Are these changes tested?

Yes. I updated the existing residual evaluator expectations for both double and float identity partitions and ran the targeted test file in a WSL-native checkout:

pytest tests/expressions/test_residual_evaluator.py -q

Result:

11 passed in 0.05s

Additional checks:

ruff check pyiceberg/expressions/visitors.py tests/expressions/test_residual_evaluator.py
git diff --check

Both passed.

Are there any user-facing changes?

Yes. Residual evaluation for NotNaN now treats null partition values as not-NaN instead of false. Please add the changelog label if this behavior change should appear in release notes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant