Skip to content

IcebergMetadataTableProvider ignores projection in scan(): projected/aggregate queries on populated metadata tables fail #2819

Description

@gmhelmold

Apache Iceberg Rust version

main (d1b8eac)

Describe the bug

The datafusion integration's metadata table provider ignores the projection argument in TableProvider::scan() (crates/integrations/datafusion/src/table/metadata_table.rs): it always returns an IcebergMetadataScan that reports the full metadata-table schema and emits unprojected batches. (limit is also unused, but that's only a missed optimization; filters never reach scan() since supports_filters_pushdown isn't overridden — projection is the correctness issue.)

This goes unnoticed because existing tests only query empty metadata tables with SELECT * (test_metadata_table in integration_datafusion_test.rs; the $snapshots checks in insert_into.slt run before any snapshot exists). Same class as #828, which fixed this for IcebergTableScan.

To Reproduce

Any populated metadata table — e.g. against the seeded table in the crates/sqllogictest harness:

INSERT INTO default.default.test_partitioned_table VALUES (1, 'a', 'b');

SELECT count(*) FROM default.default.test_partitioned_table$snapshots;
-- DataFusion error: Internal error: Physical input schema should be the same as the
-- one converted from logical input schema. Differences:
--   - Different number of fields: (physical) 6 vs (logical) 0.

SELECT operation FROM default.default.test_partitioned_table$snapshots;
-- returns all 6 snapshot columns instead of just `operation`

Expected behavior

scan() honors projection: make IcebergMetadataScan projection-aware (report the projected schema in its plan properties and project the emitted batches), mirroring how IcebergTableScan handles projection.

Willingness to contribute

Yes — happy to submit the fix with sqllogictest regression coverage on populated metadata tables.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions