Apache Iceberg Rust version
main (d1b8eac)
Describe the bug
The datafusion integration's metadata table provider ignores the projection argument in TableProvider::scan() (crates/integrations/datafusion/src/table/metadata_table.rs): it always returns an IcebergMetadataScan that reports the full metadata-table schema and emits unprojected batches. (limit is also unused, but that's only a missed optimization; filters never reach scan() since supports_filters_pushdown isn't overridden — projection is the correctness issue.)
This goes unnoticed because existing tests only query empty metadata tables with SELECT * (test_metadata_table in integration_datafusion_test.rs; the $snapshots checks in insert_into.slt run before any snapshot exists). Same class as #828, which fixed this for IcebergTableScan.
To Reproduce
Any populated metadata table — e.g. against the seeded table in the crates/sqllogictest harness:
INSERT INTO default.default.test_partitioned_table VALUES (1, 'a', 'b');
SELECT count(*) FROM default.default.test_partitioned_table$snapshots;
-- DataFusion error: Internal error: Physical input schema should be the same as the
-- one converted from logical input schema. Differences:
-- - Different number of fields: (physical) 6 vs (logical) 0.
SELECT operation FROM default.default.test_partitioned_table$snapshots;
-- returns all 6 snapshot columns instead of just `operation`
Expected behavior
scan() honors projection: make IcebergMetadataScan projection-aware (report the projected schema in its plan properties and project the emitted batches), mirroring how IcebergTableScan handles projection.
Willingness to contribute
Yes — happy to submit the fix with sqllogictest regression coverage on populated metadata tables.
Apache Iceberg Rust version
main (
d1b8eac)Describe the bug
The datafusion integration's metadata table provider ignores the
projectionargument inTableProvider::scan()(crates/integrations/datafusion/src/table/metadata_table.rs): it always returns anIcebergMetadataScanthat reports the full metadata-table schema and emits unprojected batches. (limitis also unused, but that's only a missed optimization; filters never reachscan()sincesupports_filters_pushdownisn't overridden — projection is the correctness issue.)This goes unnoticed because existing tests only query empty metadata tables with
SELECT *(test_metadata_tableinintegration_datafusion_test.rs; the$snapshotschecks ininsert_into.sltrun before any snapshot exists). Same class as #828, which fixed this forIcebergTableScan.To Reproduce
Any populated metadata table — e.g. against the seeded table in the
crates/sqllogictestharness:Expected behavior
scan()honorsprojection: makeIcebergMetadataScanprojection-aware (report the projected schema in its plan properties and project the emitted batches), mirroring howIcebergTableScanhandles projection.Willingness to contribute
Yes — happy to submit the fix with sqllogictest regression coverage on populated metadata tables.