Skip to content

DataFusion INSERT fills omitted columns with NULL instead of the column's write-default #2803

Description

@moomindani

Apache Iceberg Rust version

0.6.0 (latest version)

Describe the bug

The spec requires writers to use a column's write-default when the writer does not supply a value for the column. IcebergTableProvider does not implement DataFusion's TableProvider::get_column_default, so DataFusion's insert planner falls back to NULL for columns omitted from an INSERT. Rows written through DataFusion into a table whose schema declares write-default values silently get NULL instead of the default.

For comparison, iceberg-java delegates write-default application to the engine in the same way: its Spark integration exposes write-default as the engine-level column default (TypeToSparkType), and Spark materializes the value during INSERT planning. The DataFusion integration is the corresponding place in iceberg-rust.

To Reproduce

-- table schema: id int (required), category string with write-default 'general'
INSERT INTO t (id) VALUES (1);
SELECT category FROM t;  -- returns NULL, expected 'general'

Expected behavior

IcebergTableProvider implements get_column_default, mapping each top-level column's write-default literal to a DataFusion expression, so the insert planner fills omitted columns with the declared default.

Part of #2411 (Default Values).

Willingness to contribute

I can contribute a fix for this bug independently

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions