fix: handle mixed dictionary/plain ArrowScan batches#3667

Open

GayathriSrividya wants to merge 2 commits into

apache:mainfrom

GayathriSrividya:fix/arrowscan-mixed-dictionary-string

GayathriSrividya commented Jul 17, 2026

Contributor

closes #3260

Summary

decode dictionary-encoded columns in ArrowScan.to_table before concatenation
keep permissive concat behavior for other schema variations
add a regression test covering mixed plain-string and dictionary-encoded batches

Testing

/opt/homebrew/bin/ruff format pyiceberg/io/pyarrow.py tests/io/test_pyarrow.py
/opt/homebrew/bin/ruff check --fix pyiceberg/io/pyarrow.py tests/io/test_pyarrow.py
PYTHONPATH=. uv run --project . pytest tests/io/test_pyarrow.py -k mixed_dictionary_and_plain_string_batches -xvs

Gayathri Srividya Rajavarapu added 2 commits

July 17, 2026 18:38


          fix: handle mixed dictionary/plain ArrowScan batches

db4918f


          fix: preserve requested dictionary columns

b88b709

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet