Skip to content

fix(parquet): pass storage_options as **kwargs to fsspec.filesystem, fix mutable default args#837

Open
Tejas5405 wants to merge 1 commit into
Lightning-AI:mainfrom
Tejas5405:fix/s3-storage-options-kwargs
Open

fix(parquet): pass storage_options as **kwargs to fsspec.filesystem, fix mutable default args#837
Tejas5405 wants to merge 1 commit into
Lightning-AI:mainfrom
Tejas5405:fix/s3-storage-options-kwargs

Conversation

@Tejas5405

Copy link
Copy Markdown

Problem

index_parquet_dataset() crashes with a TypeError when passing storage_options for S3 or any cloud filesystem:

TypeError: filesystem() takes 1 positional argument but 4 were given

The bug is in src/litdata/utilities/parquet.py line 153 — storage_options (a dict) is unpacked with * (which iterates keys as positional args) instead of ** (which passes key-value pairs as keyword args):

# Before ❌ — unpacks dict keys as positional args
self.fs = fsspec.filesystem(provider, *self.storage_options)

# After ✅ — passes key-value pairs as keyword args
self.fs = fsspec.filesystem(provider, **self.storage_options)

Additional fixes

Two places used a mutable default argument ({}) for storage_options, which is a well-known Python antipattern — a single dict is shared across all calls if not overridden:

# Before ❌
storage_options: dict | None = {}

# After ✅
storage_options: dict | None = None

Fixed in ParquetCloudIndexer.__init__ (line 81) and get_parquet_indexer_cls() (line 307). The existing None{} guard in the base class handles the empty-dict default correctly.

Reproduction

import litdata as ld

ld.index_parquet_dataset(
    "s3://bucket/path/to/parquet/",
    storage_options={
        "key": "ACCESS_KEY",
        "secret": "SECRET_KEY",
        "endpoint_url": "https://s3.example.com",
    },
)
# Before fix: TypeError: filesystem() takes 1 positional argument but 4 were given
# After fix:  Works correctly

Closes #835

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

index_parquet_dataset() fails with TypeError when passing storage_options for S3

1 participant