Skip to content

inspect.files() and the other file metadata tables raise ArrowInvalid when a snapshot has no manifests #4269

Open

Description

@mbutrovich

Apache Iceberg version

0.12.0 (latest release)

Please describe the bug 🐞

Appending an empty Arrow table creates a snapshot whose manifest list has no manifests. After that, inspect.files(), data_files(), delete_files(), all_files(), all_data_files(), and all_delete_files() all raise ArrowInvalid: Must pass at least one table. I expected each of them to return an empty table with the files schema.

import tempfile

import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog

warehouse = tempfile.mkdtemp()
catalog = SqlCatalog("t", uri=f"sqlite:///{warehouse}/catalog.db", warehouse=f"file://{warehouse}")
catalog.create_namespace("n")
df = pa.table({"x": pa.array([], pa.int64())})
table = catalog.create_table("n.t", schema=df.schema)
table.append(df)
table.inspect.files()  # ArrowInvalid: Must pass at least one table
Traceback (most recent call last):
  File "repro.py", line 12, in <module>
    table.inspect.files()  # ArrowInvalid: Must pass at least one table
    ^^^^^^^^^^^^^^^^^^^^^
  File "pyiceberg/table/inspect.py", line 900, in files
    return self._files(snapshot_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyiceberg/table/inspect.py", line 888, in _files
    return pa.concat_tables(results)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 6344, in pyarrow.lib.concat_tables
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Must pass at least one table

overwrite() with an empty table after a non-empty append also leaves a snapshot with no manifests, and files() fails the same way.

_files reads one table per manifest and ends with pa.concat_tables(results), so an empty manifest list calls pa.concat_tables([]). _all_files does the same at L981. Both already return an empty table one level up: _files when the table has no current snapshot (L876-L877), and _all_files when the table has no snapshots (L968-L970). Could they return the same empty table when there are no manifests?

This reproduces on 0.12.0 and on main at e3564a5.

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions