Skip to content

Confusing behavior now required to to refresh the files of a listing table #19573

Closed
Feature
#19616
Closed
Confusing behavior now required to to refresh the files of a listing table#19573
Feature
#19616
Assignees
jizezhang
Labels
enhancementNew feature or request

Description

@alamb
alamb
opened on Dec 31, 2025

Is your feature request related to a problem or challenge?

As part of #19366, @BlakeOrth added a session (global) cache for the results of calling LIST on a remote directory.

This has a benefit that now this cache is visible, and we can report on it, and make it more aligned with the other session scoped caches. However, it has an unfortunate side effect, namely that there is no good way to force a refresh of the files that back an external table.

For example:

-- calls LIST to get the files
create external table foo...
drop table foo;
-- reuses the cached file list, but previously would actually call LIST to get a (potentially updated) version of the file. 
create external table foo ...

Previously, the cache was local to each ListingTable and thus was recreated on each call to CREATE EXTERNAL TABLE. This means that a user could force a refresh of the file list by recreating the table. Reusing the same cached list I think is pretty confusing.

@jizezhang has helpfully volunteered to help with this feature and has another PR queued up so we merged #19366 and will fix this after the fact

Describe the solution you'd like

I think the caches should still be (logically) table scoped -- so that when a CREATE EXTERNAL TABLE command is issues, it will actually make a call to LIST to see the current contents of the remote table

Describe alternatives you've considered

One idea is to have "sub caches" or something that are all table scoped but that the session level cache has a handle to (so it can report on them)

@BlakeOrth also proposed:

it would potentially be reasonable to treat a DROP command like INSERT where we manually invalidate the cache entries for that table's path.

Additional context

No response

Activity

  1. alamb
    added the
    Feature
    issue type on Dec 31, 2025
  2. alamb
    added
    enhancementNew feature or request
    on Dec 31, 2025

  3. jizezhang commented on Dec 31, 2025

    @jizezhang
    jizezhang
    on Dec 31, 2025
    Contributor

    take

  4. github-actions
    assigned
    jizezhang
    on Dec 31, 2025

  5. jizezhang commented on Dec 31, 2025

    @jizezhang
    jizezhang
    on Dec 31, 2025
    Contributor

    Hi @alamb @BlakeOrth , those might uncommon cases, but I wonder what the expected behavior would be if

    • Two tables happen to be created with the same path, e.g. 
      > create external table test
stored as parquet
location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/type=infrastructure/';
0 row(s) fetched. 
Elapsed 2.955 seconds.

> create external table test2
stored as parquet
location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/type=infrastructure/';
0 row(s) fetched. 
Elapsed 0.741 seconds.
      By "table scoped", do we expect cache reuse in this case? If not, do we expect the cache key to be logically some "table identifier" instead of just table path?
    • Two statements using the same path directly, e.g. 
      > select count(*) from 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/type=infrastructure/';
+-----------+
| count(*)  |
+-----------+
| 142969564 |
+-----------+
1 row(s) fetched. 
Elapsed 2.305 seconds.

> select count(*) from 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/type=infrastructure/';
+-----------+
| count(*)  |
+-----------+
| 142969564 |
+-----------+
1 row(s) fetched. 
Elapsed 0.037 seconds.
      In this case, do we expect cache reuse? or how would we interpret "table scoped" in this case?

    Thanks a lot!

  6. alamb commented on Jan 1, 2026

    @alamb
    alamb
    on Jan 1, 2026
    ContributorAuthor

    By "table scoped", do we expect cache reuse in this case? If not, do we expect the cache key to be logically some "table identifier" instead of just table path?

    I personally think that the cache should be table scoped -- so in this case I propose that the test2 table does not reuse the cache

    In my mind the core usecase the cache is when querying partitioned tables -- so that when querying on partition columns, querying with LIST is not required. @BlakeOrth mostly completed this use case already, but there is no way to introspect (see) what is cached without a session scoped object

    However, by creating a session scoped object we now have this unintended sharing

  7. alamb commented on Jan 1, 2026

    @alamb
    alamb
    on Jan 1, 2026
    ContributorAuthor

    (thank you for working on this ticket, by the way, it is most appreciated)

  8. jizezhang commented on Jan 3, 2026

    @jizezhang
    jizezhang
    on Jan 3, 2026
    Contributor

    Hi @alamb @BlakeOrth I was experimenting with ways to make the cache table scoped, and would like to get some high level feedback from you.

    I have a draft PR #19616 for one approach I considered, which is to continue using a session level cache as is currently but make the cache key table scoped. The PR contains a short description and some cli testing results to verify that.

    Another approach I tried is to attach an independent cache to each table (e.g. add a field in ListingTable struct) and have a session level map from table reference to table cache to track and display at session level. However as I was experimenting, I found it a bit tricky to handle user-provided ListFileCache implementation, which is currently given as a Arc<dyn ListFilesCache>. To support independent table scoped cache, I think we may need some kind of ListFilesCacheFactory instead?

    Please let me know your thoughts/suggestions, and correct me if anything does not make sense. Thank you!

  9. alamb commented on Jan 5, 2026

    @alamb
    alamb
    on Jan 5, 2026
    ContributorAuthor

    I have a draft PR #19616 for one approach I considered, which is to continue using a session level cache as is currently but make the cache key table scoped. The PR contains a short description and some cli testing results to verify that.

    Another approach I tried is to attach an independent cache to each table (e.g. add a field in ListingTable struct) and have a session level map from table reference to table cache to track and display at session level. However as I was experimenting, I found it a bit tricky to handle user-provided ListFileCache implementation, which is currently given as a Arc<dyn ListFilesCache>. To support independent table scoped cache, I think we may need some kind of ListFilesCacheFactory instead?

    I agree this one sounds tricky and unecessary -- if we can make #19616 work let's go with that one

    Edit: I reviewed #19616 and I think it looks good to me

  10. github-merge-queue
    added a commit that references this issue on Jan 8, 2026

    Make default ListingFilesCache table scoped (#19616)

    e6049de
  11. alamb
    closed this as completedin #19616on Jan 8, 2026
  12. alamb
    added a commit that references this issue on Jan 8, 2026

    Make default ListingFilesCache table scoped (apache#19616)

    aa3d413
  13. xudong963
    added a commit that references this issue on Jan 9, 2026

    [branch-52] Backport `list_files_cache`, and make default ListingFile…

    1ac18a3
  14. alamb
    added a commit that references this issue on Jan 15, 2026

    [branch-52] fix: expose `ListFilesEntry` (#19818)

    2073f04
  15. github-merge-queue
    added a commit that references this issue on Jan 15, 2026

    fix: expose `ListFilesEntry` (#19804)

    c173de1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

Feature

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions