[Bug] Evicting a RESTTokenFileIO delegate does not close its plugin FileIO #10522

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

Master at b24751731a22fa9788205897b6554f247a3fe300.

Compute Engine

Engine independent: any reader or writer of a REST catalog table with data-token.enabled.

Minimal reproduce step

1. Access an OSS table through a REST catalog with data-token.enabled=true. RESTTokenFileIO creates one delegate per data token with file-io.allow-cache=false, so the delegate owns its file systems instead of sharing cached ones.
2. Keep the job running across token refreshes. Each refresh creates a new delegate, and the previous one stays in RESTTokenFileIO's FILE_IO_CACHE until it is evicted (10 hours without access, or more than setFileIOCacheMaximumSize entries, 1,000 by default).
3. On eviction the removal listener calls close() on the delegate. The delegate returned by FileIO.get(oss://...) is OSSLoader.OSSPluginFileIO, and PluginFileIO does not override FileIO.close(), whose default does nothing.

What doesn't meet your expectations?

OSSFileIO.close(), which closes its file systems when allow-cache is false, is never reached. The AliyunOSSFileSystem instances of every evicted delegate are left open, together with their OSS clients and thread pools, and a long-running job accumulates one set per refreshed token.

The same applies to the other loaders built on PluginFileIO (S3, Azure, GS, OBS, COSN).

Anything else?

Forwarding close() from PluginFileIO to the plugin FileIO is not enough on its own: an evicted delegate can still back streams that are open, such as a long read or a multipart upload, and closing its file systems on eviction would break them. Closing after the last stream would need RESTTokenFileIO to track the streams opened through it, including those opened by callers that take the delegate from fileIO() directly.

This came up in the review of #10178 and was left out of that PR.

Are you willing to submit a PR?

• I'm willing to submit a PR!