[server] Table and partition metadata may become inconsistent after drop and recreation #4741

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

1.0.0 (latest release)

Please describe the bug 🐞

A previous PR #4545 CI failed: https://github.com/apache/fluss/actions/runs/39093673328/job/117575521595?pr=4545

After some investigations, a possible reason could be:
Currently, Fluss may identify deleted tables and partitions by their table paths or partition names when processing UpdateMetadata requests.

This can lead to incorrect metadata cleanup when a table or partition is dropped and recreated with the same name before the previous deletion notification is processed.

For example:

1. A table db.t (table ID 100) with partition p1 (partition ID 200) is created.
2. The table or partition is dropped, but its metadata deletion notification is delayed.
3. A new table db.t (table ID 101) with partition p1 (partition ID 201) is created.
4. The delayed deletion notification for the old table or partition arrives.

Since the old and new objects share the same path or name, the deletion may incorrectly remove metadata belonging to the newly created object.

Additionally, stale partition metadata may remain in the TabletServer metadata cache after its parent table is deleted or recreated, potentially exposing old partition metadata through the new table path.

Expected behavior: Metadata deletion should only affect the original table or partition. Recreated objects with different IDs must remain unaffected, and stale partition metadata must not be exposed through a newly created table.

Solution

• Preserve the original table ID and partition ID in metadata deletion notifications, and perform cleanup based on these IDs rather than paths or names.
• Keep table and partition metadata mappings consistent when objects are deleted or recreated.
• Prevent newly created tables from inheriting stale partition metadata associated with a previous table ID.
• Ensure deletion notifications are not applied as regular replica metadata updates.
• Maintain compatibility with legacy deletion notifications during rolling upgrades.
• Add regression tests covering table/partition recreation and delayed deletion notifications.

Are you willing to submit a PR?

• I'm willing to submit a PR!