Repository navigation
Unshipped blocks when out of order writes are enabled #5402
Description
Describe the bug
Unshipped blocks are shown in the cortex_ingester_oldest_unshipped_block_timestamp_seconds metric and are also visible in the ingester storage when out of order writes are enabled with the configuration introduced in #4964. Blocks are accumulating on the ingester as long as the config is set.
To Reproduce
- Start Cortex 1.15.2
- Allow out of order writes using newly introduced configuration parameters introduced in Support out of order samples ingestion #4964
- Perform Write operations
Expected behavior
Expecting to see no unshipped blocks on the ingester and have the metric cortex_ingester_oldest_unshipped_block_timestamp_seconds at value 0.
Environment
Additional Context
Tested with following two combinations of configurations and they produced the same result.
out_of_order_time_window: 30m
out_of_order_cap_max: 32
and
out_of_order_time_window: 30m
out_of_order_cap_max: 32
skip_blocks_with_out_of_order_chunks_enabled: true
Metrics:
cortex_ingester_shipper_uploads_total shows that block uploads are being done
cortex_ingester_shipper_upload_failures_total does not show any failures
cortex_compactor_runs_completed_total shows that compactions are being done
cortex_compactor_runs_failed_total shows no failed compactions
Logs:
There are no errors in Cortex component logs.
Only logs that could point to something are ingester logs regarding blocks overlapping, for example
caller=compact.go:698 org_id=fake msg="Found overlapping blocks during compaction" ulid=01H2SSZF807FMM2FD52HFGA3N7
Alerts:
CortexIngesterHasUnshippedBlocks alert from cortex-jsonnet is triggered as there are unshipped blocks available.
Activity
disambiguationuk commentedon Jun 15, 2023
I was just about to raise this same bug
danieljosephchambers commentedon Jun 15, 2023
We have tested various values for
out_of_order_time_windowincluding very short ones like10mand longer ones like2wand the problem persists.alanprot commentedon Jun 15, 2023
I will try to take a look on this this week!
cc @yeya24
yeya24 commentedon Jun 19, 2023
Is it because ingester shipper didn't upload compacted blocks? https://github.com/cortexproject/cortex/blob/master/pkg/ingester/ingester.go#L2031
yeya24 commentedon Jun 21, 2023
Also raised thanos-io/thanos#6462 on Thanos side.
I think make shipper upload compacted blocks works, but it might cause other issues (since we cannot identify compacted blocks generated by OOO or others)
disambiguationuk commentedon Jul 11, 2023
Is there anything outstanding that's blocking the merge still?
yeya24 commentedon Aug 3, 2023
@disambiguationuk I think we can merge this now #5416, I just need to rebase and resolve conflicts.
With this change https://github.com/cortexproject/cortex/pull/5495/files#diff-e1032332627c413a3010c66b54b22b6e9835cf152fa339e40cf0b11204f7241fR2043 we should be able to upload dynamically
AmerSelimovic commentedon Oct 20, 2023
Any updates on this fix, is it still being worked on?
yeya24 commentedon Oct 24, 2023
Hi @AmerSelimovic, sorry for the delay. The fix should be ready but I want to see if I can verify it first in our testing environment. I should get it done this week.
And if you are willing to test some prebuilt image, it would be very helpful
yeya24 commentedon Oct 29, 2023
@AmerSelimovic Actually I believe the bug is already fixed. If the tenant has OOO time window > 0 enabled, shipper should upload compacted blocks.
What we are trying to add in #5416 is to turn on/off shipper uploading compacted blocks dynamically in case OOO feature is enabled/disabled during runtime.
If OOO is enabled when ingester starts, all blocks can be uploaded successfully.
AmerSelimovic commentedon Nov 9, 2023
Hi @yeya24.
Not sure what do you propose fixed the reported bug?
Because issues were also happening with
out_of_order_time_window: 30mYou think it is okay with this change https://github.com/cortexproject/cortex/pull/5495/files#diff-e1032332627c413a3010c66b54b22b6e9835cf152fa339e40cf0b11204f7241fR2043
yeya24 commentedon Nov 9, 2023
The fix is to always upload compacted blocks in ingester so OOO compacted blocks can be uploaded to object store
yeya24 commentedon Nov 9, 2023
Btw https://github.com/cortexproject/cortex/releases/tag/v1.16.0-rc.0 is out. Feel free to try it out and see if it fixes this issue
yeya24 commentedon Apr 25, 2024
https://github.com/cortexproject/cortex/releases/tag/v1.17.0-rc.0 is out. It should address this issue completely as overlapped blocks will not be compacted by Prometheus anymore. Compactor will handle that.
Ingester: Add `upload_compacted_blocks_enabled` config to ingester to…
Ingester: Add `upload_compacted_blocks_enabled` config to ingester to…
Ingester: Add `upload_compacted_blocks_enabled` config to ingester to…