Right now this project works by statically creating a pool of connection (say 100), and then for each of those collections there is a queue. From that queue, n concurrent requests are pipelined (10) together to upstream.
The allocation of those requests to connections and their queues is currently round-robin. However, the optimal solution requires to always have each client at their full pipeline, and a request always ready to go, so that the TCP window of the connection stays wide open.
The problem is that one failing connection can bring down the whole pipeline, and it's quite a catastrophic event, because one failure can have n requests failing.
Also, slow requests have a huge impact on the pipeline size, as 1 requests that takes 500ms to reply will delay all the subsequent requests of 500ms.
Tuning the number of concurrent connections can swing the throughput up to 100% in my tests. I think we should find a nice algorithm to place the requests.
@lucamaraschi @delvedor do you have any idea to propose from your AI studies? This looks like a complex Knapsack problem
Right now this project works by statically creating a pool of connection (say 100), and then for each of those collections there is a queue. From that queue, n concurrent requests are pipelined (10) together to upstream.
The allocation of those requests to connections and their queues is currently round-robin. However, the optimal solution requires to always have each client at their full pipeline, and a request always ready to go, so that the TCP window of the connection stays wide open.
The problem is that one failing connection can bring down the whole pipeline, and it's quite a catastrophic event, because one failure can have n requests failing.
Also, slow requests have a huge impact on the pipeline size, as 1 requests that takes 500ms to reply will delay all the subsequent requests of 500ms.
Tuning the number of concurrent connections can swing the throughput up to 100% in my tests. I think we should find a nice algorithm to place the requests.
@lucamaraschi @delvedor do you have any idea to propose from your AI studies? This looks like a complex Knapsack problem