You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It takes a long time to serve pages. It doesn't timeout, but the delays are very noticeable. This started happening few days ago. Is there some hiccups in the system?
I was told that we think that some background process causes FD exhaustion (see #1463 for example). I looked for potential causes a few weeks ago, but didn't spot anything suspicious aside of #1451.
I'm not sure what the right repo is, but I think this issue is about how hackage-server is deployed rather than anything to do with the service software itself.
That said, there have been lots of related issues. Here's one I hit recently:
Error 503 error:0A000126:SSL routines::unexpected eof while reading
error:0A000126:SSL routines::unexpected eof while reading
Error 54113
The slowdown happens every night around midnight GMT. We've tried some various changes and audits to the hackage codebase but not found much. We certainly should try to increase the available FDs on the server as a stopgap, and also audit it for any other process outside of hackage that may run at that time.
Thank you for that datapoint. Nonetheless the primary slowdown, which we receive alerts over, and causes significant trouble, occurs around midnight GMT. In general, we do not serve most of hackage directly, but behind a fastly cache, so any intermittent slowdowns may well have to do with our proxying and not servers under our own control, or other general broader network issues.
It seems getting worse, for the past week I observe the same symptoms as @phadej: Hackage is unreasonable slow (or intermittently throwing error 503) even during day time.
Reacted by Andrea Vezzosi, Wen Kokke, Mitchell Dalvi Rosen, carlwr and Sylvain Henry
I get slow responses at any time of day as well as some 503 timeouts. For instance, the timeout in the screen capture below happened around 15:30 GMT today. I've seen multiple such timeouts today and overall loads have been incredibly slow.
Fetching http://hackage.haskell.org/package/os-string-2.0.7/os-string-2.0.7.tar.gz...
Fetching http://hackage.haskell.org/package/os-string-2.0.7/revision/1.cabal...
Other error (attempt 1): timed out
Other error (attempt 2): timed out
Other error (attempt 3): timed out
Traceback (most recent call last):
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 519, in <module>
main()
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 460, in main
fetch_from_plan(plan, rootdir)
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 370, in fetch_from_plan
download_with_retry(url, output_path)
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 153, in download_with_retry
raise Exception(f"Failed to download after {retries} attempts.")
Exception: Failed to download after 3 attempts.
Fetching http://hackage.haskell.org/package/network-uri-2.6.4.2/network-uri-2.6.4.2.tar.gz...
Fetching http://hackage.haskell.org/package/network-uri-2.6.4.2/revision/1.cabal...
Other error (attempt 1): timed out
Other error (attempt 2): timed out
Other error (attempt 3): timed out
Traceback (most recent call last):
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 519, in <module>
main()
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 460, in main
fetch_from_plan(plan, rootdir)
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 370, in fetch_from_plan
download_with_retry(url, output_path)
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 153, in download_with_retry
raise Exception(f"Failed to download after {retries} attempts.")
Exception: Failed to download after 3 attempts.
Fetching http://hackage.haskell.org/package/network-uri-2.6.4.2/network-uri-2.6.4.2.tar.gz...
Fetching http://hackage.haskell.org/package/network-uri-2.6.4.2/revision/1.cabal...
Other error (attempt 1): timed out
Other error (attempt 2): timed out
Other error (attempt 3): timed out
Traceback (most recent call last):
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 519, in <module>
main()
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 460, in main
fetch_from_plan(plan, rootdir)
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 370, in fetch_from_plan
download_with_retry(url, output_path)
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 153, in download_with_retry
raise Exception(f"Failed to download after {retries} attempts.")
Exception: Failed to download after 3 attempts.
First, we have not done anything to the hackage server itself since a redeploy Jan 10. However, from the logs, it appears that a drastic increase in 503s began on Feb 14
This coincided with a rather sharp increase in overall requests
and also in hit ratio (the amount of different content being hit
So I wonder if the issue is arising because of increased load?
That said, if we zoom in on just the past few days, it definitely seems the main issue still arises at midnight UTC -- it is just that there may be smaller issues at other times as well
and if we contrast to a month prior we still see the peaks, but there's no intermittent issues between them
Something to bear in mind is that the above is only for the hackage domain (fastly stats). The same server now also serves the hackage-content domain. The former domain is basic webpages and also of course indexes and tarballs. The content domain serves haddocks and the like. There, we can see the regime shift much more starkly
Cache hits plummet, traffic balloons.
Requests per second effectively 10x
and most tellingly, http 1 requests (associated with programmatic traffic) completely overwhelm http2 requests (associated with users using browsers)
So my operating theory is we're getting mildly dos'd by bots again. I think in this case, things coincide with the rise of agentic bots as well as the big ai crawlers, and that's likely making this worse. More speculatively, we are also likely getting hit by bots that refuse to accept cached results, and insist on getting the very latest, which could cause the hit ratio to decline so drastically.
If we have an automated process kicking in a midnight, can we move that to another time? If that's not directly related, then moving that to another time might prevent two sources of load from accidentally coinciding. But also, I wonder if something in that process is invalidating the caching, and causing the increase in origin traffic. If so, then the spike in traffic will move to the new time along with it, and tell us something.
A few days ago while building I noticed cabal-install taking a long time for downloads, pausing for a long time waiting for a download to start. I would expect that the archive files being downloaded would already be in the proxy cache, so that's a little odd, but I'm not sure from the above discussion is we've looked at load on the cache or just on the origins.
Is there anything in cabal-install that does something at most once a day (such as, looking for a new index file), where the notion of a day resets at midnight UTC? If so, we could be in part doing this to ourselves, by invalidating local caching worldwide all at the same time. (Of course, people would have to be doing builds at midnight in order for this to matter, but CI processes might run hourly, or daily at midnight.)
Also, do we have user-agent stats? That would tell us if it's bots (as long as they are well-behaved bots).
Would the HF purchasing more server capacity be a way out of this?
We could make this a priority of this year's infrastructure spending.
We'd have to do some architectural work to figure out how to even utilize e.g. a second server, especially in a way that would be superior to what we already get from varnish/fastly caching. I'm not sure where I'd even begin there.
I think we should start by investigating the mirror service, which could be an issue, and along with that tightening up our robots.txt which is wildly permissive.
edit: mirroring happens frequently, its not the issue. we do have three midnight services (logrotation etc) we could stagger to experiment with however.
bootstrap: running /usr/local/.ghcup/ghc/9.4.8/bin/ghc-9.4.8 --info
Fetching sources to bootstrap cabal-install with GHC 9.4.8 at /usr/local/.ghcup/ghc/9.4.8/bin/ghc-9.4.8...
Fetching http://hackage.haskell.org/package/os-string-2.0.7/os-string-2.0.7.tar.gz...
Fetching http://hackage.haskell.org/package/os-string-2.0.7/revision/0.cabal...
Other error (attempt 1): timed out
Other error (attempt 2): timed out
Other error (attempt 3): timed out
Traceback (most recent call last):
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 519, in <module>
main()
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 460, in main
fetch_from_plan(plan, rootdir)
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 370, in fetch_from_plan
download_with_retry(url, output_path)
File "/home/runner/work/cabal/cabal/bootstrap/bootstrap.py", line 153, in download_with_retry
raise Exception(f"Failed to download after {retries} attempts.")
Exception: Failed to download after 3 attempts.
Error: Process completed with exit code 1.
The infra team is currently pushing a configuration change that will hopefully prevent as many automated crawlers/scrapers from hammering Hackage.
As a longer term solution, the Haskell Foundation will be purchasing a more capable server for Hackage to replace the current server. As @gbaz noted, there is actual design and engineering required to make Hackage multi-host. That may be necessary one day, but for now we are just going to move Hackage to a more powerful/capable server.
Reacted by FTS, Simon Michael, Marco Z and Will Badart
I observed the fastly logs tonight after moving some scheduled services on the hackage box to see if the slowdown coincided with a move to scheduled services. It did not. However, I noted that fastly recorded a change in character of traffic at the point of slowdown.
Average object sizes shrink, hit ratio increases. So I don't think requests necessarily are increasing vastly, but it does look like we are perhaps getting periodically crawled in a way that explores the entire site including all generated pages? This would hit all our expired content and make the server have to work much harder to generate it all...
I.e. the new theory is that the periodic slowdown, like the newer problems, has indeed been periodic, but not a symptom of any internal issues so much as a crawler.
Maybe we can examine logs specifically at the slowdown point to see if there's any suspicious user-agents?
Logged onto hackage during the midnight UTC downtime, ran lsof -p [HACKAGE PID] | wc -l and got... 141547, which is mighty high.
eyeballing it appears to all be tcp.
by comparison, a recovered hackage-server will tend to fluctuate between say 240 and 600 at most.
We may also want to consider load-shedding, because hackage-server gets "stuck" when under sufficient load in a way it seems unable to recover easily from.
As of today, the server is completely unresponsive. Could we do anything about it at all? Like, shut down GET /package/*/upload-time and other secondary endpoints?
Activity
Bodigrim commentedon Feb 20, 2026
I was told that we think that some background process causes FD exhaustion (see #1463 for example). I looked for potential causes a few weeks ago, but didn't spot anything suspicious aside of #1451.
chreekat commentedon Feb 23, 2026
I'm not sure what the right repo is, but I think this issue is about how hackage-server is deployed rather than anything to do with the service software itself.
That said, there have been lots of related issues. Here's one I hit recently:
The url works fine for me now (https://hackage.haskell.org/package/QuickCheck/docs/Test-QuickCheck.html#v:genericShrink).
gbaz commentedon Feb 23, 2026
The slowdown happens every night around midnight GMT. We've tried some various changes and audits to the hackage codebase but not found much. We certainly should try to increase the available FDs on the server as a stopgap, and also audit it for any other process outside of hackage that may run at that time.
phadej commentedon Feb 24, 2026
I got a slow response right now (trying to load
splitmixpackage page), the time is nowhere near midnight GMT.gbaz commentedon Feb 24, 2026
Thank you for that datapoint. Nonetheless the primary slowdown, which we receive alerts over, and causes significant trouble, occurs around midnight GMT. In general, we do not serve most of hackage directly, but behind a fastly cache, so any intermittent slowdowns may well have to do with our proxying and not servers under our own control, or other general broader network issues.
Bodigrim commentedon Feb 24, 2026
It seems getting worse, for the past week I observe the same symptoms as @phadej: Hackage is unreasonable slow (or intermittently throwing error 503) even during day time.
wenkokke commentedon Feb 26, 2026
I get slow responses at any time of day as well as some 503 timeouts. For instance, the timeout in the screen capture below happened around 15:30 GMT today. I've seen multiple such timeouts today and overall loads have been incredibly slow.
philderbeast commentedon Feb 26, 2026
I've seen the first byte timeout too in the past week or two.
Bodigrim commentedon Mar 2, 2026
There are continuous outages any time of day causing CI failures like https://github.com/haskell/cabal/actions/runs/22579924829/job/65424134282?pr=11529#step:6:18
Same at https://github.com/haskell/cabal/actions/runs/22587804980/job/65437503670?pr=11559#step:6:27:
and at https://github.com/haskell/cabal/actions/runs/22587804980/job/65437503645?pr=11559#step:6:27:
How do we prioritise this issue? CC @jmct
gbaz commentedon Mar 2, 2026
Thanks for the report. This looks rather different than the other thing we've been diagnosing and we'll try to step up priority on looking at it.
gbaz commentedon Mar 2, 2026
First, we have not done anything to the hackage server itself since a redeploy Jan 10. However, from the logs, it appears that a drastic increase in 503s began on Feb 14
This coincided with a rather sharp increase in overall requests
and also in hit ratio (the amount of different content being hit
So I wonder if the issue is arising because of increased load?
gbaz commentedon Mar 2, 2026
That said, if we zoom in on just the past few days, it definitely seems the main issue still arises at midnight UTC -- it is just that there may be smaller issues at other times as well
and if we contrast to a month prior we still see the peaks, but there's no intermittent issues between them
gbaz commentedon Mar 2, 2026
Something to bear in mind is that the above is only for the hackage domain (fastly stats). The same server now also serves the hackage-content domain. The former domain is basic webpages and also of course indexes and tarballs. The content domain serves haddocks and the like. There, we can see the regime shift much more starkly
Cache hits plummet, traffic balloons.
Requests per second effectively 10x
and most tellingly, http 1 requests (associated with programmatic traffic) completely overwhelm http2 requests (associated with users using browsers)
So my operating theory is we're getting mildly dos'd by bots again. I think in this case, things coincide with the rise of agentic bots as well as the big ai crawlers, and that's likely making this worse. More speculatively, we are also likely getting hit by bots that refuse to accept cached results, and insist on getting the very latest, which could cause the hit ratio to decline so drastically.
gbaz commentedon Mar 2, 2026
From cloudflare, dns queries over the past month corroborate
jmct commentedon Mar 3, 2026
Would the HF purchasing more server capacity be a way out of this?
We could make this a priority of this year's infrastructure spending.
jeffstyr commentedon Mar 3, 2026
If we have an automated process kicking in a midnight, can we move that to another time? If that's not directly related, then moving that to another time might prevent two sources of load from accidentally coinciding. But also, I wonder if something in that process is invalidating the caching, and causing the increase in origin traffic. If so, then the spike in traffic will move to the new time along with it, and tell us something.
A few days ago while building I noticed cabal-install taking a long time for downloads, pausing for a long time waiting for a download to start. I would expect that the archive files being downloaded would already be in the proxy cache, so that's a little odd, but I'm not sure from the above discussion is we've looked at load on the cache or just on the origins.
jeffstyr commentedon Mar 3, 2026
Two other thoughts.
Is there anything in cabal-install that does something at most once a day (such as, looking for a new index file), where the notion of a day resets at midnight UTC? If so, we could be in part doing this to ourselves, by invalidating local caching worldwide all at the same time. (Of course, people would have to be doing builds at midnight in order for this to matter, but CI processes might run hourly, or daily at midnight.)
Also, do we have user-agent stats? That would tell us if it's bots (as long as they are well-behaved bots).
gbaz commentedon Mar 3, 2026
We'd have to do some architectural work to figure out how to even utilize e.g. a second server, especially in a way that would be superior to what we already get from varnish/fastly caching. I'm not sure where I'd even begin there.
I think we should start by investigating the mirror service, which could be an issue, and along with that tightening up our robots.txt which is wildly permissive.
edit: mirroring happens frequently, its not the issue. we do have three midnight services (logrotation etc) we could stagger to experiment with however.
philderbeast commentedon Mar 6, 2026
The bootstrapping test is failing today, https://github.com/haskell/cabal/actions/runs/22779049461/job/66079782780#step:7:32
gbaz commentedon Mar 7, 2026
We're getting absolutely hammered right now, we should move on some mitigations
jmct commentedon Mar 7, 2026
Hey folks, quick update.
The infra team is currently pushing a configuration change that will hopefully prevent as many automated crawlers/scrapers from hammering Hackage.
As a longer term solution, the Haskell Foundation will be purchasing a more capable server for Hackage to replace the current server. As @gbaz noted, there is actual design and engineering required to make Hackage multi-host. That may be necessary one day, but for now we are just going to move Hackage to a more powerful/capable server.
gbaz commentedon Mar 8, 2026
I observed the fastly logs tonight after moving some scheduled services on the hackage box to see if the slowdown coincided with a move to scheduled services. It did not. However, I noted that fastly recorded a change in character of traffic at the point of slowdown.
Average object sizes shrink, hit ratio increases. So I don't think requests necessarily are increasing vastly, but it does look like we are perhaps getting periodically crawled in a way that explores the entire site including all generated pages? This would hit all our expired content and make the server have to work much harder to generate it all...
I.e. the new theory is that the periodic slowdown, like the newer problems, has indeed been periodic, but not a symptom of any internal issues so much as a crawler.
Maybe we can examine logs specifically at the slowdown point to see if there's any suspicious user-agents?
gbaz commentedon Mar 8, 2026
and in those logs, in the period where we hit peak problems, but not elsewhere, i see lines like this:
Mar 08 03:54:50 hackage-origin hackage-server[599069]: 127.0.0.1 - [08/Mar/2026:03:54:49 +0000] "GET /package/ieee-utils-0.2/upload-time 1.0" 200 -1 "" "python-requests/2.32.3"I.e. a python package pulling upload-time information for each package. And I suspect upload-time may be an unoptimized path as well...
gbaz commentedon Mar 9, 2026
Logged onto hackage during the midnight UTC downtime, ran
lsof -p [HACKAGE PID] | wc -land got... 141547, which is mighty high.eyeballing it appears to all be tcp.
by comparison, a recovered hackage-server will tend to fluctuate between say 240 and 600 at most.
We may also want to consider load-shedding, because hackage-server gets "stuck" when under sufficient load in a way it seems unable to recover easily from.
simonmichael commentedon Mar 10, 2026
Adding just a few more data points to @gbaz's lovely charts, showing a view of external performance:
https://haskell-status.org monitors 6 different kinds of Hackage url, and here are the detail pages for each:
Bodigrim commentedon Mar 13, 2026
As of today, the server is completely unresponsive. Could we do anything about it at all? Like, shut down
GET /package/*/upload-timeand other secondary endpoints?gbaz commentedon Mar 13, 2026
We're currently investigating migrating to a server with more RAM, which might do the job.