Repository navigation
fetch hangs indefinitely during second request to a certain webpage when using HTTP/2 #3433
Description
Bug Description
When accessing the webpage https://www.marketscreener.com using fetch two times with a delay of 1 second or more in between, the Promise returned by the second fetch never resolves.
Reproducible By
In REPL, when undici is installed in ./node_modules in the current directory:
$ NODE_DEBUG=undici node
Welcome to Node.js v22.5.1.
Type ".help" for more information.
> const { Agent, fetch } = require('undici')
undefined
> const agent = new Agent({ allowH2: true })
undefined
> (await fetch("https://www.marketscreener.com", { dispatcher: agent })).status
UNDICI 87852: connecting to www.marketscreener.com using https:undefined
UNDICI 87852: connected to www.marketscreener.com using https:h2
(node:87852) [UNDICI-H2] Warning: H2 support is experimental, expect them to change at any time.
(Use `node --trace-warnings ...` to show where the warning was created)
UNDICI 87852: received response to GET https://www.marketscreener.com// - HTTP 200
200
> UNDICI 87852: trailers received from GET https://www.marketscreener.com// // ← this pops up approximately one second after the promise resolves and the status code is printed
> (await fetch("https://www.marketscreener.com", { dispatcher: agent })).status
// …after that, nothing happensExpected Behavior
The Promise resolves to the response object.
Logs & Screenshots
See above and below.
Environment
macOS 14.5 (build 23F79)
Node.js 22.5.1
Additional context
I am able to reproduce that issue on different architectures (aarch64, x86_64), different operating systems (macOS, Linux), different environments (Docker, Bare Metal), and different internet connections (residential, data center). Unfortunately, I have not found another URL other than the one mentioned above with which the issue is reproducable, and I am not in control of the webserver serving that page, I am just connecting to it as part of a project of mine.
When manually speaking HTTP/1.1 to the server, I notice that approximately 1 second (same timespan as observed above when UNDICI […]: trailers received from … pops up) after the body is delivered and immediately before the socket is closed, the server sends the word closed:
Details
$ openssl s_client -connect www.marketscreener.com:443
Connecting to 213.182.38.210
[ …lots of TLS information here ]
GET / HTTP/1.1
Host: www.marketscreener.com
HTTP/1.1 200 OK
server: nginx
[ …more headers here ]
transfer-encoding: chunked
1C0F
[ …HTML content here]
0
closed
$I do not know whether that behavior is conform with the HTTP standard (nonetheless I believe that fetch still should not hang indefinitely if something like that occurs even if the server is violating a standard) or even whether the server sends this closed over HTTP/2 as well. The server advertises itself as nginx, and I was unable to find a documented behavior of nginx similar to what I observe.
Activity
KhafraDev commentedon Aug 2, 2024
@metcoder95
metcoder95 commentedon Aug 4, 2024
Hi @marvinruder,
After debugging the use case you shared, I can tell you that this is a somehow expected behavior given the current design of the
Agent.What is happening is that, as the connection is not a
keep-alive, the server immediately after a second, sends aGOAWAYframe to the client, hinting to gracefully shutdown the connection with the remote origin.So, if you are able to trigger two or more requests within a timeframe of < 1s, they will be sent concurrently within the same connection as per H2 specification, but if waiting more than that time window the server immediately hints to close the connection.
Sadly, this lefts the session unusable and it is needed to create a new connection from start with the remote origin. Currently this is not supported by the
Agentclass, but you can use aPoolinstead. Maybe is detecting this network situations can be something nice to add to theAgentclass.See following example for reproduction:
marvinruder commentedon Aug 9, 2024
@metcoder95 Thanks for the detailed answer! It would be great if the
Agentclass was able to detect these cases one day. For the moment I will create short-lived H2 agents more frequently in my use case.metcoder95 commentedon Aug 12, 2024
I'll close this issue as completed, but please feel free to open a new one for discussing this possible feature to the
Agentclass