Skip to content

fetch hangs indefinitely during second request to a certain webpage when using HTTP/2 #3433

Closed

Description

@marvinruder

Bug Description

When accessing the webpage https://www.marketscreener.com using fetch two times with a delay of 1 second or more in between, the Promise returned by the second fetch never resolves.

Reproducible By

In REPL, when undici is installed in ./node_modules in the current directory:

$ NODE_DEBUG=undici node
Welcome to Node.js v22.5.1.
Type ".help" for more information.
> const { Agent, fetch } = require('undici')
undefined
> const agent = new Agent({ allowH2: true })
undefined
> (await fetch("https://www.marketscreener.com", { dispatcher: agent })).status
UNDICI 87852: connecting to www.marketscreener.com using https:undefined
UNDICI 87852: connected to www.marketscreener.com using https:h2
(node:87852) [UNDICI-H2] Warning: H2 support is experimental, expect them to change at any time.
(Use `node --trace-warnings ...` to show where the warning was created)
UNDICI 87852: received response to GET https://www.marketscreener.com// - HTTP 200
200
> UNDICI 87852: trailers received from GET https://www.marketscreener.com// // ← this pops up approximately one second after the promise resolves and the status code is printed

> (await fetch("https://www.marketscreener.com", { dispatcher: agent })).status

// …after that, nothing happens

Expected Behavior

The Promise resolves to the response object.

Logs & Screenshots

See above and below.

Environment

macOS 14.5 (build 23F79)
Node.js 22.5.1

Additional context

I am able to reproduce that issue on different architectures (aarch64, x86_64), different operating systems (macOS, Linux), different environments (Docker, Bare Metal), and different internet connections (residential, data center). Unfortunately, I have not found another URL other than the one mentioned above with which the issue is reproducable, and I am not in control of the webserver serving that page, I am just connecting to it as part of a project of mine.

Edit: Apparently, the behavior described below is an output from OpenSSL and not sent by the server.

When manually speaking HTTP/1.1 to the server, I notice that approximately 1 second (same timespan as observed above when UNDICI […]: trailers received from … pops up) after the body is delivered and immediately before the socket is closed, the server sends the word closed:

Details
$ openssl s_client -connect www.marketscreener.com:443
Connecting to 213.182.38.210

[ …lots of TLS information here ]
GET / HTTP/1.1
Host: www.marketscreener.com

HTTP/1.1 200 OK
server: nginx
[ …more headers here ]
transfer-encoding: chunked

1C0F
[ …HTML content here]
0

closed
$

I do not know whether that behavior is conform with the HTTP standard (nonetheless I believe that fetch still should not hang indefinitely if something like that occurs even if the server is violating a standard) or even whether the server sends this closed over HTTP/2 as well. The server advertises itself as nginx, and I was unable to find a documented behavior of nginx similar to what I observe.

Activity

  1. KhafraDev commented on Aug 2, 2024

    @KhafraDev
    Member
  2. metcoder95 commented on Aug 4, 2024

    @metcoder95
    Member

    Hi @marvinruder,

    After debugging the use case you shared, I can tell you that this is a somehow expected behavior given the current design of the Agent.

    What is happening is that, as the connection is not a keep-alive, the server immediately after a second, sends a GOAWAY frame to the client, hinting to gracefully shutdown the connection with the remote origin.
    So, if you are able to trigger two or more requests within a timeframe of < 1s, they will be sent concurrently within the same connection as per H2 specification, but if waiting more than that time window the server immediately hints to close the connection.

    Sadly, this lefts the session unusable and it is needed to create a new connection from start with the remote origin. Currently this is not supported by the Agent class, but you can use a Pool instead. Maybe is detecting this network situations can be something nice to add to the Agent class.

    See following example for reproduction:

    const { setTimeout: sleep } = require('node:timers/promises')
    const { request, fetch, Agent } = require('undici')
    
    const agent = new Agent({ allowH2: true })
    const delay = process.env.DELAY ?? 1000
    
    fetch('https://www.marketscreener.com', { dispatcher: agent })
      .then(res => {
        console.log('1st request', res.status)
    
        return sleep(delay)
      })
      .then(() => {
        console.log('dispatching after sleep for', delay, 'ms')
        return fetch('https://www.marketscreener.com', { dispatcher: agent })
      })
      .then(res => {
        console.log('2nd request', res.status)
      })
      .catch(console.error)
  3. marvinruder commented on Aug 9, 2024

    @marvinruder
    Author

    @metcoder95 Thanks for the detailed answer! It would be great if the Agent class was able to detect these cases one day. For the moment I will create short-lived H2 agents more frequently in my use case.

  4. metcoder95 commented on Aug 12, 2024

    @metcoder95
    Member

    I'll close this issue as completed, but please feel free to open a new one for discussing this possible feature to the Agent class

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions