Compatibility between EnvHttpProxyAgent and DNS interceptor

[SuperOleg39]

Hello!

I have a question about EnvHttpProxyAgent and dns interceptor.

Can you help me to understand, how I can handle relation between this two features?

We have a custom setup with replacible default undici Agent, and extensible interceptors list, like this:

const agent = new Agent(options).compose(...(interceptors ?? []));

Then, we have separate http proxy module where default Agent is replaced, like this:

const agent = new EnvHttpProxyAgent({
    httpProxy: proxyEnv,
    httpsProxy: proxyEnv,
    noProxy: noProxyEnv,
    ...options,
  }).compose(...(interceptors ?? []));

The problem is, the same interceptors list is used, and DNS cache interceptor automatically passed into EnvHttpProxyAgent.compose list. After than, requests hostname is resolved to IP before proxy, and regular proxy env rules with hostnames can't be applied.

So, my question is more about how I need to treat relationship between this two:

• these features are mutually exclusive
• or I miss somethig and these features can work together?

[SuperOleg39]

On the one hand, I still want to cache DNS lookups for hosts from NO_PROXY list

[metcoder95]

By design coincidence, they are mutually exclusive, it was not meant to be like that from the beginning.

Nonetheless, sadly I'm unsure whereas a backtrack can be easily implemented. If this stills the desire, a intermediate interceptor can be added to do that explicit backtrack.

I can think of extending the dns interceptor to add a hint to the opaque object by adding the original DNS hostname instead of the resolved IP.

[SuperOleg39]

By design coincidence, they are mutually exclusive, it was not meant to be like that from the beginning.

Nonetheless, sadly I'm unsure whereas a backtrack can be easily implemented. If this stills the desire, a intermediate interceptor can be added to do that explicit backtrack.

I can think of extending the dns interceptor to add a hint to the opaque object by adding the original DNS hostname instead of the resolved IP.

Thanks for the answer!

Looks like for us, simple workaround to filter DNS interceptor before EnvHttpProxyAgent.compose should work.

[pacocartones]

Late to this, but the mechanism is worth pinning down — "mutually exclusive" turns out to be more specific than it sounds, and the thing you gave up (a DNS cache for NO_PROXY hosts) is still reachable without any change to undici.

Why composing them breaks it. EnvHttpProxyAgent takes the proxy decision from opts.origin:

// lib/dispatcher/env-http-proxy-agent.js
[kDispatch] (opts, handler) {
  const url = new URL(opts.origin)
  const agent = this.#getProxyAgentForUrl(url)

and the DNS interceptor overwrites exactly that field before the dispatch reaches it, keeping the original host only in a header:

// lib/interceptor/dns.js
const dispatchOpts = {
  ...origDispatchOpts,
  servername: origin.hostname,
  origin: newOrigin.origin,                                   // ← now an IP
  headers: withHostHeader(origin.host, origDispatchOpts.headers)
}

Because .compose() puts the interceptor above the agent, #shouldProxy never sees a hostname to match no_proxy against. Measured on main (undici 8.10.0, node v24.14.1) with a local origin, a local proxy, HTTP_PROXY set and NO_PROXY=localhost:

A  EnvHttpProxyAgent alone                     body=ORIGIN,ORIGIN  proxyHits=0  dnsLookups=0
B  .compose(dns())                             body=PROXY,PROXY    proxyHits=2  dnsLookups=1
    proxy received: http://127.0.0.1:25434/ (Host: localhost:25434)

Two things there are worth separating. The first is the no_proxy miss you reported. The second is that the proxy is then handed an absolute-form request line built from the IP, with Host: localhost — so on the proxied path the DNS interceptor is not merely inert: it changes what the proxy is asked to fetch, while not changing which socket is opened (that one is the proxy's). Gating the interceptor by hostname does not help either — I tried applying dns() only for no_proxy hosts and it still lands in the proxy, because rewriting the origin is itself what flips the decision.

Getting both. Take the proxy decision above the rewrite instead of below it, and give each branch its own dispatcher, so dns() only ever sits on the direct one:

function proxyRouter ({ shouldProxy, direct, proxied }) {
  return () => function routerDispatch (opts, handler) {
    const url = opts.origin.constructor === URL ? opts.origin : new URL(opts.origin)
    return (shouldProxy(url.hostname) ? proxied : direct).dispatch(opts, handler)
  }
}

const direct  = new Agent().compose(interceptors.dns())
const proxied = new ProxyAgent({ uri: process.env.HTTP_PROXY })
const dispatcher = proxied.compose(proxyRouter({
  shouldProxy: (h) => !noProxySet.has(h),
  direct,
  proxied
}))

Same harness, same NO_PROXY=localhost:

no_proxy host  http://localhost    body=ORIGIN,ORIGIN   proxyHits+=0
proxied host   http://127.0.0.1    body=PROXY,PROXY     proxyHits+=2

DNS lookups performed for 4 requests (2 no_proxy + 2 proxied): 1

Direct hosts stay direct and are DNS-cached; proxied hosts reach the proxy with the hostname intact. The cost is that shouldProxy is now yours rather than EnvHttpProxyAgent's: subdomain suffixes, ports, * and bracketed IPv6 are all handled in #parseNoProxy/#shouldProxy and would need mirroring for full parity. My harness only exercises http:; I have not checked the CONNECT path.

And the smaller fix this implies for undici, as an alternative to the opaque backtrack: EnvHttpProxyAgent builds its no-proxy agent internally —

this[kNoProxyAgent] = new Agent(agentOpts)

— with no way to compose onto that one specifically. A hook that let interceptors be attached to the inner no-proxy agent alone would make the two features composable, because the ordering problem disappears once the interceptor sits below the decision instead of above it. Happy to open an issue with the harness if that direction is interesting.

Full harness — two files, run from a checkout of main, no install needed (undici has no runtime deps)

repro-5303.js — cases A/B/C above:

'use strict'
const http = require('node:http')
const { once } = require('node:events')
const { EnvHttpProxyAgent, interceptors, request } = require('./index.js')

async function servers () {
  const origin = http.createServer((req, res) => res.end('ORIGIN'))
  origin.listen(0)
  await once(origin, 'listening')

  const proxyHits = []
  const proxy = http.createServer((req, res) => {
    proxyHits.push(`${req.url} (Host: ${req.headers.host})`)
    res.end('PROXY')
  })
  proxy.listen(0)
  await once(proxy, 'listening')

  return { origin, proxy, proxyHits }
}

function countingLookup (counter) {
  return (originUrl, opts, cb) => {
    counter.n++
    cb(null, [{ address: '127.0.0.1', family: 4 }])
  }
}

function onlyForHostnames (isDirect, interceptor) {
  return (dispatch) => {
    const wrapped = interceptor(dispatch)
    return function conditionalInterceptor (opts, handler) {
      const hostname = opts.origin == null
        ? null
        : (opts.origin.constructor === URL ? opts.origin : new URL(opts.origin)).hostname
      return hostname != null && isDirect(hostname)
        ? wrapped(opts, handler)
        : dispatch(opts, handler)
    }
  }
}

async function main () {
  const { origin, proxy, proxyHits } = await servers()
  const originPort = origin.address().port
  const proxyPort = proxy.address().port
  const target = `http://localhost:${originPort}/`
  const httpProxy = `http://127.0.0.1:${proxyPort}`
  const noProxy = 'localhost'

  console.log(`undici       : ${require('./package.json').version}`)
  console.log(`node         : ${process.version}`)
  console.log(`target       : ${target}`)
  console.log(`HTTP_PROXY   : ${httpProxy}`)
  console.log(`NO_PROXY     : ${noProxy}`)
  console.log('')

  const cases = [
    ['A  EnvHttpProxyAgent alone', (agent) => agent],
    ['B  .compose(dns())', (agent, c) => agent.compose(interceptors.dns({ lookup: countingLookup(c) }))],
    ['C  .compose(onlyForHostnames(noProxy, dns()))', (agent, c) =>
      agent.compose(onlyForHostnames(
        (h) => h === 'localhost' || h.endsWith('.localhost'),
        interceptors.dns({ lookup: countingLookup(c) })
      ))]
  ]

  for (const [label, build] of cases) {
    proxyHits.length = 0
    const counter = { n: 0 }
    const agent = build(new EnvHttpProxyAgent({ httpProxy, noProxy }), counter)

    const bodies = []
    for (let i = 0; i < 2; i++) {
      const res = await request(target, { dispatcher: agent })
      bodies.push(await res.body.text())
    }
    await agent.close()

    console.log(
      `${label.padEnd(46)} body=${bodies.join(',').padEnd(13)} ` +
      `proxyHits=${proxyHits.length}  dnsLookups=${counter.n}`
    )
    if (proxyHits.length) console.log(`    proxy received: ${proxyHits[0]}`)
  }

  origin.close()
  proxy.close()
}

main().catch((err) => { console.error(err); process.exit(1) })

repro-5303b.js — the router:

'use strict'
const http = require('node:http')
const { once } = require('node:events')
const { Agent, ProxyAgent, interceptors, request } = require('./index.js')

async function servers () {
  const origin = http.createServer((req, res) => res.end('ORIGIN'))
  origin.listen(0)
  await once(origin, 'listening')

  const proxyHits = []
  const proxy = http.createServer((req, res) => {
    proxyHits.push(req.url)
    res.end('PROXY')
  })
  proxy.listen(0)
  await once(proxy, 'listening')

  return { origin, proxy, proxyHits }
}

function countingLookup (counter) {
  return (originUrl, opts, cb) => {
    counter.n++
    cb(null, [{ address: '127.0.0.1', family: 4 }])
  }
}

function proxyRouter ({ shouldProxy, direct, proxied }) {
  return () => function routerDispatch (opts, handler) {
    const url = opts.origin.constructor === URL ? opts.origin : new URL(opts.origin)
    return (shouldProxy(url.hostname) ? proxied : direct).dispatch(opts, handler)
  }
}

async function main () {
  const { origin, proxy, proxyHits } = await servers()
  const originPort = origin.address().port
  const proxyPort = proxy.address().port
  const httpProxy = `http://127.0.0.1:${proxyPort}`
  const noProxy = new Set(['localhost'])

  const counter = { n: 0 }
  const direct = new Agent().compose(interceptors.dns({ lookup: countingLookup(counter) }))
  const proxied = new ProxyAgent({ uri: httpProxy })
  const router = proxied.compose(proxyRouter({
    shouldProxy: (h) => !noProxy.has(h),
    direct,
    proxied
  }))

  console.log(`undici     : ${require('./package.json').version}`)
  console.log(`node       : ${process.version}`)
  console.log(`HTTP_PROXY : ${httpProxy}`)
  console.log(`NO_PROXY   : ${[...noProxy].join(',')}`)
  console.log('')

  const targets = [
    ['no_proxy host  http://localhost', `http://localhost:${originPort}/`],
    ['proxied host   http://127.0.0.1', `http://127.0.0.1:${originPort}/`]
  ]

  for (const [label, target] of targets) {
    const before = proxyHits.length
    const bodies = []
    for (let i = 0; i < 2; i++) {
      const res = await request(target, { dispatcher: router })
      bodies.push(await res.body.text())
    }
    console.log(`${label.padEnd(34)} body=${bodies.join(',').padEnd(15)} proxyHits+=${proxyHits.length - before}`)
  }
  console.log(`\nDNS lookups performed for 4 requests (2 no_proxy + 2 proxied): ${counter.n}`)

  await Promise.all([direct.close(), proxied.close()])
  origin.close()
  proxy.close()
}

main().catch((err) => { console.error(err); process.exit(1) })