Repository navigation
[Bug]: Agent recordings of 3200×1800 frames at the 60 fps setting reach 45 fps on a 32-thread host and 10 fps on an i7-4790S, even with PR #25075; tab capture inside Chrome reaches 59 and 20 fps for less CPU #26683
Description
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
- Set Browser recording frame rate to 60.
- Have an agent record a server-browser tab of a page that changes every frame, for example an animation or a scrolling list, with
preview_recording_startandpreview_recording_stop. At the 1600×900 viewport, frames are 3200×1800, since the browser runs at device scale 2. - Count the frames of the MP4 with
ffprobe -count_frames, and watch the CPU.
Outside the app: the scripts in https://github.com/Vantrongs/t3code-recording-tab-capture drive the pinned chrome-headless-shell 156.0.8078.70 the way startRecording does, with PR #25075's frame delivery. On Linux, record.mjs records through the screencast and record-ext.mjs through in-browser tab capture; on Windows, run.ps1 runs both with bench.py, and the GPU checks below with gpu.py.
Expected behavior
A recording at the 60 fps setting holds about 60 fps.
Actual behavior
Agent recordings go through the JPEG screencast:
- Chromium compresses each frame to JPEG.
- CDP sends the frame to the server as base64.
- Playwright passes the string to an encoder page.
- The encoder page decodes the JPEG again with
createImageBitmapand draws it onto a canvas. MediaRecorderencodescanvas.captureStream()(recordingEncoderScript).
PR #25075 (for #24979) removes the per-character base64 decode and the stall on acks. With it, a 32-thread host still records 3200×1800 at 45 fps, and smaller machines lose most of the frames at either setting. The table below shows this in the "Screencast" column. With T3's launch flags (--disable-gpu), each case was recorded twice for 4 s:
| Machine | Frames | Setting | Screencast (PR #25075) | Tab capture + cursor |
|---|---|---|---|---|
| i7-4790S, 8 threads, Windows 10 | 3200×1800 | 60 | 9.6–9.7 fps, 6.5–6.6 cores | 20.1–20.4 fps, 5.0 cores |
| 3200×1800 | 30 | 9.6 fps, 6.6 cores | 19.7–19.8 fps, 4.9–5.0 cores | |
| 1600×900 | 60 | 23.2–23.7 fps, 6.6–6.7 cores | 59.4 fps, 2.8–2.9 cores | |
| 1600×900 | 30 | 22.5–22.9 fps, 6.5–6.7 cores | 30 fps, 1.8–1.9 cores | |
| Ryzen 9 9950X, 6 vCPUs, Windows 11 VM | 3200×1800 | 60 | 20.2–23.0 fps, 5.6 cores | 53.5–53.8 fps, 4.0 cores |
| 3200×1800 | 30 | 24.1–25.3 fps, 5.6 cores | 21.9–30 fps, 2.6–4.2 cores | |
| 1600×900 | 60 | 27.4–55.7 fps, 3.5–4.8 cores | 50.5–59.4 fps, 1.5–5.2 cores | |
| 1600×900 | 30 | 29.3–29.5 fps, 3.2–3.4 cores | 30 fps, 1.2–2.0 cores | |
| Ryzen 9 9950X, 32 threads, Linux | 3200×1800 | 60 | 44.9–45.8 fps, 5.4–5.5 cores | 59.4–59.6 fps, 2.4–2.8 cores |
| 3200×1800 | 30 | 29.2 fps, 5.1 cores | 30 fps, 1.4 cores | |
| 1600×900 | 60 | 59.3–59.5 fps, 2.0 cores | 59.8 fps, 0.6–0.7 cores | |
| 1600×900 | 30 | 30.2 fps, 1.9–2.0 cores | 30 fps, 0.4–0.5 cores |
On Linux, the CPU column counts Chrome plus the Node process that relays the frames. On Windows, it is the whole machine's busy time, since the relay there was a separate Python process; the idle background was about 0.4 cores. At 3200×1800 the screencast keeps nearly every thread of the 8-thread and 6-vCPU machines busy, at either setting.
The cost does not come from H.264. On the Linux host, the encoder page alone records pre-decoded 3200×1800 frames at 59.6 fps. Per frame, passing the string through evaluate takes about 5 ms and createImageBitmap about 11 ms. Chromium sends about 58 JPEG frames a second, and the encoder takes the newest one when it is free. Those frames reach the canvas unevenly, so captureStream gets two of them in some vsync periods and none in others.
In-browser tab capture avoids all of it. That is how the desktop app records (captureTabMediaStream). Frames go from the compositor to a video track and into MediaRecorder. The "Tab capture + cursor" column ran Chrome for Testing 156.0.8078.70 in headless mode with the same flags. It drew a cursor onto each captured frame before encoding, as the agent cursor needs: each frame went through MediaStreamTrackProcessor, was drawn onto an OffscreenCanvas with the cursor, and passed on through MediaStreamTrackGenerator with its original timestamp. Every frame of the 3200×1800 tab-capture videos I checked was distinct (mpdecimate dropped none). A gradient page showed that the frames cover the whole viewport.
How the cursor is drawn matters. With a canvas compositor like the desktop's recordingCompositor (requestVideoFrameCallback → canvas → captureStream), 3200×1800 at 60 reached 57.2–58.1 fps on the Linux host, 46.5 fps in the VM and 19.0 fps on the i7-4790S. Without a cursor, it reached 59.6, 58.3 and 25.6 fps.
The GPU is a separate question (#26523), but it decides how the cursor should be drawn. On the Linux host with ANGLE on Vulkan, tab capture with the cursor took 1.1 cores at 3200×1800 and 60 fps, at 59.5–59.8 fps, while the screencast stayed at 43.9–45.9 fps and 5.2–5.3 cores. On Windows, with Chrome's default ANGLE on D3D11, 3200×1800 at 60, each case recorded twice for 4 s (on the VM after warming the encoder, see below):
| Machine | Flags | No cursor | Cursor as an element in the page | MediaStreamTrackProcessor cursor |
|---|---|---|---|---|
| i7-4790S, HD 4600 | GPU | 32.2–32.3 fps, 2.2–2.3 cores | 31.3–31.7 fps, 2.1–2.2 cores | 1.1–1.6 fps, 1.8–2.0 cores |
--disable-gpu |
25.5–25.6 fps, 3.2–3.3 cores | 25.0–25.2 fps, 3.1 cores | 20.1–20.4 fps, 4.1–4.5 cores | |
| VM, Radeon iGPU | GPU | 58.7–59.1 fps, 0.9–1.1 cores | 55.2–59.3 fps, 1.0–2.3 cores | 36.3–51.6 fps, 1.4–1.9 cores |
--disable-gpu |
58.3–59.1 fps, 2.1–2.4 cores | 54.5–59.1 fps, 2.2–2.3 cores | 33.9–55.8 fps, 3.3–3.6 cores |
Here the cores are Chrome's processes alone. The Windows performance counters show where the GPU's work goes:
- The Radeon encodes 3200×1800 H.264 in hardware (its Video Codec engine is 46–47 % busy), so
MediaRecorderneeds half the CPU. The first recording after Chrome starts loses about 0.7 s while the encoder initializes: a 12 s recording right after launch reached 55.9 fps. CallingVideoEncoder.isConfigSupported()withhardwareAcceleration: "prefer-hardware"before the first recording removed that. - The HD 4600 gets no hardware encoder at this size. Chrome's Media Foundation encoder offers H.264 above 1920×1080 only when the driver reports
MF_VIDEO_MAX_MB_PER_SEC(mf_video_encoder_util.cc), and the HD 4600's driver 20.19.15.5171 (Intel's package 15.40.48.5171) does not:isConfigSupportedaccepts 1600×900 and rejects 3200×1800. So the frames are encoded on the CPU either way, and compositing plus capture keep the 3D engine 95–98 % busy at 32 fps. - A cursor drawn onto every frame costs a full-frame GPU copy. It keeps the Radeon's 3D engine 91–93 % busy, and on the HD 4600
new VideoFrame(offscreenCanvas)stalls at 1.1–1.6 fps. A canvas compositor fares no better (22.5–30.7 fps on the HD 4600). A cursor element in the recorded page (position: fixed,pointer-events: none, the topz-index) costs nothing per frame and recorded at the no-cursor rate on both machines. One of its two GPU runs on the VM was encoded in software (Video Codec engine 7 % busy, 2.3 cores); I did not find why. Being part of the page, it would have to be hidden from screenshots and snapshots. - ANGLE on Vulkan gave 6–7 fps on the Radeon, and on OpenGL 31 fps; D3D11 is the right backend there.
At 1920×1080, the largest size Chrome encodes in hardware on the HD 4600, the i7-4790S shows what the GPU does once the encoder is available (60 fps setting, cores of Chrome's processes):
| 1920×1080 at 60, i7-4790S | GPU | --disable-gpu |
|---|---|---|
| Screencast (PR #25075) | 16.3–16.6 fps, 4.8 cores | 16.0–16.3 fps, 4.7–4.9 cores |
| Tab capture, no cursor | 57.3–59.6 fps, 0.5–2.5 cores | 57.2–58.9 fps, 2.1–2.7 cores |
| Tab capture, cursor as an element in the page | 57.2–59.8 fps, 0.4–3.7 cores | 56.9–59.8 fps, 1.8–2.2 cores |
Tab capture, MediaStreamTrackProcessor cursor |
33.3–39.2 fps, 1.0–2.0 cores | 51.4–53.0 fps, 3.9–4.2 cores |
| Tab capture, canvas cursor | 42.7–44.9 fps, 1.6–1.7 cores | 43.6–46.9 fps, 4.0–4.3 cores |
Outside Chrome, the screencast kept the machine busy for another 1.9–2.1 cores, mostly the process relaying its frames. With the GPU, MediaRecorder and VideoEncoder encode in hardware: the MP4s come out at 6.7–7.6 Mbit/s, near the 6.2 Mbit/s target, against OpenH264's 2.1–2.2 Mbit/s, and the 3D engine is 64–71 % busy. In 18 of those 32 recordings without a per-frame cursor, Chrome took 0.4–1.0 cores. In the other 14 it took 1.6–3.7 cores, at the same frame rate, GPU load and CPU clock (% Processor Performance 83–113 %), spread over the GPU, browser and renderer processes; I did not find what switches it. A cursor drawn onto every frame costs more on this GPU than on the CPU: the 3D engine goes to 85–98 % and the recording drops to 33–45 fps.
What tab capture needs, from these checks:
- A different browser build.
chrome-headless-shellrejectsgetDisplayMediawithNotSupportedError, and it does not load an unpacked extension. Chrome for Testing runs in headless mode with T3's launch flags, on Linux and on Windows. Its linux64 zip is 201 MB against the shell's 126.6 MB. It links four more libraries directly,libcairo,libcups,libpangoand NSS'slibsmime3, sot3 browser setupwould have to install three more packages on hosts that lack them. Its macOS zips hold relative links inside the app bundle, which T3's unpacker rejects today. getDisplayMediaadds an infobar. A recorder page in the tab's own context can pick the tab with--auto-select-tab-capture-source-by-title=<token>, which matches a substring of the title. Chrome then shows its "sharing this tab" infobar, which takes 56 CSS px of the window. With headless Chrome's 87 px toolbar, the window's content area ends up 143 px shorter than Playwright's emulated viewport. The page keeps its emulated size, but the frame misses its bottom 143 px until the tab navigates. When I made the window 143 px taller than the viewport, the frames matched it exactly: before and after a cross-origin navigation, and with another tab in front.- A
chrome.tabCaptureextension shows no infobar. That is what the table used, loaded with--load-extensionand--allowlisted-extension-id; without the latter,getMediaStreamIdfails with "Extension has not been invoked for the current page". It does not see tabs in off-the-record contexts (browser.newContext(), which T3 uses for incognito and isolated tabs), even with"incognito": "spanning". I did not try allowing it in incognito through the profile. Agent tabs open in the default persistent profile once a client has reported its profiles, so the extension reaches most agent recordings. - The capture records the window, not the emulated viewport. Playwright emulates T3's viewport inside a window that keeps its own size, and Chrome scales the page into it. Tab capture also keeps one frame size and letterboxes into it, unless the size bounds exceed 1 px and differ in aspect ratio (
SelectResolutionPolicyFromCandidates). With the window's content area set to the viewport (Browser.setContentsSize) and bounds of 3×2 to 7680×7680, the frames kept the page at the scale and position ofPage.captureScreenshot, and they followed viewport changes the way screencast frames do. - Desktop-attached tabs render in Electron, not in this browser. They would keep the screencast or use the desktop's own capture.
A smaller change to the screencast path, if it stays: at the 30 fps setting, PR #25075 acks every frame at once. Chromium then compresses about 58 JPEG frames a second, and half of them are discarded. Delaying each ack to one frame interval after the previous one cut 3200×1800 at 30 on the Linux host from 5.22 to 3.36–3.39 cores. The recording went from 29.1–29.2 to 27.5–28.2 fps.
Impact
Major degradation or frequent failure
Version or commit
main at 125b619 with PR #25075's delivery (3ba653f); chrome-headless-shell 156.0.8078.70 (T3's pinned version) for the screencast; Chrome for Testing 156.0.8078.70 for tab capture
Environment
- Intel Core i7-4790S (4 cores, 8 threads), Intel HD Graphics 4600, 8 GB, Windows 10 Pro.
- Windows 11 Pro VM with 6 vCPUs on an AMD Ryzen 9 9950X, with the Ryzen's Radeon iGPU passed through.
- Linux (NixOS) on the same Ryzen 9 9950X (32 threads), NVIDIA RTX 2060 SUPER with the 615.78.08 driver; load average 1.0–1.6 during the runs.
Node 24.21 with Playwright 1.60.0 on Linux, Python 3.12 with Playwright on Windows.
Activity
qlaffont commentedon Nov 10, 2026
Another data point on a recent nightly, Linux, 16 threads.
Setup: T3 0.1.9-nightly.20261110.4452,
chrome-headless-shell157.0.8103.38, Debian 13, Intel Core Ultra 9 285H (16 threads, 62 GB). The Arc iGPU (/dev/dri/renderD128, VA-API drivers installed) is unused because of--disable-gpu. Recording frame rate setting left at the default (30). Frame counts come fromffprobepacket timestamps of the MP4 returned bypreview_recording_stop.Agent recordings (
preview_recording_start/stop), page changing every frame (arequestAnimationFrameloop toggling a 2 px element, plus a CSS transition):Same 1920×1080 page without the rAF loop (only a CSS transition and idle time): 4.7 fps on average, with gaps of up to 2.4 s. In a real 56 s agent recording of a web app at 1920×1080, the video had 336 frames (6 fps on average) and three holes of 2.2 to 3.3 s. Each hole matched the time the agent spent between two tool calls, when the page did not repaint.
Standalone, same
chrome-headless-shellbinary, Playwright with a 1920×1080 viewport,Page.startScreencast(JPEG, quality 90), each frame acked on receipt: 60 fps at scale 1 and at scale 2. This confirms the analysis above: on this host, Chromium delivers 3840×2160 frames at full rate, and the loss is in the relay, decode and canvas path intoMediaRecorder.Workaround we use for now: outside T3, write each screencast frame to disk with its
metadata.timestamp. Then build an ffmpeg concat list with the real durations and encode at a constant 30 fps (scale=1920:1080:flags=lanczos,fps=30,libx264). This captures 44–60 fps during motion. The output has a constant frame rate, and idle periods are held instead of dropped.+1 for in-browser tab capture, or at least a constant-rate output that repeats the last frame while the page is idle.