Repository navigation
Expand Perf Suite Codebases, Expand Perf Suite Operations #44033
Description
The perf suite code typically was added for specific scenarios, but if you're trying to get a sense of how sourcemap emit has changed, it's hard given that not all projects use that functionality.
I think we should
- Have perf suites projects enable emit
- Have perf suites projects enable source map emit
- Have perf suites projects enable declaration emit
Activity
weswigham commentedon May 10, 2021
Ron Buckton (@rbuckton) we can turn on declaration emit, source maps, and declaration maps for most (if not all) perf projects (and remove
noEmitOnError), but doing so is going to invalidate historic relationships for the data, so if we want to, we'll have to regenerate some historical data - are we OK with doing that?MartinJohns commentedon May 11, 2021
Wesley Wigham (@weswigham) Perhaps you can do this additionally?
DanielRosenwasser commentedon May 11, 2021
(CC Andrew Casey (@amcasey))
Martin Johns (@MartinJohns) can you elaborate?
MartinJohns commentedon May 11, 2021
Daniel Rosenwasser (@DanielRosenwasser) Continue to run the perf suite as is, but additionally run the perf suite with declaration emit, source maps and declaration maps enabled. Basically produce two data sets.
rbuckton commentedon May 13, 2021
In our current setup, we run benchmarks against a snapshot of a specific project, using the settings specific to that project. This is intended to cover different scenarios, such as:
Compiler-Unionsheavily uses union types to stress test those scenarios).That way we aren't only focused on everything being a "kitchen sink" build, and we can catch performance regressions in non-"kitchen sink" areas (i.e., we don't want to slow down in code paths where we don't emit source maps). The current approach is somewhat randomized, but it covers most of the bases. I wouldn't want to have to run a benchmark against the same project for every permutation of every feature, as that doesn't scale.
What we can do is add new scenarios that point to the same sources but with different settings. That way old runs wouldn't be invalidated. So in that case we might have a "TFS+SourceMaps" scenario, etc.
amcasey commentedon Aug 2, 2021
Related: #44251
24 remaining items
DanielRosenwasser commentedon Aug 1, 2023
Drizzle? (see #54939)
jakebailey commentedon Aug 30, 2023
As of today, our benchmark suite can now run
bun. For example: #52656 (comment)This is particularly interesting as it uses an entirely different engine (JSC) than Node/Deno (V8).
bun0.8.1 is currently broken when runningtsc(oven-sh/bun#4327), but one can run 0.7.3 and it works.tsservertests don't work, nor dotsserverstartup tests, but given I'm using an old release, hopefully it is fixed in a future release.I'm not yet sure if this should be a part of our main test suite; probably not until we increase our capacity a bit more.
Next up would be using VS Code's electron build via
ELECTRON_RUN_AS_NODE, which has different performance behaviors due to pointer compression and memory caging; we don't currently test it, yet there's probably more CPU cycles spent intsservervia VS Code than any other way to run TypeScript!Jarred-Sumner commentedon Aug 30, 2023
This is fixed in oven-sh/bun#4378 (currently the canary build) which will become the release build (v0.8.2) tomorrow or the next day. The error was in some code assuming
/dev/fd/${n}is mounted which it is not mounted by default in minimal Linux distributions like alpine. Our CI machine uses Ubuntu Server which mounts/dev/by default, which is why we didn't catch it sooner. We will add an integration test that runstsc --help.The last few lines of output of
strace tsc --helpwhentscis run in bun:Note how it is no longer calling openat() on
/dev/fd/1in v0.8.2jakebailey commentedon Aug 30, 2023
Thanks for the fix! There are other things that are broken (as noted above), but I have yet to go reproduce them. I'll send some bug reports once I get that sorted.
Jolg42 commentedon Aug 30, 2023
Hello from Prisma 👋🏼
Let us know if you're interested to hear from us about some complex types to test. It could be interesting to have some types used in Prisma Client Extensions, for example.
The performance test suite is located at https://github.com/microsoft/typescript-benchmarking? but is currently private? Do you plan to make it public one day? (Of course you might have reasons to not make it public, I'm curious here)
jakebailey commentedon Aug 30, 2023
That repo is currently only build infra, which I plan to make public in short order as there's nothing sensitive.
The actual tool that does the benchmarking isn't ready to be made public and that's where the benchmarks currently live. I'm hoping I can fix it up to pull benchmarks from the public repo but it's not really made for multiple places to store benchmarks, or benchmarks that have to be cloned from elsewhere. But that's just details and I'm working on it. The new hosts were just a lot easier to add!
jakebailey commentedon Aug 30, 2023
perf test vscodeis now live: #55267 (comment)Next up is adding some more benchmarks and making public what I can make public quickly, e.g. the infra itself (but not the tool, sorry).
jakebailey commentedon Jan 8, 2024
I'm happy to say that the new perf infra is now live and public: https://github.com/microsoft/typescript-benchmarking
I've been waiting to make this public as I did not yet have a good mechanism to support public benchmarks, but I got it all working in the past few days and now it's ready to go. The first public benchmarks are
vscode, as well as self tests of TypeScript compiling itself (bothtsc -p ./src/compileandtsc -b ./src).Note that the public benchmarks are still split off from our old internal benchmarks; we have a lot more perf capacity than we used to have but still not enough to start adding lots that always run.
For now, they can be run via the
perf test publicpreset. An example: #56987 (comment)This is the first time we've had new, actually error-free benchmarks in years, so I'm very excited.
jakebailey commentedon Feb 2, 2024
I'm going to close this issue in favor of microsoft/typescript-benchmarking#32 on the benchmarking repo.
This thread was refocused toward benchmark suggestions, but now that the repo that holds them is public, discussion and PRing can happen there.