Compilation is unbelievably slow

Hi, for some reason we have performance problems when using ts-node in our mocha tests. It takes about 500 ms to compile an empty ts file. I added some console.logs to measure the time to run this very line: const output = service.getEmitOutput(fileName) and this line takes 500 ms to run even though the content of the file is an empty string (the code variable on the line 314 is empty string). Actually, all our tests files take too long to process, while the production files take a few milliseconds to compile. So any *.spec.ts takes about 500 ms to compile, a regular *.ts file takes about 20 ms. Do you have any idea what could be the root cause or how should I debug it more?

We use latest mocha, ts-node and typescript, but we tried some old versions too and the problem persists. The tsconfig.json:

{
  "compileOnSave": false,
  "compilerOptions": {
    "outDir": "dist",
    "rootDir": ".",
    "declaration": true,
    "target": "es2017",
    "lib": ["es2017"],
    "module": "commonjs",
    "moduleResolution": "node"
  },
  "exclude": [
    "node_modules",
    "dist"
  ]
}

@blakeembrey

I have also experienced this issue when running alsatian tests after upgrading to ts-node v7. I've found I can resolve the issue by specifying the --files CLI flag.

The problem seems to stem from the fact that the test framework requires each .test.ts file in turn, and each call to require appears to compile every module in that file's dependency tree from scratch. Specifying --files reverts to the old behaviour of loading all files on startup (and presumably caching them in memory).

@blakeembrey does this sound like a plausible diagnosis, and can you see any better solution?

Have you tried --transpile-only mode?

@cspotcode

I've run into similar issue today. After ts-node upgrade from 7.0.1 to 8.0.1 my mocha test suite became very slow (32s vs 4s). Adding TS_NODE_TRANSPILE_ONLY=true resolved my issue. Thanks @cspotcode

--transpile-only does speed things up considerably for me. I don't want to lose the type checking though!

I tried all the proposed solutions. The files: true makes execution significantly faster! Turning off type checking also made things faster. I just had to turn off typeCheck and turn on transpileOnly (typeCheck: false, transpileOnly: true). This is good enough workaround, but it seems that the default behavior is still weird.

@davidgruar

That's intereseting, files: false is definitely meant to make things faster rather than slower. Does anyone have a large enough project that's running slow and want to either 1. investigate if the reason @davidgruar mentions appears true or 2. allow me temporary access to look into further myself?

Ideally files: false loads a single file and spiders out so it is possible that this is a slight issue with performance. It's also possible that the previous version was hitting the cache heavily, if you upgraded from v7 to v8 (v8 removed the cache because of type issues). We could also investigate adding caching back but it needs to be much smarter than the previous version to solve transitive type changes.

I've hit this issue too after updating from 7.0.1 to 8.0.1.

From what I can tell when files: false the tsconfig.json file gets changed in memory to remove the files and includes attributes.

ts-node/src/index.ts

Line 484 in 544851e

config.files = []

This seems to create an invalid config (when using TypeScript 3.2.4):

error TS18002: The 'files' list in config file 'tsconfig.json' is empty.

Removing the files and includes attributes altogether however seems to resolve the issue.

If anyone can give me temporary access to a slow repo, let me know! I can quickly take a look into the issue and happy to sign any NDA, etc you need. If not, can I get a sense of peoples tsconfig.json that's slow - are you relying on files, include, exclude, rootDir or other to compile with TypeScript?

Let's try with reactions:

❤️ files
🚀 include
👀 exclude
😕 rootDir
🎉 Other

8.0.2 hasn't made any difference for me. The fact it's adding files: [] in to the tsconfig.json still kills performance.

I can't share our project unfortunately - but here's our tsconfig.json:

{
	"compileOnSave": true,
	"compilerOptions": {
		"emitDecoratorMetadata": true,
		"experimentalDecorators": true,
		"forceConsistentCasingInFileNames": true,
		"lib": [
			"es2017"
		],
		"module": "commonjs",
		"moduleResolution": "node",
		"newLine": "LF",
		"noEmitOnError": true,
		"noImplicitReturns": true,
		"noUnusedLocals": true,
		"noUnusedParameters": false,
		"outDir": "dist",
		"removeComments": true,
		"rootDir": ".",
		"sourceMap": true,
		"strict": true,
		"strictFunctionTypes": false,
		"suppressImplicitAnyIndexErrors": true,
		"target": "es6"
	}
}

@garethflowers

@garethflowers Are you using mocha or something else that makes the explanation in #754 (comment) make sense?

Alternatively, can you try populating files = ['one.ts file here'] so it's valid, but still overridden, and let me know if that changes anything?

@blakeembrey

@blakeembrey I've made a minimal repro at https://github.com/davidgruar/tsnode-perf-repro - please check it out and have a play. Loading just three test files with fairly few dependencies takes more than a second, and the more imports you add, the longer it takes.

It doesn't seem like a huge overhead when you time the entire program, instead of just requires:

There is a difference, however, so for testing it might make the most sense to enable --files whereas running an application it doesn't - there's only a single input and TypeScript pre-processes everything anyway. I'll look into it more though because the difference is clear and it would be nice to bring the dynamic files number down.

Ok, perfect. It does look like TypeScript makes 4x the number of requests to the filesystem when files: false. Looking into whether there's some issue with how TypeScript is functioning here and how this could be improved.

i've found several settings contributing to slow compile times, im pretty sure the issue is related to settings that will delete the default setting of ignore: node_modules. IE: allowJs. Others that may also (just from my memory of guess/check) are removeComments, emitDecoratorMetadata, and a few others. I've also noticed that if you glob without file ext like include: src//* instead of src//*.ts it can cause slowdowns , idk if its chokadir watching fontfiles that may be in there or something. using isolatedModules is the sure-fire way to speed things up though.

@jeremy-coleman

@jeremy-coleman Sorry, I couldn't quite follow. Can you please let me know who you/what are responding to here?

Edit: I don't think any of those are related to the root cause of issues mentioned in this thread, that's all.

In my case with .mocharc.js, replace:
require: ["ts-node/register", ...]
with:
require: ["ts-node/register/transpile-only", ...]
resolved my issue.

@O-Q

@O-Q thank you, man. That helps me to start test faster!

Could this be a solution?
https://github.com/a7ul/esbuild-node-tsc

@thousandsofraccoons ts-node's "swc" integration is the recommended solution here. It skips typechecking and uses the blazing-fast swc transpiler. You can use it today, check the docs links below for instructions.

https://typestrong.org/ts-node/docs/performance#skip-typechecking
https://typestrong.org/ts-node/docs/transpilers#bundled-swc-integration

It is promoted out of "experimental" status in our next, upcoming release. #1536

Closing because some solutions are already available (--transpiler), other aspects are tracked by other open issues (incremental builds), and unfortunately this thread has not yielded good reproductions with which we can test.

--transpiler with swc yields the fastest compilation when skipping typechecking, beating out even a fully-cached solution

Docs have a dedicated page with performance recommendations: https://typestrong.org/ts-node/docs/performance

FYI swc seems to be incompatible with sequelize-typescript. I found the issue sec#1160 which mentions this, but it's possible that there are more undocumented issues preventing the two from working together.

Running with --transpile-only alone doesn't seem to help much for our tests execution.

We've seen a lot of cases where someone thought they were using transpileOnly, but actually they weren't. So that's always a possibility whenever it doesn't seem like transpileOnly is making much impact: perhaps it's not even turned on.

Is this an issue with cross-file metadata, in other words, is it compatible or incompatible with isolatedModules?

@eyalroth

@eyalroth

FYI swc seems to be incompatible with sequelize-typescript

Yeah, I believe only tsc (and ts-node without SWC) can handle emitDecoratorMetadata correctly. I tried almost all solutions such as babel, esbuild, swc but they doesn't emit decorator metadata correctly. I checked it by reading compiled code and it seems impossible.

Thus, I stopped using decorators. Moved from mikro-orm to objection, Type-GraphQL to Nexus, remove tsyringe, then moved from ts-node to esbuild. Now things became much better.

@cspotcode

We've seen a lot of cases where someone thought they were using transpileOnly, but actually they weren't. So that's always a possibility whenever it doesn't seem like transpileOnly is making much impact: perhaps it's not even turned on.

@cspotcode I tried all the options -- cli argument, environment variable and tsconfig.json 🤷

Is this an issue with cross-file metadata, in other words, is it compatible or incompatible with isolatedModules?

I... have no idea. Looks like @acro5piano knows more about this 😄

Compilers such as babel, swc, and esbuild do not understand type information. If type information from one file affects the emitted code from another file, it will fail to emit the code you're expecting using these tools. TypeScript has an option isolatedModules which will raise typechecking errors to warn you that your code is incompatible with these tools.

https://www.typescriptlang.org/tsconfig/#isolatedModules

This is an important detail to understand: these errors are raised when using TypeScript to typecheck, not when using babel, swc, or esbuild to compile your code! The assumption here is that you run typechecking as part of your automated testing even when you use a non-typechecking compiler. Think of the typechecker as a lint step.

Unfortunately, the reason I closed this issue still applies here:
Without any reproductions for us to look at, we can't keep guessing at what may or may not be happening in someone's project. Given a reproduction, we can take a look.

Another thing worth mentioning:
It's understandable that many users cannot share a proprietary codebase. But they might be able to create a fake codebase with just as many lines of code, using the exact same configuration files, but all the code is copy-pasted nonsense. If a fake codebase exhibits similar performance, it gives us something concrete to look at.

It's understandable that many users cannot share a proprietary codebase. But they might be able to create a fake codebase with just as many lines of code, using the exact same configuration files, but all the code is copy-pasted nonsense. If a fake codebase exhibits similar performance, it gives us something concrete to look at.

Of course! I'm planning on creating such a fake repo. I just need time :)

Awesome, thank you! Feel free to share it here or create a separate issue if that would help to focus the conversation.

The strangest thing happened. I undid all the configuration changes I made yesterday, npm ci-ed, added--transpile-only alone, and now it seems that the tests start running much faster.

Sorry for all the fuss.

chore: use mocha transpile-only

Navigation Menu

Compilation is unbelievably slow #754

150 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Compilation is unbelievably slow #754

Description

Activity

davidgruar commented on Jan 23, 2019

cspotcode commented on Jan 23, 2019

mazswojejzony commented on Jan 24, 2019

davidgruar commented on Jan 24, 2019

lukashavrlant commented on Jan 24, 2019

blakeembrey commented on Jan 25, 2019

garethflowers commented on Jan 25, 2019

blakeembrey commented on Jan 26, 2019

garethflowers commented on Jan 26, 2019

blakeembrey commented on Jan 26, 2019

davidgruar commented on Jan 26, 2019

blakeembrey commented on Jan 26, 2019

blakeembrey commented on Jan 27, 2019

jeremy-coleman commented on Jan 27, 2019

blakeembrey commented on Jan 27, 2019

150 remaining items

O-Q commented on Jun 18, 2021

leoblum commented on Jun 23, 2021

multipliedtwice commented on Nov 11, 2021

cspotcode commented on Nov 11, 2021

eyalroth commented on Nov 16, 2021

cspotcode commented on Nov 16, 2021

acro5piano commented on Nov 16, 2021

eyalroth commented on Nov 16, 2021

cspotcode commented on Nov 16, 2021

cspotcode commented on Nov 16, 2021

eyalroth commented on Nov 16, 2021

cspotcode commented on Nov 16, 2021

eyalroth commented on Nov 17, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions