AOT build fails due to "JavaScript heap out of memory"

Bug Report or Feature Request (mark with an `x`)

- [X ] bug report -> please search issues before submitting
- [ ] feature request

Versions.

@angular/cli: 1.0.0
node: 7.7.4
os: win32 x64
@angular/animations: 4.0.0
@angular/common: 4.0.0
@angular/compiler: 4.0.0
@angular/core: 4.0.0
@angular/flex-layout: 2.0.0-rc.1
@angular/forms: 4.0.0
@angular/http: 4.0.0
@angular/material: 2.0.0-beta.2
@angular/platform-browser: 4.0.0
@angular/platform-browser-dynamic: 4.0.0
@angular/router: 4.0.0
@angular/cli: 1.0.0
@angular/compiler-cli: 4.0.0

Repro steps.

ng build --prod --aot

The log given by the failure.

/node_modules/@angular/material/aut 92% chunk asset optimization
<--- Last few GCs --->

[6328:0000016AB4D09F00]  1775597 ms: Mark-sweep 1411.2 (1513.5) -> 1411.0 (1513.5) MB, 1240.6 / 0.0 ms  last resort


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0000034B0DD266A1 <JS Object>
    2: optimize [0000000FABC02311 <undefined>:5480] [pc=00000082575652C8](this=000000633199A279 <an AST_Call with map 0000014753FA0291>,compressor=00000226DBDECA61 <a Compressor with map 0000032B1EC8F829>)
    3: before [0000000FABC02311 <undefined>:5463] [pc=0000008258C1818D](this=00000226DBDECA61 <a Compressor with map 0000032B1EC8F829>,node=000000633199A279 <an AST_Call with map 0000014753...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Desired functionality.

I would like to be able the app using AOT.

Mention any other details that might be useful.

I have tried changing ng.cmd to this:

@IF EXIST "%~dp0\node.exe" (
  "%~dp0\node.exe" --max_old_space_size=8192 "%~dp0\..\@angular\cli\bin\ng" %*
) ELSE (
  @SETLOCAL
  @SET PATHEXT=%PATHEXT:;.JS;=;%
  node --max_old_space_size=8192 "%~dp0\..\@angular\cli\bin\ng" %*
)

and ngc.cmd to:

@IF EXIST "%~dp0\node.exe" (
  "%~dp0\node.exe" --max_old_space_size=8192 "%~dp0\..\@angular\compiler-cli\src\main.js" %*
) ELSE (
  @SETLOCAL
  @SET PATHEXT=%PATHEXT:;.JS;=;%
  node --max_old_space_size=8192 "%~dp0\..\@angular\compiler-cli\src\main.js" %*
)

I have 8GB swap file. The build fails after 20-25 minutes standing on 92% progress.

@jmcclanahan

@jmcclanahan @filipesilva

ng serve : SUCCESS
ng build : SUCCESS

ng build --prod : FAIL
ng build --prod --build-optimizer=false : FAIL
ng build --aot : FAIL
ng build --aot --build-optimizer : FAIL

output as per #5618 (comment)

It also takes several 'minutes' hung at 92% before producing an error.

How do you suggest I produce a production build?

@filipesilva

Hi @filipesilva

What version of the CLI and Angular are you on?
1.5
Does this only happen on ng build --prod?
No, with --aot and --build-optimizer it fails too (see below)
Do you still get the memory error with these build commands:
For the commands that fail, it can be intermittent - ie. fail 4/5 times, but pass on the 5th
ng build --build-optimizer=false: Passes
ng build --aot: Passes
ng build --aot --build-optimizer: Fails

I can get these to pass 100% of the time by increasing the amount of available memory (ie: node --stack_size=4048 node_modules/.bin/ng build --prod | OR by --build-optimizer=false)

I've created a simple repo of the problem here: https://github.com/seanlandsman/build-optimiser-issue

Steps to reproduce:
npm install
npm run prod (note: this fails most of the time, but very occasionally it will pass)

thanks

I don't think increasing the node --stack_size is a good idea. All that will do is mask the real issue here. The build should not be gobbling up so much memory. There is a fundamental problem that needs to be fixed.

Think about CI, some environments don't have so much memory. How this "feature" pass tests? Why issue is closed?

The issue is not "closed". It's both open and marked urgent.
No one is suggesting that increasing the stack size is a permanent solution
angular-cli developers have chimed in indicating they're taking the issue seriously and requested feedback on which builds were having issues.

If folks want to complain about memory requirements, they can. If folks want to say it's unreasonable this bug exists, they can. If folks want to encourage people to no longer use Angular, they can. The developers are already looking at this, no amount of vitriol will make that faster. In the mean time, add a swap volume, increase the stack size, and subscribe to the issue for updates -- which unfortunately will likely be mostly complaints for a while.

Not sure if this is just my experience or not, but in cases where I had template errors (used a non-public variable in template, used the wrong type for an @Input(), etc.) I would see the memory usage climb really high, and then crash the build. My suggestion is to make extra sure that there are no template errors. This can be done by invoking ngc directly on the project. Not sure if that will help, just sharing my experience.

@magemello

Hey all,

I've been running a bunch of memory profiles on projects to try and figure out what the cause is. I'm comparing projects using CLI 1.5.0 + Angular 5.0.0 + Typescript 2.4.2 against CLI 1.4.9 + Angular 4.4.6 + Typescript 2.3.2. Special thanks to everyone that provided repros, that helps a lot.

@magemello I'm having some trouble building your project. It seems to be a couple of missing style variables (like $alfresco-typography). I assume I have to build these before the app. Can you tell me what the missing steps are? This is what I did:

git clone https://github.com/Alfresco/alfresco-ng2-components
cd alfresco-ng2-components
cd demo-shell-ng2
git checkout development
npm install
npm run ng -- build --prod --build-optimizer # errors on Undefined variable: "$alfresco-typography".

@seanlandsman I could not reproduce with the repository you linked. I got the same results with 1.4.9 as I did with 1.5.0. Builds seemed to finish in roughly the same amount of time, and consume a maximum of 500 MB of memory. Bundle sizes were also the same (more on that in a second). Can you think of anything I might be missing? I'm on Windows 10, node 8.9.1.

@teabagp using your repro (#8457 (comment)) I saw both a bundle size increase (500kB to 4MB) in 1.5.0 and a memory usage increase (from 580 to 1080 MB). This is the only community repro I really have at the moment, if someone else has a project I can look at please let me know.

The last repro is specially relevant because it shows this problem even in a small project. Those of you that ran into memory problems that went away after increasing the memory limit, can you tell me if your bundles also got significantly bigger?

Based on @teabagp's repro, this is what I can currently say:

this seems to be related to the library used (devextreme and devextreme-angular),
memory usage seems normal up until the "92% asset optimization" phase, then dramatically increases,
switching build optimizer off doesn't change this,
switching to the old uglify doesn't change this,
I don't think it's directly related to the new TS compiler since that runs at the 10% building modules phase and memory usage seems normal then.

At the moment I don't think the 1.5 memory usage increase is directly related to either Build Optimizer or Sourcemaps. Both of these increase memory usage by virtue of doing more thing. So bigger projects should indeed use more memory. We should still keep this under control but for the 1.5 specific problem it doesn't seem to be the root cause.

@filipesilva

@filipesilva

Those of you that ran into memory problems that went away after increasing the memory limit, can you tell me if your bundles also got significantly bigger?

Yes, main bundle increased by roughly 4MB. Sadly the project isn't open source, but it's pretty complex anyway the other project is a better example to try diagnose I think.

@filipesilva

Hi @filipesilva,

sorry for not providing you more information on how to run it correctly:

cd scripts
./update-version.sh -v 2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432
cd ..
cd demo-shell-ng2
npm install
npm run ng -- build --prod --build-optimizer 


> Alfresco-Angular2-Demo@1.10.0-beta6 ng /Users/mromano/dev/alfresco-ng2-components/demo-shell-ng2
> ng "build" "--prod" "--build-optimizer"

 92% chunk asset optimization

@magemello

@magemello I'm on windows and unfortunately that script doesn't seem to be very windows friendly even with gitbash...

kamik@T460p MINGW64 /d/sandbox/memory/alfresco-1.4/scripts (development)
$ ./update-version.sh -v 2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432
====== New version 2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432 =====
====== UPDATE COMPONENTS ======
====== clean lock file ng2-alfresco-core ======
rm: cannot remove '/d/sandbox/memory/alfresco-1.4/scripts/../ng2-components/ng2-alfresco-core/package-lock.json': No such file or directory
====== UPDATE COMPONENT ng2-alfresco-core ======
====== UPDATE PACKAGE VERSION of  to 2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432 version in all the package.json ======
sed: can't read s/"version": "[0-9]\.[0-9]\.[0-9]"/"version": "2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432"/g: No such file or directory
====== UPDATE DEPENDENCY VERSION of ng2-alfresco-core to ~2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432 in ng2-alfresco-core======
sed: can't read s/"ng2-alfresco-core": "[0-9]\.[0-9]\.[0-9]"/"ng2-alfresco-core": "2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432"/g: No such file or directory
sed: can't read s/"ng2-alfresco-core": "~[0-9]\.[0-9]\.[0-9]"/"ng2-alfresco-core": "~2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432"/g: No such file or directory
====== UPDATE DEPENDENCY VERSION of ng2-alfresco-datatable to ~2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432 in ng2-alfresco-core======
sed: can't read s/"ng2-alfresco-datatable": "[0-9]\.[0-9]\.[0-9]"/"ng2-alfresco-datatable": "2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432"/g: No such file or directory
sed: can't read s/"ng2-alfresco-datatable": "~[0-9]\.[0-9]\.[0-9]"/"ng2-alfresco-datatable": "~2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432"/g: No such file or directory
====== UPDATE DEPENDENCY VERSION of ng2-alfresco-upload to ~2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432 in ng2-alfresco-core======
sed: can't read s/"ng2-alfresco-upload": "[0-9]\.[0-9]\.[0-9]"/"ng2-alfresco-upload": "2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432"/g: No such file or directory
sed: can't read s/"ng2-alfresco-upload": "~[0-9]\.[0-9]\.[0-9]"/"ng2-alfresco-upload": "~2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432"/g: No such file or directory
====== UPDATE DEPENDENCY VERSION of ng2-activiti-diagrams to ~2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432 in ng2-alfresco-core======
sed: can't read s/"ng2-activiti-diagrams": "[0-9]\.[0-9]\.[0-9]"/"ng2-activiti-diagrams": "2.0.0-6a24c6ef754f24212bdd7120d5cdbe47b83c3432"/g: No such file or directory

Is there something I can do even if hacky to get around the need for it? Ideally without removing the global styles as I'm also investigating if style processing is using a lot of memory.

@hanvyj

@hanvyj thanks for confirming that!

@filipesilva

@filipesilva
We started experiencing this pretty significantly in CLI 1.3 & it got worse with 1.4 and so we downgraded to 1.2.6 and haven't had any problems since. we're still running ng 4.4.6 with cli 1.2.6 and it seems fine with default settings.

I don't have a ton of time to dig into this right now so i'm sorry this won't be super helpful but just wanted to let you know where the version numbers went wrong.

We are locked into the following specific versions (along with material 2.0.0-beta.12). if we change JUST cli to 1.3 or 1.4 (and presumably 1.5) we see the issue:

  "dependencies": {
    "@angular/animations": "4.4.6",
    "@angular/cdk": "2.0.0-beta.12",
    "@angular/common": "4.4.6",
    "@angular/compiler": "4.4.6",
    "@angular/core": "4.4.6",
    "@angular/forms": "4.4.6",
    "@angular/http": "4.4.6",
    "@angular/material": "2.0.0-beta.12",
    "@angular/platform-browser": "4.4.6",
    "@angular/platform-browser-dynamic": "4.4.6",
    "@angular/router": "4.4.6",
    "@ngrx/store": "4.0.3",
    "@ngrx/store-devtools": "4.0.0",
    "core-js": "2.4.1",
    "dexie": "1.5.1",
    "font-awesome": "4.7.0",
    "fs-extra": "2.0.0",
    "hammerjs": "2.0.8",
    "lodash": "4.5.0",
    "lscache": "1.0.5",
    "moment": "2.17.1",
    "net": "1.0.2",
    "ng-diff-match-patch": "2.0.6",
    "ng2-file-upload": "1.2.0",
    "ng2-select": "1.2.0",
    "ngx-infinite-scroll": "0.5.1",
    "normalizr": "2.1.0",
    "primeng": "4.1.3",
    "rxjs": "5.4.3",
    "stompjs": "2.3.3",
    "zone.js": "0.8.14"
  },
  "devDependencies": {
    "@angular/cli": "1.2.6",
    "@angular/compiler-cli": "4.4.6",
    "@compodoc/compodoc": "1.0.3",
    "@types/jasmine": "2.5.53",
    "@types/jasminewd2": "2.0.2",
    "@types/lodash": "4.14.65",
    "@types/lscache": "1.0.27",
    "@types/node": "6.0.60",
    "@types/normalizr": "2.0.16",
    "agentkeepalive": "^3.3.0",
    "codelyzer": "3.1.2",
    "jasmine-core": "2.6.2",
    "jasmine-spec-reporter": "4.1.0",
    "karma": "1.7.0",
    "karma-chrome-launcher": "2.1.1",
    "karma-cli": "1.0.1",
    "karma-coverage-istanbul-reporter": "1.2.1",
    "karma-firefox-launcher": "1.0.0",
    "karma-jasmine": "1.1.0",
    "karma-jasmine-html-reporter": "0.2.2",
    "karma-phantomjs-launcher": "1.0.2",
    "node-sass": "4.5.3",
    "phantomjs-prebuilt": "2.1.7",
    "protractor": "5.1.2",
    "suitcss-components-grid": "3.0.3",
    "suitcss-utils-size": "1.1.0",
    "ts-node": "3.2.0",
    "tslint": "5.3.2",
    "typescript": "2.4.2"
  }

We were hoping to upgrade soon so glad to see some focus on the issue. Thanks so much!

I have executed it for you:

git checkout dev-filipesilva 
cd demo-shell-ng2
npm install
npm run ng -- build --prod --build-optimizer

@filipesilva

@filipesilva

I've just tried on a clean install and it still fails for me. I'm on OSX Sierra and am running node v7.7.3

I also note that if I run ng build --prod --build-optimizer=false it will run successfully repeatedly, whereas running just ng build --prod will fail most of the time.

I'm not sure what I can add here - is there some sort of profiling I can run on my side and provide the output?

thanks

this is also affecting me, using angular 5 and cli 1.5, ive had to switch off the build optimizer to get my builds to pass without a out of memory error :(

@seanlandsman

@seanlandsman do the bundle sizes remain the same?

@filipesilva

@filipesilva

With "node --stack_size=4048 node_modules/.bin/ng build --prod" I get:

1.4K 14 Nov 20:43 inline.aa760cefdeaeea9e038f.bundle.js
835K 14 Nov 20:43 main.05e5301d42766024fc03.bundle.js
60K 14 Nov 20:43 polyfills.e6475d2787bb3154d59c.bundle.js
48K 14 Nov 20:43 styles.9b2fb464224d0a79781d.bundle.css

With "./node_modules/.bin/ng build --prod --build-optimizer=false" I get:

1.4K 14 Nov 20:45 inline.d7125ae6ed51e3e3af81.bundle.js
4.9K 14 Nov 20:45 main.47d0056e07ee5cb9c483.bundle.js
60K 14 Nov 20:45 polyfills.ad37cd45a71cb38eee76.bundle.js
48K 14 Nov 20:45 styles.9b2fb464224d0a79781d.bundle.css
951K 14 Nov 20:45 vendor.799da157f41ee457e4bd.bundle.js

Interestingly you can see the main.xx.js got much bigger with in the first run, but the vendor file disappeared. This might be expected?

One further bit of information - I just tried this on an older Mac running El Capitan (node version v7.7.3) that I've not used for a while. When I run "node_modules/.bin/ng build --prod" it runs just fine, repeatedly, and much faster than on my current machine. Hopefully this is helpful.

thanks

@filipesilva

@filipesilva our build also got increased by 3MB or 4MB like the last example you said, taking a long time after the 92%, and we also use the devextreme libraries. Can't share the project repo unfortunately but I could provide our package.json if neeeded.

@filipesilva

@filipesilva try to reduce the stack_size to reproduce the @seanlandsman case.
node --stack_size=512 node_modules/.bin/ng build --prod gets me that error on my mac.

@filipesilva

@filipesilva
What version of the CLI and Angular are you on?
1.5 and 5.0.0 respectivley
Does this only happen on ng build --prod?
Only when the --aot flag is use.
Do you still get the memory error with these build commands:
ng build --build-optimizer=false
no
ng build --aot
yes
ng build --aot --build-optimizer
yes
If you have a project that I can look at, can you link it please?
Sorry, but our application is private per our corporate requirements.

@jmcclanahan

@jmcclanahan @filipesilva

ng serve : SUCCESS
ng build : SUCCESS

ng build --prod : FAIL
ng build --prod --build-optimizer=false : FAIL
ng build --aot : FAIL
ng build --aot --build-optimizer : FAIL

output as per #5618 (comment)

It also takes several 'minutes' hung at 92% before producing an error.

Is there any workaround to bypass it?
It is not working with the --max_old_space_size=8192

I have the same problem. I also recognized that the size of my application with ng serve drastically increased after the update to angular 5. With angular 4 the vendor.bundle.js created by ng serve used to be 10MB and with angular 5 it is now 30MB.

ng serve -> works
ng build -> works
ng build --prod -> fails
ng build --prod --build-optimizer=false -> fails
ng build --aot -> fails
ng build --aot --build-optimizer=false -> fails

@filipesilva

@filipesilva we all could also help debugging the webpack build, what tool are you using to debug it?
I was trying like that, but I'm not sure is the best way:

node-nightly --inspect-brk ./node_modules/@angular/cli/bin/ng build --prod

If you give me more instructions here or on Gitter I will be more than happy to help.

Heya, using the repros you all provided I believe I've found the main problem with 1.5. It has to do with how the new pipeline moved decorator removal into Build Optimizer.

For those interested in a more detailed explanation, this is what happens:

When you build with --aot, we use TypeScript transforms in Build Optimizer to remove Angular decorators (stuff like @Component and @NgModule) from both your app and libraries (they aren't needed after AOT).
In 1.4 we didn't use Build Optimizer for that, we used manual removal on your app only.
Due to a bug, decorators are removed but their imports stay in the code.
These imports make some third party libs be wholly included, instead of only partially included.
You can check this yourself by comparing ng build --aot when using CLI1.4.9+NG4 versus CLI1.5.0+NG5.
In the devextreme lib case, there was almost 9megs of extra code included.
Extra code means a lot more work for Uglify, which makes it consume a lot more memory.

There might be more factors contributing to the excessive memory usage but currently this is the approach I'm following as it's the most promising one. I hope to have a PR with a fix soon.

@filipesilva

@filipesilva awesome news! Thanks for sharing your progress - very much appreciated!

Some of us are experiencing this in non-latest-version setups and others all the way back in cli 1.2 and 1.3 setups. Do you believe this is just a natural limitation and expected behavior due to the size of the projects/dependencies? Is the default <2GB RAM allocation just naturally too small for compiling bigger projects?

@radoslavpetranov

@radoslavpetranov I have been thinking about that (@randyaa said the same thing as you), and trying to figure out what a good approach is.

Something that springs to mind is to split up into multiple processes. But upon further scrutiny that approach doesn't really help in the memory usage problem, since multiple process will use at least the same total memory, probably more due to inefficiencies and duplication.

The 1.5 problem highlighted how much Build Optimizer actually helps keep memory usage down by trimming imports (and thus project size). So maybe that's an overall better approach. It does imply that a project must be able to use Build Optimizer though, and there might still be problems with it (code manipulation is tricky).

Another approach might be to took a look at the Uglify memory usage in large bundles. Maybe there room for config tuning or something. But it's hard to say since those tools really rely on a semantic understanding of code bundles to work, so one does expect a lot of memory used on larger bundles.

Something else that I've been thinking about is to actually have a low memory setting, where disk/cpu/total time is traded for a smaller memory footprint. This is just a very high level idea that I haven't explored at all. It would help on limited environments like CI, whereas for dev environments you really want the tradeoffs that give you better rebuild/speeds instead.

For really large projects you just can't get around the fact there are more performant languages than JS and that build tools that can use them effectively (like Bazel) will offer a better experience.

@filipesilva

@filipesilva - Thanks for investigating this. Here's another data point, fwiw:

What version of the CLI and Angular are you on?
1.5.0 and 5.0.0
Does this only happen on ng build --prod?
Seems to be building fine without the --prod flag

Do you still get the memory error with these build commands:

ng build --build-optimizer=false
no
ng build --aot
no
ng build --aot --build-optimizer
no

If you have a project that I can look at, can you link it please?
Sorry, private repository and project

@filipesilva

@filipesilva Thanks for the update and explanation, however we experience this with our projects using cli 1.4.7 and angular 4.4.5 (and early versions too). Currently to build our production version we use the following:

ng build --environment=prod --aot --sourcemaps=false --base-href=\"/ui/\" --preserve-symlinks --output-hashing=all --extract-css=true

We then pass the output to Uglify to minify and mangle the JS and CSS. This process works correctly and we do not hit memory issues, but when we build with ng build --prod that is when we run into memory issues.

We also do not have the devextreme lib. I can't share our code, but I'm happy to install other versions of the CLI and post build results from different commands. Our application is approx 450 components and 290 modules.

@filipesilva

Hello @filipesilva
I don't know exactly if this my comment might be useful, but I will report some tests that I did.

I have an angular-cli (1.4.1), angular (4.4.4), typescript (2.3.4) project.
This project contains approximately 200 services and more than 700 components, distributed in at least 15 modules.

When running ng build --prod I noticed a few things.
In this specific project, 0 to 1.9 GB of memory occurs up to the first 10% of the compilation.
(I'm using the saas files from bootstrap).

From the 11% up to the 89% compilation, my memory goes up to less than 3 GB.

From 90% to 92% the memory is used in 4.6 GB.

And finally at 95% the consumption goes up to the maximum limit allowed.
In my case now, up to the 6 GB of RAM I've bound.

To make it compile, I had to configure virtual memory (swap).
I have only 8 GB RAM.

I increased to 12 GB, 16 GB, until finally I was able to compile the project by setting 32 GB RAM.

Consumption of NodeJS hit 28 GB at these same 95% emitting.

Unfortunately this project is private and would not be able to make it available.

Tests requested:

ng build --build-optimizer=false works! 1.5 GB max.

ng build --aot error!

To try to help, I created a very simple project.
In it, there are 10 modules created.
In each module, there are 50 components created.
In each component there is only one HTML with output "works!".

There is no CSS, there is no third-party npm component.
A simple project like this is consuming 1.6 GB of RAM for the AOT build.

I believe that consumption is a little high because it contains absolutely nothing in the project.
If you wanted to download the repository to test, it is zipped in the link below.
http://www.codegenerator.com.br/downloads/ng-heap-memory.zip

Thanks in advance for your help.

fix(@ngtools/webpack): remove app decorators on AOT

@mattem

I've put up a PR that should specifically address the memory usage regression when using 1.5 (#8506). Once it's merged and released I'll ping you all here to see if it helped across your projects.

@tolitech your report of how memory usage changed with build stage was very interesting and useful. Below the 92% stage what happens is TypeScript compilation and loading files into Webpack. After that it's mostly Uglify and Build-Optimizer.

But since you are on 1.4.1 you shouldn't have Build Optimizer by default with --prod. Just to double check, can you confirm that you are not using --build-optimizer?

In your report you mention using what I consider a large project (200 services and 700 components). Definitely not the largest around but still big. I'm not surprised at a 3GB memory usage for TS compilation and loading all app files.

But when it starts getting really bad is after that, when you end up needing at least 28GB at the 92% phase. This to me tells me the bottleneck is Uglify, or maybe some other part of our build pipeline that I'm not seeing right now. But it definitely bears investigating, which I'll do.

Running the small ng-heap-memory project as is I saw a maximum of 1.4GB ram being used. Using the PR this dropped to around 800MB, so it seems to be helping there. I agree that the 500 components are essentially empty, but it doesn't surprise me that just the sheer number of them causes high memory consumption on a build. It's a valuable example by the way, and thank you for making it.

@mattem that's a very interesting approach, and doubly interesting is that you don't suffer much memory issues with it. Can I ask you to give me some concrete numbers for comparison please? Maximum memory usage on a CLI prod build versus CLI non-prod plus manual Uglify. I'd also like to know your uglify version and settings please.

@filipesilva

Hello, @filipesilva
Yes, I confirm that I'm using without --build-optimizer.
Just ng build --prod.

And thanks for the feedback.

@filipesilva

@filipesilva i used same workaround i described it in my comment #5618 (comment)

command
ng build --app homeapp -aot=true -e=prod -sm=false -ec=true --named-chunks=false -oh=all --build-optimizer=true -vc=true

my memory usage:
0 % -18 % <= 828 mb
18% -89 <= 1140 mb
90 - 100 < 1253 mb

difference between --build-optimizer=true and false 90 mb

for uglify used gulp-uglify^3.0.0 default settings 1058 mb
also i used gulp-postcss for uglify and autoprefixer all latest version

@angular 5.0.1
@angular/cli 1.5.0

@filipesilva

@filipesilva Using ng build --prod gave the OOM error, however running the CLI with node --max_old_space_size=8192 it did complete, with the following memory usage (observed from looking at Activity Monitor in MacOS):

~ 14% 800mb real / 5.5gb virtual
~ 69% 1.5gb real / 6.4gb virtual
~ 90% 1.8gb real / 6.65gb virtual
~ 92% 2.3gb real / 7gb virtual
~ 94% 2.6gb real / 7.12gb virtual

Running with ng build --environment=prod --aot --sourcemaps=false --base-href="/ui/" --preserve-symlinks --output-hashing=all --extract-css=true (no increasing max_old_space) tops out at 1.49gb real / 6.23gb virtual. This is running with @angular/cli@1.4.7 and @angular/*@4.4.5

We use gulp-uglify@3.0.0, giving us uglify-js@3.0.28, setting compress and mangle to true.
The output is also run though loose-envify, uglifycss, and htmlmin
The gulp task peaks at 868mb real / 5.57gb virtual.

fix(@ngtools/webpack): remove app decorators on AOT

According to changelog the above mentioned fix is included in 1.5.1+. It still fails for ng build --prod with 1.5.2 :(

95% emitting
<--- Last few GCs --->

[4980:000001B75DBAB8D0] 268298 ms: Mark-sweep 1399.5 (1698.5) -> 1399.5 (1698.5) MB, 483.4 / 0.0 ms allocation failure GC in old space requested
[4980:000001B75DBAB8D0] 268868 ms: Mark-sweep 1399.5 (1698.5) -> 1399.5 (1629.5) MB, 569.7 / 0.0 ms last resort GC in old space requested
[4980:000001B75DBAB8D0] 269387 ms: Mark-sweep 1399.5 (1629.5) -> 1399.5 (1600.0) MB, 518.8 / 0.0 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 00000185B7C25EC1
1: DoJoin(aka DoJoin) [native array.js:~95] [pc=0000002F55357679](this=00000075A3182311 ,p=0000001FB1A5B831 <JSArray[937]>,q=937,E=00000075A31823B1 ,A=00000185B7C6EC69 <String[1]: >,z=00000075A3182421 )
2: Join(aka Join) [native array.js:~120] [pc=0000002F550C6489](this=00000075A3182311 ,p=0000001FB1A5B831 <JSArray[937]>,q=937,A=00000185B7C6EC69 <Str...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Still encountering this issue with angular cli 1.5.2

`ng build --aot --prod
95% emitting
<--- Last few GCs --->

209426 ms: Mark-sweep 1512.7 (1406.8) -> 1512.5 (1406.8) MB, 453.1 / 0.0 ms [allocation failure] [GC in old space requested].
209905 ms: Mark-sweep 1512.5 (1406.8) -> 1513.6 (1403.8) MB, 479.5 / 0.0 ms [last resort gc].
210359 ms: Mark-sweep 1513.6 (1403.8) -> 1514.7 (1403.8) MB, 453.6 / 0.0 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x4353ac3fa99
1: SparseJoinWithSeparatorJS(aka SparseJoinWithSeparatorJS) [native array.js:~75] [pc=0x1e05abc97c4a] (this=0x4353ac04241 ,w=0x1a58c504381 <JS Array[1078]>,F=0x1a58c5043a1 <JS Array[1078]>,x=1078,I=0x37d7075bc209 <JS Function ConvertToString (SharedFunctionInfo 0x4353ac5dbc9)>,J=0x4353ac4a369 <String[1]: >)
2: DoJoin(aka DoJoin) [native array.js:~129] [pc=0x1e05abb3c1b6] (thi...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/usr/local/bin/node]
2: node::FatalException(v8::Isolate*, v8::Localv8::Value, v8::Localv8::Message) [/usr/local/bin/node]
3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
4: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
5: v8::internal::Runtime_SparseJoinWithSeparator(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
6: 0x1e0596e060c7
Abort trap: 6
Jays-MacBook-Pro:ngx-dynamic-dashboard-framework jayhamilton$ ng --version

_                      _                 ____ _     ___

/ \ _ __ __ _ _ | | __ _ _ __ / | | | |
/ △ \ | ' \ / _ | | | | |/ _ | '| | | | | | |
/ ___ | | | | (| | || | | (| | | | || | | |
// __| ||_, |_,||_,|| _|||
|___/

Angular CLI: 1.5.2
Node: 6.11.1
OS: darwin x64
Angular: 5.0.1
... animations, common, compiler, compiler-cli, core, forms
... http, platform-browser, platform-browser-dynamic, router

@angular/cdk: 2.0.0-beta.12
@angular/cli: 1.5.2
@angular/material: 2.0.0-beta.12
@angular-devkit/build-optimizer: 0.0.33
@angular-devkit/core: 0.0.20
@angular-devkit/schematics: 0.0.36
@ngtools/json-schema: 1.1.0
@ngtools/webpack: 1.8.2
@schematics/angular: 0.1.5
typescript: 2.6.1
webpack: 3.8.1
`

@jayhamilton

@jayhamilton +1 swimlane/ngx-datatable#1104 (comment)

@PeS82

Hey all, 1.5.2 includes the fix for the 1.5 memory usage regression. This should help users that had projects that did not have memory problems on 1.4 but started having them in 1.5.

Projects that were already having memory problems in 1.4 should see no change. I'm still looking into the balooning memory requirements for the 92% bundle optimizing phase.

@PeS82 @jayhamilton @finalxcode you reported that you're still running into memory problems with 1.5.2. Was this on projects that did not have memory problems in 1.4.x?

@filipesilva

@filipesilva With angular cli 1.4.x it used to work, then when i upgrade angular/cli to 1.5.2, it have memory problems, i have to use command node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod --aot to build

@filipesilva

@filipesilva with cli 1.5.2 my project fails at 95%. With angular cli 1.4.x it used to work.

@filipesilva

@filipesilva @JackArlington same for me. Our project used to build (prod) on 1.4.x and since 1.5.0 does not anymore. 1.5.2 didnot improve the situation. When I increase the memory limit it does compile, but uses aprox 5GB on emitting.

On a side not the huge memory consuption occurs on "95% emitting"

@filipesilva

@filipesilva @Shadowlauch I have just recognized that the machine I tested before had node 6.x upgrading it to 8.9.1 solved the problem and ng build --prod --aot works now.
Thank you for the help.

@seanlandsman

To be clear, I am not aware of any memory issue related specifically to Node 6. I have been testing with Node 8.9.1 myself because I couldn't get memory timelines using Chrome Devtools with Node 6.

Maybe that's part of the equation here... I expect newer nodes to be faster and more memory efficient but perhaps there's more to it.

For instance, I couldn't repro the problem on @seanlandsman's project. @seanlandsman what version of node were you using?

@JackArlington

I had node 6.x aswell and after upgrading to 8.9.1 it works now. @JackArlington thanks for the tip!

@filipesilva

@filipesilva I'm on node 7.7.3

Edit: I've just upgrade the test repo to 1.5.2 and retried - it still fails I'm afraid.

Did you try running with "node --stack_size=512 node_modules/.bin/ng build --prod" to see if this helps reproduce what I'm seeing?

thanks again

@filipesilva

@filipesilva
I do not get heap out of memory error anymore with cli 1.5.2. My node version is 7.5.0.
However the build still takes about 8 - 10 minutes to finish.

@filipesilva

@filipesilva I have updated the version and I'm still having the same problem :(

ng build --prod --aot

92% chunk asset optimization                                                             
<--- Last few GCs --->

[70119:0x103000800]   221253 ms: Mark-sweep 1412.4 (1539.6) -> 1412.4 (1527.1) MB, 2772.9 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2773 ms) last resort 
[70119:0x103000800]   223828 ms: Mark-sweep 1412.4 (1527.1) -> 1412.4 (1527.1) MB, 2574.0 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1174077a8799 <JSObject>
    2: add_parameter [0x35ed60282311 <undefined>:2953] [bytecode=0x2a011d5e84c1 offset=69](this=0x1dee67b6c479 <Object map = 0x13abb57fc8b1>,token=0x3e75c751c6b9 <AST_Token map = 0x2edf31c836a1>)
    4: binding_element(aka binding_element) [0x35ed60282311 <undefined>:3236] [bytecode=0x2a011d5e6669 offset=2234](this=0x35ed60282311 <undefined>,used_parameters=0x1dee67b6c479 <Object map = 0x13abb57...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

Here are the versions which I'm using:

    _                      _                 ____ _     ___
   / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
  / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
 / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
/_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
               |___/
    
Angular CLI: 1.5.2
Node: 8.5.0
OS: darwin x64
Angular: 5.0.2
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, router

@angular/cdk: 5.0.0-rc0
@angular/cli: 1.5.2
@angular/flex-layout: 2.0.0-beta.10
@angular/material: 5.0.0-rc0
@angular-devkit/build-optimizer: 0.0.33
@angular-devkit/core: 0.0.20
@angular-devkit/schematics: 0.0.36
@ngtools/json-schema: 1.1.0
@ngtools/webpack: 1.8.2
@schematics/angular: 0.1.5
typescript: 2.4.2
webpack: 3.8.1

If you give me enough instruction on how you debug the webpack build I can help find the source of the problem....

Can confirm that it's fixed the problem for us. Using 1.5.2 and Node 9.2.0 - no longer need to increase Node memory.

With the new 1.5.2 release, my memory problem and bundle size haven been fixed, it still takes some time stuck at the 92% but works in the end.

@seanlandsman

@seanlandsman I tested your repros again with --stack_size=512 and --stack-size=8048 and only saw memory consumption in the 400-500MB range. The default seems to be --stack_size=984 btw.

I did see something else though:

kamik@T460p MINGW64 /d/sandbox/memory/aggrid-1.5 (master)
$ npm run prod

> test-ag-grid@0.0.0 prod D:\sandbox\memory\aggrid-1.5
> node --stack_size=512 ./node_modules/@angular/cli/bin/ng build --prod

Date: 2017-11-17T10:46:12.052Z
Hash: db122ac832a501fd0cf8
Time: 31443ms
chunk {0} main.67d8219d9d4c87c59477.bundle.js (main) 830 kB [initial] [rendered]
chunk {1} polyfills.e6475d2787bb3154d59c.bundle.js (polyfills) 61.1 kB [initial] [rendered]
chunk {2} styles.9b2fb464224d0a79781d.bundle.css (styles) 49 kB [initial] [rendered]
chunk {3} inline.d67e755da47b31497a24.bundle.js (inline) 1.45 kB [entry] [rendered]

ERROR in ./node_modules/ag-grid/dist/lib/gridOptionsWrapper.js
Module build failed: RangeError: Maximum call stack size exceeded
    at getJSDocsWorker (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:8167:33)
    at getJSDocs (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:8163:13)
    at getJSDocTags (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:8148:27)
    at getJSDocParameterTags (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:8213:20)
    at Object.isRestParameter (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:8295:28)
    at symbolToParameterDeclaration (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:29262:41)
    at D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:29215:89
    at Array.map (<anonymous>)
    at signatureToSignatureDeclarationHelper (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:29215:55)
    at createTypeNodeFromObjectType (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:29024:49)
    at createAnonymousTypeNode (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:28988:42)
    at typeToTypeNodeHelper (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:28938:28)
    at createTypeNodesFromResolvedType (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:29177:67)
    at createTypeNodeFromObjectType (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:29035:35)
    at createAnonymousTypeNode (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:28988:42)
    at typeToTypeNodeHelper (D:\sandbox\memory\aggrid-1.5\node_modules\typescript\lib\typescript.js:28938:28)
 @ ./node_modules/ag-grid/dist/lib/columnController/columnUtils.js 11:27-59
 @ ./node_modules/ag-grid/main.js
 @ ./node_modules/ag-grid-angular/dist/ng2FrameworkFactory.js
 @ ./src/app/app.component.ngfactory.js
 @ ./src/app/app.module.ngfactory.js
 @ ./src/main.ts
 @ multi ./src/main.ts

This seems consistent with what --stack-size actually does (nodejs/node#6360). It doesn't increase total available memory, but rather the the size of each stack region that node uses. It that increasing it can solve issues with a lot of recursion (but that code should try to not have that).

The symptoms seem to be different on OSX (memory bust) and Windows (maximum call stack size exceeded). I don't know why that is.

I think the problem you're experiencing is solely related to Build Optimizer as you posited, and is the one tracked in angular/devkit#240. Still relevant for this thread mind you, but the specific issue is that one.

@changLiuUNSW were your build times with 1.4.x shorter, and if so, by how much?

@magemello I'll ping you on the gitter you provided and see what we can do in your project.

@filipesilva

@filipesilva
I do not have a chance to try 1.4.x because we used CLI 1.3.0 in our project before
Angular CLI 1.3.0 with Angular 4.2.4

ng build --prod
[14:08:02][Step 4/15] Time: 236558ms
[14:08:02][Step 4/15] chunk {0} polyfills.6bb91be4d06bd2c9e250.bundle.js (polyfills) 217 kB {4} [initial] [rendered]
[14:08:02][Step 4/15] chunk {1} main.71c6b1116dc39af75599.bundle.js (main) 1.59 MB {3} [initial] [rendered]
[14:08:02][Step 4/15] chunk {2} styles.21cad12ac9e3527f0f7d.bundle.css (styles) 166 kB {4} [initial] [rendered]
[14:08:02][Step 4/15] chunk {3} vendor.dfcfc494ca724bf884bf.bundle.js (vendor) 653 kB [initial] [rendered]
[14:08:02][Step 4/15] chunk {4} inline.f1134fc323101279dbaf.bundle.js (inline) 1.46 kB [entry] [rendered]

Angular CLI 1.5.2 with Angular 5.0.1

ng build --prod
[22:20:39][Step 4/15] Time: 530465ms
[22:20:39][Step 4/15] chunk {0} polyfills.953b2ea9f096cadfdccc.bundle.js (polyfills) 147 kB [initial] [rendered]
[22:20:39][Step 4/15] chunk {1} main.e2cfa60d1ec303d72a6a.bundle.js (main) 1.77 MB [initial] [rendered]
[22:20:39][Step 4/15] chunk {2} styles.b49f8f5ee25381105567.bundle.css (styles) 108 kB [initial] [rendered]
[22:20:39][Step 4/15] chunk {3} inline.2ae12ef65c8a9f6162ba.bundle.js (inline) 1.49 kB [entry] [rendered]

I understand in 1.5, --build-optimize is enable by default for production build, but 5 minutes difference between 1.5 and 1.3 is not good.

@changLiuUNSW

@changLiuUNSW I agree 5 minutes difference is a long time. Can you try --build-optimizer=false for now please?

@filipesilva

@filipesilva
Angular CLI 1.5.2 with Angular 5.0.1

ng build --prod --build-optimizer=false
Time: 163265ms
chunk {0} polyfills.cdff46cd9b1a8d656218.bundle.js (polyfills) 147 kB [initial] [rendered]
chunk {1} main.d1916123bd0e4aa2858f.bundle.js (main) 1.34 MB [initial] [rendered]
chunk {2} styles.d1f7c9f576103c62cb95.bundle.css (styles) 108 kB [initial] [rendered]
chunk {3} vendor.1fbdf23b6f274bb4005e.bundle.js (vendor) 639 kB [initial] [rendered]
chunk {4} inline.58b71d550a47f85197a5.bundle.js (inline) 1.5 kB [entry] [rendered]

6 minutes difference with build-optimizer=true

@filipesilva

Thank you @filipesilva for your help, I did the following:

Set Angular dependencies to 5.0.2
Set Angular CLI to 1.5.2
Delete node_modules & package-lock.json
npm install

and now the result is the following:

npm run ng -- build works
ng build --aot fails
node --max_old_space_size=30000 node_modules/.bin/ng build --aot works
ng build --prod --build-optimizer=false fails
node --max_old_space_size=30000 node_modules/.bin/ng build --prod --build-optimizer=false works
node --max_old_space_size=30000 node_modules/.bin/ng build --prod fails

So, in general raise the max_old_space_size fix my issue at exception of the "ng build --prod", in this case my problem seems to be related to this: angular/angular#20115.
In other words the problem seems to be the presence of ts files in my libraries and aot geting angry about it :), the solution for this point seems to be put ts file in the .npmignore and exclude it from the packages. (haven't test it yet)

I'm just reporting the finding, hopying it helps others....all the credit goes to @filipesilva.

@changLiuUNSW

@changLiuUNSW @tolitech can I ask you to do some more advanced debugging if you have time? I don't have access to your projects but they seem good examples.

@changLiuUNSW Your project doesn't seem to have memory problems anymore but it is taking very long to build with Build Optimizer.

I think one of the build optimizer transformations is taking a very long time on your codebase, but I can't know which ones. I can tell you how to disable each one individually though.

You can do this in ./node_modules/@angular-devkit/build-optimizer/src/build-optimizer/build-optimizer.js. From line 63 to line 82, each transform is individually added to the getTransforms array. You can comment out each of the getTransforms.push( and that transform will not be added.

If you can do this for each one and time me how it impacts build time, I might be able to figure out which of the transforms is taking too long.

@tolitech your project has problems with memory usage with 1.4.x, and you have already confirmed it does not use build optimizer. You reported that memory usage went up to 28gigs during a prod build.

I'd like to ask you to do a couple of things. First, can you tell me the memory usage of just ng build please?

Then, I'd like you to try to manually disable Uglify. This can be done in ./node_modules/@angular/cli/models/webpack-configs/production.js, line 111-117 (the new webpack.optimize.UglifyJsPlugin({ call). Comment those lines, and try to do a production build again (with increased memory limits). What was the maximum memory used?

I understand that debugging these things take time and appreciate that you take the time to do it. I'm doing the same things on projects I have access to but each project is different so it's very difficult to zone in on the exact issue without having access to the code. Asking you to do some tests is the next best thing.

Running ng build --aot --prod yielded the following

Date: 2017-11-17T18:23:24.062Z
Hash: 7dff5d083d5a19785434
Time: 558874ms
chunk {0} polyfills.e8bddc586f67afdffd0d.bundle.js (polyfills) 65.6 kB [initial] [rendered]
chunk {1} main.ba0bc9f17d63868a89a3.bundle.js (main) 2.49 MB [initial] [rendered]
chunk {2} styles.cb450a88292b46f1511b.bundle.css (styles) 81 kB [initial] [rendered]
chunk {3} inline.0002f507274cf434cef2.bundle.js (inline) 1.45 kB [entry] [rendered]

took nearly 10 minutes but the build did complete and didn't exceed the 1.3GB RAM mark now. By the looks of it, the bundle also decreased in size. Before I was getting a main bundle of 1.98 MB and a vendor bundle of 1.51 MB and now I only seem to have a single main bundle of 2.49MB.

The dev bundle went from vendor 16MB and main 4.61MB to vendor 5.91MB and main 1.87MB and it didn't exceed the 800MB RAM mark.

This was generated with the following setup

Angular CLI: 1.5.2
Node: 6.11.0
OS: win32 x64
Angular: 5.0.2
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, router

@angular/cdk: 5.0.0-rc0
@angular/cli: 1.5.2
@angular/flex-layout: 2.0.0-beta.10-4905443
@angular/material: 5.0.0-rc0
@angular-devkit/build-optimizer: 0.0.33
@angular-devkit/core: 0.0.20
@angular-devkit/schematics: 0.0.36
@ngtools/json-schema: 1.1.0
@ngtools/webpack: 1.8.2
@schematics/angular: 0.1.5
typescript: 2.6.1
webpack: 3.8.1

I'll update my node to the latest 8.* version and I'll see if this changes anything in terms of RAM consumption.

@filipesilva

@filipesilva I was looking at the package.json for Angular:
"engines": { "node": ">=6.9.5 <7.0.0", "npm": ">=3.10.7 <4.0.0", "yarn": ">=1.0.2 <2.0.0" },
I've always taken the engines section to mean that those were the engines supported. Is running Node 8 officially supported or am I reading this all wrong?

Well, interestingly, node 8.9.1 did complete the bundle nearly twice as fast (~5min 50 seconds) compared to node 6.11 and it did use about ~150-200MB less ram throughout the final (92%-95%) stages of the process.

Date: 2017-11-17T20:14:47.417Z
Hash: fd0232ebf712617a9dad
Time: 351019ms
chunk {0} polyfills.e8bddc586f67afdffd0d.bundle.js (polyfills) 65.6 kB [initial] [rendered]
chunk {1} main.ba0bc9f17d63868a89a3.bundle.js (main) 2.49 MB [initial] [rendered]
chunk {2} styles.cb450a88292b46f1511b.bundle.css (styles) 81 kB [initial] [rendered]
chunk {3} inline.0002f507274cf434cef2.bundle.js (inline) 1.45 kB [entry] [rendered]

Angular CLI: 1.5.2
Node: 8.9.1
OS: win32 x64
Angular: 5.0.2
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, router

So although the compilation time reduction was very nice, the node version doesn't seem to play a major role in the RAM consumption issue (at least on my machine/project).

EDIT:

Running ng build --aot --prod --build-optimizer=false finished 3 times faster than ng build --aot --prod but it did use about 150-200MB more RAM.

Date: 2017-11-17T20:29:38.921Z
Hash: 94cb3178b3be74bd0397
Time: 126625ms
chunk {0} polyfills.e8bddc586f67afdffd0d.bundle.js (polyfills) 65.9 kB [initial] [rendered]
chunk {1} styles.cb450a88292b46f1511b.bundle.css (styles) 81 kB [initial] [rendered]
chunk {2} main.55c058ed7857dda94b2e.bundle.js (main) 1.8 MB [initial] [rendered]
chunk {3} vendor.ba5bb536a884f0ae6767.bundle.js (vendor) 1.33 MB [initial] [rendered]
chunk {4} inline.39aa19fa644263353f85.bundle.js (inline) 1.45 kB [entry] [rendered]

So all in all build optimizer seems to actually help RAM consumption at the expense of time.

@filipesilva

Hello @filipesilva
I've done some more tests and would like to share it with you.

To be a fair test, I restarted my laptop at all times when I changed a line in the production.js file and after running ng build -prod.

In all tests, the virtual memory added was 32 GB.
The laptop has 8 GB physical RAM.

In the ng.cmd file the configured command was --max_old_space_size = 32768 in all cases.

It is important to remember that in this test, the total memory allocated is not exactly the total memory required by Node.js, since it would be the total used by the computer in all of its processes.

However, there was no program or process running, and in all tests the computer was restarted.

Let's go to the tests:

Test 1
Configuration:

Total memory allocated by the computer: 36.5 GB

Waiting time to complete: 26 min 24 secs

Test 2
Configuration:

Total memory allocated by the computer: 34 GB

Waiting time to complete: 20 min 52 secs

Test 3
Configuration:

Total memory allocated by the computer: 7.9 GB

Waiting time to complete: 10 min 03 secs

Test 4
Configuration:

Total memory allocated by the computer: 8.8 GB

Waiting time to complete: 13 min 59 secs

Node v8.9.1
Angular CLI 1.4.1

By my tests, I have to conclude that in my specific case, it is the new webpack.optimize.ModuleConcatenationPlugin() command line that is dramatically increasing my memory need.

With this line enabled, my consumption goes from 8.8 GB to 36.5 GB as indicated by the windows task manager.

If you need more tests, just give me instructions.
I work more with .NET, but with instructions I believe I can test.

Thank you.

@amikitevich

@tolitech thanks lot for trying that! I think you're on to something.

The commit that introduced ModuleConcatenationPlugin was released in 1.3.0-rc.0 (fe85750).

This matches up with what @amikitevich @randyaa and @radoslavpetranov said: their memory problems started with 1.3.0.

Using ModuleConcatenationPlugin is important for two reasons:

it makes javascript loading in the browser faster (https://medium.com/webpack/webpack-3-official-release-15fd2dd8f07b)
it enables smaller bundles when used with Uglify and Build Optimizer

But it also seems to be causing excessive memory usage in some cases.

I'm a bit at a loss at what I can do about this now. ModuleConcatenationPlugin might use a lot of memory by design or it might just be a bug, but it's hard to tell without being able to test it myself.

To users that started having memory problems after the 1.3x update: can you try doing a build disabling ModuleConcatenationPlugin manually (same instructions as I gave @tolitech in #5618 (comment)) and tell me if it helps? If it does help and your project is public, can you tell me where I can see it?

@tolitech can you tell me which libraries you use in your app? Maybe I can try including them in a project and can reproduce it like that.

running ng build --aot --prod with new webpack.optimize.ModuleConcatenationPlugin() commented out

Date: 2017-11-19T08:34:58.624Z
Hash: 9c0b60b77822f45ea9fd
Time: 158811ms
chunk {0} main.dc03843e6c0ae1671bc2.bundle.js (main) 2.61 MB [initial] [rendered]
chunk {1} polyfills.a0160642e7aef2ebb253.bundle.js (polyfills) 65.6 kB [initial] [rendered]
chunk {2} styles.cb450a88292b46f1511b.bundle.css (styles) 81 kB [initial] [rendered]
chunk {3} inline.4c5cd4d92e0d15ed3e92.bundle.js (inline) 1.45 kB [entry] [rendered]

running ng build --aot --prod with new webpack.optimize.ModuleConcatenationPlugin() activated

Date: 2017-11-19T08:42:05.090Z
Hash: 8e9b1a1739224a7ca0ad
Time: 336440ms
chunk {0} polyfills.e8bddc586f67afdffd0d.bundle.js (polyfills) 65.6 kB [initial] [rendered]
chunk {1} main.4f0ad6b15ea65c6ef0b0.bundle.js (main) 2.49 MB [initial] [rendered]
chunk {2} styles.cb450a88292b46f1511b.bundle.css (styles) 81 kB [initial] [rendered]
chunk {3} inline.9ebe9dea2365b402bb45.bundle.js (inline) 1.45 kB [entry] [rendered]

No difference in RAM consumption on my project.

@filipesilva

@filipesilva My project is public, and I am facing exactly the same issue after I upgrade Angular-CLI a few days ago. You can have a try on 'ng build -prod' via https://github.com/alvachien/achihui

@alvachien

@alvachien I tried your project with 1.5.2 and it ng build --prod built correctly with a peak of 1.2GB memory usage. Did you try 1.5.2? There's a fix there that isn't in 1.5.0.

@markmaynard what you're looking at is the the package.json for the angular/angular repo: https://github.com/angular/angular/blob/master/package.json#L10-L13. That repro is what we call a monorepo, where multiple packages are under the same repository. The engine requirements you see there are for the monorepro itself: the requirements to build the Angular packages. But not the requirements for using the Angular packages. If you check out package.json for @angular/core (in your node_modules for instance), there won't be any engine requirements.

@mattem @WandererInVoids I tried running a few experiments in a project with a lot of libs, to compare memory usage with and without a separate Uglify process:

ng build --prod: peak 1.1GB memory, 4.1MB main bundle
ng build --prod with Uglify manually commented off: peak 0.8GB memory, 10.5MB main bundle
Uglify ran manually on main bundle: peak 0.7GB memory, 4.2MB uglified main bundle

CLI used was 1.5.2. It's important to note that I used the same Uglify we use in the CLI (uglify-es@3.1.5, the one included by uglifyjs-webpack-plugin@1.0.0), and that I also used the same flags we use in the CLI: --compress pure_getters,passes=3 --mangle.

Flags here are really important because these affect the memory usage and end result. See mishoo/UglifyJS#2113 for some conversation about memory usage in Uglify.

From the numbers I got I wouldn't say there is a problem with how we use Uglify. The difference (1.1GB vs 0.8+0.7) is larger than each process individually but is smaller than both together. I expect this to be because of all the files/sourcemaps loaded into memory.

But it's worth mentioning that by not running --prod you are also not running ModuleConcatenationPlugin. Maybe in your projects that's the actual problem and not Uglify. Can you try disabling as well? (see #5618 (comment) on how to disable it).

@filipesilva

Hello @filipesilva
I have tried to create a very similar model of what I use so that the tests can be performed with greater precision.

In this application there are basically 5 main pages (category, status, severity, project and issue) that are replicated 10 times within the same module. We have 50 pages per module. 10 modules totaling 500 pages were created.

For me, a page is composed of 3 components, the entry point, the grid and the form.
So we have 1,500 components created.

There are several other components created and used by the application, base components that are inherited via typescript, pagination, menu, security, shared module, directives, guards, models, resolvers, among other elements.

Not all npm referenced libraries are being used in this example, but I believe the model is quite useful for testing.

In this example, my maximum memory consumption hit 23 GB.
Somewhat below my actual case, because although this example has many more components, the components created in my real case are more complex.

I do not know if I exaggerated the number of elements created for the test.
Although hundreds of services have been created (one for each page), all modules and page replications call the same five Api urls (category, status, severity, project and issue).

Both the API and the angular project are published and working in:
Api: http://clientes.tolitech.com.br/ngTestApi
Api Documentation: http://clientes.tolitech.com.br/ngTestApi/api-docs
Angular: http://clientes.tolitech.com.br/ngtest/

To download the source code:
http://www.codegenerator.com.br/downloads/ng-test.zip

@tolitech whoa this is perfect, thank you so much! I'll use it to run some benchmarks and see what I can find. Will also try and trim it down to do a webpack issue for ModuleConcatenationPlugin so we can get the ball rolling there. This is great!

@filipesilva

@filipesilva due your comment i commented out that lines memory usage decreased from 5.5 gb to 2.5 gb

@WandererInVoids

@WandererInVoids ok, that's a good sign. I'll keep pressing on this line of investigation.

@tolitech using your repo (CLI 1.4.1/ Angular 4.4.6 / TS 2.3.3) :

ng build peaks at 2.2GB ram
ng build --prod peaks at 18.5GB ram

The memory usage per progress % looks like this:

0-11% - 3gb peak (ngtools/webpack)
11%-89% - 5.2gb peak (loading and parsing ~4.3k modules into webpack)
90%-94% - 8.0gb peak (module concatenation, uglify)
95-100% - 18.5gb peak (emitting)

I also tried to upgrade the repro to use CLI 1.5.2/ Angular 5.0.2 / TS 2.4.2.

ng build peaks at 2.3GB ram
ng build --prod peaks at 22.0GB ram

The memory usage per progress % looks like this:

0-11% - 3.3gb peak (ngtools/webpack, build-optimizer)
11%-89% - 4.0gb peak (loading and parsing ~4.3k modules into webpack)
90%-94% - 8.6gb peak (purify, module concatenation, uglify)
95-100% - 22.0gb peak (emitting)

Note: using CLI 1.5 + Angular 5 defaults to using Build Optimizer on prod builds. I also noticed that the 92% phase took a lot longer than before like @changLiuUNSW reported, while memory usage kinda plateaued. It makes me think that PurifyPlugin (part of Build Optimizer) is what takes so long here.

For comparison, I also tried compiling the project using ngc and it peaked at 3.3gb used memory. Comparing that to the 0-11% phase it looks like our compiler plugin doesn't add a significant overhead. tsc itself peaks 830MB used memory.

The project using Angular 5 can be found at https://github.com/filipesilva/angular-cli-test-repo.

Here's a breakdown of the project files by extension:

1551 html
   1 ico
   8 jpg
   5 json
   5 png
  46 scss
2637 ts

Looking at the peak memory usage numbers I can say a couple of things:

Memory usage from compilation (0-11% phase) seems reasonable for project size.
The 11%-89% phase takes a while loading everything but it's also not very surprising.
I expected the 90%-94% phase to take the lion's share of memory usage but that wasn't the case at all.
The 95-100% phase is where memory is being massively used up.

I'm going to follow up with the Webpack folks to see what's actually happening in that last phase.

@tolitech you've been a massive help getting to the bottom of this, I can't thank you enough 💯

I don't get how 3.3 Gb memory for 1500 HTML files is reasonable. That's 2 Mb per file. Do these numbers contain collectable garbage or is this what's actually claimed until the end of the build?

@filipesilva

@filipesilva My setup is like this:
CLI 1.5.2/ Angular 5.0.2 / TS 2.4.2
I use a few 3rd party packages.

ng build --prod : fails with the JavaScript heap out of memory error
ng --env=build --aot : works

I ran into this issue. I was able to get a successful build with
node --max_old_space_size=8192 node_modules/.bin/ng build --prod

@emilio-martinez

To add on @emilio-martinez's answer, on Windows use:
node --max_old_space_size=8192 node_modules/@angular/cli/bin/ng build --prod

ng build --aot --prod
 95% emittingFATAL ERROR: MarkCompactCollector: semi-space copy, fallback in old gen Allocation failed - JavaScript heap out of memory

Angular CLI: 1.5.2
Node: 8.9.1
OS: win32 x64
Angular: 5.0.2
... animations, common, compiler, compiler-cli, core, forms
... http, platform-browser, platform-browser-dynamic
... platform-server, router

@angular/cdk: 5.0.0-rc0
@angular/cli: 1.5.2
@angular/material: 5.0.0-rc0
@angular-devkit/build-optimizer: 0.0.33
@angular-devkit/core: 0.0.20
@angular-devkit/schematics: 0.0.36
@ngtools/json-schema: 1.1.0
@schematics/angular: 0.1.5
typescript: 2.4.2
webpack: 3.8.1

@Tragetaschen

@tolitech FYI I opened a webpack issue for the ModuleConcatenationPlugin memory usage: webpack/webpack#5992.

@Tragetaschen it's not just the HTML files, it's all the TS and code transformations too. The amount of memory used by the plugin (which is developed in this repo) is reasonable insofar as it is not a big overhead, 3 vs 3.3gb, over the ngc compiler itself (which is developed in angular/angular).

@filipesilva

Thanks, @filipesilva

Taking advantage of the moment, I do not know if this would be the best time, but is there any technique for development to be improved?

Example: In a large project like this, a simple change in HTML or TS ends up taking a long time to reflect the onscreen settings. (ng serve -o running).

@tolitech I'd like to talk about the different factors influencing dev rebuilds in big projects but would much prefer to do it in a separate issue. It's something that gets asked about frequently and it would be good to have discussion on it.

Can you open an issue called "Total rebuild time in medium and large projects", share your experience (rebuild time in the console but also time until app is ready in the browser), and ping me there please?

@filipesilva

@filipesilva I have tried commenting out the ModuleConcaternationPlugin and rerun our builds. In short our project still runs into memory issues.

I don't know if this makes a difference, but the way our projects are currently setup we have a number of internal libraries that while in development, use npm link to link.

Memory peaks at 2.51gb real / 7.26 virtual when running with the ModuleConcaternationPlugin commented out.
Full output:

Matts-MacBook-Pro:mediator-ui matt$ node --max_old_space_size=8192 ./node_modules/.bin/ng build --prod --preserve-symlinks
 14% building modules 37/38 modules 1 active ...pace/html/mediator-ui/src/styles.scssTemplate parse warnings:
The <template> element is deprecated. Use <ng-template> instead ("      <span *ngIf="!itemTemplate">{{field ? option[field] : option}}</span>
                        [WARNING ->]<template *ngIf="itemTemplate" [pTemplateWrapper]="itemTemplate" [item]="option"></template>
        "): ng:///Users/matt/Documents/workspace/html/mediator-ui/node_modules/primeng/components/autocomplete/autocomplete.d.ts.AutoComplete.html@22:24                           Date: 2017-11-21T13:58:28.746Z                                                       Hash: 683f2d7486502f8e507a
Time: 252811ms
chunk {0} 0.3a309735936ea414db80.chunk.js (common) 581 kB {20}  [rendered]
chunk {1} 1.8caff8c57661c1307d98.chunk.js () 2.55 kB {20}  [rendered]
chunk {2} 2.b3d1517bf29cec331f56.chunk.js () 179 kB {20}  [rendered]
chunk {3} 3.dd257d4e2f9e4640b2d4.chunk.js () 164 kB {20}  [rendered]
chunk {4} 4.e2edee33bd5ed8a53124.chunk.js () 116 kB {20}  [rendered]
chunk {5} 5.13f271efca121303c5fc.chunk.js () 83.2 kB {20}  [rendered]
chunk {6} 6.08af5ef8c2fe3e09267a.chunk.js () 36.8 kB {20}  [rendered]
chunk {7} 7.8ec810fd30855fef71f7.chunk.js () 30 kB {20}  [rendered]
chunk {8} 8.15f033cf28671ef4988a.chunk.js () 29.5 kB {20}  [rendered]
chunk {9} 9.151ee32b9cc676f2034e.chunk.js () 31.1 kB {20}  [rendered]
chunk {10} 10.cb4d037fcc697b055929.chunk.js () 42.7 kB {20}  [rendered]
chunk {11} 11.f2584b57685d1109a976.chunk.js () 37.5 kB {20}  [rendered]
chunk {12} 12.fe05dccec88da4ae948c.chunk.js () 29.4 kB {20}  [rendered]
chunk {13} 13.2028975a01820ba8496f.chunk.js () 6.1 kB {20}  [rendered]
chunk {14} 14.3be6220b9707e6312847.chunk.js () 3.12 kB {20}  [rendered]
chunk {15} 15.5113d2a9539e99c7ad7f.chunk.js () 6.15 kB {20}  [rendered]
chunk {16} 16.915b0b2f0ed403ce914b.chunk.js () 5.66 kB {20}  [rendered]
chunk {17} 17.485ba9c96f0e3aa081f0.chunk.js () 3.06 kB {20}  [rendered]
chunk {18} 18.db6e52da7740aeca6bb0.chunk.js () 13 kB {20}  [rendered]
chunk {19} polyfills.efb98b65fb45848a1cc3.bundle.js (polyfills) 170 kB {23} [initial] [rendered]
chunk {20} main.4a056bb640e0da824df3.bundle.js (main) 49 kB {22} [initial] [rendered]
chunk {21} styles.e50be95f68f4bf889ab4.bundle.css (styles) 263 kB {23} [initial] [rendered]
chunk {22} vendor.f23a5bbcd6f969d5bec4.bundle.js (vendor) 5.1 MB [initial] [rendered]
chunk {23} inline.35b8b1727b921c4ee6ae.bundle.js (inline) 1.94 kB [entry] [rendered]

I then tried with the UglifyJsPlugin commented out:
Memory peaks at 1.54gb real / 6.35 virtual.

Matts-MacBook-Pro:mediator-ui matt$ ng build --prod --preserve-symlinks
 14% building modules 37/38 modules 1 active ...pace/html/mediator-ui/src/styles.scssTemplate parse warnings:
The <template> element is deprecated. Use <ng-template> instead ("      <span *ngIf="!itemTemplate">{{field ? option[field] : option}}</span>
                        [WARNING ->]<template *ngIf="itemTemplate" [pTemplateWrapper]="itemTemplate" [item]="option"></template>
        "): ng:///Users/matt/Documents/workspace/html/mediator-ui/node_modules/primeng/components/autocomplete/Date: 2017-11-21T12:31:03.567Z                                                       Hash: a93f716c1a9cd5e34227
Time: 205392ms
chunk {0} 0.7f726b90dbfda77c6fd2.chunk.js (common) 1.78 MB {20}  [rendered]
chunk {1} 1.b2415c8c032817feee41.chunk.js () 9.49 kB {20}  [rendered]
chunk {2} 2.5dbd68c68fe419faaac6.chunk.js () 20.2 kB {20}  [rendered]
chunk {3} 3.f65ad52f00741668ecea.chunk.js () 325 kB {20}  [rendered]
chunk {4} 4.c60ce468892788b134f0.chunk.js () 11.2 kB {20}  [rendered]
chunk {5} 5.0e510c127917b57b977e.chunk.js () 112 kB {20}  [rendered]
chunk {6} 6.17a1081f0b32fa32114a.chunk.js () 116 kB {20}  [rendered]
chunk {7} 7.0db6a1bd5ef8cd7389e2.chunk.js () 145 kB {20}  [rendered]
chunk {8} 8.08dea29d9d56c08ac135.chunk.js () 133 kB {20}  [rendered]
chunk {9} 9.6c20abd2e257ccb42f3a.chunk.js () 112 kB {20}  [rendered]
chunk {10} 10.245dd4aebc6e15998e11.chunk.js () 455 kB {20}  [rendered]
chunk {11} 11.3a63d64df6659b591bc7.chunk.js () 91.8 kB {20}  [rendered]
chunk {12} 12.4bdb534a025025ab67c5.chunk.js () 28.8 kB {20}  [rendered]
chunk {13} 13.58f953974b1a1cab76cd.chunk.js () 27 kB {20}  [rendered]
chunk {14} 14.a295fb06cc46ee37d6dd.chunk.js () 18 kB {20}  [rendered]
chunk {15} 15.573a4c4817db899cbaa5.chunk.js () 522 kB {20}  [rendered]
chunk {16} 16.1bef7d4db5d387785c53.chunk.js () 209 kB {20}  [rendered]
chunk {17} 17.2892448d1217c3964316.chunk.js () 10.2 kB {20}  [rendered]
chunk {18} 18.8a52fcf63d5fb567be17.chunk.js () 80.8 kB {20}  [rendered]
chunk {19} polyfills.efb98b65fb45848a1cc3.bundle.js (polyfills) 497 kB {23} [initial] [rendered]
chunk {20} main.1b1138cc3dcdfe7392ca.bundle.js (main) 174 kB {22} [initial] [rendered]
chunk {21} styles.e50be95f68f4bf889ab4.bundle.css (styles) 263 kB {23} [initial] [rendered]
chunk {22} vendor.2bca18ae1189b0ecdba1.bundle.js (vendor) 14.4 MB [initial] [rendered]
chunk {23} inline.8e041fa7d36f79764f2f.bundle.js (inline) 6.41 kB [entry] [rendered]

The build completed without the need to add additional memory to Node.

I also tried running with node --inspect and attaching the Chrome debugger to grab some heap snapshots and memory profiles, but it in my tests that caused the CLI to hang during the compilation stage with 1 module remaining.

The other part that is hammering us is the time taken by the compile for tests, we have had to continually increase timeouts in the Karma conf, and the time taken for a page refresh with tests (during development) is quite high, mainly due to that ~15mb vendor bundle.

@filipesilva

@filipesilva I get that there is "stuff" that's generated, that's why I asked if these are clean numbers or if remaining garbage from the actual processing is still contained in that number.
If that's the actual number of bytes claimed, then I still don't get it: Assuming 2Kb per HTML file on average(!) (which is still much more than I would expect - I have 1 kb on average) then each single byte from those files (spaces, angular brackets, text characters, etc.) will generate 1000 bytes of transformed / generated content. I just cannot believe that's the case.

@Tragetaschen

@Tragetaschen the numbers I got were just looking at the task manager and keeping track of what the maximum used memory was. I'm not sure why you are focusing on the HTML templates though. ngc does more than just compile templates.

@mattem

@mattem if you're already reaching 1.54GB without Uglify/ModuleConcaternationPlugin, then it looks like your project as is needs more than the default node max memory. At how much memory does your project peak when doing just ng build?

@mattem

@mattem can you open an issue called "Total rebuild test time in medium and large projects", similar to the one I asked @tolitech to open in #5618 (comment)? It's a broader topic as well.

@Tragetaschen

@Tragetaschen I measured tsc peak as well and it's 830MB. Bear in mind however that ngc both compiles templates, transforms code, and generate new TS files. So total TS files processed would be around double. I pinged the team about the difference between ngc and tsc.

@filipesilva

@filipesilva Running ng build uses 1.31gb real / 5.99 virtual peak.

Here is the full output from the command

Matts-MBP:mediator-ui matt$ ng build
Date: 2017-11-21T18:59:28.538Z                                                            
Hash: 5c68c24623b719d5f1b0
Time: 100214ms
chunk {brand-registration-task.module} brand-registration-task.module.chunk.js, brand-registration-task.module.chunk.js.map () 17.4 kB {main}  [rendered]
chunk {common} common.chunk.js, common.chunk.js.map (common) 562 kB {main}  [rendered]
chunk {custom-report-task-module} custom-report-task-module.chunk.js, custom-report-task-module.chunk.js.map () 7.4 kB {main}  [rendered]
chunk {episode-registration-task.module} episode-registration-task.module.chunk.js, episode-registration-task.module.chunk.js.map () 17.5 kB {main}  [rendered]
chunk {feature-not-available-view.module} feature-not-available-view.module.chunk.js, feature-not-available-view.module.chunk.js.map () 4.32 kB {main}  [rendered]
chunk {home-view.module} home-view.module.chunk.js, home-view.module.chunk.js.map () 5.46 kB {main}  [rendered]
chunk {inline} inline.bundle.js, inline.bundle.js.map (inline) 5.83 kB [entry] [rendered]
chunk {job-management-dashboard-task-module} job-management-dashboard-task-module.chunk.js, job-management-dashboard-task-module.chunk.js.map () 8.61 kB {main}  [rendered]
chunk {main} main.bundle.js, main.bundle.js.map (main) 17 MB {vendor} [initial] [rendered]
chunk {material-registration-task.module} material-registration-task.module.chunk.js, material-registration-task.module.chunk.js.map () 18.4 kB {main}  [rendered]
chunk {package-editor-task-module} package-editor-task-module.chunk.js, package-editor-task-module.chunk.js.map () 18.3 kB {main}  [rendered]
chunk {placing-registration-task.module} placing-registration-task.module.chunk.js, placing-registration-task.module.chunk.js.map () 18.5 kB {main}  [rendered]
chunk {playlist-task.module} playlist-task.module.chunk.js, playlist-task.module.chunk.js.map () 23.4 kB {main}  [rendered]
chunk {polyfills} polyfills.bundle.js, polyfills.bundle.js.map (polyfills) 534 kB {inline} [initial] [rendered]
chunk {series-registration-task.module} series-registration-task.module.chunk.js, series-registration-task.module.chunk.js.map () 17.5 kB {main}  [rendered]
chunk {session.module} session.module.chunk.js, session.module.chunk.js.map () 18.4 kB {main}  [rendered]
chunk {spot-check-task-module} spot-check-task-module.chunk.js, spot-check-task-module.chunk.js.map () 6.3 kB {main}  [rendered]
chunk {styles} styles.bundle.js, styles.bundle.js.map (styles) 318 kB {inline} [initial] [rendered]
chunk {tasks.module} tasks.module.chunk.js, tasks.module.chunk.js.map () 26.2 kB {main}  [rendered]
chunk {timeline-task-module} timeline-task-module.chunk.js, timeline-task-module.chunk.js.map () 52.3 kB {main}  [rendered]
chunk {user-settings-view.module} user-settings-view.module.chunk.js, user-settings-view.module.chunk.js.map () 31.8 kB {main}  [rendered]
chunk {vendor} vendor.bundle.js, vendor.bundle.js.map (vendor) 5.07 MB [initial] [rendered]
chunk {views.module} views.module.chunk.js, views.module.chunk.js.map () 91.1 kB {main}  [rendered]
chunk {workflow-dashboard-task-module} workflow-dashboard-task-module.chunk.js, workflow-dashboard-task-module.chunk.js.map () 1.76 MB {main}  [rendered]

We don't run into memory issues (yet, I guess) when running development builds, we are generally experiencing the same as what @tolitech has already mentioned. (I'm keeping an keen eye on the progress of the Bazel building and the work being done there, happy to be a guinea pig for any of that)

@filipesilva

@filipesilva Just to be sure: It's not that I care about the HTML compilation step per se, it's that you claimed 3.3 GiB memory consumption just for HTML and NgModule processing "reasonable for project size". Using a thousand times the memory compared to the input data size while the resulting bundle (even without any further optimizations) is over an order of magnitude smaller just isn't reasonable.

Sure, there are bigger problems when it comes to memory consumption right now and I wish I could me more (well, any kind of) constructive, but the entire compilation process currently takes one or two orders of magnitude (!) more memory than I would call "reasonable".

@Tragetaschen

@Tragetaschen I think Felipe is commenting on your math. That step doesn't just involve the HTML file but includes the typescript and CSS files too. In order to properly uglify your entire project is brought into memory, so you can't just count the HTML. Additionally, some of your logic may cause duplication of subsets of your Dom, exploding the size. This is a complicated task that will take a good amount of RAM. We should use as little as possible, but we can't expect miracles.

@filipesilva

@filipesilva Opened issue #8579 for test rebuild times

@filipesilva

I am also having the same issue. I can share my code to you @filipesilva

My --prod build works but slow. When I added --stats-json it ran out of memory. I got it working by doing this:

"build:prod:myapp:stats": "node --max_old_space_size=4096 node_modules/@angular/cli/bin/ng build --prod --app=myapp --stats-json"

fix(@angular/cli): update webpack, uglifyjs-webpack-plugin

@changLiuUNSW

We're updating the Webpack version used in the CLI in #8689 to include the fix for excessive memory usage in ModuleConcatenationPlugin (webpack/webpack#5997).

This will be available in the next release of Angular CLI and should address most of the reports of excessive memory usage in 1.3.x and up.

I'd like to reiterate that it is expected that projects will reach the memory limit after a certain size. The bigger your project is, the more memory it will need to be built especially in --prod.

For those cases the recommendation is that you run the CLI with an increased memory limit via a npm script:

    "ng-high-mem": "node --max_old_space_size=8192 ./node_modules/@angular/cli/bin/ng"

Then you can do npm run ng-high-mem -- build --prod to do a production build. Note the double dash (--), it is used in npm scripts to separate args.

In this script the limit is increased to 8gigs. You can use a bigger number if you need.

Meanwhile we will keep on making improvements to reduce the memory footprint of the Angular CLI build system.

@changLiuUNSW I've opened a separate issue to track the excessive build time increase when using --build-optimizer: angular/devkit#305.

@filipesilva

Thanks for taking the time to look into this in such detail @filipesilva! Much appreciated!

The problem is, this is my webpack.config:

`const path = require('path');
const webpack = require('webpack');
const AngularCompilerPlugin = require('@ngtools/webpack').AngularCompilerPlugin;
const merge = require('webpack-merge');
const CheckerPlugin = require('awesome-typescript-loader').CheckerPlugin;

module.exports = (env) => {
// Configuration in common to both client-side and server-side bundles
//const isDevBuild = !(env && env.prod);
const isDevBuild = true;
const sharedConfig = {
stats: { modules: false },
context: __dirname,
resolve: { extensions: [ '.js', '.ts' ] },
output: {
filename: '[name].js',
publicPath: 'dist/' // Webpack dev middleware, if enabled, handles requests for this URL prefix
},
module: {
rules: [
{ test: /.ts$/, include: /ClientApp/, use: isDevBuild ? ['awesome-typescript-loader?silent=true', 'angular2-template-loader'] : '@ngtools/webpack' },
{ test: /.html$/, use: 'html-loader?minimize=false' },
{ test: /.css$/, use: [ 'to-string-loader', isDevBuild ? 'css-loader' : 'css-loader?minimize' ] },
{ test: /.(png|jpg|jpeg|gif|svg)$/, use: 'url-loader?limit=25000' }
]
},
plugins: [new CheckerPlugin()]
};

// Configuration for client-side bundle suitable for running in browsers
const clientBundleOutputDir = './wwwroot/dist';
const clientBundleConfig = merge(sharedConfig, {
    entry: { 'main-client': './ClientApp/boot.browser.ts' },
    output: { path: path.join(__dirname, clientBundleOutputDir) },
    plugins: [
        new webpack.DllReferencePlugin({
            context: __dirname,
            manifest: require('./wwwroot/dist/vendor-manifest.json')
        })
    ].concat(isDevBuild ? [
        // Plugins that apply in development builds only
        new webpack.SourceMapDevToolPlugin({
            filename: '[file].map', // Remove this line if you prefer inline source maps
            moduleFilenameTemplate: path.relative(clientBundleOutputDir, '[resourcePath]') // Point sourcemap entries to the original file locations on disk
        })
    ] : [
        // Plugins that apply in production builds only
        new webpack.optimize.UglifyJsPlugin(),
        new AngularCompilerPlugin({
            tsConfigPath: './tsconfig.json',
            entryModule: path.join(__dirname, 'ClientApp/app/app.module.browser#AppModule'),
            exclude: ['./**/*.server.ts']
        })
    ])
});

// Configuration for server-side (prerendering) bundle suitable for running in Node
const serverBundleConfig = merge(sharedConfig, {
    resolve: { mainFields: ['main'] },
    entry: { 'main-server': './ClientApp/boot.server.ts' },
    plugins: [
        new webpack.DllReferencePlugin({
            context: __dirname,
            manifest: require('./ClientApp/dist/vendor-manifest.json'),
            sourceType: 'commonjs2',
            name: './vendor'
        })
    ].concat(isDevBuild ? [] : [
        // Plugins that apply in production builds only
        new AngularCompilerPlugin({
            tsConfigPath: './tsconfig.json',
            entryModule: path.join(__dirname, 'ClientApp/app/app.module.server#AppModule'),
            exclude: ['./**/*.browser.ts']
        })
    ]),
    output: {
        libraryTarget: 'commonjs',
        path: path.join(__dirname, './ClientApp/dist')
    },
    target: 'node',
    devtool: 'inline-source-map'
});

return [clientBundleConfig, serverBundleConfig];

};
`

So, I am not using ModuleConcatenationPlugin and I still get the same error. That is why I offered my code

@dobrinsky

@dobrinsky how many TS files do you have in your project? Do you know what is peak memory usage?

I have 240. Will get up to 400 probably until I finish the project.

No, I do not know. I am a beginner...I do not know how to find that out

I usually run the build process, and then just look at the task manager to see how much memory the process is using. Peak memory usage is just the biggest number I saw for memory usage while the process was running.

:)))) I didn't knew it was that simple. Thought I need some software or something...

Anyway, the strangest thing happened:

I started with:

Then Visual Studio started using memory in the build process:

Then of course, Node started using too:

But my peak was 68%, with just 1.67 GB of memory usage

So, this is definitely something else

Upgrading to Angular-CLI 1.5.5, Node to 8.9.1 and Yarn to 1.3.2 Fixed that issue for me.
And running Yarn did the fix better than npm.

I am not using Yarn. I have Node 9.2, but Angular-CLI 1.5.4.

I will try with 1.5.5

@filipesilva

Thanks @filipesilva for your post above re: "ng-high-mem". I confirmed with my project that this does work and I no longer run out of memory during prod build with aot and build-optimizer on. Without this I ran out of memory.

package.json scripts:
"ng-high-mem": "node --max_old_space_size=8192 ./node_modules/@angular/cli/bin/ng"
Now this works:
npm run ng-high-mem -- build --prod --aot=true --build-optimizer=true

Note in your msg above you wrote:
"Then you can do npm run ng-high-mem -- --prod to do a production build."
But I think you missed "build" in there:
npm run ng-high-mem -- build --prod

@filipesilva

Hi @filipesilva,

In our project we are having the out of memory issue with below version
@angular/cli: 1.2.0
@angular: 4.2.5

We are near production and we expect solution for this. Due to some constraints we couldn't able to migrate to angular 5 for now.

Is there any solution will be available for this?

@dobrinsky

@dobrinsky have you tried running your webpack config with node --max_old_space_size=8192 to increase the memory limit? What is the maximum you see then? If it's just above the maximum you saw before it might just be that you need that little extra bit of memory.

@JasonPaape thanks, I added it now!

@Jayaraj8905 at this point your best bet is to either use that script (ng-high-mem) or update. 1.2.0 was a long time ago and we don't publish fixes for it anymore.

Using ng-high-mem option with @angular/cli 1.5.4 solved it for me.
Using @angular/cli 1.5.5, I saw 30-40% less memory usage, but still memory outage, so still ng-high-mem required for my build.

Now using ng-high-mem (with 8192) it is using a lot more memory. Is it using more when it has more memory available?

Please bear in mind that the webpack update is not in yet. It will be in either 1.5.6 or 1.6.0.

Hopefully that will also speed up the production build a lot. I just migrated from the old Angular 2 way (ngc aot + rollup) to angular/cli and the build time grow from 4m to about 12m. :-(

fix(@angular/cli): update webpack, uglifyjs-webpack-plugin

I'm seeing this since the migration to angular 5 + angular-cli 1.5.5 on my CI build machine.
Thanks for the help of appveyor team I was able to use the --max_old_space_size on the node cmd file.
Relevant info can be found in the following discussion (which links here):
https://help.appveyor.com/discussions/problems/10192-out-of-memory-when-using-angular-cli-15-angular-5
And in this issue in my repository.
IsraelHikingMap/Site#567

fix(@angular/cli): update webpack, uglifyjs-webpack-plugin

I have a same issue with @angular/cli@1.5.5

ng build -prod
 95% emitting                                                                             
<--- Last few GCs --->

  113209 ms: Mark-sweep 853.8 (1439.3) -> 857.6 (1439.3) MB, 271.2 / 0.0 ms [allocation failure] [scavenge might not succeed].
  113472 ms: Mark-sweep 857.6 (1439.3) -> 861.5 (1439.3) MB, 263.2 / 0.0 ms [allocation failure] [scavenge might not succeed].
  113756 ms: Mark-sweep 861.5 (1439.3) -> 876.5 (1423.3) MB, 284.4 / 0.0 ms [last resort gc].
  114039 ms: Mark-sweep 876.5 (1423.3) -> 891.6 (1423.3) MB, 282.5 / 0.0 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x31d99d6cfb51 <JS Object>
    1: DoJoin(aka DoJoin) [native array.js:~129] [pc=0xc7b98c68639] (this=0x31d99d604381 <undefined>,w=0x23bc0f2ffd31 <JS Array[590]>,x=590,N=0x31d99d6043c1 <true>,J=0x31d99d6ae4d1 <String[1]:  >,I=0x31d99d6b46e1 <JS Function ConvertToString (SharedFunctionInfo 0x31d99d652dc9)>)
    2: Join(aka Join) [native array.js:180] [pc=0xc7b9864f2b2] (this=0x31d99d604381 <undefined>,w=0x23bc0f2ffd31 <JS ...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [ng]
 2: 0x10983ec [ng]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [ng]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [ng]
 5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [ng]
 6: v8::internal::Runtime_StringBuilderJoin(int, v8::internal::Object**, v8::internal::Isolate*) [ng]
 7: 0xc7b939092a7
Aborted (core dumped)

adding to this line
"build-prod": "node --max_old_space_size=8192 ./node_modules/@angular/cli/bin/ng build --prod",
to package.json helped me.

@filipesilva

@filipesilva I saw a new version of webpack has been release days ago which ships your performance tuning code, when will angular-cli release a new version refer to the latest version of webpack? Thank you.

@alvachien

@alvachien In next version of Angular CLI, 1.5.6 or 1.6.0, is what he said just a little up in this thread.

Judging by latest releases I would assume that this could happen this week unless there is something in particular blocking it.

@kisdaniel

@kisdaniel 's solution/workaround seems to work fine. its sorted my issues

Increasing --max_old_space_size works for me, but after an hour or so, my RAM (32Gb) usage hits 70% or 80%, incremental builds slow down, and node or ng serve crashes.

@wolfhoundjesse

@wolfhoundjesse are you using AOT on rebuilds?

Is there any ETA on release with aforementioned deps update? Fighting with this issue in our CI which doesn't have much RAM for now, so this fix is urgently required (I know I can hack package.json to point directly at master, but this is a hack I want to avoid).

It should be in this weeks release, since we usually release once per week.

@krassx

@krassx for now, try @kisdaniel 's fix, works fine for an 8gb ram machine

@mast3rd3mon

@mast3rd3mon, It doesn't help that much as we have 4GB RAM on our build machine. Will try to play with the limit to make it work.

@krassx

@krassx maybe try the same command but for 4gb ram? so 4096 not 8192

Tried that. And got that out of memory error. It does work on my local machine (macOS, 8GM RAM) but fails on build machine (Linux, 4GB).

@krassx

@krassx try disable build optimizer and source map, can u share yours build command?

node --max_old_space_size=8192 ./node_modules/@angular/cli/bin/ng build --target=production --aot --output-hashing=bundles --build-optimizer=true

This is what I use locally with no errors. But it fails on CI machine as I noted before.

@krassx

@krassx try this for production u can just use prod
node --max_old_space_size=4096 --stack_size=512 ./node_modules/@angular/cli/bin/ng build --prod -vc --build-optimizer=false
refer to this page for more option https://github.com/angular/angular-cli/wiki/build

Ok, will give it a try. Thank you.

@WandererInVoids

@WandererInVoids, disabling build optimizer did the trick. Thank you for the advice.

@krassx

@krassx your welcome

fix(@angular/cli): update webpack, uglifyjs-webpack-plugin

@angular/cli@1.6.0-rc.2 is now released and includes the memory fix on Webpack's side.

@filipesilva

@filipesilva I'm using node --max_old_space_size=16384 node_modules/.bin/ng serve --aot. I'll upgrade to rc.2 now.

@filipesilva

@filipesilva I am sorry , I am traveling and missed your question to me. To reply, yes, my problems started after updating to 1.5.0 and plain ng build --prod failed with any 1.5.x. I had to build, as many others, with raised memory node --max-old-space-size=8192 node_modules\@angular\cli\bin\ng build --prod.

Having said that at the same time as I updated to 1.5 I included 3rd party library that probably sent the build process over the edge.

And now the good news. I have just updated to 1.6 and the build finishes just fine with plain old ng build --prod.

Thanks.

I can confirm this issue, it looks a serious one.

It happens only when AOT is enabled.

So it will work fine if I disable AOT and run: ng build --prod --aot false

Angular CLI: 1.5.5
Node: 8.9.1
OS: linux x64
Angular: 5.0.5
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, router

@angular/cli: 1.5.5
@angular-devkit/build-optimizer: 0.0.34
@angular-devkit/core: 0.0.22
@angular-devkit/schematics: 0.0.39
@ngtools/json-schema: 1.1.0
@ngtools/webpack: 1.8.5
@schematics/angular: 0.1.9
typescript: 2.4.2
webpack: 3.8.1

1.6.0 seems to have fixed it for us. Our app compiled without extending the memory.

@elclanrs

@elclanrs not here, (1.6.0)

apparently, upgrading to latest cli(1.6)/angular(5.1)/typescript(2.5) solved the issue

I upgraded to cli 1.6 and angular 5.1 and now our app could build under 1.7 gigs

@filipesilva

@filipesilva a bit late, but I finally got around to testing ag-Grid with 1.6.0 and it all builds fine now.

Thanks for your hard work on this

@filipesilva

@filipesilva I tried the 1.6.0, it seems my project work fine, thank you.

@filipesilva

@filipesilva don`t work for me but build time and memory usage decreased

Now it’s time to switch to Bazel/Rollup

I upgraded to cli 1.6 and angular 5.1.
The max memory the production build now using is around 1.3 GB. So no heap errors anymore.

Strange thing is that the process is 'hanging' at '92% chunk asset optimization' for 7 minutes of the total of 9 minutes buildtime.
What is happening there?

@Ristaaf

@Ristaaf thank you very much, your solution works perfect

We still have this issue. We had this issue since angular@2 when I first time used angular CLI. Then I switched to angular starter and everything worked. Now Im trying to use angular cli again on the same project (upgrade from angular 4) and now I still have same issue with angular 5.1 and cli 1.6.2.

We have a really huge project and we are using angular on this project from early versions of angular since 2013.

When I simply use ng build build is going 1 hour. Here is a log.

When I use ng build I always have an error (after few minutes of build):

ng build
 10% building modules 3/3 modules 0 active                                         
<--- Last few GCs --->

[30628:0x102804600]   151498 ms: Mark-sweep 1405.8 (1664.5) -> 1405.8 (1664.5) MB, 1634.9 / 0.0 ms  allocation failure GC in old space requested
[30628:0x102804600]   152767 ms: Mark-sweep 1405.8 (1664.5) -> 1405.7 (1616.5) MB, 1268.5 / 0.0 ms  last resort 
[30628:0x102804600]   154066 ms: Mark-sweep 1405.7 (1616.5) -> 1405.7 (1601.0) MB, 1298.4 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x14c9d8b1cef1 <JSObject>
    1: set [native collection.js:~247] [pc=0x2fa4affd9607](this=0x166929284d21 <Map map = 0x20d66a1183b9>,p=0xb60e511a3c9 <NodeObject map = 0x78bae661b01>,x=0x3ce8d86d6ce9 <Object map = 0x3647e82a689>)
    3: record [/Users/pleerock/www/yakdu/ng5.yakdu.dev/node_modules/@angular/compiler-cli/src/transformers/node_emitter.js:135] [bytecode=0x214b10b7f429 offset=37](this=0x166929284ce9 <_NodeEmitte...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/usr/local/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/usr/local/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node]
 4: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
 5: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [/usr/local/bin/node]
 6: v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node]
 7: 0x2fa4afe040dd
Abort trap: 6

If you want to know how many code and files we have:

cloc ./src/
    2705 text files.
    2680 unique files.                                          
https://github.com/AlDanial/cloc v 1.66  T=12.20 s (219.4 files/s, 17125.4 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
TypeScript                    2662          21207          15921         143472
CSS                              4           4084            203          18682
SASS                             6            772            305           4144
JSON                             3              0              0             68
HTML                             1              8              5             25
-------------------------------------------------------------------------------
SUM:                          2676          26071          16434         166391
-------------------------------------------------------------------------------

@pleerock

@pleerock have you tried using the command to increase max memory, on both prod and dev builds?

If your project is near to the maximum memory it will need a lot of GC and that slows down things. Increasing the max memory even when you're not hitting it (like in dev) should help.

@filipesilva

@filipesilva thanks, it works. Both prod and dev builds. And build time is still 1 hour. Im glad that it works this way, however having to wait 1 hour each time is a problem. Also, its a big problem to run ng serve. When we run it, it takes same 1 hour to start serving which makes development process completely impossible.

@pleerock

@pleerock yeah 1h for ng serve is really odd. I have never seen such a large build time in any projects I've handled.

I have reports of slow prod builds of that length (#6795) but haven't heard of anything like that just for ng serve... I use https://github.com/filipesilva/angular-cli-test-repo as a benchmark for larger projects and it has around 2400 TS files, but still only takes a few minutes.

Is your repro public or is there a way that I can see it?

Another option would be to take a CPU profile (there's some instructions here #8259). But I wouldn't be surprised if the devtools crash, they usually do on huge jobs. Maybe a CPU profile of just a few minutes would be feasible?

Is your repro public or is there a way that I can see it?

no, its not public. If you can we can make a call and I can show you and maybe we can use teamviewer or similar things.

BTW, 76% basic chunk optimization step takes most of time (99%). If I run ng serve and change a file it takes same 1 hour to rebuild everything.

CPU profile of first 10 minutes of 76% basic chunk optimization process (ng serve):

CPU-20180103T195000.cpuprofile.zip

Navigation Menu

Repository navigation

AOT build fails due to "JavaScript heap out of memory" #5618

Bug Report or Feature Request (mark with an `x`)

Versions.

Repro steps.

The log given by the failure.

Desired functionality.

Mention any other details that might be useful.

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

AOT build fails due to "JavaScript heap out of memory" #5618

Description

Bug Report or Feature Request (mark with an x)

Versions.

Repro steps.

The log given by the failure.

Desired functionality.

Mention any other details that might be useful.

Activity

JonathanYates commented on Nov 14, 2017

seanlandsman commented on Nov 14, 2017

JonathanYates commented on Nov 14, 2017

MichalEmbiq commented on Nov 14, 2017

joecot commented on Nov 14, 2017

josephliccini commented on Nov 14, 2017

filipesilva commented on Nov 14, 2017

hanvyj commented on Nov 14, 2017

magemello commented on Nov 14, 2017

filipesilva commented on Nov 14, 2017

filipesilva commented on Nov 14, 2017

randyaa commented on Nov 14, 2017

magemello commented on Nov 14, 2017

seanlandsman commented on Nov 14, 2017

rajravat commented on Nov 14, 2017

filipesilva commented on Nov 14, 2017

seanlandsman commented on Nov 14, 2017

Dunos commented on Nov 14, 2017

dzonatan commented on Nov 14, 2017

tim-white87 commented on Nov 14, 2017

alvachien commented on Nov 15, 2017

JackArlington commented on Nov 15, 2017

magemello commented on Nov 15, 2017

filipesilva commented on Nov 15, 2017

radoslavpetranov commented on Nov 15, 2017

filipesilva commented on Nov 15, 2017

erlloyd commented on Nov 15, 2017

mattem commented on Nov 15, 2017

eduardotolino commented on Nov 15, 2017

filipesilva commented on Nov 15, 2017

eduardotolino commented on Nov 15, 2017

WandererInVoids commented on Nov 15, 2017

mattem commented on Nov 16, 2017

PeS82 commented on Nov 17, 2017

jayhamilton commented on Nov 17, 2017

finalxcode commented on Nov 17, 2017

filipesilva commented on Nov 17, 2017

finalxcode commented on Nov 17, 2017

JackArlington commented on Nov 17, 2017

Shadowlauch commented on Nov 17, 2017

JackArlington commented on Nov 17, 2017

filipesilva commented on Nov 17, 2017

Shadowlauch commented on Nov 17, 2017

seanlandsman commented on Nov 17, 2017

changLiuUNSW commented on Nov 17, 2017

magemello commented on Nov 17, 2017

jinder commented on Nov 17, 2017

Dunos commented on Nov 17, 2017

filipesilva commented on Nov 17, 2017

changLiuUNSW commented on Nov 17, 2017

filipesilva commented on Nov 17, 2017

changLiuUNSW commented on Nov 17, 2017

magemello commented on Nov 17, 2017

filipesilva commented on Nov 17, 2017

radoslavpetranov commented on Nov 17, 2017

markmaynard commented on Nov 17, 2017

radoslavpetranov commented on Nov 17, 2017

eduardotolino commented on Nov 18, 2017

filipesilva commented on Nov 18, 2017

filipesilva commented on Nov 18, 2017

radoslavpetranov commented on Nov 19, 2017

alvachien commented on Nov 20, 2017

filipesilva commented on Nov 20, 2017

eduardotolino commented on Nov 20, 2017

filipesilva commented on Nov 20, 2017

WandererInVoids commented on Nov 20, 2017

filipesilva commented on Nov 20, 2017

Tragetaschen commented on Nov 20, 2017

kdubious commented on Nov 20, 2017

emilio-martinez commented on Nov 21, 2017

AndreaBarbasso commented on Nov 21, 2017

Bug Report or Feature Request (mark with an `x`)