Files created right before watching starts make watching go into a loop

I've noticed this strange behaviour where watcher loops for 10 seconds due to this code:

if(data) {
  var ts = data[0] === data[1] ? data[0] + FS_ACCURENCY : data[0];
  if(ts > startTime)
    watcher.emit("change", data[1] + FS_ACCURENCY);
}

watchpack/lib/DirectoryWatcher.js

Lines 202 to 205 in 7efdd93

    
           if(data) { 
        
           	var ts = data[0] === data[1] ? data[0] + FS_ACCURENCY : data[0]; 
        
           	if(ts > startTime) 
        
           		watcher.emit("change", data[1] + FS_ACCURENCY);

What I'm doing in my script is basically creating a file in a temporary directory /temp/entry.js, creating a webpack compiler and run compiler.watch(). Now for 10 seconds (which is the default value of FS_ACCURENCY) this is what's happening:

compilation succeeds
Watching.prototype._done gets called
recalling Watching.prototype.watch
which immediately triggers a change event for the /temp/entry.js file, because it's < 10 seconds old
back to step 1

Any pointers on how to fix this would be helpful. For example, this could be fixed by modifying the above code to this, but I'd need to stare at this for longer to understand what this block of code is conceptually responsible for.

if(ts > startTime + FS_ACCURENCY) {

FS_ACCURENCY should automatically adjust to your file system accuracy. This may not happen fast enough if you have few files and the files are created unlucky on a timestamp modulo 10s.

The watching may loop in a unlucky case, but this should not result in a different compilation hash. I. e. the webpack-dev-server doesn't trigger a update if the hash is equal.

Ah, but Im experiencing this issue 100% of the time, because I am
programatically creating the entry.js file and starting the watch process
on it right away. So for 10s it keeps rebuilding continuously.
On Fri, 6 May 2016 at 13:12, Tobias Koppers notifications@github.com
wrote:

FS_ACCURENCY should automatically adjust
https://github.com/webpack/watchpack/blob/master/lib/DirectoryWatcher.js#L230-L239
to your file system accuracy. This may not happen fast enough if you have
few files and the files are created unlucky on a timestamp modulo 10s.

The watching may loop in a unlucky case, but this should not result in a
different compilation hash. I. e. the webpack-dev-server doesn't trigger a
update if the hash is equal.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#25 (comment)

Would be nice to resolve this without having to fork webpack or delay
watching for 10s. I wonder if I could create a file with an older
timestamp..
On Fri, 6 May 2016 at 18:29, Karolis Narkevičius karolis.n@gmail.com
wrote:

Ah, but Im experiencing this issue 100% of the time, because I am
programatically creating the entry.js file and starting the watch process
on it right away. So for 10s it keeps rebuilding continuously.
On Fri, 6 May 2016 at 13:12, Tobias Koppers notifications@github.com
wrote:

FS_ACCURENCY should automatically adjust
https://github.com/webpack/watchpack/blob/master/lib/DirectoryWatcher.js#L230-L239
to your file system accuracy. This may not happen fast enough if you have
few files and the files are created unlucky on a timestamp modulo 10s.

The watching may loop in a unlucky case, but this should not result in a
different compilation hash. I. e. the webpack-dev-server doesn't trigger a
update if the hash is equal.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#25 (comment)

@sokra

@sokra "FS_ACCURENCY" only adjusts itself when you get an actual onChange event. When you first fire compiler.watch(), and the compilation is fast enough (say, one file with no bundled dependencies), assuming you saved the file in the last few seconds, watchpack will keep emitting change events until the difference gets large enough.

When chokidar is using fswatch (OSX), FS_ACCURENCY eventually settles on 2000 if you trigger some real change events (not sure why, fsevent timestamps are multiples of 1000).

Maybe FS_ACCURENCY should get a first adjustment on the nextTick handler; at least that would reduce the worst case to 2s as opposed to 10s. If the build is really fast (say, ~400ms per build), you can easily get 20+ rebuilds at the start if you touched any file before starting the watch process.

It'd also be nice if it got renamed to FS_PRECISION, but that's going off-topic :)

We are having the same problem. A workaround we found is to manually change the mtime of the file before it gets picked up by Webpack, but it feels like just adding a hack on top of a hack.

This problem has been haunting our team for a while. Files are being stored on doInitialScan() 10 seconds in the future and this generates multiple webpack-dev-middleware builds until the future becomes the past. It seems to me that watchpack should compare mtimes between nextTick()s instead of trying to guess if the mtime is in the future or in the past.

Watchpacks strategy is: if not sure if it's a change, thread it as change. With low fs accuracy files could have changed even if mime is equal.

Normally multiple rebuilds shouldn't cause issues and this cause should be cached by the caller by comparing hash. I.e. the webpack CLI doesn't display stats output for equal hash.

Nevertheless this may could be fixed in watch pack by storing the time of the initial scan/watcher installation and useing this time.

How is FS_ACCURENCY helping in that case? If the filesystem doesn't provide the precision necessary to detect the change, FS_ACCURENCY will only add X milliseconds to that lack of precision. If mtime didn't change because of lack of precision, adding FS_ACCURENCY to it won't make it any different. The only reason I can imagine is when trying to guess if the mtime is in the future or not, and what I propose is to stop doing that and just act upon the change in mtime itself.

FYI, here's an extract of what we see if we have a recent change before starting webpack-dev-middleware. I put a loop printing the current date in the background and added some console.logs for debugging purposes showing arguments for the compiler plugins. Note that the hash remains the same and the multiple builds happening until startTime is greater than mtime + FS_ACCURENCY:

Tue 18 Oct 2016 11:33:20 EDT
webpack built a6a4935a84d4fc7fddd3 in 3823ms
Hash: a6a4935a84d4fc7fddd3
Version: webpack 2.1.0-beta.22
Time: 3823ms
                               Asset     Size  Chunks             Chunk Names
13ed6d2a0f1addfda9c8530374554c10.png    12 kB          [emitted]
007abab9d8f8ae99042650e9311bdc82.png  16.1 kB          [emitted]
7f86a225ce5b3f8879ef567e8ef37743.png  14.2 kB          [emitted]
           0.a6a4935a84d4fc7fddd3.js  14.5 kB       0  [emitted]
           1.a6a4935a84d4fc7fddd3.js  23.1 kB       1  [emitted]
         app.a6a4935a84d4fc7fddd3.js  1.46 MB       2  [emitted]  app
                         favicon.ico  7.41 kB          [emitted]
                       vendor.dll.js  3.92 MB          [emitted]
                          index.html  6.61 kB          [emitted]
Child html-webpack-plugin for "index.html":
                                   Asset     Size  Chunks             Chunk Names
    13ed6d2a0f1addfda9c8530374554c10.png    12 kB          [emitted]
                              index.html  1.46 MB       0
webpack: bundle is now VALID.
Tue 18 Oct 2016 11:33:21 EDT
invalid
{ '0': '/Users/fcarasso/Dev/console-proto/src/main.js',
  '1': 1476804803000 }
webpack: bundle is now INVALID.
invalidAsync
{ '0':
   Watching {
     startTime: 1476804802113,
     invalid: false,
     error: null,
     stats:
      Stats {
        compilation: [Object],
        hash: 'a6a4935a84d4fc7fddd3',
        startTime: 1476804797495,
        endTime: 1476804801318 },
     handler: [Function],
     watchOptions: { aggregateTimeout: 200 },
     compiler:
      Compiler {
        _plugins: [Object],
        outputPath: '/Users/fcarasso/Dev/console-proto/dist',
        outputFileSystem: [Object],
        inputFileSystem: [Object],
        recordsInputPath: undefined,
        recordsOutputPath: undefined,
        records: [Object],
        fileTimestamps: [Object],
        contextTimestamps: [Object],
        resolvers: [Object],
        parser: [Object],
        options: [Object],
        context: '/Users/fcarasso/Dev/console-proto',
        name: 'client',
        watchFileSystem: [Object],
        _lastCompilationFileDependencies: [Object],
        _lastCompilationContextDependencies: [] },
     running: true,
     watcher: null },
  '1': [Function: next] }
invalid
{}
webpack building...
webpack built a6a4935a84d4fc7fddd3 in 227ms
Hash: a6a4935a84d4fc7fddd3
Version: webpack 2.1.0-beta.22
Time: 227ms
                               Asset     Size  Chunks       Chunk Names
13ed6d2a0f1addfda9c8530374554c10.png    12 kB
007abab9d8f8ae99042650e9311bdc82.png  16.1 kB
7f86a225ce5b3f8879ef567e8ef37743.png  14.2 kB
           0.a6a4935a84d4fc7fddd3.js  14.5 kB       0
           1.a6a4935a84d4fc7fddd3.js  23.1 kB       1
         app.a6a4935a84d4fc7fddd3.js  1.46 MB       2       app
Child html-webpack-plugin for "index.html":
                                   Asset     Size  Chunks       Chunk Names
    13ed6d2a0f1addfda9c8530374554c10.png    12 kB
                              index.html  1.46 MB       0
webpack: bundle is now VALID.
invalid
{ '0': '/Users/fcarasso/Dev/console-proto/src/main.js',
  '1': 1476804803000 }
webpack: bundle is now INVALID.
invalidAsync
{ '0':
   Watching {
     startTime: 1476804802571,
     invalid: false,
     error: null,
     stats:
      Stats {
        compilation: [Object],
        hash: 'a6a4935a84d4fc7fddd3',
        startTime: 1476804802113,
        endTime: 1476804802340 },
     handler: [Function],
     watchOptions: { aggregateTimeout: 200 },
     compiler:
      Compiler {
        _plugins: [Object],
        outputPath: '/Users/fcarasso/Dev/console-proto/dist',
        outputFileSystem: [Object],
        inputFileSystem: [Object],
        recordsInputPath: undefined,
        recordsOutputPath: undefined,
        records: [Object],
        fileTimestamps: [Object],
        contextTimestamps: [Object],
        resolvers: [Object],
        parser: [Object],
        options: [Object],
        context: '/Users/fcarasso/Dev/console-proto',
        name: 'client',
        watchFileSystem: [Object],
        _lastCompilationFileDependencies: [Object],
        _lastCompilationContextDependencies: [] },
     running: true,
     watcher: null },
  '1': [Function: next] }
invalid
{}
webpack building...
Tue 18 Oct 2016 11:33:22 EDT
webpack built a6a4935a84d4fc7fddd3 in 220ms
Hash: a6a4935a84d4fc7fddd3
Version: webpack 2.1.0-beta.22
Time: 220ms
                               Asset     Size  Chunks       Chunk Names
13ed6d2a0f1addfda9c8530374554c10.png    12 kB
007abab9d8f8ae99042650e9311bdc82.png  16.1 kB
7f86a225ce5b3f8879ef567e8ef37743.png  14.2 kB
           0.a6a4935a84d4fc7fddd3.js  14.5 kB       0
           1.a6a4935a84d4fc7fddd3.js  23.1 kB       1
         app.a6a4935a84d4fc7fddd3.js  1.46 MB       2       app
Child html-webpack-plugin for "index.html":
                                   Asset     Size  Chunks       Chunk Names
    13ed6d2a0f1addfda9c8530374554c10.png    12 kB
                              index.html  1.46 MB       0
webpack: bundle is now VALID.
invalid
{ '0': '/Users/fcarasso/Dev/console-proto/src/main.js',
  '1': 1476804803000 }
webpack: bundle is now INVALID.
invalidAsync
{ '0':
   Watching {
     startTime: 1476804803016,
     invalid: false,
     error: null,
     stats:
      Stats {
        compilation: [Object],
        hash: 'a6a4935a84d4fc7fddd3',
        startTime: 1476804802571,
        endTime: 1476804802791 },
     handler: [Function],
     watchOptions: { aggregateTimeout: 200 },
     compiler:
      Compiler {
        _plugins: [Object],
        outputPath: '/Users/fcarasso/Dev/console-proto/dist',
        outputFileSystem: [Object],
        inputFileSystem: [Object],
        recordsInputPath: undefined,
        recordsOutputPath: undefined,
        records: [Object],
        fileTimestamps: [Object],
        contextTimestamps: [Object],
        resolvers: [Object],
        parser: [Object],
        options: [Object],
        context: '/Users/fcarasso/Dev/console-proto',
        name: 'client',
        watchFileSystem: [Object],
        _lastCompilationFileDependencies: [Object],
        _lastCompilationContextDependencies: [] },
     running: true,
     watcher: null },
  '1': [Function: next] }
invalid
{}
webpack building...
webpack built a6a4935a84d4fc7fddd3 in 207ms
Hash: a6a4935a84d4fc7fddd3
Version: webpack 2.1.0-beta.22
Time: 207ms
                               Asset     Size  Chunks       Chunk Names
13ed6d2a0f1addfda9c8530374554c10.png    12 kB
007abab9d8f8ae99042650e9311bdc82.png  16.1 kB
7f86a225ce5b3f8879ef567e8ef37743.png  14.2 kB
           0.a6a4935a84d4fc7fddd3.js  14.5 kB       0
           1.a6a4935a84d4fc7fddd3.js  23.1 kB       1
         app.a6a4935a84d4fc7fddd3.js  1.46 MB       2       app
Child html-webpack-plugin for "index.html":
                                   Asset     Size  Chunks       Chunk Names
    13ed6d2a0f1addfda9c8530374554c10.png    12 kB
                              index.html  1.46 MB       0
webpack: bundle is now VALID.
Tue 18 Oct 2016 11:33:23 EDT
Tue 18 Oct 2016 11:33:24 EDT
Tue 18 Oct 2016 11:33:25 EDT
^C

We are experiencing the endless rebuilding problem even for files that weren't created shortly before watching started. This is a problem that has affected us with Webpack 1 as well as 2, and occurs sporadically but often enough that it makes working with webpack-dev-server a chore and it almost makes one long for the days of grunt-watch and BrowserSync. ;) Here is a link to our config:
https://gist.github.com/mischkl/cf7ab4b10df9763aa698a5b885f700cd

This is on Windows 7 Enterprise x64 with Node 6.9.1, and seems to primarily happen when IntelliJ IDEA is open.

I should also note that we have seen the problem when using angular-cli with the default settings as well. The fact that it only seems to affect a subset of users makes me strongly suspect that this is a Windows/Intellij/WebStorm-related issue.

For anyone else tearing out hair over this, backdating the relevant files by ten seconds is an awful hack but it worked for me:

var f = path.resolve('someFile')
var now = Date.now() / 1000
var then = now - 10
fs.utimes(f, then, then, function (err) { if (err) throw err })

Update - This fix stopped working for me at some point, but Yassky's solution further down the page works great.

@AndyHall

You just made my day @AndyHall

I confirm that his solution works like a charm!

fix(watch): Add workaround for the webpack watch issue

multiple watches on start is out of my hands has to do with webpack/w…

Same here, check #53. My explanation why this happens is very detailed.

Watchpacks strategy is: if not sure if it's a change, thread it as change. With low fs accuracy files could have changed even if mime is equal.

Normally multiple rebuilds shouldn't cause issues and this cause should be cached by the caller by comparing hash. I.e. the webpack CLI doesn't display stats output for equal hash.

Nevertheless this may could be fixed in watch pack by storing the time of the initial scan/watcher installation and useing this time.

I am looking forward to the official solution to this problem.

fix($core): know issue: webpack/watchpack#25

old issue

problem is kind of expected behavior

we reduced the startup filesystem accuracy to 1 second which should reduce the problem

Given the number of people complaining, "expected behavior" seems inappropriate as a classification.

Why not just get rid of FS_ACCURACY?

I still have this issue and this cannot be the expected behavior...

I'm having this issue too and the suggested workarounds don't work. Can you reopen this?

Same issue here. The only fix for me was to add the plugin: https://github.com/yessky/webpack-mild-compile

i'm experiencing this bug, but with webpack 5. i've try using webpack-mild-compile but no hope. any idea how to resolve this issue?

i'm experiencing this bug, but with webpack 5. i've try using webpack-mild-compile but no hope. any idea how to resolve this issue?
Could you provide reproduceable demo project? I'd try to fix it recently
Try webpack-mild-compile@3.4.0

having this issue right now, the solutions don't seem to work (that is if I'm doing them correctly)

Having the same issue with webpack 5

@fenomas

Popping in here to say that I have a similar problem because I need to manipulate a file Webpack is watching before the watch compile occurs. The only solution that has worked for me is to combine the solution proposed by @fenomas with the information I found in webpack/webpack#7997 by @jods4. By changing the mtime of the file in the invalid sync hook, the loop seems to be prevented.

Edit for context:

I'm generating my watched file first in the environment hook to catch the initial compile and non-watch related compiles
I then rerun the generation scripts in the invalid hook
I then update the mtime in the invalid hook immediately after

plugins: [
  {
    apply: (compiler) => {
      compiler.hooks.environment.tap('environment', () => {
        // ... Manipulate watched file
      })
      compiler.hooks.invalid.tap('invalid', (fileName, changeTime) => {
        // ... Manipulate watched file again
        // Update the mtime
        var my_manipulated_file = path.resolve('path/to/my/manipulated/file')
        var now = Date.now() / 1000
        var then = now - 100
        fs.utimes(my_manipulated_file, then, then, function (err) { if (err) throw err })
    },
  },
]

Hi. Lost a workday because of this. Please, consider fixing it. Thanks.

Its again happening in Angular 16 related. Kindly fix this permanently. thank you.

Navigation Menu

Repository navigation

Files created right before watching starts make watching go into a loop #25

43 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

	if(data) {
	var ts = data[0] === data[1] ? data[0] + FS_ACCURENCY : data[0];
	if(ts > startTime)
	watcher.emit("change", data[1] + FS_ACCURENCY);

Files created right before watching starts make watching go into a loop #25

Description

Activity

sokra commented on May 6, 2016

KidkArolis commented on May 6, 2016

KidkArolis commented on May 6, 2016

Jessidhia commented on Jul 14, 2016

rochdev commented on Oct 18, 2016

hrimhari commented on Oct 18, 2016

sokra commented on Oct 19, 2016

hrimhari commented on Oct 19, 2016

hrimhari commented on Oct 19, 2016

mischkl commented on Nov 22, 2016

fenomas commented on Mar 20, 2017

atinux commented on Mar 25, 2017

trsh commented on Jul 25, 2017

43 remaining items

yuchonghua commented on Jan 6, 2019

sokra commented on Jul 10, 2019

hrimhari commented on Jul 10, 2019

cliedelt commented on Jul 19, 2019

turnerhayes commented on May 30, 2020

imagina commented on Nov 27, 2020

afazzdev commented on Jun 25, 2021

yessky commented on Jul 5, 2021

hedonicadapter commented on Sep 2, 2021

bardsol commented on Dec 1, 2021

JoshBeveridge commented on Jul 27, 2022

dashcode commented on Aug 29, 2023

oshilaRightS commented on Sep 7, 2023

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions