.mjs extension trade-offs revision

I obviously can't speak for the whole community, but it seems like a lot of people are not happy with .mjs.

One of the main arguments to keep .js is that if we can detect 99% of cases where we CAN tell if is it CJS or ESM (or where we just know what to do), we may just call rest 1% edge cases and deal with it.

We can even come up with some linter rules and/or workarounds to simply teach people to do the right thing.

I want to start off with the 99% fallacy here. We are talking about module graphs. If 1% of your graph is wrong it can affect 100% of your graph. Let us keep that in mind.

Lets also try and enumerate where things differ between CJS, Script, and ESM whenever we talk about the problematic situations.

Lets also try and enumerate use cases where those situation might occur whenever we accept or dismiss them as being valid or invalid.

Let's start with simple stuff.
I think one of the most common patterns in node is to require() a bunch of core modules and export either one object with some API, or just export a plain function:

// Full CJS
const fs = require('fs');
const path = require('path');

const api = function(...params) {
  // ... 
};

module.exports = {
  api: api
};

Which in ESM I guess translates to something like:

// Full ESM
import fs from 'fs';
import path from 'path';

const api = function(...params) {
  // ... 
};

export {
  api: api
};

Are there any problems so far in that particular example?

Parsing (focus on when ESM is not a parse error but others are):

Source	Script	CJS	ESM
`var arguments`	global	local	parse error
`var eval`	global	local	parse error
`import("x")`	ok	ok	ok
`import "x";`	parse error	parse error	ok
`export {};`	parse error	parse error	ok
`with({}){}`	ok	ok	parse error
`<!--\n`	ok	ok	parse error
`-->\n`	ok	ok	parse error
`0777`	ok	ok	parse error
`delete x`	ok	ok	parse error
`try {} catch (eval) {}`	ok	ok	parse error
`try {} catch (arguments) {}`	ok	ok	parse error
`(function (_, _) {})`	ok	ok	parse error
`eval = eval`	ok	ok	parse error
`arguments = []`	ok	ok	parse error
`implements`	ok	ok	parse error
`interface`	ok	ok	parse error
`let`	ok	ok	parse error
`package`	ok	ok	parse error
`private`	ok	ok	parse error
`protected`	ok	ok	parse error
`public`	ok	ok	parse error
`static`	ok	ok	parse error
`yield`	ok	ok	parse error
`return`	parse error	ok	parse error
`await`	ok	ok	parse error

Eval differences (overlap parses and runs, but diff results)

Source	Script	CJS	ESM
`this`	global	module	undefined
`var x`	global	local	local
`(function (){return this}())`	global	global	undefined
`(function (x) {x = 1; return arguments[0];})()`	1	1	undefined
`(function () {return typeof this;}).call(1)`	"object"	"object"	"number"
`var x = 0; eval('var x = 1'); x`	1	1	0
`__filename`	global	local	global
`__dirname`	global	local	global
`require`	global	local	global
`exports`	global	local	global
`module`	global	local	global
`arguments`	global	local	global

Timing differences (not a problem, listed for posterity)

Source	Timing	Hoisted	Blocking
`require('foo');`	sync	no	yes
`import "foo";`	untimed (async generally)	yes	yes
`import('foo');`	async	no	no

Potential relevant use cases w/o ESM specific declarations:

In all cases where only import declarations are used import() may be used instead. This list really is about things that do not export values.

Mutation of Globals
- Polyfills
Usage of Globals
- Configuration of globals (source-map-support, some config scripts)
- JSONP-like calling of globals
Prefetching (wants to be unordered parallel so uses import())
Init scripts (CLI etc.)
REPL like text (either for local testing or things like vm.runInContext)
- Anything really that uses the completion value of the text

Potential relevant use cases that may combine source texts

Relevant to when import or export declarations may be added/removed.

Optimizers / Build Scripts
- Inlining of dependencies
- Concatenation of files
Editing a file
- Copy/Paste from internet

Potential relevant use cases w/o `package.json`

One off Scripts
- Bash-like utils
- Education situations
Configuration scripts
- Shared .rc style files
Movable scripts (npm bin scripts have similar but not same)
Serverless-like single file situations
Eval from CLI (STDIN, "-e", "-p", REPL)
- including node myapp.js

Potential tooling that lacks `package.json` capabilities

Scripts without JSON libraries
- nvm in particular
OS-like file associations
Code editors

Potential relevant use cases w/o a file extension

Eval from CLI (STDIN, "-e", "-p", REPL)
- excluding node myapp.mjs

Potential tooling that lacks file extension capabilities

None known yet.

Mandates

Resolution

Resolution in ESM is URL based and should be 100% compatible in non-error cases with the web specification. This is known to have minor differences with CJS.

Forward Path

A path for CJS to being using ESM modules must exist.

It must be possible for CJS files to import() ESM.
- Since this is available in the Script goal, this is automatically fulfilled.

Backwards Path

A path for ESM to be created that uses legacy or CJS files must exist.

It must be possible to import CJS files.
- Some things such as Node core are unable to be upgraded to ESM due to design or usage differences. Other Modules such as meow use things that are not present/do not translate into ESM like module.parent (due to parallel loading, idempotency requirement, etc.).
- Some things like "deep" linking is in use and necessary to support like "lodash/chunk"
There must be an upgrade path for ESM to be safe against CJS becoming ESM.
- Generally achieved by ensuring CJS files are a facade with only a single default export
Mixed mode situations (both ESM and CJS in same app/package) must be supported. Note: again, there may be cases where code can never be updated to ESM.

ESM only future

It should be possible for the ecosystem to move to be ESM only for newly written code.

Whatever path is taken, it should be considered debt if files are still easily or accidentally able to be CJS.
Whatever path is taken, it should be ready for the so called "3rd goal" problem.
- Relevant to WASM that is looking to use same stuff as JS
- Could be relevant with other things that have been proposed like a "use pure" situation if syntax or semantic changes are required.

@YurySolovyov

@YurySolovyov can you clarify what you mean by "Are there any problems so far in that particular example?" I mean, they do act similar but aren't the same.

I mainly meant that they don't introduce ambiguity, right?
Even if we'll have to parse them >1 times, we can with 100% certainty tell which is which, right?

I am not asking in this issue if ESM can fulfill a use case, I'm asking about use cases where ambiguity is possible / exists. Take for example a simple prefetching script:

import('./something-for-later-1')
import('./something-for-later-2')
import('./something-for-later-3')
import('./something-for-later-4')

Such text is either a Script, CJS, or a Module, but could rely on something pretty easily that causes it to cease functioning:

import(`${__dirname}/something-for-later-1`)

@YurySolovyov

@YurySolovyov for

const fs = require('fs');
const path = require('path');

const api = function(...params) {
  // ... 
};

module.exports = {
  api: api
};

We can make a good guess that it is CJS if we parse for require() but not all CJS uses require(). We need to define what mechanism are you using to perform these guesses.

It also would need to ensure no local variable named require exists.

Also in theory that CJS can parse and eval just fine in Module.

Will the absence of import/export be a better sign of CJS than trying to parse requires ?

If import and export declarations do not exist, it could be either ESM, [Script], or CJS. It is better to say import and export declarations (not import()) signifies something that currently is only able to parse in ESM.

Detecting CJS really won't work due to amount of overlap in that direction, detecting [something that is only] ESM is possible though.

So we can't just always start with ESM and then fallback to CJS if that failed?

Looking at all these parse errors in the table above, I don't think many of them are useful or just make a lot of sense in general.

I have a question though about let, yield, return and await: is this about top-level ones or what?
I can't imagine a module system where you can't use these in function's body.

No, default needs to be CJS since that is what existing backwards compat needs.

I have a question though about let, yield, return and await: is this about top-level ones or what?
I can't imagine a module system where you can't use these in function's body.

return works in CJS at top level.

let, yield, await, etc. are reserved words in ESM but not in the other goals.

Looking at all these parse errors in the table above, I don't think many of them are useful or just make a lot of sense in general.

Indeed! thats why https://github.com/bmeck/UnambiguousJavaScriptGrammar went to TC39!

Eval errors are much more insidious. Too many chats going on right now for me to complete.

To note as well: the guess cannot change over time. Once you ship the guessing mechanism it will stay backwards compat (so if something guesses CJS, it will always guess CJS even 10 years from now)

Again, I must be able to import CJS - otherwise users of my module will be forced to deal with a breaking change were I to migrate to ESM, which means I won't migrate to ESM - and neither will many module authors. That will kill the future we all want where ESM is the only module format.

I do not wish to endorse a specific path forward, but perhaps I can clarify a point where I see parties talking past one another.

These two claims seem to keep coming up, without a resolution.

Whatever node does, it should allow a package author to upgrade to ESM with breaking consumers.
With ".mjs", an author can't simply transpile because consumers still have to know to get the default property of the package exports.

Making it possible for a consumer not to know the format of a module is difficult. It is unreasonable to expect this without some work from authors. ESM distinguishes between the default export and the module namespace while CJS does not.

The following pattern requires a little bit of work, but address the issue for some cases.

index.js

// Notice that the default export is lifted here, for CJS consumers.
// This is taking care of interopRequireDefault for non-babel, CJS consumers.
module.exports = require('./lib/index.js').default;

index.mjs

export * from './lib/index.mjs';

This approach does require that the author ensure that the default export of ./lib is its namespace, otherwise the results are different.

The situation is a little trickier when consumers want to do deep imports/requires. My intuition is that there is a pattern that would work for this, but would need to thinking further to see exactly how.

Well said. For deep imports, it'd be the same thing; for each entry point, include a .js file that requires and extracts .default - or for maximum backwards compat, author in ES5 CJS and manually create a .mjs file for each entry point.

Again, I must be able to import CJS - otherwise users of my module will be forced to deal with a breaking change were I to migrate to ESM, which means I won't migrate to ESM - and neither will many module authors. That will kill the future we all want where ESM is the only module format.

Right now CJS cannot take the shape of all ESM when imported thats why efforts are going on with tooling about how to fully support this. There is some push back because it would encourage people to just ship CJS+pragma in some people's eyes.

With ".mjs", an author can't simply transpile because consumers still have to know to get the default property of the package exports.

This isn't related to .mjs but to the fact that import is allowed to load CJS. There are similar effect to JSON files and C++ addons as well. I think this relates to my previous comment, but your solution seems workable.

I would like to see better first class support for generating Module Namespaces so that people can implement various module types land as we get user feedback from the loader hooks, as this would allow more rich integration with other module systems. That will take time though.

loader,docs,test: set named exports based on keys from `module.exports`

Is this --esm flag ever going to happen before somebody will just fork node again?

BTW: I am currently using reify as a workaround but .mjs is wrong decision in the long term.

Given the following: index.js, es6moduleA.js
If I symlink es6moduleA.js as es6moduleA.mjs, then
index.js can successfully import "es6moduleA.mjs"
(nodejs --harmony --experimental-modules --preserve-symlinks ) and es6moduleA.js is still usable by browsers.

But as soon as es6moduleA.js itself import 'es6moduleB.js', you're stuck:

Either it's imported using the .js extension (code works with browsers, file is served application/javascript by Apache, Nginx, php built-in webserver, ... ...)
Either it's renamed and imported using .mjs and javascript-code start to be node.js-only because most tooling is not compliant with this new extension and because JS-engines must not consider ES6 modules served using a wrong mimetype

Currently, a brand-new purely ES6 project, following the language syntax (using *.js file extension and working in most browsers) would not only just fail under node.js but there is currently no command-line parameter to make it work.

Extensions are not part of the language; there’s nothing “correct” about using .js except for what it conveys by convention: that it’s a JavaScript Script.

Browsers ignore extensions entirely; the point of them is to inform the server how to parse the file (to tell it what kind of file it is, to inform the mime type etc).

Following this reasoning servers are in charge of dealing with extensions -> mimetype (w.r.t to IETF)
but they do not support mjs (and from the PoV I guess there is a few reason to support it).
Still node.js, which is not a server does not even offer an option to be extension-agnostic.

Given that there’s nothing “incorrect” in not using .js, a simple --esm is a must have so that even script.jpeg could be interpreted as an ES6 module.

They don't support it yet; at one time, long ago, they didn't support .js either. As soon as node ships .mjs unflagged, all the servers that want to stay relevant will ship support for it.

node certainly could take any extension - or even an extensionless file - and choose to parse it however it wants (including obeying a command-line argument). However, that wouldn't be very useful, and it would almost certainly hide bugs and mistakes.

Minor note that several MIME DBs (and so servers like express by proxy) have the updated MIME that has gone through IETF with the new extension. Also the goal parameter has been added to further disambiguate .js which now has 3 well known meanings for IANA.

…

On Jan 27, 2018 11:48 AM, "Jordan Harband" ***@***.***> wrote: They don't support it *yet*; at one time, long ago, they didn't support .js either. As soon as node ships .mjs unflagged, all the servers that want to stay relevant will ship support for it. node certainly *could* take any extension - or even an extensionless file - and choose to parse it however it wants (including obeying a command-line argument). However, that wouldn't be very useful, and it would almost certainly hide bugs and mistakes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#57 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOUo0UUV0Ouv0vmtuCLzZkocTgE_93Kks5tO2FdgaJpZM4NYUQA> .

Nevermind, I just discovered the (much awaited) PR nodejs/node#18392

just on a lighter aesthetic aspect, I'd have liked .es

@caub

@caub Folks are already using mjs, IMO if Node chooses to use a new extension, it's far too late to try proposing a change.

@tbranyen

@tbranyen I personally don't know anybody who does, they use reify/esm

Navigation Menu

Repository navigation

.mjs extension trade-offs revision #57

Parsing (focus on when ESM is not a parse error but others are):

Eval differences (overlap parses and runs, but diff results)

Timing differences (not a problem, listed for posterity)

Potential relevant use cases w/o ESM specific declarations:

Potential relevant use cases that may combine source texts

Potential relevant use cases w/o `package.json`

Potential tooling that lacks `package.json` capabilities

Potential relevant use cases w/o a file extension

Potential tooling that lacks file extension capabilities

Mandates

Resolution

Forward Path

Backwards Path

ESM only future

224 remaining items

index.js

index.mjs

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

.mjs extension trade-offs revision #57

Description

Activity

bmeck commented on May 11, 2017

YurySolovyov commented on May 11, 2017

bmeck commented on May 11, 2017

Parsing (focus on when ESM is not a parse error but others are):

Eval differences (overlap parses and runs, but diff results)

Timing differences (not a problem, listed for posterity)

Potential relevant use cases w/o ESM specific declarations:

Potential relevant use cases that may combine source texts

Potential relevant use cases w/o package.json

Potential tooling that lacks package.json capabilities

Potential relevant use cases w/o a file extension

Potential tooling that lacks file extension capabilities

Mandates

Resolution

Forward Path

Backwards Path

ESM only future

bmeck commented on May 11, 2017

YurySolovyov commented on May 11, 2017

bmeck commented on May 11, 2017

bmeck commented on May 11, 2017

bmeck commented on May 11, 2017

bmeck commented on May 11, 2017

YurySolovyov commented on May 11, 2017

bmeck commented on May 11, 2017

bmeck commented on May 11, 2017

YurySolovyov commented on May 11, 2017

bmeck commented on May 11, 2017

bmeck commented on May 11, 2017

224 remaining items

ljharb commented on Oct 15, 2017

tilgovi commented on Oct 16, 2017

index.js

index.mjs

ljharb commented on Oct 16, 2017

bmeck commented on Oct 16, 2017

cztomsik commented on Dec 12, 2017

drzraf commented on Jan 27, 2018

ljharb commented on Jan 27, 2018

drzraf commented on Jan 27, 2018

ljharb commented on Jan 27, 2018

bmeck commented on Jan 27, 2018

drzraf commented on Jan 29, 2018

caub commented on May 13, 2018

tbranyen commented on May 14, 2018

cztomsik commented on May 15, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Potential relevant use cases w/o `package.json`

Potential tooling that lacks `package.json` capabilities