Skip to content

rootDirs should merge outputsΒ #44321

Open

Description

Suggestion

πŸ” Search Terms

rootDirs merge output

Related: #9875

βœ… Viability Checklist

My suggestion meets these guidelines:

  • This wouldn't be a breaking change in existing TypeScript/JavaScript code
  • This wouldn't change the runtime behavior of existing JavaScript code
  • This could be implemented without emitting different JS based on the types of the expressions
  • This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, new syntax sugar for JS, etc.)
  • This feature would agree with the rest of TypeScript's Design Goals.

⭐ Suggestion

rootDirs should merge outputs instead of emitting to the folders listed in rootDirs. This would be consistent with rootDir.

πŸ“ƒ Motivating Example

Starting point: single root

Let's say I start with a project with a src/ folder:

.
└── src
    β”œβ”€β”€ index.ts
    β”œβ”€β”€ lib
    β”‚   └── lib.ts
    └── test
        └── test.ts

It has one rootDir and this config:

{
  "compilerOptions": {
    "outDir": "./",
    "rootDir": "./src",
  },
  "include": ["src/**/*.ts"],
  "exclude": []
}

When I compile, the output goes to ./ and mirrors the structure of ./src:

.
β”œβ”€β”€ lib
β”‚   └── lib.js
β”œβ”€β”€ src
β”‚   β”œβ”€β”€ index.ts
β”‚   β”œβ”€β”€ lib
β”‚   β”‚   └── lib.ts
β”‚   └── test
β”‚       └── test.ts
└── test
    └── test.js

All well and good.

Friction point: add generated files

Now I want to add some generated .ts files to my project and keep them outside of src/ for easier source control.

I put a .graphql source file in my src/ tree, and set up the generator to output to gen/:

.
β”œβ”€β”€ gen
β”‚   └── lib
β”‚       └── schema.ts
└── src
    β”œβ”€β”€ index.ts
    β”œβ”€β”€ lib
    β”‚   β”œβ”€β”€ lib.ts
    β”‚   └── schema.graphql
    └── test
        └── test.ts

Now I want the same output structure as before, but I want to compile the sources in gen/ and output them as if I had "rootDir": "./gen".

The first thing I try is to replace "rootDir": "./src" with "rootDirs": ["./src", "./gen"]:

{
  "compilerOptions": {
    "outDir": "./",
    "rootDirs": ["./src", "./gen"],
  },
  "include": ["src/**/*.ts", "gen/**/*.ts"],
  "exclude": []
}

but this unfortunately defaults rootDir to ./, putting src/ and /gen in the output paths and writing .js files as siblings to their .ts sources:

.
β”œβ”€β”€ gen
β”‚   └── lib
β”‚       β”œβ”€β”€ schema.js
β”‚       └── schema.ts
└── src
    β”œβ”€β”€ index.js
    β”œβ”€β”€ index.ts
    β”œβ”€β”€ lib
    β”‚   β”œβ”€β”€ lib.js
    β”‚   β”œβ”€β”€ lib.ts
    β”‚   └── schema.graphql
    └── test
        β”œβ”€β”€ test.ts
        └── test.ts

This is very much not what I want or expect, and a big difference from the behavior of rootDir.

What I expect is that rootDirs acts like a multi-value rootDir, and for each input the root dir that it's in is stripped from the output path.

One additionally confusing aspect of rootDir vs rootDirs is that "rootDir": "./src" is not equivalent to "rootDirs": ["./src"]. The former will cause output files to be siblings to inputs.

It looks like to work around this we need to set "outDir": "./build", and then add an additional build step just to copy the files from ./build/src and ./build/gen into ./. This obviously can work but it's more of a complication than just a simple step: --watch will no longer work correctly unless you wire up the copy step to automatically run when ./build files change. For a project with simple build scripts, this is a pretty big leap in complexity.

Starting package.json:

  "scripts": {
    "build": "tsc --build"
  },

Desired addition of a generator:

  "scripts": {
    "build": "npm run build:graphql && npm run build:ts",
    "build:ts": "tsc --build",
    "build:graphql": "graphql-codegen"
  },

Actual addition of a generator, with broken --watch:

  "scripts": {
    "build": "npm run build:graphql && npm run build:ts && npm run build:copy",
    "build:ts": "tsc --build",
    "build:graphql": "graphql-codegen",
    "build:copy": "cp -r build/{src,gen}/*"
  },

Actual addition of a generator, with working --watch:

πŸ€·β€β™‚οΈ I have to go figure this out still. Also, I'm not sure if this will badly interact with a monorepo, composite projects, and --build. The cross-package import paths are going to be different from the compiler output.

Potential workaround

I tried setting "rootDir": "./(src|gen)", hoping to tell tsc to strip those paths from the output. No luck as that path isn't interpreted as a pattern. That could potentially be a non-breaking way to add merging.

πŸ’» Use Cases

Compiling generated files into a common output tree.

Activity

  1. justinfagnani commented on May 28, 2021

    @justinfagnani
    Author

    Also, I'm not sure if this will badly interact with a monorepo, composite projects, and --build. The cross-package import paths are going to be different from the compiler output.

    Well, so far it does look like adding the copy steps interferes with composite projects.

    To test I made a change to a shared project that causes a break in a project that references it while I was running tsc --build --watch in the dependent project.

    When I make the change to the source file, tsc recompiles, but with no errors. (Because the files actually imported by the dependent project are the copies and they haven't changed yet?) If I run the copy step the imported files do change, but the .tsbuildinfo file hasn't, so tsc doesn't see a change still. If I then kill the watch build and run tsc --build -f the error is picked up.

    edit: More complications with a monorepo / composite setup. On a clean build, when a dependent project builds it'll run tsc on the referenced project, but not invoke the copy step, so the imports in the dependent project fail to resolve.

    It seems like rootDirs and composite projects just don't mix? It'd be great to see a working example as requested in #37257 I think things would just work if rootDirs merged the roots as requested here.

  2. RyanCavanaugh commented on Jun 3, 2021

    @RyanCavanaugh
    Member

    Stating the obvious, probably: We can't just change this and break everyone's existing builds, so there'd have to be some new opt-in

    Less obvious, perhaps: This is tricky. We'd start to have to deal with what to do with conflicting output names. It's also unclear how relative-path module specifiers should work in this world -- does importing "./foo" mean we'd scan every directory at the same relative point in the hierarchy, or would you only be able to import relative filenames from the input-side path despite there being a larger set of paths that would actually work at runtime?

  3. justinfagnani commented on Jun 3, 2021

    @justinfagnani
    Author

    Yeah, changing rootDirs would be a breaking change, I was assuming there'd need to be an opt-in.

    I think there are good answers to the questions you raise:

    • Conflicting output names should be disallowed
    • Relative imports should resolve within the merged folder hierarchy

    This is how other src/gen overlay systems I've used work. It makes imports behave as if the generated files were generated directly into the source folders, which has the essential benefit of abstracting away whether a file is checked-in or generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Awaiting More FeedbackThis means we'd like to hear from more people who would be helped by this featureSuggestionAn idea for TypeScript

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions