secretlinter and jscpd run ordrer is not respected

[lvjp]

Describe the bug
Sporadically, when jscpd and secretlinter are run in parallel, you can encounter this error :

[Error: ENOENT: no such file or directory, open '/github/workspace/megalinter-reports/copy-paste/html/jscpd-report.json'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/github/workspace/megalinter-reports/copy-paste/html/jscpd-report.json'
}

To Reproduce
Run a lot of time megalinter

Expected behavior
Only run secretlinter after a successful run of jscpd

Screenshots

2024-09-09T13:21:14.0247818Z Processing linters on [4] parallel cores… (can be decreased with variable PARALLEL_PROCESS_NUMBER in case of performance issues)
2024-09-09T13:21:15.4280679Z ##[group]�[32m✅ Linted [MARKDOWN] files with [markdown-link-check] successfully - (0.76s)�[0m (expand for details)
2024-09-09T13:21:15.4283160Z - Using [markdown-link-check v3.12.2] https://megalinter.io/8.0.0/descriptors/markdown_markdown_link_check
2024-09-09T13:21:15.4284636Z - MegaLinter key: [MARKDOWN_MARKDOWN_LINK_CHECK]
2024-09-09T13:21:15.4285628Z - Rules config: [.markdown-link-check.json]
2024-09-09T13:21:15.4286686Z - Number of files analyzed: [1]
2024-09-09T13:21:15.4287796Z ##[endgroup]
2024-09-09T13:21:15.4784250Z ##[group]�[32m✅ Linted [REPOSITORY] files with [devskim] successfully - (1.25s)�[0m (expand for details)
2024-09-09T13:21:15.4796006Z - Using [devskim v1.0.33] https://megalinter.io/8.0.0/descriptors/repository_devskim
2024-09-09T13:21:15.4797212Z - MegaLinter key: [REPOSITORY_DEVSKIM]
2024-09-09T13:21:15.4798234Z - Rules config: [.devskim.json]
2024-09-09T13:21:15.4799303Z ##[endgroup]
2024-09-09T13:21:16.0565321Z ##[group]�[32m✅ Linted [COPYPASTE] files with [jscpd] successfully - (1.26s)�[0m (expand for details)
2024-09-09T13:21:16.0623931Z - Using [jscpd v4.0.5] https://megalinter.io/8.0.0/descriptors/copypaste_jscpd
2024-09-09T13:21:16.0625633Z - MegaLinter key: [COPYPASTE_JSCPD]
2024-09-09T13:21:16.0626460Z - Rules config: [.jscpd.json]
2024-09-09T13:21:16.0627661Z ##[endgroup]
2024-09-09T13:21:16.5622894Z ##[group]�[31m❌ Linted [REPOSITORY] files with [secretlint]: Found 1 error(s) - (0.63s)�[0m (expand for details)
2024-09-09T13:21:16.5625067Z - Using [secretlint v8.2.4] https://megalinter.io/8.0.0/descriptors/repository_secretlint
2024-09-09T13:21:16.5626366Z - MegaLinter key: [REPOSITORY_SECRETLINT]
2024-09-09T13:21:16.5627249Z - Rules config: [.secretlintrc.json]
2024-09-09T13:21:16.5628121Z --Error detail:
2024-09-09T13:21:16.5629474Z [Error: ENOENT: no such file or directory, open '/github/workspace/megalinter-reports/copy-paste/html/jscpd-report.json'] {
2024-09-09T13:21:16.5630885Z   errno: -2,
2024-09-09T13:21:16.5631570Z   code: 'ENOENT',
2024-09-09T13:21:16.5632194Z   syscall: 'open',
2024-09-09T13:21:16.5633206Z   path: '/github/workspace/megalinter-reports/copy-paste/html/jscpd-report.json'
2024-09-09T13:21:16.5634769Z }
2024-09-09T13:21:16.5635007Z 
2024-09-09T13:21:16.5635717Z ##[endgroup]

Additional context
I run megalinter on GitHub Actions with this configuration :

permissions: {}

jobs:
  mega-linter:
    runs-on: ubuntu-22.04

    steps:
      - name: Checkout Code
        uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7

      - name: MegaLinter
        id: ml
        uses: oxsecurity/megalinter@c217fe8f7bc9207062a084e989bd97efd56e7b9a # v8.0.0
        env:
          VALIDATE_ALL_CODEBASE: true
          # GO needs go1.23 which is not supported
          # SPELL is a little bit tricky
          DISABLE: GO,SPELL

      - name: Archive Megalinter artifacts
        if: success() || failure()
        uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
        with:
          name: MegaLinter reports
          path: |
            megalinter-reports
            mega-linter.log

[nvuillam]

@lvjp you can bypass the issue by ignoring megalinter-reports in .jscpd.json

This is the case with default .jscpd.json shipped with MegaLinter :)

https://github.com/oxsecurity/megalinter/blob/main/TEMPLATES/.jscpd.json

[lvjp]

I think that the problem is not for jscpd but in secretlint.

[nvuillam]

@lvjp you're totally right, sorry :)

You can define a .secretlintignore file containing **/megalinter-reports/** :)

[lvjp]

You can define a .secretlintignore file containing **/megalinter-reports/** :)

@nvuillam I think that this definition should be in the default .secretlintignore to be able to directly use it out of the box with the default behavior.

[nvuillam]

@lvjp 100% agree with you, but today MegaLinter architecture contain default config files, not default ignore files :/

It will be faster for you to define one in your repo than to wait that we implement it here ^^ ( except if you're interested to make a Pull Request of course ^^ )

[lvjp]

It's a little bit confusing for me since in the file TEMPLATES/.jscpd.json whe can found :

{"ignore": ["**/node_modules/**"]}

Why we cannot create the file TEMPLATES/.secretlintignore with only content **/megalinter-reports/** ?

[nvuillam]

The architecture does not allow that for now ^^
(the name of ignore file parameter is not configurable)

[lvjp]

Ok, I see.

Maybe we can close this issue and add the label wontfix

[nvuillam]

It can be a future evolution :)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed in 14 days if no further activity occurs.
Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[trevor-vaughan]

Hi folks, I'd like to re-open this one after hitting it a few times in the wild and determining that (I think) it's a TOCTOU issue.

Basically, I keep ending up with CI (or local) runs that will randomly fail and it appears to be the secretlint + jscpd ENOENT entry described by this bug.

I've reproduced this (it's a race condition so it's difficult) on 8a18f34.

It looks like the fix is twofold:

1. Megalinter should exclude any MegaLinter-owned output directories from all linters, including project/repository-scoped ones
2. During SARIF aggregation, a non-zero exiting linter should not be ingested into yaml.safe_load() but should instead be pushed to stderr

---

Information post dumpster dive with my good friend Claude 🤖:

Why the .secretlintignore workaround from this thread doesn't hold

The advice here was to add **/megalinter-reports/** to .secretlintignore. In practice the failing consumer already has exactly that, and it still fails — because an ignore pattern filters a directory snapshot, it is not a lock. The crash is inside @secretlint/source-creator → createRawSource → readFile, which runs after discovery and ignore filtering. For a path to reach readFile it was in the final target set (so it passed the ignore filter at glob time) and then vanished before the read. Two distinct secretlint failure modes make this concrete:

• File filtered out / absent at glob time → Error: Not found target files (exit 2).
• File in the target set but gone at read time → ENOENT in createRawSource (what CI shows).

So the ignore file cannot close this — it's a race against jscpd writing into the same tree, widened by MegaLinter deleting the copy-paste/ folder on jscpd success.

Why PR #7847 doesn't cover this either

#7847 (merged) excludes REPORT_OUTPUT_FOLDER from linting, but only inside MegaLinter.list_files_all() — the os.walk file list that feeds file / list_of_files linters. secretlint runs in project mode, where Linter.py:1505-1511 appends the literal glob **/* to the secretlint CLI and never consults MegaLinter's file list. The report-folder exclusion is bypassed for project-mode scanners, so this specific case is untouched.

Confirmed source pointers (current main)

1. jscpd's report is always produced when reports are enabled, regardless of the user's .jscpd.json reporters:
   • megalinter/linters/JsCpdLinter.py build_lint_command injects --output {report_folder}/copy-paste/; the descriptor forces --reporters console,html. CLI args override the config file.
   • JsCpdLinter.complete_text_reporter_report shutil.rmtrees copy-paste/ on success — written during lint, then deleted, widening the window.
2. secretlint scans the report tree and project mode ignores FILTER_REGEX_EXCLUDE:
   • repository.megalinter-descriptor.yml: cli_lint_mode: project, cli_lint_mode_project_extra_args_after: ["**/*"], --no-gitignore.
   • megalinter/Linter.py:1505-1511: project mode appends the glob, never the filtered self.files.
3. The misleading secondary error:
   • megalinter/Linter.py:1671 yaml.safe_load(stdout) on the Node stack trace (Error: ENOENT: ... fails at column 14).
   • megalinter/Linter.py:1713-1720: the except sets total_result = 1 and logs "Error while getting total warnings from SARIF output" — turning a crash with zero real findings into a failing count.

[trevor-vaughan]

@nvuillam Looks like I'll need to bump you directly on this to re-open as I don't have the ability to do so 😅