Skip to content

node-daily-coverity failing on test-packetnet-ubuntu1604-x64-1 #2183

Closed
Closed
node-daily-coverity failing on test-packetnet-ubuntu1604-x64-1#2183
Labels
build-agenda

Description

@richardlau
richardlau
opened on Feb 18, 2020

https://ci.nodejs.org/job/node-daily-coverity/ is failing on https://ci.nodejs.org/computer/test-packetnet-ubuntu1604-x64-1/

image

e.g. https://ci.nodejs.org/job/node-daily-coverity/1421/console

+ getconf _NPROCESSORS_ONLN
+ V=1 cov-build --dir cov-int make -j 8
/tmp/jenkins5860511461133452759.sh: 6: /tmp/jenkins5860511461133452759.sh: cov-build: not found

According to https://github.com/nodejs/build/blob/97b004a672d17820976ea8eac91b4dde8a6a4697/ansible/MANUAL_STEPS.md#jenkins-workspace installing Coverity Build Tool for Linux x64 is a manual step. I've logged into Coverity but I can't download the build tool as apparently I need admin access to at least one project:
image

https://ci.nodejs.org/job/node-daily-coverity/1422/ ran on test-packetnet-ubuntu1604-x64-2 and passed.

Activity

  1. github-actions commented on Dec 15, 2020

    @github-actions
    github-actionsbot
    on Dec 15, 2020

    This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

  2. github-actions
    added
    stale
    on Dec 15, 2020
  3. github-actions
    closed this as completedon Jan 14, 2021
  4. richardlau
    reopened this on Jul 6, 2021

  5. richardlau commented on Jul 6, 2021

    @richardlau
    richardlau
    on Jul 6, 2021
    MemberAuthor

    This is still failing as the Coverity Build Tool isn't available on the machines running the job:

    cov-build: not found

    I've applied to join the Node.js project on Coverity. As I originally posted it appears you need to have admin access to at least one project to be able to download the Coverity Build Tool. I'm not sure who the owners of the Node.js project on Coverity are -- hopefully they're still involved and able to respond to the application.

  6. AshCripps
    removed
    stale
    on Jul 6, 2021

  7. mhdawson commented on Aug 4, 2021

    @mhdawson
    mhdawson
    on Aug 4, 2021
    Member

    @richardlau is this the issue where you can update who you reached out to (@jbergstroem ?)

  8. richardlau commented on Aug 4, 2021

    @richardlau
    richardlau
    on Aug 4, 2021
    MemberAuthor

    Yes, it is -- I sent an email to @jbergstroem (and copying the build email alias) back on 9 July but I think he was out of office at the time.

  9. richardlau
    added
    build-agenda
    on Aug 4, 2021

  10. mhdawson commented on Aug 11, 2021

    @mhdawson
    mhdawson
    on Aug 11, 2021
    Member

    @richardlau might make sense to follow up with @jbergstroem again.

  11. richardlau commented on Aug 26, 2021

    @richardlau
    richardlau
    on Aug 26, 2021
    MemberAuthor

    @jbergstroem has added me to the Node.js project on Coverity so I can now access the downloads for the Coverity build tools. I've proceeded to download and install them on the three jenkins-workspace hosts and updated the job so that the PATH points to the installation (as per https://github.com/nodejs/build/blob/master/ansible/MANUAL_STEPS.md#jenkins-workspace).

    I've also updated the hosts (via rerunning our Ansible playbook against them) to install gcc-8 and updated the job to use the select-compiler.sh script.

    Test build: https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-coverity/1979/console

  12. richardlau commented on Aug 26, 2021

    @richardlau
    richardlau
    on Aug 26, 2021
    MemberAuthor

    Build failed 😞.

    image

    From the corresponding build log:
    https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-coverity/1979/console

    [WARNING] No files were emitted. This may be due to a problem with your configuration
or because no files were actually compiled by your build command.
Please make sure you have configured the compilers actually used in the compilation.
 For more details, please look at: 
    /home/iojs/build/workspace/node-daily-coverity/cov-int/build-log.txt
+ ./node --version
+ git rev-parse --short HEAD
+ VERSION=v17.0.0-pre-48655e17e1
+ FILE=nodejs-v17.0.0-pre-48655e17e1.tar.gz
+ tar -czf nodejs-v17.0.0-pre-48655e17e1.tar.gz cov-int
+ set +x
Build successfully submitted.
+ rm -rf nodejs-v17.0.0-pre-48655e17e1.tar.gz cov-int

    Unfortunately the job has deleted the cov-int directory and the build-log.txt inside it.

  13. richardlau commented on Aug 26, 2021

    @richardlau
    richardlau
    on Aug 26, 2021
    MemberAuthor

    The issue is that by using a binary other than gcc/g++, e.g. we use gcc-8/g++-8, Coverity fails to detect the compiler invocations. The recommendation is to use cov-configure but when we try to do that in a build it attempts to write to /var/cov-analysis-linux64-2020.09/config which the iojs user cannot do:

    [ERROR] Could not create the path to the generated config: /var/cov-analysis-linux64-2020.09/config/gcc-config-0: mkdir: Permission denied

    I ran (as root)

    /var/cov-analysis-linux64-2020.09/bin/cov-configure --comptype gcc --compiler gcc-8

    but ended up with ERRORS like the following in cov-init/build-log.txt

    [ERROR] This gcc-8 compiler command specifies the following arguments which should be marked as required in your configuration:
        -m64 -m64
Please reconfigure this compiler as follows:
        cov-configure -co gcc-8 -- -m64 -m64
or create a template configuration for it (recommended).

    so I ran (again as root)

    /var/cov-analysis-linux64-2020.09/bin/cov-configure -co gcc-8 -- -m64 -m64
/var/cov-analysis-linux64-2020.09/bin/cov-configure -co g++-8 -- -m64 -m64 -fno-rtti -fno-exceptions -std=gnu++14

    and this appears to have resulted in a build where the upload was accepted by Coverity, which now currently shows
    image

    I'll wait and see if we get a plausible analysis and look at updating the instructions for setting this up. Or maybe try setting up on new system where gcc/g++ are new enough to compile current Node.js master (perhaps Ubuntu 20.04?).

  14. richardlau commented on Sep 1, 2021

    @richardlau
    richardlau
    on Sep 1, 2021
    MemberAuthor

    I did some local tests and running in Ubuntu 20.04 allows us to use the default gcc/g++ binaries (gcc 9) without additional Coverity build tool configuration. Furthermore we don't get the previously mentioned ERRORs in the Coverity build log. So Ubuntu 20.04 is a potential option.

    However I've also been able to get the Coverity CI job to work in the existing Ubuntu 18.04 jenkins-workspace's using symbolic links and PATH manipulation. Essentially:

    1. Run select-compiler to set up CC and CXX for the version of Node.js being built (e.g. master for the daily runs).
    2. Filter out the ccache symlink directory from PATH.
    3. Determine the path to the actual compiler binary (e.g. /usr/bin/g++) from the CC/CXX env vars set by select-compiler.
    4. Create symlinks called gcc and g++ to the actual compiler binaries.
    5. Set PATH so that it contains the path to the Coverity build tools (as per the existing MANUAL_STEPS instruction) and
      a. The ccache symlink directory (so ccache is enabled).
      b. Our directory containing the gcc and g++ symlinks. This needs to come between the ccache symlink directory and the directory containing the actual compiler binary (e.g. /usr/bin) so that ccache picks up our symlink.
    6. Reset CC and CXX to gcc and g++ respectively -- Coverity will not detect ccache gcc (for example).

    I've changed the CI job to do those steps. No further additional Coverity build tool configuration is required (I cleared and reinstalled the tool to check). A test build with this setup has completed, submitted to Coverity and is currently being analyzed:
    https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-coverity/2015/console (v17.0.0-pre-73d5f8a843).

    The ERRORs in the Coverity build log are gone and as a result we process more "units",
    e.g. compare
    https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-coverity/2015/console

    Emitted 3260 C/C++ compilation units (99%) successfully
[WARNING] Recoverable errors were encountered during 203 of these C/C++ compilation units.

3260 C/C++ compilation units (99%) are ready for analysis
The cov-build utility completed successfully.

    to this morning's daily:
    https://ci.nodejs.org/view/Node.js%20Daily/job/node-daily-coverity/1997/console

    Emitted 1189 C/C++ compilation units (99%) successfully
[WARNING] Recoverable errors were encountered during 203 of these C/C++ compilation units.

1189 C/C++ compilation units (99%) are ready for analysis
The cov-build utility completed successfully.

  15. richardlau commented on Sep 1, 2021

    @richardlau
    richardlau
    on Sep 1, 2021
    MemberAuthor

    Good news is that I think the Coverity CI build is now fixed (🎉). We may need to fix up the exclusion patterns on the Coverity analysis side, currently:
    https://scan.coverity.com/projects/node-js?tab=analysis_settings
    image
    which doesn't cover all of the deps:
    image

    but that should probably be a separate issue.

  16. richardlau
    closed this as completedon Sep 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    build-agenda

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions