Repository navigation
Benchmark CI is using the test-double benchmarker for some HTTP benchmarks #16557
Description
- Version: master
See https://ci.nodejs.org/job/benchmark-node-micro-benchmarks/22/console
Refs: #15752
{{ message }}
See https://ci.nodejs.org/job/benchmark-node-micro-benchmarks/22/console
Refs: #15752
Activity
joyeecheung commentedon Oct 27, 2017
I am not sure why but the CI is using the test-double benchmarker to run some HTTP benchmarks...does the CI machine have
wrkorautocannonsetup? @nodejs/buildrefack commentedon Oct 27, 2017
So the machine does not have
nodenornpminstalled...There are a few options:
joyeecheung commentedon Oct 28, 2017
@refack
I think the easiest & quickest way is to go ahead an install wrk there. I can update it from time to time if I just need to ssh there and do this (IIUC? I guess all I need to do is
sudo apt-get install wrkfor now and probably need to build it from source if important updates are not available fromapt), although I don't expect a lot of important updates of wrk.Safest solution would probably be using
autocannoninstalled with the Node.js binary/npm that it has built/pulled, if the network config allows that.Not quite sure what
3. Add to Jenkins jobmeans?AndreasMadsen commentedon Oct 30, 2017
I would prefer
wrk. It has lower variance so it will give us better results. It will also be easier to compare runs becausewrkdoesn't depend on node.mhdawson commentedon Nov 1, 2017
We really should have ansible scripts that configure the 2 new benchmark machines. Manual installs will only persist until we have to re-install the machines.
If that is not possible, at the very least we should have a PR adding documentation to the benchmark repo that documents what needs to be manually installed for the the job in question.
mhdawson commentedon Nov 1, 2017
In cases of installation of npm's (non-global) I have done that install in the job itself. For things that need to be installed with apt-get I don't think they should be in the jobs as they change the machine configuration.
refack commentedon Nov 1, 2017
The ideal is for the jobs to be self-sufficient. Since
node,npmandwrkcan be installed withoutaptIMHO the installation should go in https://github.com/nodejs/benchmarking/blob/master/experimental/benchmarks/community-benchmark/run.sh (this is what the job runs)Even better is to migrate to a pipeline similar to nodejs/build#956
mhdawson commentedon Nov 3, 2017
@refack either of those 2 suggestions sound good to me.
gareth-ellis commentedon Nov 28, 2017
Hello,
I don't like the idea of having a static copy of node on the machine, as some jobs add node to the path, and then run tests - if we have another node on the path, we could risk running the wrong one (if for example the node we've just built failed, and wasn't picked up; jobs would keep running but on the static version of node).
I think either adding to an ansible script so its installed globally, or we can include it in the package. I've just had a go at building on a local machine, and it takes a couple of minutes to build there - so I think what might be best is to add to an ansible script to checkout a specific version, build it, and then we could add it to the same location we have jmeter etc, and the benchmarking run.sh can add wrk to the path for its runs.
Does that sound like an idea?
joyeecheung commentedon Nov 28, 2017
@gareth-ellis Sorry I am not quite following, I think adding wrk to the machine does not require a static copy of node? Or am I missing something?
joyeecheung commentedon Nov 28, 2017
@gareth-ellis Oh is it for installing autocannon? I think if we choose wrk instead (as suggested in #16557 (comment)) we would not have to worry about having a node in the path at all. Personally I prefer wrk as well.
(If it comes from #16557 (comment), I meant installing autocannon using one of the node built for benchmarking, probably the master one, not installing a separate copy of node just for autocannon)
gareth-ellis commentedon Nov 28, 2017
Yes, sorry. It seemed the conversation was suggesting autocannon which would/may require a static node installed, or wrk which does not. I prefer the idea of not having static copies of node around, so wrk seems a good way forward.
mhdawson commentedon Nov 28, 2017
We should add a comment to this issue nodejs/build#868 if we install wrk manually on the machines
gareth-ellis commentedon Nov 29, 2017
Is there ansible scripts for the benchmarking machine already? Seems would be quite an easy thing to write in ansible, so if there's somewhere I can add the bits required to install wrk via ansible, I'm happy to submit a PR
mhdawson commentedon Nov 29, 2017
@gareth-ellis unfortunately we don't have an ansible script for the benchmarking machines yet. See discussion here: nodejs/build#868
AndreasMadsen commentedon Jan 5, 2018
Could you guys just install
wrkfor now? It is super annoying that we can't run http benchmarks.Regarding the update issue, we have the same issue for the
Rpackages so I don't see what is new. I respect the need to automate this, but this is taking too long. The lastwrkupdate was almost a year ago, so I think we can survive no automated updates in the meantime.mhdawson commentedon Jan 5, 2018
I manually installed wrk on the 2 machines. We really need to make some progress on the automation side but since we've not managed to do it for too long I went ahead and did the install.
AndreasMadsen commentedon Jan 5, 2018
Thanks :)
joyeecheung commentedon Jan 14, 2018
I think this can be closed now. Ideally the benchmark jobs should be migrated to ansible but that should be an issue in the build repo.