Repository navigation
grafana/monitoring: request for github client credentials #2370
Description
As part of improving monitoring, I've set up a grafana, influxdb and telegraf instance at Joyent, available at https://grafana.nodejs.org. I would like to set up a github oauth2 client to handle ACL, similar to how we do authorization with jenkins.
cc @mmarchini
Activity
jbergstroem commentedon Jun 30, 2020
To elaborate, we will collect vm basics (cpu, ram, disk, net, ..) per host and use grafana to both make dashboards available (to build and likely a larger crowd) as well as setting up alerts to notify people once things are not working as intended.
AshCripps commentedon Jun 30, 2020
Will this be targetting a subset of machines or do you plan to roll it out to all machines?
jbergstroem commentedon Jun 30, 2020
Telegraf supports all kinds of architectures, so this basically boils down to incorporating the ansible role as well as getting it deployed across the fleet. Right now I did this as a "make it work"-work, but will add the automation bits at some point.
For now, I will focus on critical machines:
www,ci,ci-release,backup,unencryptedandgh-bot(I'm almost done)As you can imagine, we can do much, much more with this setup other than monitoring such as graphing jenkins build times over time or whathaveyou. I think allowing interested users to create their own dashboards (and share with the broader community) would be a great goal.
MylesBorins commentedon Jun 30, 2020
If you are looking to get a +1 for using a GitHub app you should open an issue on http://github.com/nodejs/admin to ask permission with a link to the app you plan to install (assuming I understood the request appropriately)
mmarchini commentedon Jun 30, 2020
I suggested Johan open an issue here first to share more detailed context (and to let folks know this is being worked on), and then reference it on nodejs/admin.
FWIW I'm +1 on this effort, it's something I wanted to implement a while back but never got the time to do.
mhdawson commentedon Jul 2, 2020
Thanks for the heads up. Once we can log in I'd be interested in getting access.
AshCripps commentedon Jul 9, 2020
@jbergstroem I'd be happy to help with this, I also think it would be good to get the monitoring host itself into ansible so the machine can be recreated easily in the event of disaster.
mmarchini commentedon Jul 21, 2020
Not sure if there's anything that can be configured on Grafana (don't know if GitHub allows this level of granularity), but the OAuth requests read-only permission to all orgs it can, not only to
nodejs.jbergstroem commentedon Jul 22, 2020
Hey all - just an update: it works but for it to scale we really need the enterprise plugin to "sync" teams similarly to the jenkins plugin. I got help reaching out to the grafana team and they will help us out! I had a few days off but will be back in action from tomorrow and will finish setting this up. After it has been done I would really appreciate all help we can get:
github-actions commentedon May 19, 2021
This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.
AshCripps commentedon May 19, 2021
Adding the never stale label as this will still be useful to have - espically the alerting for the files like the rootfs filling up as it did last night - #2592 (comment)
AshCripps commentedon May 20, 2021
I wouldnt mind have a go at setting up the alerting if someone from @nodejs/build-infra wouldn't mind sharing the admin password to the grafana with me.
richardlau commentedon May 20, 2021
AFAIK the grafana admin password wasn't added to secrets. cc @jbergstroem
jbergstroem commentedon May 20, 2021
Will add -- done!
jbergstroem commentedon May 20, 2021
The ACL for our grafana is inherited via the github groups btw.
AshCripps commentedon May 20, 2021
@jbergstroem oh so does that mean we should have admin rights already? or is that for infra members only
jbergstroem commentedon May 20, 2021
I don't think
adminnecessarily, but you should be able to create/edit dashboards which also implies alerting. Let me know if that's not the case.AshCripps commentedon May 20, 2021
I see to only have a view role - doesnt let me edit dashboards at all and dashboards is the only thing in my side menu
jbergstroem commentedon May 20, 2021
I can't quite figure out how the inheritance from the group works; logged in as admin and gave your user admin rights. As we scale we can also assign "editor" roles
AshCripps commentedon May 20, 2021
Great that works for me now, thanks!
AshCripps commentedon May 21, 2021
Got a basic alert setup, should trigger when the machines hit 95% disk full (let me know if thats too high). Currently it will post a message to
#nodejs-build-infra-alertsin the openjs slack (thanks to Brian for helping me set up the integration).I made a new chart below the current disk usage to show a percentage and alert on that
I did this to stop it constant triggering cause the other graph uses total as well which would causing it to trigger.