Repository navigation
Perfetto in Node.js #277
Description
Perfetto (source) is an open-source tracing system that has been adopted in Android and Chromium. We are proposing to introduce it into Node as well; in short, it would replace some of Node's current implementation for writing trace events.
What it is
The website has more detail, but for our purposes Perfetto is a (C++) library that defines interactions between producers and consumers of trace events (and other instrumentation data). Producers and consumers don't communicate directly but through a central "tracing service". Trace events are flushed from producers to the tracing service via a shared memory buffer.
Perfetto supports, and provides an implementation for, communication between producers/consumers and the tracing service over IPC, via protobufs. I believe this is largely meant for OS-wide trace collection. For Node, we would most likely not use IPC, and just have everything be in-process, which Perfetto also supports.
The trace events implementation in Chromium is currently using Perfetto.
How Node benefits from it
Perfetto would replace the current mechanism for buffering and writing trace events. It would be arguably more stable than the current trace event writer implementation in Node, and also provide a standardized set of config options (such as buffer size, file options) for trace events. It would also be easier to reason about having multiple consumers for trace events (writing to file thru command line options; inspector; JS API) as this is intrinsic to Perfetto's design.
The current system for creating trace events would not change; the trace event macros as they are now would remain the same.
Current plans
I'm currently working on a demo, and can talk and answer questions at the Diagnostics meeting tomorrow and in future sessions.
Activity
hashseed commentedon Feb 19, 2019
Do you think it makes sense to give a short presentation at the Diagnostics summt?
kjin commentedon Feb 20, 2019
@hashseed Yep! I won't be able to make it in person but should be able to present remotely at that time.
mcollina commentedon Feb 20, 2019
Definitely, this is important.
Can you please include in your presentation:
joyeecheung commentedon Feb 20, 2019
Thanks for the explanation, looking forward to your presentation!
There were descriptions about Perfetto being able to address some threading issues that comes from the current trace events implementation. I am curious, how would this framework integrate with the workers? Will it enable us to control e.g. configurations of trace events in workers? Currently only the main thread is allowed to do that, because the tracing controller is global and per-process, but if Perfetto is designed for OS-wide trace collection and provides communication with IPC, maybe there is a different story for workers?
mhdawson commentedon Feb 20, 2019
What platforms does Perfetto support?
kjin commentedon Feb 21, 2019
@mcollina Thanks for the pointers! I don't anticipate any change for adding trace events, so I don't think there would be any new/changed interactions with async_hooks.
@joyeecheung I believe there is a task queue abstraction in Perfetto that would be good for managing trace events coming from different threads, I will be looking into it more over the next few days.
@mhdawson It seems like wherever Chromium can run, so there might be issues on non-Windows/macOS/Linux builds. Maybe we could make it conditional, or otherwise see how much work it takes to add Perfetto support for other platforms.
cjihrig commentedon Feb 21, 2019
and
If trace_events and perfetto can coexist, then feel free to ignore the following the question.
If trace_events and perfetto can't both be used at the same time, wouldn't platform support for perfetto be an issue? trace_events are currently targeting being a tier 1 supported tool, and platform support would bump perfetto to at least tier 2.
ofrobots commentedon Feb 21, 2019
I think of Trace Events as the API we use in core, and expose via the command line flags. I view Perfetto to be the implementation details of how it is the trace buffer is implemented and managed. They will co-exist.
mhdawson commentedon Feb 22, 2019
@ofrobots even if they co-exist, having a different implementation on some platforms does not necessarily make sense to me as it may end up being neglected.
I think we should include seeing if it builds on all of the Node.js supported platforms as part of the integration. I suspect in many cases the additional require changes might not be that significant. We would want a path to getting those upstreamed.
jasnell commentedon Feb 22, 2019
major +1 on seeing this move forward.
As I understand it, perfetto is a completely replacement for the underlying trace events implementation. All of the existing macro and API hooks continue to work, but instead of directing the output to the existing JSON writer or inspector target, it writes to perfetto. This means that how we instrument for trace events does not change, but how we consume them would. This would have an impact for tools like clinicjs but there's a massive upside that makes it worth it.
As @ofrobots mentions, there should be no change here, at least on the event production side.
mmarchini commentedon Feb 25, 2019
I think it would be interesting to understand (from a user perspective) what are the advantages of using Perfetto. What can we do with Perfetto that we can't do with the current trace events implementation? Any chance we can have a Perfetto demonstration in the Summit?
hashseed commentedon Feb 25, 2019
Not sure whether a working prototype will be ready. @kjin can probably speak to that. But I expect that the presentation on Perfetto will bring clarity about the benefits.
mmarchini commentedon Feb 25, 2019
A demonstration on what Perfetto can do on Chromium would be enlightening too.
ofrobots commentedon Feb 25, 2019
Perfetto is an implementation detail on how the trace buffers are handled, so from the user-perspective, the only visible change will be 'less crashy'.
kjin commentedon Mar 7, 2019
Presentation: https://docs.google.com/presentation/d/1iS3f4BfOmbdCO2AXFIi9Sii6jDQy4za2CSN2cuVEedM
Architecture diagram: https://drive.google.com/file/d/1oEZ5MnsJbp1_SIDYGLm0xQAfRvibsxGz (open with https://draw.io, there are four pages)
Code: https://github.com/kjin/node/tree/perfetto (cannot be built directly, must be built via node-ci: https://github.com/kjin/node-ci/tree/perfetto)
Will get back to this thread on:
kjin commentedon Mar 8, 2019
@mhdawson Re: binary size -- I saw the binary size rise from ~47.5 to ~48.8 MB in my local build after adding Perfetto. For the record, the Perfetto CLI (a separate build target in Perfetto) is ~1.15MB.
@jasnell IIUC when you were mentioning the
TraceObjectclass, were you referring to the rigidity of the current default V8 tracing controller in thatTraceObjectcannot dynamically include additional fields, and so might not be able to support arbitrary additional trace event fields? If so I believe using Perfetto's writing implementation might be able to help there -- as protobuf messages written by the Perfetto Trace Writer can contain arbitrary new fields.kjin commentedon Apr 17, 2019
A working repo has been created: https://github.com/nodejs/perfetto-nodejs-wip
mhdawson commentedon Jul 3, 2019
@kjin any update on this front?
kjin commentedon Jul 3, 2019
Yep, the Perfetto API has changed significantly since I opened the issue (to better suit the Node use case, in fact), which is why I haven't done anything in the way of opening a PR yet. But we (incl. the V8 and Perfetto teams at Google) are still actively working on this. Apologies for the lack of updates!
mmarchini commentedon Sep 11, 2019
Removing from the Diagnostics agenda since there has been no updates in the last few months. Feel free to re-add to the agenda if anyone wants to discuss this in the next meeting.
kjin commentedon Sep 25, 2019
Hey @mmarchini (and others), sorry for the silence here. Unfortunately I don't have the bandwidth to move forward with this project at this point.
If anybody has an interest in picking this up, please let me know and I can point you towards the right people to contact. Thanks!
jasnell commentedon Sep 25, 2019
Absolutely want to see this move forward but no bandwidth on my end either. I'll ask around a few folks. It would be helpful, @kjin, to get a quick summary on where the work left off, what is still needed to do, and how one would get started.
github-actions commentedon Jul 17, 2020
This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.