Description
While working on performance optimization features in lucene, I see we usually end up writing our own JMH micro-benchmark(we have a lot of them now) or run existing luceneutil benchmarks locally. And regressions are often only caught later during nightlies.
I was wondering if we could provide a way for users to trigger lightweight benchmarks directly from a PR? Someone could write a command like below, and benchmarks would automatically run in the background and post results on the PR comparing against the baseline:
/benchmark jmh <filter> # Run an existing JMH benchmark
/benchmark jmh "Write a benchmark to test my PR changes" # passing a prompt to AI model,
# it writes a temporary benchmark that doesn't need to checked in
/benchmark search <task-filter> # luceneutil search-only (on top of existing data)
/benchmark full <task-filter> # luceneutil index + search (default 1M/10M docs?)
/benchmark full <task-filter> --data wikimedium5m # explicit corpus size
This would make it easy for anyone(author or reviewer) to run benchmarks and catch regressions before merge.
I know this requires a dedicated machine to run the benchmarks, so I'm not sure how that part would work. Maybe something similar to what we already have for the luceneutil nightlies or a self-hosted GitHub Actions runner?
I'm not sure if folks have already thought about this, but I'd love to hear opinions.
Description
While working on performance optimization features in lucene, I see we usually end up writing our own JMH micro-benchmark(we have a lot of them now) or run existing luceneutil benchmarks locally. And regressions are often only caught later during nightlies.
I was wondering if we could provide a way for users to trigger lightweight benchmarks directly from a PR? Someone could write a command like below, and benchmarks would automatically run in the background and post results on the PR comparing against the baseline:
This would make it easy for anyone(author or reviewer) to run benchmarks and catch regressions before merge.
I know this requires a dedicated machine to run the benchmarks, so I'm not sure how that part would work. Maybe something similar to what we already have for the luceneutil nightlies or a self-hosted GitHub Actions runner?
I'm not sure if folks have already thought about this, but I'd love to hear opinions.