Repository navigation
Performance: datahub export filter is O(n*m) over a 3x-duplicated subject id array #1412
Description
Current Implementation and Limitations
handleExportSelection in apps/web/src/routes/_app/datahub/index.tsx:227 downloads the whole group's export from the server and then filters it in the browser down to the subjects currently listed in the table:
const listedSubjects = table
.getPrePaginationRowModel()
.rows.flatMap((row) => row.getVisibleCells().map((cell) => removeSubjectIdScope(cell.row.original.id)));
const filteredData = data.filter((dataEntry) => listedSubjects.includes(dataEntry.subjectId));Three separate problems in these four lines:
1. listedSubjects contains one entry per visible cell, not per row. The flatMap iterates row.getVisibleCells() but ignores cell entirely and reads cell.row.original.id — the same value for every cell in the row. The master table has three data columns, so the array holds at least three identical copies of every subject id, and removeSubjectIdScope() is called three times per row to produce them.
2. listedSubjects.includes(...) inside data.filter(...) is O(n × m). includes on an array is a linear scan. With n export rows and m entries in listedSubjects, this is n × m string comparisons — and because of (1), m is already 3× larger than it needs to be.
3. The filtering happens on the client at all. The groupId is the only thing sent to the server (params: { groupId: currentGroup?.id }); every other active filter — the sex filter, the date-of-birth range, the search string, the "with records only" toggle — is applied after the full group export has crossed the network. A user who filters down to 10 subjects still downloads the export for all of them.
The same anti-pattern appears at logs.tsx:141 in the audit log download, which builds its payload from the client's in-memory row model.
Measurements
Combining with the measurements from the export endpoint issue: a group with 25,000 instrument records produces 750,000 export rows and a 111 MiB response body. The master table for that group lists 5,000 subjects, so listedSubjects holds 15,000 strings (3 × 5,000 duplicates).
That makes the client-side filter 750,000 × 15,000 = 1.1 × 10¹⁰ string comparisons in the worst case (a row whose subject is not listed scans the entire array). This runs synchronously on the main thread, after the browser has already parsed a 111 MiB msgpack payload. In practice the tab will be unresponsive for a long time or crash.
Deduplicating and using a Set reduces the same work to 750,000 hash lookups — a reduction of roughly four orders of magnitude — and is a two-line change.
Associated Application Components
Client, Server
Proposed Solution
1. Immediate, low-risk fix — dedupe and use a Set:
const listedSubjects = new Set(
table.getPrePaginationRowModel().rows.map((row) => removeSubjectIdScope(row.original.id))
);
const filteredData = data.filter((dataEntry) => listedSubjects.has(dataEntry.subjectId));This drops the redundant getVisibleCells() traversal, calls removeSubjectIdScope once per row instead of three times, and turns the O(n × m) filter into O(n). It changes no behaviour.
2. Real fix — filter server-side. Send the listed subject ids (or, better, the filter criteria themselves) to /v1/instrument-records/export so the server only produces the rows the user asked for. This composes directly with the streaming rework proposed in the export endpoint issue: with a subjectIds parameter the export becomes proportional to what is being exported rather than to the size of the group.
3. Apply the same fix to the audit log download (logs.tsx:140), which has the same "build the export from the client's row model" shape and additionally captures a stale table via useCallback(..., []).
Estimated Difficulty
Low
Priority
High
Activity
joshunrau commentedon Jul 20, 2026
@gdevenyi Claude can do this
gdevenyi commentedon Jul 20, 2026
Correction to the original report: the duplication factor is 4×, not 3×. I measured it against the real table rather than counting columns by eye — the master table has three data columns plus the row-actions column libui injects when
rowActionsis set:#1425 fixes item 1, measured at 3,070 ms → 17 ms for 200,000 export rows against 5,000 listed subjects (identical results, 167,000 matched either way).
Item 2 is still open — the filtering remains client-side, so every active filter is applied only after the whole group's export has crossed the network. That is better done alongside the streaming rework in #1407 than by adding a
subjectIdsparameter to an endpoint that is about to be restructured.Item 3 (the same pattern in the audit log download) is handled in #1419.
perf(datahub): match export rows against a set of listed subject ids
perf(datahub): match export rows against a set of listed subject ids
perf(datahub): match export rows against a set of listed subject ids
perf(web): match export rows against a set of listed subject ids
perf(web): match export rows against a set of listed subject ids
perf(web): match export rows against a set of listed subject ids
perf(web): match export rows against a set of listed subject ids
joshunrau commentedon Oct 26, 2026
Partially fixed on
main(d86740e). Items 1 and 3 are done: #1425 (508e7ab) and #1419 have merged. Item 2 is still open. d4bf375 moved the handler toapps/web/src/routes/_app/datahub/subjects/index.tsx:342. It still sends onlygroupId(:334-336) and filters the response in the browser (:358-360).Item 2 does not need to wait for #1407
The streaming in #1407 changes how rows leave the process, not what the request asks for. None of it has landed: c363e31 and e2007a7 edited the pipeline in place. Do item 2 as its own PR after #1807, which edits both
handleExportSelectionandqueryRecordsRaw.It is also a correctness fix
Both sides of the browser filter compare unscoped ids.
getListedSubjectIds(apps/web/src/utils/table.ts:13-15) and the worker'ssubjectId(export-worker.js:103,:125) both go throughremoveSubjectIdScope, so ids that differ only in scope compare equal:Group_A$P001andGroup_B$P001both becomeP001. An admin with no current group exports without agroupId. If they narrow the table to one of the two (by sex, say, since the id search matches both), the file still holds both subjects' records.$(A $ in a group name makes every custom subject ID in that group display and export as the same value, and names differing only by space vs underscore share subjects #1725), every custom ID unscopes to the same value. Listing one subject exports all of them.Matching the stored
subjectIdin Mongo avoids both cases, whatever #1725 decides.Options
testing/src/specs/datahub.spec.ts:174pins. The ids must go in a body. Node's defaultmaxHeaderSizeis 16 KiB and the URL counts toward it: a local probe gets a 431 at 200 hashed ids in axios'ssubjectIds[]=form. libnest 9.0.0 setsbodyLimitto 50 MiB (src/app/app.container.ts:31), and 5,000 hashed ids are about 335 KB of JSON.index.tsx:474-479,:509-518,:532-542,:565-567,:584-594). The id search matches the unscoped id. The record count and last-collected date come fromsummarizeBySubject, which skips pending records (instrument-records.service.ts:375). The export would have to repeat that aggregation and reimplementremoveSubjectIdScopein Mongo. Any drift between the two filters exports rows the table does not show.Recommendation: option 1. Option 2 pays off only if the subjects table itself moves to server-side filtering and paging, as the audit log did in #1419.
Plan
packages/schemas/src/instrument-records/instrument-records.ts: add$ExportInstrumentRecordsDatawithgroupId?and a requiredsubjectIdswith.min(1), so an empty list is a 400, not an empty export.instrument-records.controller.ts: add@Post('export')with@HttpCode(HttpStatus.OK), the same@RouteAccessandMsgpackInterceptor, and the body schema imported as a value. Leave the GET (:78-84) as the whole-group export.instrument-records.service.ts: passsubjectIdsthroughexportRecordstoqueryRecordsRaw. PutsubjectId: { $in: subjectIds }and thegroupIdcondition in one$matchahead of the first$lookup(:590).subjectIdleads@@index([subjectId, instrumentId, date])(schema.prisma:187). That is the "$matchfirst" half of Performance: /instrument-records/export buffers the whole dataset in memory (12.8s, +2.9 GiB RSS, 122 MiB body at 200k records) #1407 item 3, not the ability-derived half. Keep the ability filter at:693-696: the ids narrow the export, they do not authorize it.getListedSubjectIdsreturnsrow.original.id, the scoped id.getExportRecordsposts{ groupId, subjectIds }, and thedata.filtergoes. When the table lists nothing, show the existing "No entries to export" error without making a request.datahub.spec.ts:171-173andauthorization.spec.ts:554-555.apps/web/src/services/axios.ts:37retries onlyget/head/options, so the POST loses the retry on dropped connections and 502/503/504 that the GET gets today. The export is read-only, so a newmetaopt-in would be safe.Tests
subjectIdsis rejected. Controller: the POST forwardssubjectIdsand the ability.instrument-records.service.test.ts, beside thegroupId$matchtest at:908:it('should match the requested subjects before the first $lookup, so a narrowed export joins only their records').apps/web/src/utils/__tests__/table.test.tsx:71: "should strip the group scope" flips to keeping the scope.subjects/__tests__/index.test.tsx: the axios mock (:24) andresolveExport(:130-132) stub onlyget.:444-454asserts that the POST body carries only the listed subject.:466-475and:505-515stop testing a browser filter. Add a case where an empty table makes no request.datahub.spec.ts:174passes either way, because it reads the downloaded file. Capture the response withpage.waitForResponseand assert that its bytes lack the filtered-out id, asauthorization.spec.ts:577-579does for group ids.Open questions
authorization.spec.ts:572, and b82fe8b removed routes no client calls. External scripts may still use it.With option 1, the remaining work is Medium difficulty, not High.
Triage note written with Claude Code. Code references point at
main@d86740e.