getodk / getodk/central

Increase batch size when streaming client audit logs

Open
#1,249 1 comment 0 reactions 0 assignees View on GitHub
backend performance
Dominant language
JavaScript
Stars
226
Forks
235
Avg merge
2d 22h
Merged PRs (30d)
19

Description

The `client_audits` table can get pretty big. One form that I've encountered had 16K submissions, but 1.2 million `client_audits` rows. It took 2 minutes to download the submissions.csv.zip for this form, most of which was spent streaming from `client_audits` (via `ClientAudits.streamForExport()`). The resulting submissions.csv.zip was 15 MB, and the combined client audit log CSV file was 105 MB unzipped.

It'd be nice to improve the performance of this streaming. It seems possible that a form could have even more `client_audits` rows. One idea I have for improving the performance is increasing the batch size when streaming from Postgres. Configuring the batch size isn't documented in the Slonik readme, but there was a PR for it here: gajus/slonik#322. However, the PR was released with 27.1.0, which is ahead of the version of Slonik that we're on.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.