NASA-AMMOS / NASA-AMMOS/plandev
Add maxRequestSize env var to merlin-server to set Javalin HTTP config
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 128
- Forks
- 33
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 12
Description
Users are running into a case where they need to upload data that is larger than the accepted current default size for Javalin. This ticket is to expose an environment variable to merlin-server so users can set the Javalin HttpConfig maxRequestSize based on their use case. We should make the default larger than 1 Mb. We should test a large payload (> 10 Mb) works after this update.
For example when using the addExternalDataset action, Hasura makes a POST request to the merlin-server /addExternalDataset endpoint. The default value for the maxRequestSize config is 1 Mb and external datasets will often be much larger than 1 Mb. Payloads larger than 1 Mb returns a 413 Payload Too Large.
Additionally we should investigate if there are env vars on the Hasura side to support larger payloads. What even is the upper limit of payload size for Hasura actions? I can't find this documented anywhere.
More questions to think about when working on this ticket are:
- Can we support streaming very large payloads into a Hasura action and into
merlin-server? - If we cannot support streaming, can we support submitting data in batches? This would involve either updating the
addExternalDatasetaction to accept batches, or adding anupdateExternalDatasetaction to include batch semantics.
See the internal Slack thread here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in merlin-server/src/main/java/gov/nasa/jpl/aerie/merlin/server/AerieAppDriver.java near the Javalin configuration, then inspect deployment/hasura/metadata/actions.yaml and the addExternalDataset endpoint. Determine how the environment variable and larger default should affect Javalin's maxRequestSize, and verify that payloads over 10 MB succeed. Also investigate the Hasura action limits and whether streaming or batching is required.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100