NASA-AMMOS / NASA-AMMOS/plandev

Add maxRequestSize env var to merlin-server to set Javalin HTTP config

Open
#515 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

feature soon
Dominant language
Java
Stars
128
Forks
33
Avg merge
1d 17h
Merged PRs (30d)
12

Description

Users are running into a case where they need to upload data that is larger than the accepted current default size for Javalin. This ticket is to expose an environment variable to merlin-server so users can set the Javalin HttpConfig maxRequestSize based on their use case. We should make the default larger than 1 Mb. We should test a large payload (> 10 Mb) works after this update.

For example when using the addExternalDataset action, Hasura makes a POST request to the merlin-server /addExternalDataset endpoint. The default value for the maxRequestSize config is 1 Mb and external datasets will often be much larger than 1 Mb. Payloads larger than 1 Mb returns a 413 Payload Too Large.

Additionally we should investigate if there are env vars on the Hasura side to support larger payloads. What even is the upper limit of payload size for Hasura actions? I can't find this documented anywhere.

More questions to think about when working on this ticket are:

  • Can we support streaming very large payloads into a Hasura action and into merlin-server?
  • If we cannot support streaming, can we support submitting data in batches? This would involve either updating the addExternalDataset action to accept batches, or adding an updateExternalDataset action to include batch semantics.

See the internal Slack thread here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in merlin-server/src/main/java/gov/nasa/jpl/aerie/merlin/server/AerieAppDriver.java near the Javalin configuration, then inspect deployment/hasura/metadata/actions.yaml and the addExternalDataset endpoint. Determine how the environment variable and larger default should affect Javalin's maxRequestSize, and verify that payloads over 10 MB succeed. Also investigate the Hasura action limits and whether streaming or batching is required.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.