Users can't resolve 500 errors on collections
Nobody has claimed this yet.
- Dominant language
- Elixir
- Stars
- 296
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 50
Description
I went back and forth on whether this should be a "bug" or an "enhancement" but I went with bug because with the current implementation, when a collection item isn't added properly because it's too big (more than 1,000,000 chars?) the user has no way of knowing why or fixing it.
This is on version v2.13.1
I have reproduced this locally on main:
- Yes
- No
To Reproduce Steps to reproduce the behavior:
- Try to add a value to collections that's more than 1,000,000 characters
- Note the error in the run logs
- Note the error in the web logs (not visible to the user)
Expected behavior A clear and concise description of what you expected to
happen.
I'd like some indication of why the collections.set() call fails. Ideally, if there's a "too many characters" error we'd show that limit to the end-user.
Screenshots If applicable, add screenshots to help explain your problem.
Additional context Add any other context about the problem here.
Here's the current error we hit when someone tries to add a big string:
INFO 2025-06-17T09:40:21.736221606Z [resource.labels.containerName: global-web] [info] POST /collections/metadata_mappings
INFO 2025-06-17T09:40:21.791090424Z [resource.labels.containerName: global-web] [info] Sent 500 in 52ms
INFO 2025-06-17T09:40:21.791996228Z [resource.labels.containerName: global-web] [error] #PID<0.815728.0> running LightningWeb.Endpoint (connection #PID<0.815727.0>, stream id 1) terminated
INFO 2025-06-17T09:40:21.792029087Z [resource.labels.containerName: global-web] Server: app.openfn.org:80 (http)
INFO 2025-06-17T09:40:21.792035342Z [resource.labels.containerName: global-web] Request: POST /collections/metadata_mappings
INFO 2025-06-17T09:40:21.792039383Z [resource.labels.containerName: global-web] ** (exit) an exception was raised:
INFO 2025-06-17T09:40:21.792044949Z [resource.labels.containerName: global-web] ** (Postgrex.Error) ERROR 22001 (string_data_right_truncation) value too long for type character varying(1000000)
INFO 2025-06-17T09:40:21.792049179Z [resource.labels.containerName: global-web] (ecto_sql 3.12.1) lib/ecto/adapters/sql.ex:1096: Ecto.Adapters.SQL.raise_sql_call_error/1
INFO 2025-06-17T09:40:21.792054832Z [resource.labels.containerName: global-web] (ecto_sql 3.12.1) lib/ecto/adapters/sql.ex:967: Ecto.Adapters.SQL.insert_all/9
INFO 2025-06-17T09:40:21.792058626Z [resource.labels.containerName: global-web] (ecto 3.12.5) lib/ecto/repo/schema.ex:59: Ecto.Repo.Schema.do_insert_all/7
INFO 2025-06-17T09:40:21.792063949Z [resource.labels.containerName: global-web] (ecto 3.12.5) lib/ecto/multi.ex:919: Ecto.Multi.apply_operation/4
INFO 2025-06-17T09:40:21.792067976Z [resource.labels.containerName: global-web] (ecto 3.12.5) lib/ecto/multi.ex:897: Ecto.Multi.apply_operation/5
INFO 2025-06-17T09:40:21.792071771Z [resource.labels.containerName: global-web] (elixir 1.18.3) lib/enum.ex:2546: Enum."-reduce/3-lists^foldl/2-0-"/3
INFO 2025-06-17T09:40:21.792075502Z [resource.labels.containerName: global-web] (ecto 3.12.5) lib/ecto/multi.ex:870: anonymous fn/5 in Ecto.Multi.apply_operations/5
INFO 2025-06-17T09:40:21.792079188Z [resource.labels.containerName: global-web] (ecto_sql 3.12.1) lib/ecto/adapters/sql.ex:1400: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the collections.set() call and the POST /collections/metadata_mappings entry point, then trace how the Postgrex string_data_right_truncation error is handled. Reproduce the oversized collection value and verify that the user receives a clear explanation of the character limit instead of a 500 error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir, postgresql
- Domain
- backend, database
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100