OpenFn / OpenFn/lightning

Users can't resolve 500 errors on collections

Open
#3,312 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

I went back and forth on whether this should be a "bug" or an "enhancement" but I went with bug because with the current implementation, when a collection item isn't added properly because it's too big (more than 1,000,000 chars?) the user has no way of knowing why or fixing it.

This is on version v2.13.1

I have reproduced this locally on main:

  • Yes
  • No

To Reproduce Steps to reproduce the behavior:

  1. Try to add a value to collections that's more than 1,000,000 characters
  2. Note the error in the run logs
  3. Note the error in the web logs (not visible to the user)

Expected behavior A clear and concise description of what you expected to
happen.

I'd like some indication of why the collections.set() call fails. Ideally, if there's a "too many characters" error we'd show that limit to the end-user.

Screenshots If applicable, add screenshots to help explain your problem.

Image

Additional context Add any other context about the problem here.

Here's the current error we hit when someone tries to add a big string:

INFO 2025-06-17T09:40:21.736221606Z [resource.labels.containerName: global-web] [info] POST /collections/metadata_mappings
INFO 2025-06-17T09:40:21.791090424Z [resource.labels.containerName: global-web] [info] Sent 500 in 52ms
INFO 2025-06-17T09:40:21.791996228Z [resource.labels.containerName: global-web] [error] #PID<0.815728.0> running LightningWeb.Endpoint (connection #PID<0.815727.0>, stream id 1) terminated
INFO 2025-06-17T09:40:21.792029087Z [resource.labels.containerName: global-web] Server: app.openfn.org:80 (http)
INFO 2025-06-17T09:40:21.792035342Z [resource.labels.containerName: global-web] Request: POST /collections/metadata_mappings
INFO 2025-06-17T09:40:21.792039383Z [resource.labels.containerName: global-web] ** (exit) an exception was raised:
INFO 2025-06-17T09:40:21.792044949Z [resource.labels.containerName: global-web] ** (Postgrex.Error) ERROR 22001 (string_data_right_truncation) value too long for type character varying(1000000)
INFO 2025-06-17T09:40:21.792049179Z [resource.labels.containerName: global-web] (ecto_sql 3.12.1) lib/ecto/adapters/sql.ex:1096: Ecto.Adapters.SQL.raise_sql_call_error/1
INFO 2025-06-17T09:40:21.792054832Z [resource.labels.containerName: global-web] (ecto_sql 3.12.1) lib/ecto/adapters/sql.ex:967: Ecto.Adapters.SQL.insert_all/9
INFO 2025-06-17T09:40:21.792058626Z [resource.labels.containerName: global-web] (ecto 3.12.5) lib/ecto/repo/schema.ex:59: Ecto.Repo.Schema.do_insert_all/7
INFO 2025-06-17T09:40:21.792063949Z [resource.labels.containerName: global-web] (ecto 3.12.5) lib/ecto/multi.ex:919: Ecto.Multi.apply_operation/4
INFO 2025-06-17T09:40:21.792067976Z [resource.labels.containerName: global-web] (ecto 3.12.5) lib/ecto/multi.ex:897: Ecto.Multi.apply_operation/5
INFO 2025-06-17T09:40:21.792071771Z [resource.labels.containerName: global-web] (elixir 1.18.3) lib/enum.ex:2546: Enum."-reduce/3-lists^foldl/2-0-"/3
INFO 2025-06-17T09:40:21.792075502Z [resource.labels.containerName: global-web] (ecto 3.12.5) lib/ecto/multi.ex:870: anonymous fn/5 in Ecto.Multi.apply_operations/5
INFO 2025-06-17T09:40:21.792079188Z [resource.labels.containerName: global-web] (ecto_sql 3.12.1) lib/ecto/adapters/sql.ex:1400: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the collections.set() call and the POST /collections/metadata_mappings entry point, then trace how the Postgrex string_data_right_truncation error is handled. Reproduce the oversized collection value and verify that the user receives a clear explanation of the character limit instead of a 500 error.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir, postgresql
Domain
backend, database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.