OpenFn / OpenFn/kit

CLI slugify drops non-ASCII characters from job and workflow identifiers

Open
#1,388 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

CLI
Dominant language
TypeScript
Stars
21
Forks
23
Avg merge
1d 22h
Merged PRs (30d)
17

Description

Came across this while investigating lightning#4577 (allowing special characters in workflow step names). Flagging it here because it's a pre-existing bug in @openfn/project that affects the CLI's pull/deploy round-trip, and it'll get more user-visible once Lightning loosens its own validation.

The slugify() helper in packages/project/src/util/slugify.ts is:

text.replace(/\W/g, ' ').trim().replace(/\s+/g, '-').toLowerCase()

JavaScript's \W is ASCII-only, so any non-Latin character gets treated as a separator. Running it on a few realistic inputs:

  • Vérifier l'état du patient becomes v-rifier-l-tat-du-patient
  • café becomes caf
  • résumé becomes r-sum
  • Anything written entirely in a non-Latin script collapses to an empty string

The problem is that slugify() isn't just used for URLs or display. It's used as identity. In parse/from-app-state.ts it sets the local workflow id (line 117), the local job id (line 164), and the keys used for edge references (lines 144 and 187). In serialize/to-app-state.ts it's the workflow key at line 60 and a fallback job key at line 155. So on pull, a non-Latin name becomes a collapsed or empty id locally, and on deploy those collapsed ids flow back as keys and break edge references. Two jobs in the same script would even collide to the same id.

This already affects workflow names today, since Lightning has no charset restriction on workflows. Step names being restricted was the only reason we hadn't seen it more. Once lightning#4577 merges, step names will be free too, and users with non-English names will hit it immediately when they try to pull their project via the CLI.

The fix direction I'd suggest: stop using slugify for identity. Keep the original string or the UUID as the local key. Slugification is still fine for filenames or URLs, but it shouldn't be what decides whether two things are the same thing.

For testing, the golden path is round-tripping a project with non-ASCII names (French accents, Arabic, Chinese, an apostrophe or two) and asserting the deployed state matches what Lightning sent. A worthwhile edge case is two jobs whose names share the same slugified form, to make sure they stay distinct locally.

Related: OpenFn/lightning#4577, OpenFn/apollo#446.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/project/src/util/slugify.ts, then trace its identity uses in packages/project/src/parse/from-app-state.ts and packages/project/src/serialize/to-app-state.ts. Add or run round-trip coverage using French, Arabic, Chinese, and apostrophe-containing names, including two jobs with the same slugified form. Done means pulled and deployed state preserves distinct non-ASCII workflow and job identifiers and their edge references.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.