kestra-io / kestra-io/plugin-serdes

Error on field name reading with serdes IonToParquet

Open
#414 0 comments 0 reactions 0 assignees View on GitHub
area/plugin good first issue
Dominant language
Java
Stars
1
Forks
20
Avg merge
1d 18h
Merged PRs (30d)
36

Description

### Describe the issue

Si le nom des champs d'origine contient certaines caractères spéciaux, la conversion échoue (here it's dashes)
[exemple_origin.csv](https://github.com/user-attachments/files/30392618/exemple_origin.csv) converted in ion :
`Illegal character in: fichier-code-nature-voie-ficoba3-20230811`
```
org.apache.avro.SchemaParseException: Illegal character in: fichier-code-nature-voie-ficoba3-20230811
at org.apache.avro.Schema.validateName(Schema.java:1694)
at org.apache.avro.Schema$Field.(Schema.java:576)
at org.apache.avro.Schema$Field.(Schema.java:615)
at io.kestra.plugin.serdes.avro.infer.InferAvroSchema.inferField(InferAvroSchema.java:151)
at io.kestra.plugin.serdes.avro.infer.InferAvroSchema.inferField(InferAvroSchema.java:81)
at io.kestra.plugin.serdes.avro.infer.InferAvroSchema.lambda$inferAvroSchemaFromIon$0(InferAvroSchema.java:54)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
at reactor.core.publisher.FluxLimitRequest$FluxLimitRequestSubscriber.onNext(FluxLimitRequest.java:99)
at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:113)
at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:880)
at reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:805)
at reactor.core.publisher.FluxCreate$SerializedFluxSink.next(FluxCreate.java:163)
at java.base/java.util.Iterator.forEachRemaining(Unknown Source)
at io.kestra.core.serializers.FileSerde.lambda$readAll$0(FileSerde.java:159)
at reactor.core.publisher.FluxCreate.subscribe(FluxCreate.java:97)
at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
at reactor.core.publisher.Mono.block(Mono.java:1778)
at io.kestra.plugin.serdes.avro.infer.InferAvroSchema.inferAvroSchemaFromIon(InferAvroSchema.java:63)
at io.kestra.plugin.serdes.parquet.IonToParquet.run(IonToParquet.java:172)
at io.kestra.plugin.serdes.parquet.IonToParquet.run(IonToParquet.java:40)
at io.kestra.worker.WorkerTaskCallable.doCall(WorkerTaskCallable.java:85)
at io.kestra.worker.AbstractWorkerCallable.call(AbstractWorkerCallable.java:66)
at io.kestra.worker.WorkerSecurityService.callInSecurityContext(WorkerSecurityService.java:11)
at io.kestra.worker.DefaultWorker.lambda$callJob$0(DefaultWorker.java:1038)
at io.kestra.core.trace.NoopTracer.inCurrentContext(NoopTracer.java:16)
at io.kestra.worker.DefaultWorker.callJob(DefaultWorker.java:1034)
at io.kestra.worker.DefaultWorker.runAttempt(DefaultWorker.java:972)
at io.kestra.worker.DefaultWorker.run(DefaultWorker.java:753)
at io.kestra.worker.DefaultWorker.handleTask(DefaultWorker.java:383)
at io.kestra.worker.DefaultWorker.lambda$run$8(DefaultWorker.java:301)
at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:104)
at reactor.core.publisher.Mono.block(Mono.java:1779)
... 17 more
```

If this cannot be fixed as is, is it possible to convert the special characters into underscores beforehand?

### Environment

- Kestra Version: `1.3.29`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with io.kestra.plugin.serdes.avro.infer.InferAvroSchema.inferField, where the stack trace shows the invalid field name is passed to Avro, and inspect io.kestra.plugin.serdes.parquet.IonToParquet.run for the conversion entry point. Reproduce the failure with the attached CSV and converted Ion input, then verify that field names containing dashes are handled consistently and the Ion-to-Parquet conversion completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.