kestra-io / kestra-io/plugin-serdes
Error on field name reading with serdes IonToParquet
- Dominant language
- Java
- Stars
- 1
- Forks
- 20
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 36
Description
### Describe the issue
Si le nom des champs d'origine contient certaines caractères spéciaux, la conversion échoue (here it's dashes)
[exemple_origin.csv](https://github.com/user-attachments/files/30392618/exemple_origin.csv) converted in ion :
`Illegal character in: fichier-code-nature-voie-ficoba3-20230811`
```
org.apache.avro.SchemaParseException: Illegal character in: fichier-code-nature-voie-ficoba3-20230811
at org.apache.avro.Schema.validateName(Schema.java:1694)
at org.apache.avro.Schema$Field.(Schema.java:576)
at org.apache.avro.Schema$Field.(Schema.java:615)
at io.kestra.plugin.serdes.avro.infer.InferAvroSchema.inferField(InferAvroSchema.java:151)
at io.kestra.plugin.serdes.avro.infer.InferAvroSchema.inferField(InferAvroSchema.java:81)
at io.kestra.plugin.serdes.avro.infer.InferAvroSchema.lambda$inferAvroSchemaFromIon$0(InferAvroSchema.java:54)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
at reactor.core.publisher.FluxLimitRequest$FluxLimitRequestSubscriber.onNext(FluxLimitRequest.java:99)
at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:113)
at reactor.core.publisher.FluxCreate$BufferAsyncSink.drain(FluxCreate.java:880)
at reactor.core.publisher.FluxCreate$BufferAsyncSink.next(FluxCreate.java:805)
at reactor.core.publisher.FluxCreate$SerializedFluxSink.next(FluxCreate.java:163)
at java.base/java.util.Iterator.forEachRemaining(Unknown Source)
at io.kestra.core.serializers.FileSerde.lambda$readAll$0(FileSerde.java:159)
at reactor.core.publisher.FluxCreate.subscribe(FluxCreate.java:97)
at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
at reactor.core.publisher.Mono.block(Mono.java:1778)
at io.kestra.plugin.serdes.avro.infer.InferAvroSchema.inferAvroSchemaFromIon(InferAvroSchema.java:63)
at io.kestra.plugin.serdes.parquet.IonToParquet.run(IonToParquet.java:172)
at io.kestra.plugin.serdes.parquet.IonToParquet.run(IonToParquet.java:40)
at io.kestra.worker.WorkerTaskCallable.doCall(WorkerTaskCallable.java:85)
at io.kestra.worker.AbstractWorkerCallable.call(AbstractWorkerCallable.java:66)
at io.kestra.worker.WorkerSecurityService.callInSecurityContext(WorkerSecurityService.java:11)
at io.kestra.worker.DefaultWorker.lambda$callJob$0(DefaultWorker.java:1038)
at io.kestra.core.trace.NoopTracer.inCurrentContext(NoopTracer.java:16)
at io.kestra.worker.DefaultWorker.callJob(DefaultWorker.java:1034)
at io.kestra.worker.DefaultWorker.runAttempt(DefaultWorker.java:972)
at io.kestra.worker.DefaultWorker.run(DefaultWorker.java:753)
at io.kestra.worker.DefaultWorker.handleTask(DefaultWorker.java:383)
at io.kestra.worker.DefaultWorker.lambda$run$8(DefaultWorker.java:301)
at io.micrometer.core.instrument.internal.TimedRunnable.run(TimedRunnable.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:104)
at reactor.core.publisher.Mono.block(Mono.java:1779)
... 17 more
```
If this cannot be fixed as is, is it possible to convert the special characters into underscores beforehand?
### Environment
- Kestra Version: `1.3.29`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with io.kestra.plugin.serdes.avro.infer.InferAvroSchema.inferField, where the stack trace shows the invalid field name is passed to Avro, and inspect io.kestra.plugin.serdes.parquet.IonToParquet.run for the conversion entry point. Reproduce the failure with the attached CSV and converted Ion input, then verify that field names containing dashes are handled consistently and the Ion-to-Parquet conversion completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100