[source-dynamodb] Crashes with integer cursor in incremental mode
- Langage dominant
- Python
- Étoiles
- 22.1k
- Forks
- 5.3k
- Métriques de merge des PR
- Métriques de PR en attente
Description
### Connector Name
source-dynamodb
### Connector Version
0.3.11
### What step the error happened?
During the sync
### Relevant information
When running the source-dynamodb in incremental mode with a cursor of type integer the connector crashes with the following exception:
```
15:42:04.038 [main] INFO io.airbyte.integrations.source.dynamodb.DynamodbSource -- cursor type: ["null","integer"]
15:42:04.071 [main] INFO io.airbyte.integrations.source.dynamodb.DynamodbSource -- cursor primitive: integer
15:42:04.094 [main] ERROR io.airbyte.cdk.integrations.base.AirbyteExceptionHandler -- Something went wrong in the connector. See the logs for more details.
java.lang.IllegalArgumentException: No enum constant io.airbyte.protocol.models.JsonSchemaPrimitiveUtil.JsonSchemaPrimitive.INTEGER
at java.base/java.lang.Enum.valueOf(Enum.java:293)
at io.airbyte.protocol.models.JsonSchemaPrimitiveUtil$JsonSchemaPrimitive.valueOf(JsonSchemaPrimitiveUtil.java:25)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.scanIncremental(DynamodbSource.java:245)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.lambda$read$1(DynamodbSource.java:129)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.read(DynamodbSource.java:132)
at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:275)
at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:173)
at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.main(DynamodbSource.java:53)
```
The issue happens in [this line](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-dynamodb/src/main/java/io/airbyte/integrations/source/dynamodb/DynamodbSource.java#L245) when trying to get the primitive for `INTEGER` which doesn't exist in the [JsonSchemaPrimitive map](https://github.com/airbytehq/airbyte-protocol/blob/main/protocol-models/src/main/java/io/airbyte/protocol/models/JsonSchemaPrimitiveUtil.java#L23).
The fix is to change [this](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-dynamodb/src/main/java/io/airbyte/integrations/source/dynamodb/DynamodbSource.java#L230) to either `number` or `integer_v1` (not sure what's the recommended one) so it can find it in the map. I've tested it with `integer_v1` and it works.
I'm happy to submit a PR, but I'm not sure if it's helpful, as I have another PR for a bug in a different connector that I'm trying to get reviewed since April. Maybe it's faster if I let someone from Airbyte to do the PR?
### Relevant log output
```shell
15:42:04.038 [main] INFO io.airbyte.integrations.source.dynamodb.DynamodbSource -- cursor type: ["null","integer"]
15:42:04.071 [main] INFO io.airbyte.integrations.source.dynamodb.DynamodbSource -- cursor primitive: integer
15:42:04.094 [main] ERROR io.airbyte.cdk.integrations.base.AirbyteExceptionHandler -- Something went wrong in the connector. See the logs for more details.
java.lang.IllegalArgumentException: No enum constant io.airbyte.protocol.models.JsonSchemaPrimitiveUtil.JsonSchemaPrimitive.INTEGER
at java.base/java.lang.Enum.valueOf(Enum.java:293)
at io.airbyte.protocol.models.JsonSchemaPrimitiveUtil$JsonSchemaPrimitive.valueOf(JsonSchemaPrimitiveUtil.java:25)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.scanIncremental(DynamodbSource.java:245)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.lambda$read$1(DynamodbSource.java:129)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.read(DynamodbSource.java:132)
at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:275)
at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:173)
at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.main(DynamodbSource.java:53)
```
### Contribute
- [x] Yes, I want to contribute
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/10800
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.