airbytehq / airbytehq/airbyte

[source-dynamodb] Crashes with integer cursor in incremental mode

Ouverte
#71,171 5 commentaires 1 réaction 0 personnes assignées Voir sur GitHub
area/connectors autoteam community connectors/source/dynamodb needs-triage team/extensibility type/bug
Langage dominant
Python
Étoiles
22.1k
Forks
5.3k
Métriques de merge des PR
Métriques de PR en attente

Description

### Connector Name

source-dynamodb

### Connector Version

0.3.11

### What step the error happened?

During the sync

### Relevant information

When running the source-dynamodb in incremental mode with a cursor of type integer the connector crashes with the following exception:
```
15:42:04.038 [main] INFO io.airbyte.integrations.source.dynamodb.DynamodbSource -- cursor type: ["null","integer"]
15:42:04.071 [main] INFO io.airbyte.integrations.source.dynamodb.DynamodbSource -- cursor primitive: integer
15:42:04.094 [main] ERROR io.airbyte.cdk.integrations.base.AirbyteExceptionHandler -- Something went wrong in the connector. See the logs for more details.
java.lang.IllegalArgumentException: No enum constant io.airbyte.protocol.models.JsonSchemaPrimitiveUtil.JsonSchemaPrimitive.INTEGER
at java.base/java.lang.Enum.valueOf(Enum.java:293)
at io.airbyte.protocol.models.JsonSchemaPrimitiveUtil$JsonSchemaPrimitive.valueOf(JsonSchemaPrimitiveUtil.java:25)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.scanIncremental(DynamodbSource.java:245)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.lambda$read$1(DynamodbSource.java:129)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.read(DynamodbSource.java:132)
at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:275)
at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:173)
at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.main(DynamodbSource.java:53)
```

The issue happens in [this line](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-dynamodb/src/main/java/io/airbyte/integrations/source/dynamodb/DynamodbSource.java#L245) when trying to get the primitive for `INTEGER` which doesn't exist in the [JsonSchemaPrimitive map](https://github.com/airbytehq/airbyte-protocol/blob/main/protocol-models/src/main/java/io/airbyte/protocol/models/JsonSchemaPrimitiveUtil.java#L23).
The fix is to change [this](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-dynamodb/src/main/java/io/airbyte/integrations/source/dynamodb/DynamodbSource.java#L230) to either `number` or `integer_v1` (not sure what's the recommended one) so it can find it in the map. I've tested it with `integer_v1` and it works.

I'm happy to submit a PR, but I'm not sure if it's helpful, as I have another PR for a bug in a different connector that I'm trying to get reviewed since April. Maybe it's faster if I let someone from Airbyte to do the PR?

### Relevant log output

```shell
15:42:04.038 [main] INFO io.airbyte.integrations.source.dynamodb.DynamodbSource -- cursor type: ["null","integer"]
15:42:04.071 [main] INFO io.airbyte.integrations.source.dynamodb.DynamodbSource -- cursor primitive: integer
15:42:04.094 [main] ERROR io.airbyte.cdk.integrations.base.AirbyteExceptionHandler -- Something went wrong in the connector. See the logs for more details.
java.lang.IllegalArgumentException: No enum constant io.airbyte.protocol.models.JsonSchemaPrimitiveUtil.JsonSchemaPrimitive.INTEGER
at java.base/java.lang.Enum.valueOf(Enum.java:293)
at io.airbyte.protocol.models.JsonSchemaPrimitiveUtil$JsonSchemaPrimitive.valueOf(JsonSchemaPrimitiveUtil.java:25)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.scanIncremental(DynamodbSource.java:245)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.lambda$read$1(DynamodbSource.java:129)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.read(DynamodbSource.java:132)
at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:275)
at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:173)
at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)
at io.airbyte.integrations.source.dynamodb.DynamodbSource.main(DynamodbSource.java:53)
```

### Contribute

- [x] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/10800

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.