Source Snowflake : Incremental Sync Failing if integer cursor column's value is ending with '0' (zero).
- Lingua principale
- Python
- Stelle
- 22.1k
- Fork
- 5.3k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
## Environment
- **Airbyte version**: 0.38.4-alpha
- **OS Version / Instance**: macOS, Ubuntu 18.04, AWS EC2
- **Deployment**: Docker
- **Source Connector and version**: source-snowflake:0.1.12
- **Destination Connector and version**: destination-csv:0.2.9 or destination-s3:0.3.5
- **Step where error happened**: During second sync(first incremental sync)
## Current Behavior
if the integer cursor value is ending with '0' like 1230 then it sets cursor in different way(in exponential notation) : like "cursor":"1.23E+3"
```
2022-06-09 16:29:03 INFO i.a.w.DefaultReplicationWorker(run):266 - State capture: Updated state to: Optional[io.airbyte.config.State@180104ad[state={"cdc":false,"streams":[{"stream_name":"CUST_AMEYA","stream_namespace":"PUBLIC","cursor_field":["ID"],"cursor":"1.23E+3"}]}]]
```
due to which first full sync works fine but during second sync, it fails with following error.
```
2022-06-09 16:24:31 source > Exception in thread "main" java.lang.NumberFormatException: For input string: "1.23E+3"
2022-06-09 16:24:31 source > at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
2022-06-09 16:24:31 source > at java.base/java.lang.Long.parseLong(Long.java:711)
2022-06-09 16:24:31 source > at java.base/java.lang.Long.parseLong(Long.java:836)
2022-06-09 16:24:31 source > at io.airbyte.db.jdbc.AbstractJdbcCompatibleSourceOperations.setBigInteger(AbstractJdbcCompatibleSourceOperations.java:197)
2022-06-09 16:24:31 source > at io.airbyte.db.jdbc.JdbcSourceOperations.setStatementField(JdbcSourceOperations.java:78)
2022-06-09 16:24:31 source > at io.airbyte.db.jdbc.JdbcSourceOperations.setStatementField(JdbcSourceOperations.java:26)
2022-06-09 16:24:31 source > at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.lambda$queryTableIncremental$16(AbstractJdbcSource.java:276)
2022-06-09 16:24:31 source > at io.airbyte.db.jdbc.StreamingJdbcDatabase.unsafeQuery(StreamingJdbcDatabase.java:62)
2022-06-09 16:24:31 source > at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.lambda$queryTableIncremental$17(AbstractJdbcSource.java:266)
```
I tried with cursor values other than ending with 0(zero) and it works fine. It saves cursor in correct int form.
```
if cursor value is 1228 or any other then it saves value in correct integer form.
2022-06-09 16:32:04 INFO i.a.w.DefaultReplicationWorker(run):266 - State capture: Updated state to: Optional[io.airbyte.config.State@1b5c1a2f[state={"cdc":false,"streams":[{"stream_name":"CUST_AMEYA","stream_namespace":"PUBLIC","cursor_field":["ID"],"cursor":"1228"}]}]]
```
## Expected Behavior
Incremental scan should work fine with all integer cursor column values. This is very basic case of incremental sync where cursor value is ending with 0.
## Logs
First full successful scan log [logs-3222.txt](https://github.com/airbytehq/airbyte/files/8872562/logs-3222.txt) it sets cursor value in different form.
First failed Incremental scan log [logs-3224.txt](https://github.com/airbytehq/airbyte/files/8872564/logs-3224.txt)
## Steps to Reproduce
Create Table in snowflake:
```
CREATE OR REPLACE TABLE cust_ameya (
id int,
"first_name" varchar(50),
"last_name" varchar(50),
"email" varchar(100),
"timestamp" timestamp
);
```
Insert some values
```
insert into cust_ameya (id, "first_name", "last_name","email", "timestamp")
values
(1226, 'first70', 'last70', 'first70last70@gmail.com', sysdate()),
(1227, 'first71', 'last71', 'first71last71@gmail.com',sysdate()),
(1228, 'first72', 'last72', 'first72last72@gmail.com',sysdate()),
(1230, 'first73', 'last73', 'first73last73@gmail.com',sysdate()); // make sure last id is integer ending with zero
```
Create Source and destination, Create incremental append connection with 'id' column as cursor field, make sure that cursor value would be the integer ending with 0(zero)
Trigger first scan. It should run fine but cursor values in different form.
Add some rows:
```
insert into cust_ameya (id, "first_name", "last_name","email", "timestamp")
values
(1505, 'first810', 'last810', 'first810last70@gmail.com', sysdate()),
(1506, 'first80', 'last80', 'first80last70@gmail.com', sysdate());
```
Trigger sync again which is first incremental scan. You should see the failure and the errors same as attached above.
## Are you willing to submit a PR?
I am afraid, i still do not have enough command on airbyte code.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.