airbytehq / airbytehq/airbyte

[source-netsuite] bug- incremental sync silently drops records modified between sync time and account-local midnight

Abierto
#79,653 4 comentarios 0 reacciones 0 asignados Ver en GitHub
area/connectors autoteam community connectors/source/netsuite hyd-fix hyd-review needs-triage team/use type/bug
Lenguaje dominante
Python
Estrellas
22.1k
Forks
5.4k
Merge medio
5 h
PR fusionados (30 d)
671

Descripción

### Connector Name

source-netsuite

### Connector Version

0.1.27

### What step the error happened?

During the sync

### Relevant information

## Summary

`source-netsuite` incremental streams permanently lose records whose
`lastModifiedDate` falls between the sync run time and midnight in the
NetSuite account's configured timezone.

## Steps to reproduce

1. Configure a NetSuite connection with an incremental stream (e.g. `journalentry`)
2. Schedule daily syncs that run before midnight UTC (e.g. a sync running at
03:00 UTC on a PDT account, where local midnight = 07:00 UTC)
3. A record is created or modified after the sync runs but before local midnight
(e.g. at 04:15 UTC)
4. The next day's sync does not return the record

## Expected behavior

All records modified since the previous sync cursor are returned on the next sync.

## Actual behavior

Records modified in the window `(sync_run_time, account_local_midnight)` UTC are
never returned. They are permanently absent from the destination.

## Root cause

`stream_slices` calls `.date()` on the cursor, stripping the time component, and
formats slice bounds using `NETSUITE_INPUT_DATE_FORMATS` (date-only strings such as
`"03/15/2026"`). NetSuite interprets bare date strings in the account's configured
timezone, so `AFTER "03/15/2026"` on a PDT account resolves to
`2026-03-15T07:00:00Z` — not UTC midnight.

A record at `04:15Z` therefore falls in a dead zone:
- The previous sync ran at `03:00Z` and could not see it (it didn't exist yet)
- The cursor advances to `03:00Z`, which `.date()` truncates to March 15
- The next sync starts its slice from `"03/15/2026"` = `07:00Z`
- The record at `04:15Z` is before `07:00Z` and is never queried

The dead zone is `[sync_run_time, account_local_midnight)` UTC — up to 12 hours
wide depending on the account's timezone offset.

## Proposed fix

Replace `.date()` + locale format with `.replace(hour=0,...)` +
`NETSUITE_OUTPUT_DATETIME_FORMAT`, producing explicit UTC datetime bounds
(`"2026-03-15T00:00:00Z"`), unambiguous regardless of account timezone.

## Environment

- Connector: `source-netsuite:0.1.27`
- Affected streams: all `IncrementalNetsuiteStream` subclasses
- NetSuite account timezone: reproduced on `America/Los_Angeles`; affects any
account whose UTC offset means local midnight > sync run time

### Relevant log output

```shell

```

### Contribute

- [x] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12846

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.