airbytehq / airbytehq/airbyte

[source-netsuite] bug- incremental sync silently drops records modified between sync time and account-local midnight

Open
#79,653 4 comments 0 reactions 0 assignees View on GitHub
area/connectors autoteam community connectors/source/netsuite hyd-fix hyd-review needs-triage team/use type/bug
Dominant language
Python
Stars
22.1k
Forks
5.4k
Avg merge
5h
Merged PRs (30d)
671

Description

### Connector Name

source-netsuite

### Connector Version

0.1.27

### What step the error happened?

During the sync

### Relevant information

## Summary

`source-netsuite` incremental streams permanently lose records whose
`lastModifiedDate` falls between the sync run time and midnight in the
NetSuite account's configured timezone.

## Steps to reproduce

1. Configure a NetSuite connection with an incremental stream (e.g. `journalentry`)
2. Schedule daily syncs that run before midnight UTC (e.g. a sync running at
03:00 UTC on a PDT account, where local midnight = 07:00 UTC)
3. A record is created or modified after the sync runs but before local midnight
(e.g. at 04:15 UTC)
4. The next day's sync does not return the record

## Expected behavior

All records modified since the previous sync cursor are returned on the next sync.

## Actual behavior

Records modified in the window `(sync_run_time, account_local_midnight)` UTC are
never returned. They are permanently absent from the destination.

## Root cause

`stream_slices` calls `.date()` on the cursor, stripping the time component, and
formats slice bounds using `NETSUITE_INPUT_DATE_FORMATS` (date-only strings such as
`"03/15/2026"`). NetSuite interprets bare date strings in the account's configured
timezone, so `AFTER "03/15/2026"` on a PDT account resolves to
`2026-03-15T07:00:00Z` — not UTC midnight.

A record at `04:15Z` therefore falls in a dead zone:
- The previous sync ran at `03:00Z` and could not see it (it didn't exist yet)
- The cursor advances to `03:00Z`, which `.date()` truncates to March 15
- The next sync starts its slice from `"03/15/2026"` = `07:00Z`
- The record at `04:15Z` is before `07:00Z` and is never queried

The dead zone is `[sync_run_time, account_local_midnight)` UTC — up to 12 hours
wide depending on the account's timezone offset.

## Proposed fix

Replace `.date()` + locale format with `.replace(hour=0,...)` +
`NETSUITE_OUTPUT_DATETIME_FORMAT`, producing explicit UTC datetime bounds
(`"2026-03-15T00:00:00Z"`), unambiguous regardless of account timezone.

## Environment

- Connector: `source-netsuite:0.1.27`
- Affected streams: all `IncrementalNetsuiteStream` subclasses
- NetSuite account timezone: reproduced on `America/Los_Angeles`; affects any
account whose UTC offset means local midnight > sync run time

### Relevant log output

```shell

```

### Contribute

- [x] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12846

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.