airbytehq / airbytehq/airbyte

[source-netsuite] bug- incremental sync silently drops records modified between sync time and account-local midnight

Đang mở
#79,653 4 bình luận 0 reaction 0 người được giao Xem trên GitHub
area/connectors autoteam community connectors/source/netsuite hyd-fix hyd-review needs-triage team/use type/bug
Ngôn ngữ chính
Python
Star
22.1k
Fork
5.4k
Merge trung bình
5 giờ
Pull request đã merge (30 ngày)
671

Mô tả

### Connector Name

source-netsuite

### Connector Version

0.1.27

### What step the error happened?

During the sync

### Relevant information

## Summary

`source-netsuite` incremental streams permanently lose records whose
`lastModifiedDate` falls between the sync run time and midnight in the
NetSuite account's configured timezone.

## Steps to reproduce

1. Configure a NetSuite connection with an incremental stream (e.g. `journalentry`)
2. Schedule daily syncs that run before midnight UTC (e.g. a sync running at
03:00 UTC on a PDT account, where local midnight = 07:00 UTC)
3. A record is created or modified after the sync runs but before local midnight
(e.g. at 04:15 UTC)
4. The next day's sync does not return the record

## Expected behavior

All records modified since the previous sync cursor are returned on the next sync.

## Actual behavior

Records modified in the window `(sync_run_time, account_local_midnight)` UTC are
never returned. They are permanently absent from the destination.

## Root cause

`stream_slices` calls `.date()` on the cursor, stripping the time component, and
formats slice bounds using `NETSUITE_INPUT_DATE_FORMATS` (date-only strings such as
`"03/15/2026"`). NetSuite interprets bare date strings in the account's configured
timezone, so `AFTER "03/15/2026"` on a PDT account resolves to
`2026-03-15T07:00:00Z` — not UTC midnight.

A record at `04:15Z` therefore falls in a dead zone:
- The previous sync ran at `03:00Z` and could not see it (it didn't exist yet)
- The cursor advances to `03:00Z`, which `.date()` truncates to March 15
- The next sync starts its slice from `"03/15/2026"` = `07:00Z`
- The record at `04:15Z` is before `07:00Z` and is never queried

The dead zone is `[sync_run_time, account_local_midnight)` UTC — up to 12 hours
wide depending on the account's timezone offset.

## Proposed fix

Replace `.date()` + locale format with `.replace(hour=0,...)` +
`NETSUITE_OUTPUT_DATETIME_FORMAT`, producing explicit UTC datetime bounds
(`"2026-03-15T00:00:00Z"`), unambiguous regardless of account timezone.

## Environment

- Connector: `source-netsuite:0.1.27`
- Affected streams: all `IncrementalNetsuiteStream` subclasses
- NetSuite account timezone: reproduced on `America/Los_Angeles`; affects any
account whose UTC offset means local midnight > sync run time

### Relevant log output

```shell

```

### Contribute

- [x] Yes, I want to contribute

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/12846

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.