PipedreamHQ / PipedreamHQ/pipedream

[BUG] eTrusted new-review source permanently drops reviews that eTrusted indexes after the cursor advanced

Open
#21,901 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
11.7k
Forks
5.8k
Avg merge
3d 10h
Merged PRs (30d)
102

Description

**For Neople** — this blocks a live customer workflow.

App: **eTrusted** (Trusted Shops)
Component: source `etrusted-new-review` v0.0.1 (`components/etrusted/sources/new-review/new-review.mjs`, added in #21225)

**Describe the bug**

`etrusted-new-review` keeps a cursor of the newest `submittedAt` it has seen, and permanently rejects every review whose `submittedAt` is older than that cursor:

```js
// _getNextCursor: the cursor jumps to the newest submittedAt in the batch
const newestSubmittedAt = withSubmittedAt.reduce(...)

// _isNewReview: anything older than the cursor is dropped
if (reviewTs < cursorTs) { return false; }
```

The polling query is anchored to the same cursor (`submittedAfter = cursor.submittedAt - 1ms`), so such a review is never fetched again either.

The eTrusted Reviews API does not make a review readable the moment it is submitted. We consistently measure 4 to 15 minutes between `submittedAt` and the review appearing in `GET /reviews`, and moderation can add more. Within one 15 minute polling interval it is therefore normal for a batch to contain a review submitted at 15:39 while a review submitted at 15:22 is not indexed yet. The cursor advances to 15:39, the 15:22 review appears on the next poll, and it is then discarded for good.

**To Reproduce**

1. Deploy `etrusted-new-review` on a channel with real review traffic, default polling interval.
2. Let a poll run in which the newest review in the response was submitted later than a review that eTrusted has not indexed yet.
3. Wait for the next poll. The late review is now returned by the API, but no event is emitted for it.
4. Compare emitted events against `GET /reviews` for the same window: the review exists in the API and was never emitted.

**Expected behavior**

Every review returned by the API is emitted exactly once, regardless of how long eTrusted took to index it.

**Requested change**

Do not treat the newest `submittedAt` as a hard watermark. Either approach solves it:

1. Query with a lookback window: `submittedAfter = cursor.submittedAt - GRACE` (1 to 24 hours, ideally exposed as a prop) and rely on `dedupe: "unique"` plus a retained id set to avoid re-emitting.
2. Keep the cursor as it is, but stop filtering on timestamp: emit anything whose id is not in a rolling set of already emitted ids covering the grace window.

Option 1 with a configurable grace period would be ideal for us, since it also covers longer moderation delays.

**Additional context**

On one production channel we lost 3 reviews in 3 days this way. Verified against `GET /reviews`: the reviews exist, no event was ever emitted for them, and they are the only gaps in the series. The customer sees random public reviews going unanswered, with no error anywhere, which is hard to explain and hard to detect. There is no workaround inside the source, since any consumer would have to re-implement the listing to find the gaps.

Second, much smaller point on the same component: `ignoreStatements` is available as a prop, and leaving it unset means the API omits "statements", reviews that carry only a star rating and no text. On our channel that is about 55% of all reviews, so the default silently hides more than half the traffic. Setting the prop to `false` fixes it for us, but the prop description does not mention the default. Stating it ("if not set, statements are not returned") would save others the same investigation.

Docs: https://developers.etrusted.com/reference/getreviews

Contributor guide

Open the contributing guide

Research direction

Start in components/etrusted/sources/new-review/new-review.mjs, especially _getNextCursor, _isNewReview, the submittedAfter query, dedupe configuration, and the ignoreStatements prop. Reproduce the delayed-indexing case described in the issue, then verify that late reviews are emitted once and that the ignoreStatements default is documented accurately.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.