`chunk()` returns a sparse array when one event exceeds `maxKB`, stalling the event queue (regression in 2.23.0)

Open
#1,309 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
react-native, typescript
Domain
api, backend

Research direction

Start in packages/core/src/util.ts with chunk and reproduce the sparse-array case using the standalone example; then inspect batching and error aggregation in packages/core/src/plugins/SegmentDestination.ts. Done means chunk returns dense batches, resets its size accumulator for each batch, preserves the count limit, and allows oversized-event responses to be processed so the queue drains.

Written by the indexing model from the issue text.

Description

bug investigate
  • analytics-react-native version: 2.24.0 (regression introduced in 2.23.0; 2.21.3 and 2.22.0 unaffected)
  • Integrations versions (if used): @segment/analytics-react-native-plugin-advertising-id, @segment/analytics-react-native-plugin-idfa
  • React Native version: 0.83.10 (Hermes, New Architecture, Expo SDK 55)
  • iOS or Android or both? Both

chunk() in packages/core/src/util.ts can return a sparse array. Since 2.23.0 the upload path iterates its results with for...of, which does not skip holes, so a single oversized event now makes every flush throw and the event queue stops draining.

chunk assigns into an index rather than appending:

if (maxKB !== undefined) {
  rollingKBSize += sizeOf(item);
  if (rollingKBSize >= maxKB) {
    chunks[++currentChunk] = [item];   // on index 0 this skips chunks[0]
    return chunks;
  }
}

When the first item alone is >= maxKB, currentChunk goes 0 -> 1 and chunks[0] is never created. MAX_PAYLOAD_SIZE_IN_KB is 500, so any single queued event serialising to 500KB or more triggers it.

Array.prototype.map preserves the hole, Promise.all resolves it to undefined, and aggregateErrors in packages/core/src/plugins/SegmentDestination.ts then reads result.status off undefined:

const results: BatchResult[] = await Promise.all(batches.map((batch) => this.uploadBatch(batch)));
const aggregation = this.aggregateErrors(results);   // for (const result of results) { switch (result.status)

This was harmless before 2.23.0. 2.21.x used chunkedEvents.map(async (batch) => { ... }) and never iterated the results, so the hole was silently skipped and the oversized event simply never uploaded. 2.23.0 added aggregateErrors with for (const result of results), which made the same sparse array fatal.

Steps to reproduce

chunk and sizeOf are pure, so the sparse array reproduces standalone (both copied verbatim from packages/core/src/util.ts):

const sizeOf = (obj) => (encodeURI(JSON.stringify(obj)).split(/%..|./).length - 1) / 1024;

const chunk = (array, count, maxKB) => {
  if (!array.length || !count) return [];
  let currentChunk = 0, rollingKBSize = 0;
  return array.reduce((chunks, item, index) => {
    if (maxKB !== undefined) {
      rollingKBSize += sizeOf(item);
      if (rollingKBSize >= maxKB) { chunks[++currentChunk] = [item]; return chunks; }
    }
    if (index !== 0 && index % count === 0) { chunks[++currentChunk] = [item]; }
    else { if (chunks[currentChunk] === undefined) chunks[currentChunk] = []; chunks[currentChunk].push(item); }
    return chunks;
  }, []);
};

// one event over MAX_PAYLOAD_SIZE_IN_KB (500), then two normal ones
const big = { messageId: 'a', properties: { blob: 'x'.repeat(520 * 1024) } };
const batches = chunk([big, { messageId: 'b' }, { messageId: 'c' }], 100, 500);

// [ <1 empty item>, [ {messageId:'a'} ], [ {messageId:'b'} ], [ {messageId:'c'} ] ]
//   ^ the hole                            ^ 'b' and 'c' should have shared a batch;
//                                           rollingKBSize is never reset, so they don't
console.log(batches);
console.log(0 in batches);   // false  <-- hole

Promise.all(batches.map((b) => ({ status: 'success', messageIds: [] }))).then((results) => {
  for (const result of results) { void result.status; }   // TypeError
});

In-app, track any event whose serialised size is >= 500KB while it is the only or first entry in the persisted queue, then let a flush policy fire.

Expected behavior

chunk returns a dense array of non-empty batches. An item that alone exceeds maxKB gets its own batch; the server rejects it with a 4xx, default4xxBehavior: 'drop' drops it, and the queue continues to drain.

Actual behavior

errorHandler receives ErrorType.FlushError with Flush failed: TypeError: Cannot read property 'status' of undefined (Hermes wording) on every flush.

Because the throw escapes sendEvents after Promise.all has uploaded the batches but before processUploadResults runs, nothing is ever dequeued. The queue therefore never drains, the same events are re-uploaded on every flush, and the device recovers only when pruneExpiredEvents discards the events at maxTotalBackoffDuration — 12 hours by default. We saw this on roughly 200 devices across both platforms in a single release, each reporting a FlushError every 30 seconds.

There is a second, independent problem in the same function: rollingKBSize is never reset when a new chunk starts. Once the cumulative size crosses maxKB, the size branch fires for every remaining item and each one becomes its own batch, so the count limit becomes unreachable. Measured with 1200 events of about 1KB: 700 batches, 699 of them single-event, against 3 batches once fixed. That is 700 HTTP requests per flush where 3 would do.

Suggested fix

Build the chunks by appending, and reset the accumulator per chunk. This removes the hole, restores the count limit, and keeps an oversized item isolated in its own batch:

export const chunk = <T>(array: T[], count: number, maxKB?: number): T[][] => {
  if (!array.length || !count) {
    return [];
  }

  let rollingKBSize = 0;

  return array.reduce((chunks: T[][], item: T) => {
    const itemKBSize = maxKB === undefined ? 0 : sizeOf(item);
    const currentChunk = chunks[chunks.length - 1];
    const isOverMaxKB = maxKB !== undefined && rollingKBSize + itemKBSize >= maxKB;

    if (currentChunk === undefined || currentChunk.length >= count || isOverMaxKB) {
      rollingKBSize = itemKBSize;
      chunks.push([item]);
      return chunks;
    }

    rollingKBSize += itemKBSize;
    currentChunk.push(item);
    return chunks;
  }, []);
};

Happy to open a PR if that would help.

Dominant language
TypeScript
Stars
383
Forks
206
Avg merge
14h 45m
Merged PRs (30d)
11

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from segmentio/analytics-react-native

All issues in segmentio/analytics-react-native

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.