jsforce / jsforce/jsforce

[Bug] autoFetch not fetching all records in subsubquery (i.e. recursive pagination)

Open
#1,799 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
1.5k
Forks
551
Avg merge
9h 2m
Merged PRs (30d)
3

Description

My hierarchy is:

Event__c -> Blocks__r -> Selected_Blocks__r

When using autoFetch = true and ensuring that maxFetch is far above the number of records returned, I am still seeing done = false and a nextRecordsUrl property for the Selected_Blocks__r.

My code is:

import "dotenv/config";
import type { Connection } from "jsforce";
import { withSalesforceConnection } from "@/utils/salesforce-auth";

const fetchEvent = async (conn: Connection, eventId: string) => {
  const query = `
    SELECT
        Id
      , Events_Explore_Points__c
      , (
        SELECT
            Id
        FROM
            Delegates__r
        WHERE
          Delegate_Email_Address__c != null
          AND Delegate_Status__c = 'Confirmed'
      )
      , (
        SELECT
            Id
            , Speaker_Bio__c
        FROM
            Speakers1__r
        WHERE
            Confirmed_Sessions__c > 0
      )
      , (
          SELECT
              Id
            , (
                SELECT
                    Id
                  , Delegate__r.Id
                FROM
                    Selected_Blocks__r
                WHERE
                    Delegate__r.Delegate_Email_Address__c != null
                    AND Delegate__r.Delegate_Status__c = 'Confirmed'
            )
          FROM
              Blocks__r
         )
      FROM Event__c
      WHERE
          Id = '${eventId}'
    `;

  const results = await conn.query(query, {
    maxFetch: 5_000_000,
    autoFetch: true,
  });

  if (!results.records.length) {
    throw new Error("No results");
  }

  return results;
};

const main = async () => {
  const eventId = "a00Mn000015cntOIAQ";
  const result = await withSalesforceConnection(fetchEvent, eventId);

  const { records, ...metadata } = result;

  console.log("Event Metadata:", metadata);
  console.log("Delegate Size:", records[0].Delegates__r.totalSize);
  console.log("Speaker Size:", records[0].Speakers1__r.totalSize);
  console.log("Block Records:", records[0].Blocks__r.records);
};

void main().then(() => process.exit(0));

which yields the output:

Event Metadata: { totalSize: 1, done: true }
Delegate Size: 195
Speaker Size: 24
Block Records: [
  {
    attributes: {
      type: 'Block__c',
      url: '/services/data/v66.0/sobjects/Block__c/a5DMn000006bHpfMAE'
    },
    Id: 'a5DMn000006bHpfMAE',
    Selected_Blocks__r: {
      totalSize: 182,
      done: false,
      nextRecordsUrl: '/services/data/v66.0/query/0r8xx5WjMNGxMXrA0N-66',
      records: [Array]
    }
  },
  {
    attributes: {
      type: 'Block__c',
      url: '/services/data/v66.0/sobjects/Block__c/a5DMn000006bQcnMAE'
    },
    Id: 'a5DMn000006bQcnMAE',
    Selected_Blocks__r: { totalSize: 277, done: true, records: [Array] }
  },
  {
    attributes: {
      type: 'Block__c',
      url: '/services/data/v66.0/sobjects/Block__c/a5DMn0000078o5FMAQ'
    },
    Id: 'a5DMn0000078o5FMAQ',
    Selected_Blocks__r: { totalSize: 12, done: true, records: [Array] }
  }
]

Note that in the first selected block record, we have done: false and a nextRecordsUrl:

    Selected_Blocks__r: {
      totalSize: 182,
      done: false,
      nextRecordsUrl: '/services/data/v66.0/query/0r8xx5WjMNGxMXrA0N-66',
      records: [Array]
    }

I assume the relevant code in jsforce is:

https://github.com/jsforce/jsforce/blob/8ab4831540d7242bc3f653542e2cff0ad7e863a2/src/query.ts#L671-L686

https://github.com/jsforce/jsforce/blob/8ab4831540d7242bc3f653542e2cff0ad7e863a2/src/query.ts#L1171-L1199

and that this doesn't support recursive nested query pagination.

If so, would it be possible to introduce this as an option?

EDIT:

I've made pull request #1801 and have tested that it works for my query. May need some help with writing test cases.


Note also that this kind of nested queries require minimum API level 58, and has a maximum of 5 nested.

https://help.salesforce.com/s/articleView?id=release-notes.rn_api_soql_5level.htm&language=en_US&release=244&type=5

Image

Thank you!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/query.ts around lines 671–686 and 1171–1199, then compare the behavior described in pull request #1801. Done means autoFetch recursively resolves nested Selected_Blocks__r pages so their records are complete and no nested nextRecordsUrl remains; add regression coverage for the shown hierarchy if the repository’s test suite supports it.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.