googleapis / googleapis/google-cloud-node

Autopagination does not throw backend errors appropriately

Open
#7,760 0 comments 0 reactions 1 assignee Claimed by @sofisl View on GitHub
library: google-cloud-node-core priority: p2 type: bug
Dominant language
TypeScript
Stars
3.2k
Forks
712
Avg merge
2d 3h
Merged PRs (30d)
99

Description

Investigating: https://github.com/googleapis/nodejs-datacatalog/issues/296

It seems like, when autoPagination is on, and you throw an error, it does not stop getting resources. To reproduce:

1. call `searchAssets.js` in datacatalog (datacatalog has a lot of resources, we're trying to get a large response):

```
'use strict';

async function main(projectId) {
// [START data_catalog_search_assets]
// Import the Google Cloud client library.
const {DataCatalogClient} = require('@google-cloud/datacatalog').v1;
const datacatalog = new DataCatalogClient();

async function searchAssets(projectId) {
// Search data assets.

/**
* TODO(developer): Uncomment the following lines before running the sample.
*/
// const projectId = 'my_project'; // Google Cloud Platform project

// Create request.
const scope = {
includeProjectIds: [projectId],
// Alternatively, search using Google Cloud Organization scopes.
//includeOrgIds: ["google.com"],
};

const request = {
scope: scope,
pageSize: 50,
includeGcpPublicDatasets: true,
orderBy: 'relevance',
};

const [result] = await datacatalog.searchCatalog(request);

console.log(`Found ${result.length}`);
console.log('Datasets:');
result.forEach(dataset => {
console.log(dataset.relativeResourceName);
});
}
searchAssets();
// [END data_catalog_search_assets]
}
main(...process.argv.slice(2));

```

Then, in gax (in `pagedApiCaller.ts`, https://github.com/googleapis/gax-nodejs/blob/2217633d95b49279a3d00fa7b720443d89ab3086/src/paginationCalls/pagedApiCaller.ts#L91):

Add a global counter, and these statements:

```
counter++;
callback(err, resources, nextPageRequest, response);
if (counter === 5) { const err = new googleError_1.GoogleError('error'); err.code = 8; /* resource exhausted */ callback(err); return; }
```

It will continue grabbing pages indefinitely.

In addition to fixing this bug, we should add unit tests [here](https://github.com/googleapis/gax-nodejs/blob/2217633d95b49279a3d00fa7b720443d89ab3086/test/unit/pagedIteration.ts#L202
).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.