Cloud Function updating Firestore document throws CANCELLED: Call cancelled

## Environment

- OS: Firebase Cloud Functions
- Node.js version: 20.15.1
- npm version: 10.7.0
- Firebase SDK version:
- `firebase-admin`: 12.1.1,
=> `@google-cloud/firestore`: 7.9.0
- `firebase-functions`: 5.0.1

## Problem

We are experiencing a problem where one of our Cloud Functions is trying to update a Firestore document (simplified)
```
import { getFirestore } from 'firebase-admin/firestore';

const firestore = getFirestore();
await firestore.collection("accounts").doc(id).update(item);
```

This pattern is used for many updates and our app has ~80.000 writes daily.

In very rare occasions the update fails due to the following error:

```
Error: 1 CANCELLED: Call cancelled
at callErrorFromStatus (/layers/google.nodejs.yarn/yarn_modules/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
at Object.onReceiveStatus (/layers/google.nodejs.yarn/yarn_modules/node_modules/@grpc/grpc-js/build/src/client.js:193:76)
at Object.onReceiveStatus (/layers/google.nodejs.yarn/yarn_modules/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
at Object.onReceiveStatus (/layers/google.nodejs.yarn/yarn_modules/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
at /layers/google.nodejs.yarn/yarn_modules/node_modules/@grpc/grpc-js/build/src/resolving-call.js:129:78
at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
at ServiceClientImpl.makeUnaryRequest (/layers/google.nodejs.yarn/yarn_modules/node_modules/@grpc/grpc-js/build/src/client.js:161:32)
at ServiceClientImpl. (/layers/google.nodejs.yarn/yarn_modules/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
at /layers/google.nodejs.yarn/yarn_modules/node_modules/@google-cloud/firestore/build/src/v1/firestore_client.js:242:29
at /layers/google.nodejs.yarn/yarn_modules/node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
at repeat (/layers/google.nodejs.yarn/yarn_modules/node_modules/google-gax/build/src/normalCalls/retries.js:80:25)
at /layers/google.nodejs.yarn/yarn_modules/node_modules/google-gax/build/src/normalCalls/retries.js:119:13
at OngoingCallPromise.call (/layers/google.nodejs.yarn/yarn_modules/node_modules/google-gax/build/src/call.js:67:27)
at NormalApiCaller.call (/layers/google.nodejs.yarn/yarn_modules/node_modules/google-gax/build/src/normalCalls/normalApiCaller.js:34:19)
at /layers/google.nodejs.yarn/yarn_modules/node_modules/google-gax/build/src/createApiCall.js:112:30
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Caused by: Error
at WriteBatch.commit (/layers/google.nodejs.yarn/yarn_modules/node_modules/@google-cloud/firestore/build/src/write-batch.js:433:23)
at DocumentReference.update (/layers/google.nodejs.yarn/yarn_modules/node_modules/@google-cloud/firestore/build/src/reference.js:445:14)
at AccountRepository.update (/layers/google.nodejs.yarn/yarn_modules/node_modules/data-repository/lib/repositories/base/BaseRepository.js:42:43)
at AccountUpdates.onUserSettingsUpdate (/workspace/lib/account_updates/index.js:88:27)
at /workspace/lib/index.js:68:25
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
{
code: 1,
details: 'Call cancelled',
metadata: [Metadata],
note: 'Exception occurred in retry method that was not classified as transient'
}

```

The same error appeared only 3 times since 2024-07-16 and never before.

I have not been able to figure out why this is happening.
My assumption was that `@grpc/grpc-js` fails because the Firestore denies the write? But according to the logs the same document received no other updates and the update size is minimal. A rate limit or size limit wouldn't make sense to me.