Instructions for downloading C4 insufficient

What I need help with / What I was wondering

I'm trying to download C4 via the instructions given in the T5 repo. Their instructions include a link to the tfds instructions for beam. Following these instructions after the quickstart therein, I successfully start a job, but the workers that google cloud dataflow can spin up are limited to two, instead of the multiple hundreds that the T5 instructions talk about.

What I've tried so far

In keeping with the instructions for generating big datasets with Apache Beam, I followed the Google Cloud Dataflow Quickstart instructions, setting up my project, billing, credentials, etc. all through the Google Cloud Console and then creating a virtual environment, installing tensorflow, and the google cloud sdk, then setting credentials with export GOOGLE_APPLICATION_CREDENTIALS="path/to/json/from/Google-Cloud". When I then set MY_BUCKET, MY_PROJECT, and MY_REGION variables, and I actually run the instructions

pip install 'tfds-nightly[c4]'
echo 'tfds-nightly[c4]' > /tmp/beam_requirements.txt
python -m tensorflow_datasets.scripts.download_and_prepare \
  --datasets=c4/en \
  --data_dir=gs://$MY_BUCKET/tensorflow_datasets \
  --beam_pipeline_options="project=$MY_PROJECT,job_name=c4,staging_location=gs://$MY_BUCKET/binaries,temp_location=gs://$MY_BUCKET/temp,runner=DataflowRunner,requirements_file=/tmp/beam_requirements.txt,experiments=shuffle_mode=service,region=$MY_REGION"

it begins to run, but I get a "Failed:Resize Instance Group Manager" message for a 403 error like every 20 seconds

and I'm limited to 2 workers when the console output keeps saying that it's trying to scale up to 1000.

.
.
.
I1014 15:43:05.446238 140556195309312 dataflow_runner.py:248] 2020-10-14T21:43:01.141Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
I1014 15:43:05.446516 140556195309312 dataflow_runner.py:248] 2020-10-14T21:43:01.171Z: JOB_MESSAGE_DETAILED: Workers have started successfully.
I1014 15:49:42.444391 140556195309312 dataflow_runner.py:248] 2020-10-14T21:49:40.042Z: JOB_MESSAGE_BASIC: Autoscaling: Resizing worker pool from 1 to 2.
I1014 15:49:47.653243 140556195309312 dataflow_runner.py:248] 2020-10-14T21:49:45.542Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 2 based on
the rate of progress in the currently running stage(s).
I1014 15:51:37.070624 140556195309312 dataflow_runner.py:248] 2020-10-14T21:51:36.931Z: JOB_MESSAGE_BASIC: Autoscaling: Resizing worker pool from 2 to 1000.
I1014 16:39:49.413619 140556195309312 transport.py:179] Refreshing due to a 401 (attempt 1/2)
I1014 16:39:49.448023 140556195309312 client.py:795] Refreshing access_token
I1014 17:39:53.158122 140556195309312 transport.py:179] Refreshing due to a 401 (attempt 1/2)
I1014 17:39:53.191963 140556195309312 client.py:795] Refreshing access_token
I1014 18:39:54.347596 140556195309312 transport.py:179] Refreshing due to a 401 (attempt 1/2)
I1014 18:39:54.377913 140556195309312 client.py:795] Refreshing access_token
I1014 19:39:59.015963 140556195309312 transport.py:179] Refreshing due to a 401 (attempt 1/2)
I1014 19:39:59.051589 140556195309312 client.py:795] Refreshing access_token
.
.
.

I think this is a permissions issue, from the 403 error and related questions, but I followed the instructions in creating a service account and making it "Owner", so if I'm setting my credentials to the json file I got from the GC console, why would I lack permissions, how do I go about validating my credentials, etc. so as to stop getting these 403 errors and successfully spin up hundreds of workers?

Lastly, I thought this might be a quota issue, but in examining the quotas through Google Cloud Console, it seemed like the job wasn't bumping up against any quotas. Every one of them had a green checkmark and was well-short of full.

It would be nice if...

This particular issue might be outside of the scope of tfds, as it seems like a cloud issue, but the whole reason for the T5 instructions is to spin up 500 workers using apache beam and google cloud dataflow, and so if we have to do something besides the quickstart for google cloud dataflow and the instructions found in T5 in order to make it work, it would be helpful to include that information. Like in my case, there is something up with the permissions that the quickstart didn't resolve, and so I have spent many hours digging around info on credentials, authentication, permissions, quotas, etc. etc. and the instructions don't talk about any of that stuff.

Environment information

• Operating System: macOS Catalina 10.15.7
• Python version: 3.6.9
• tfds-nightly==4.0.1.dev202010140107
• tensorflow==2.3.1

Review the T5 C4 section, the TFDS Beam instructions, and the Google Cloud Dataflow Python quickstart alongside the shown download_and_prepare command. Investigate the 403 Resize Instance Group Manager failure and worker limit, then document any required permissions, credential validation, or quota steps so the documented path reliably reaches the intended worker scale.