Access token expires when accessing Google Cloud Storage (GS) objects

Start with hfile_libcurl.c and the bearer-token work from pull request #600, including the HTS_AUTH_LOCATION handling. Compare that path with Google Cloud Storage authentication and the metadata URL described here; done means establishing whether expiring metadata tokens can be refreshed for GCS access rather than only for htsget.

I’m currently using the GCS_OAUTH_TOKEN environment variable method to provide an OAuth access token to samtools in order to access GCS stored objects (see #390). Obtaining an access token is fairly easy on a Google compute VM with the command “export GCS_OAUTH_TOKEN=$(gcloud auth application-default print-access-token)”. However the application-default access token that is returned expires after 3600 seconds and therefore any long running program/script that attempts to invoke samtools after the expiration period is understandably denied access. As a workaround it’s possible to keep track of the expiration time and reissue a request for a new token prior to invoking samtools – but this is a serious inconvenience and not always possible.

I’m left wondering why htslib doesn’t just try a request the access token itself should no other means of authentication be provided? As I understand it the metadata URL for the token is http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token and you need to pass a header of "Metadata-Flavor: Google".

Example:

joe@test:/tmp$ export GCS_OAUTH_TOKEN=$(gcloud auth application-default print-access-token); echo $GCS_OAUTH_TOKEN
ya29.c.XXXXXXXXXXXXXXXXXXXXX
joe@test:/tmp$ curl -s -S https://www.googleapis.com/oauth2/v1/tokeninfo?access_token="$GCS_OAUTH_TOKEN"  | jq '.expires_in'
2442
joe@test:/tmp$ curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token
{"access_token":"ya29.c.XXXXXXXXXXXXXXXXXXXXX","expires_in":2437,"token_type":"Bearer"}

And there already seems to be code inhfile_libcurl.c that handles expiring tokens that could also be used, or at least work as a model. Then I stumbled across the Add Bearer token support to hfile_libcurl (for htsget) #600 pull, which looks to be where the code came from and does exactly what I think I was just suggesting by setting a HTS_AUTH_LOCATION token. Unfortunately though I wasn’t able to get it to work for GCS -- is this functionality only for htsget ?