fsspec / fsspec/gcsfs

service account permissions error

Open
#89 15 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
398
Forks
181
Avg merge
2d 10h
Merged PRs (30d)
36

Description

I am trying to use gcsfs with a GCS service account .json token. I created a token at https://console.cloud.google.com/iam-admin/serviceaccounts/ and assigned it the role of "Storage Admin". This should have permissions to do anything to my GCS resources. I downloaded the .json token.

I use this with gcsfs as follows:

fs = gcsfs.GCSFileSystem(project='pangeo-181919', token='/home/rpa/pangeo-bf62fe06ed97.json')
fs.buckets

I get this error:

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-14-81fa34b27c7f> in <module>()
      1 # connect to gcs
      2 fs = gcsfs.GCSFileSystem(project='pangeo-181919', token='/home/rpa/pangeo-bf62fe06ed97.json')
----> 3 fs.buckets

/home/rpa/.conda/envs/dask_distributed/lib/python3.5/site-packages/gcsfs/core.py in buckets(self)
    449     def buckets(self):
    450         """Return list of available project buckets."""
--> 451         return [b["name"] for b in self._list_buckets()["items"]]
    452 
    453     @classmethod

<decorator-gen-128> in _list_buckets(self)

/home/rpa/.conda/envs/dask_distributed/lib/python3.5/site-packages/gcsfs/core.py in _tracemethod(f, self, *args, **kwargs)
     49         logger.log(logging.DEBUG - 1, tb_io.getvalue())
     50 
---> 51     return f(self, *args, **kwargs)
     52 
     53 

/home/rpa/.conda/envs/dask_distributed/lib/python3.5/site-packages/gcsfs/core.py in _list_buckets(self)
    568         items = []
    569         page = self._call(
--> 570             'get', 'b/', project=self.project
    571         )
    572 

<decorator-gen-123> in _call(self, method, path, *args, **kwargs)

/home/rpa/.conda/envs/dask_distributed/lib/python3.5/site-packages/gcsfs/core.py in _tracemethod(f, self, *args, **kwargs)
     49         logger.log(logging.DEBUG - 1, tb_io.getvalue())
     50 
---> 51     return f(self, *args, **kwargs)
     52 
     53 

/home/rpa/.conda/envs/dask_distributed/lib/python3.5/site-packages/gcsfs/core.py in _call(self, method, path, *args, **kwargs)
    430                 time.sleep(2**retry - 1)
    431                 r = meth(self.base + path, params=kwargs, json=json)
--> 432                 validate_response(r, path)
    433                 break
    434             except (HtmlError, RequestException, GoogleAuthError) as e:

/home/rpa/.conda/envs/dask_distributed/lib/python3.5/site-packages/gcsfs/core.py in validate_response(r, path)
    156             raise FileNotFoundError(path)
    157         elif "forbidden" in m:
--> 158             raise IOError("Forbidden: %s\n%s" % (path, msg))
    159         elif "invalid" in m:
    160             raise ValueError("Bad Request: %s\n%s" % (path, msg))

OSError: Forbidden: b/
gcsfs-gyre-token@pangeo-181919.iam.gserviceaccount.com does not have storage.buckets.list access to project 464800473488.

This doesn't make sense to me. It seems like listing buckets should definitely be within the privileges of the storage admin.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported GCSFileSystem constructor and fs.buckets call, then inspect the _list_buckets, _call, and validate_response paths in gcsfs/core.py alongside the service-account role and project IDs shown in the report. Done means determining whether the forbidden response is expected for this configuration and documenting or correcting the confirmed issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
authorization, cloud
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.