microsoft / microsoft/planetary-computer-apis

Unable to dump collection from azurite container

Open
#112 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
129
Forks
33
Avg merge
2d 20h
Merged PRs (30d)
1

Description

I've successfully deployed dev environment and check that naip collection exist. I've tried to dump collection from inner azurite storage, following steps described in docs/collection-config.md using pcapis from ./scripts/console

pcapis dump -t collection --account=devstoreaccount1 --table=collectionconfig --sas=$SAS --output=collectionconfig.json --account-url=http://*.*.*.*:10002/devstoreaccount1

and get following error:

root@ed21df10fee0:/opt/src# pcapis dump -t collection --account=devstoreaccount1 --table=collectionconfig --sas=$SAS --output=collectionconfig.json --account-url=http://*.*.*.*:10002/devstoreaccount1
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/azure/data/tables/_models.py", line 363, in _get_next_cb
    return self._command(
  File "/usr/local/lib/python3.9/site-packages/azure/data/tables/_generated/operations/_table_operations.py", line 386, in query_entities
    raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: Operation returned an invalid status 'Bad Request'
Content: {"odata.error":{"code":"InvalidInput","message":{"lang":"en-US","value":"The query condition specified in the request is invalid.\nRequestId:83d710d6-6f7e-4f91-98f6-d64218398d4a\nTime:2022-07-22T09:19:56.753Z"}}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/pcapis", line 33, in <module>
    sys.exit(load_entry_point('pccommon', 'console_scripts', 'pcapis')())
  File "/opt/src/pccommon/pccommon/cli.py", line 234, in cli
    return dump(**args)
  File "/opt/src/pccommon/pccommon/cli.py", line 67, in dump
    for (_, collection_id, col_config) in col_config_table.get_all():
  File "/opt/src/pccommon/pccommon/tables.py", line 204, in get_all
    for entity in table_client.query_entities(""):
  File "/usr/local/lib/python3.9/site-packages/azure/core/paging.py", line 128, in __next__
    return next(self._page_iterator)
  File "/usr/local/lib/python3.9/site-packages/azure/core/paging.py", line 76, in __next__
    self._response = self._get_next(self.continuation_token)
  File "/usr/local/lib/python3.9/site-packages/azure/data/tables/_models.py", line 372, in _get_next_cb
    _process_table_error(error)
  File "/usr/local/lib/python3.9/site-packages/azure/data/tables/_error.py", line 153, in _process_table_error
    _reraise_error(decoded_error)
  File "/usr/local/lib/python3.9/site-packages/azure/data/tables/_error.py", line 145, in _reraise_error
    raise decoded_error.with_traceback(exc_traceback)
  File "/usr/local/lib/python3.9/site-packages/azure/data/tables/_models.py", line 363, in _get_next_cb
    return self._command(
  File "/usr/local/lib/python3.9/site-packages/azure/data/tables/_generated/operations/_table_operations.py", line 386, in query_entities
    raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: The query condition specified in the request is invalid.
RequestId:83d710d6-6f7e-4f91-98f6-d64218398d4a
Time:2022-07-22T09:19:56.753Z
ErrorCode:InvalidInput
Content: {"odata.error":{"code":"InvalidInput","message":{"lang":"en-US","value":"The query condition specified in the request is invalid.\nRequestId:83d710d6-6f7e-4f91-98f6-d64218398d4a\nTime:2022-07-22T09:19:56.753Z"}}}

pcapis load function works correctly:

pcapis load -t collection --account=devstoreaccount1 --table=collectionconfig --sas=$SAS --file=alos-palsar-config.json --account-url=http://localhost:10002/devstoreaccount1

I can see that collection appeared in collectionconfig table in inner storage, but it is invisible in explorer and also is missing in stac api request http://*.*.*.*:8080/stac/collections/alos-palsar-mosaic

{"code":"NotFoundError","description":"No collection with id 'alos-palsar-mosaic' found!"}

P. S. Still having a lot of troubles as a lack of documentation maybe I can contact someone from your team, and contribute a better tutorial for beginners in future?)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the dump command from docs/collection-config.md using ./scripts/console, then inspect pccommon/cli.py around dump and pccommon/tables.py around get_all. Compare the failing dump path with the working load command and the Azure Table query. Done means collectionconfig can be dumped and the loaded collection appears in the explorer and STAC API.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.