apache / apache/beam

Python: ReadFromDatastore Embedded Entities

Open
#20,057 0 comments 1 reaction 0 assignees View on GitHub
bug gcp io P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

[Issue 8405](https://issues.apache.org/jira/browse/BEAM-8405) discussed the possibility to support embedded entities when using the conversion methods to/from of the client entity type.

This feature was added in [PR 9805]([https://github.com/apache/beam/pull/9805)], which was shipped in 2.17.

However, there seems to be an issue when Datastore embedded entities do not have a key.
Keys in embedded entities are optional.

Because of this, when using the apache_beam.io.gcp.datastore.v1new.datastoreio.ReadFromDatastore transform, the pipeline fails with the following stacktrace:
```

File "apache_beam/runners/common.py", line 780, in apache_beam.runners.common.DoFnRunner.process

File "apache_beam/runners/common.py", line 440, in apache_beam.runners.common.SimpleInvoker.invoke_process

File "apache_beam/runners/common.py", line 895, in apache_beam.runners.common._OutputProcessor.process_outputs

File "/Users/quentin/Work/sensome/dev/senback/dataflow/senback-reporting-db-import-flow/venv/lib/python3.7/site-packages/apache_beam/io/gcp/datastore/v1new/datastoreio.py",
line 264, in process
yield types.Entity.from_client_entity(client_entity)
File "/Users/quentin/Work/sensome/dev/senback/dataflow/senback-reporting-db-import-flow/venv/lib/python3.7/site-packages/apache_beam/io/gcp/datastore/v1new/types.py",
line 225, in from_client_entity
value = Entity.from_client_entity(value)
File "/Users/quentin/Work/sensome/dev/senback/dataflow/senback-reporting-db-import-flow/venv/lib/python3.7/site-packages/apache_beam/io/gcp/datastore/v1new/types.py",
line 219, in from_client_entity
Key.from_client_key(client_entity.key),
File "/Users/quentin/Work/sensome/dev/senback/dataflow/senback-reporting-db-import-flow/venv/lib/python3.7/site-packages/apache_beam/io/gcp/datastore/v1new/types.py",
line 156, in from_client_key
return Key(client_key.flat_path, project=client_key.project,
AttributeError:
'NoneType' object has no attribute 'flat_path' [while running 'Read from Datastore/Read']

```

 

Imported from Jira [BEAM-9380](https://issues.apache.org/jira/browse/BEAM-9380). Original Jira may contain additional context.
Reported by: Cavalié.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.