Can't save datastore objects
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
I can't seem to save my database objects using `WriteToDatastore`, as it errors out on a strange unicode issue when trying to write a batch. Stacktrace follows:
```
File "apache_beam/runners/common.py", line 195, in apache_beam.runners.common.DoFnRunner.receive (apache_beam/runners/common.c:5142)
self.process(windowed_value)
File "apache_beam/runners/common.py", line 267, in apache_beam.runners.common.DoFnRunner.process
(apache_beam/runners/common.c:7201)
self.reraise_augmented(exn)
File "apache_beam/runners/common.py",
line 279, in apache_beam.runners.common.DoFnRunner.reraise_augmented (apache_beam/runners/common.c:7590)
raise type(exn), args, sys.exc_info()[2]
File "apache_beam/runners/common.py", line 263, in apache_beam.runners.common.DoFnRunner.process
(apache_beam/runners/common.c:7090)
self._dofn_simple_invoker(element)
File "apache_beam/runners/common.py",
line 198, in apache_beam.runners.common.DoFnRunner._dofn_simple_invoker (apache_beam/runners/common.c:5262)
self._process_outputs(element, self.dofn_process(element.value))
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/datastoreio.py",
line 354, in process
self._flush_batch()
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/datastoreio.py",
line 363, in _flush_batch
helper.write_mutations(self._datastore, self._project, self._mutations)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/helper.py", line 187,
in write_mutations
commit(commit_request)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py",
line 174, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcp/datastore/v1/helper.py",
line 185, in commit
datastore.commit(req)
File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py",
line 140, in commit
datastore_pb2.CommitResponse)
File "/usr/local/lib/python2.7/dist-packages/googledatastore/connection.py",
line 199, in _call_method
method='POST', body=payload, headers=headers)
File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py",
line 631, in new_request
redirections, connection_type)
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py",
line 1609, in request (response, content)
= self._request(conn, authority, uri, request_uri, method,
body, headers, redirections, cachekey)
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py",
line 1351, in _request (response, content)
= self._conn_request(conn, request_uri, method, body, headers)
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1273, in _conn_request
conn.request(method,
request_uri, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1039, in request
self._send_request(method,
url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1073, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 1035, in endheaders
self._send_output(message_body)
File
"/usr/lib/python2.7/httplib.py", line 877, in _send_output
msg += message_body TypeError: must be
str, not unicode
[while running 'write to datastore/Convert to Mutation']
```
My code is basically:
```
| 'convert from entity' >> beam.Map(ConvertFromEntity)
| 'write to datastore' >> WriteToDatastore(client.project)
```
Where `ConvertFromEntity` converts from a google.cloud.datastore object (which has a nice API/interface) into the underlying protobuf (which is what the beam gcp/datastore library expects):
```
from google.cloud.datastore import helpers
def ConvertFromEntity(entity):
return helpers.entity_to_protobuf(entity)
```
I assume entity_to_protobuf works fine/normally, since it's also what is used by `google/cloud/datastore/batch.py` to write a bunch of `entity_pb2.Entity` objects into the `datastore_pb2.CommitRequest.mutations[n].upsert`:
In batch.py: `put() -> _assign_entity_to_pb() -> entity_to_protobuf()`.
In datastoreio.py: `WriteToDatastore->DatastoreWriteFn.to_upsert_mutation->_Mutate.DatastoreWriteFn->helper.write_mutations`
Any idea what's going on here and why this doesn't work? Yes, I may have some unicode in my objects...but it works in my appengine DB/NDB usage. I will attempt to skip WriteToDatastore and just put unbatched entities using the datastore library and see if that goes any better for me...
Imported from Jira [BEAM-1800](https://issues.apache.org/jira/browse/BEAM-1800). Original Jira may contain additional context.
Reported by: mlambert.
Contributor guide
Research direction
Start by reproducing the WriteToDatastore batch failure with an entity containing Unicode, then read apache_beam/io/gcp/datastore/v1/datastoreio.py and helper.py around _flush_batch and write_mutations. Compare the generated commit request with the Python 2 httplib call shown in the traceback; done means Unicode-valued entities can be written in a batch without the TypeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100