TF Python producing malformed JSON for numpy style arrays.
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 928
- Forks
- 227
- PR merge metrics
- No merged PRs in 30d
Description
TF Python serializes numpy style arrays (e.g. Java NdArray) in the format 'normalizer': array([[0.2, 0.2], [0.1, 0.3]], dtype=float32)'.
This is not standard JSON, and tools, like GSON, throw a MalformedJsonException when trying to parse this.
The issue is how to best handle this in TF Java.
I have looked at GSON using customized TypeAdapter and Serilaizers/Deserializers, but I cannot get past the low level parsing throwing the MalformedJsonException. Right now I think my only alternative is to write a pre-parser to convert the array() format string to a wellformed JSON string.
For example, for the above example:
{
'normalizer': {
" '_ArraySize_' = [1,2],
" '_ArrayData_' = [[0.2, 0.2], [0.1, 0.3]],
" '_ArrayType_' = 'float32'
}
}
_ArraySize_ is the array's shape and _ArrayType_ is the datatype for the NdArray.
This format is taken from OpenJData
Also, once we settle on a way to do this, should the NdArray package be modified to include serialize/deserialize methods.
What about compatibility with TF Python?
Any suggestions?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the TF Python serialization shown in the issue and tracing how TF Java NdArray data reaches GSON parsing. Compare the proposed array representation with the OpenJData format and define the expected compatibility behavior; done means the format and serialization/deserialization ownership are agreed and malformed JSON is no longer produced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python, tensorflow
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 22/100