tensorflow / tensorflow/tflite-support
Android AudioClassifier/TensorAudio does not work with non-float Audio Data in latest Releases
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 441
- Forks
- 146
- PR merge metrics
- No merged PRs in 30d
Description
In the latest release of the TFLite Support Java library for audio classification (v0.3.1), when the audio data provided by the Android AudioRecord is of type short instead of float, the classifications do not work. I traced the issue to TensorAudio method public void load(short[] src, int offsetInShort, int sizeInShort) in which the input short array is converted to a float array and encoded in the PCM float format.
In line 194, when the conversion from short to float should be happening, there is an order of operations issue related to the casting and division that is resulting in erroneous results. Specifically, the casting is implicit in Java:
floatData[i] = src[i] / Short.MAX_VALUE;
In the line above, src[i] is a short, and Short.MAX_VALUE is a short, so the result of the division will be a short, and then because floatData[i] is a float, it is cast to a float after the division. Now, because the division is short division, the result of that division is a short of value 0, and then that value is converted to a float also 0f, but it has lost its meaning.
Instead, the casting of the short to a float must be done before doing the division to ensure the proper value is returned. Something like:
floatData[i] = ((float) src[i]) / Short.MAX_VALUE;
It appears that in the master branch, this issue has been knowingly or unknowingly fixed when fixing the offset issue, as the line is now:
floatData[i] = src[i + offsetInShort] * 1.f / Short.MAX_VALUE;
The inclusion of the * 1.f before the division is doing the work to ensure the short is cast to a float before dividing by a large number. While this fixes the issue when the Java is compiled, I believe it should also be updated to explicitly state the casting and order of operations for clarity and to ensure any changes in the future do not reintroduce this bug. Additionally, this fix has not been released, so the libraries that can be downloaded still have this bug and require handling the data properly on the app side before passing to the android library if any data other than floats are used.
As a result, I have a few requests:
- Please update the line to explicitly state the casting required to handle the separate data types
- Please spin a new release of the TFLite Support package as soon as possible that has the fix to this bug
- Consider adding new unit tests to handle cases other than float data
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Inspect tensorflow_lite_support/java/src/java/org/tensorflow/lite/support/audio/TensorAudio.java around line 194 and the load(short[] src, int offsetInShort, int sizeInShort) method, comparing the v0.3.1 code with the master implementation. Run or add unit coverage for short audio input and confirm classifications work correctly; a released package containing the fix is also requested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- android, java
- Domain
- machine-learning, mobile
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100