aws-amplify / aws-amplify/docs
Fix Bug in Amplify Predictions: Transcribing a PCM Audio byte buffer to Text code example
- Dominant language
- MDX
- Stars
- 506
- Forks
- 1.1k
- Avg merge
- 3h 14m
- Merged PRs (30d)
- 1
Description
The [original code](https://docs.amplify.aws/lib/predictions/sample/q/platform/js/#sample-react-app) used the **`microphone-stream`** package, which is designed for Node.js environments, and is not compatible with browser-based client-side applications. As a result, when trying to use this package in a React application running in the browser, it raised a "process is not defined" error.
```javascript
import mic from 'microphone-stream';
...
const startMic = new mic();
startMic.setStream(stream);
startMic.on('data', (chunk) => {
var raw = mic.toRaw(chunk);
if (raw == null) {
return;
}
audioBuffer.addData(raw);
});
```
To overcome the compatibility issue with the microphone-stream package, I replaced it with the [MediaRecorder](https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder) API provided by modern browsers. The MediaRecorder API lets us capture audio from the user's microphone directly in the browser without needing external packages.
```js
function AudioRecorder(props) {
// ...
let mediaRecorder;
let chunks = [];
async function startRecording() {
// ...
const stream = await navigator.mediaDevices.getUserMedia({ video: false, audio: true });
mediaRecorder = new MediaRecorder(stream);
chunks = []; // Reset the chunks array at the start of recording
// ...
mediaRecorder.ondataavailable = (event) => {
// Collect audio chunks as they become available
chunks.push(event.data);
};
mediaRecorder.start();
// ...
}
async function stopRecording() {
// ...
if (mediaRecorder && mediaRecorder.state !== 'inactive') {
mediaRecorder.onstop = () => {
// Concatenate all the recorded chunks into a single Blob
const audioBlob = new Blob(chunks, { type: 'audio/webm' });
// Call the finishRecording callback and pass the recorded audio Blob
const finishRecording = props.finishRecording;
if (typeof finishRecording === 'function') {
finishRecording(audioBlob);
}
};
mediaRecorder.stop();
}
// ...
}
// ...
}
```
Instead of using the `microphone-stream` buffer implementation, I used the `MediaRecorder` API to collect audio chunks in the `chunks` array. To obtain the final audio data in the form of an `ArrayBuffer,` I converted the `Blob` containing the collected chunks using a FileReader and its `readAsArrayBuffer` method.
Once I obtained the **`ArrayBuffer`** representation of the recorded audio, I used the Predictions API (**`Predictions.convert`**) to transcribe the audio to text. However, there seems to be an issue with the **`Predictions.convert`** API, as it is not transcribing the audio correctly.
```js
function convertFromBuffer(audioBlob) {
setResponse('Converting text...');
audioBlob.arrayBuffer().then((resultBuffer) => {
console.log("resultBuffer", resultBuffer);
// Use the AWS Amplify Predictions API to convert audio to text
Predictions.convert({
transcription: {
source: {
bytes: resultBuffer,
},
language: "en-GB", // other options are "en-GB", "fr-FR", "fr-CA", "es-US"
},
})
.then(({ transcription: { fullText } }) => {
setResponse(fullText);
console.log("Transcription result:", fullText);
})
.catch(err => {
setResponse(JSON.stringify(err, null, 2));
console.error("Transcription error:", err);
});
});
}
```

As you can see, the code is properly recording audio using the **`MediaRecorder`** API and obtaining the **`ArrayBuffer`** representation of the recorded audio. However, the issue lies with the **`Predictions.convert`** API, which is probably not transcribing the audio correctly and returning an empty string.
I’ve also tried sending the Recorded Audio as `base64 string`, `Buffer`, `ArrayBuffer`, and `Blob` :
```ts
export interface BytesSource {
bytes: Buffer | ArrayBuffer | Blob | string;
}
```
I’ve looked at `node_modules/@aws-amplify/predictions/lib/Providers/AmazonAIConvertPredictionsProvider.js` but i don’t see any bugs with sending data to Amazon Transcribe.
I don’t have much experience with handling audio processing and transcription so this is how far I can contribute at this point until I get further guidance.
**To Reproduce**
Steps to reproduce the behavior:
- Clone this repo: [https://github.com/Umoren/webpredictions](https://github.com/Umoren/webpredictions/tree/audio2text)
- Switch to the `audio2text` branch
- Install the dependencies `npm install`
- `amplify init` to initialize the backend
- `amplify add predictions` to add the predictions feature
- Select Transcribe text from audio option
```bash
? What would you like to convert? (Use arrow keys)
Translate text into a different language
Generate speech audio from text
> Transcribe text from audio
? Who should have access? Auth and Guest users
```
- `amplify push` when this is done.
- `npm run start` to start the app.
**Desktop (please complete the following information):**
- OS: macOS Monterey
- Browser: Chrome
- Version [115.0.5790.170 ]
Contributor guide
Research direction
Start with the Amplify Predictions transcription sample linked in the issue and compare it with the reproduction repository's audio2text branch. Review the reported MediaRecorder and Predictions.convert flow, then inspect AmazonAIConvertPredictionsProvider.js as suggested. Done means the browser-based sample records audio and returns transcribed text through Predictions.convert.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, javascript, react
- Domain
- documentation, frontend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100