Unity-Technologies / Unity-Technologies/com.unity.webrtc
[BUG]: Not able to display MediaStream from browser inside unity on RawImage as texture.
@karasusan is already working on this.
Since Mar 18, 2022.
- Dominant language
- Assembly
- Stars
- 852
- Forks
- 238
- PR merge metrics
- No merged PRs in 30d
Description
Package version
2.4.0-exp.6
Environment
* OS:Window10
* Unity version: 2019.4.17f1
* WebRTC Package : 2.4.0-exp.6
Steps To Reproduce
Grab a media stream from browers WebRTC api's and add that stream to remotepeerconnection.
Following is the JavaScript code from browser :
const localConnection = new RTCPeerConnection()
localConnection.onicecandidate = e => {
console.log(" NEW ice candidnat!! on localconnection reprinting SDP " )
console.log(JSON.stringify(localConnection.localDescription))
}
const sendChannel = localConnection.createDataChannel("sendChannel");
sendChannel.onmessage =e => console.log("messsage received!!!" + e.data )
sendChannel.onopen = e => console.log("open!!!!");
sendChannel.onclose =e => console.log("closed!!!!!!");
handleSuccess = stream => {
for (const track of stream.getTracks()) {
localConnection.addTrack(track, stream);
}
}
handleError = e => {
console.log("error : ",e)
}
await navigator.mediaDevices.getDisplayMedia({ video: true, audio: false }).then(handleSuccess, handleError);
localConnection.createOffer().then(o => localConnection.setLocalDescription(o) )
Next add the offer generated by the browser into unity and create answer, set the answer in browser and connection will be established.
Once the connection is established I can see the streams are activate in chrome://webrtc-internals/ and frames are being transmitted. to remotepeer.
Unity Side once you receive the video add the texture to rawImage : The result is grey color image has been displayed not the correct one. All other parameters are correct as I tested height and width of texture being received in unity.
Here texture assign is the code from unity:
private DelegateOnTrack remoteOntrack;
remoteOntrack = e =>
{
if (e.Track is VideoStreamTrack video)
{
video.OnVideoReceived += tex =>
{
receiveImage.texture = tex;
};
receiveVideoStream = e.Streams.First();
receiveVideoStream.OnRemoveTrack = ev =>
{
receiveImage.texture = null;
ev.Track.Dispose();
};
}
};
remotePeerConnection.OnTrack = remoteOntrack;
Current Behavior
Browser sends the displayMedia as a stream to remotepeer to unity and Texture rendered on RawImage inside unity is grey color image.
Expected Behavior
Correct displayMedia tobe displayed on the rawImage of unity scene.
Anything else?
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.