Unity-Technologies / Unity-Technologies/ml-agents

CameraSensor NullReferenceException & Black Screen

Open
#6,320 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C#
Stars
19.7k
Forks
4.5k
PR merge metrics
No merged PRs in 30d

Description

Describe the bug
Whenever using a CameraSensor Component with a game object that has an Agent script, a NullReferenceException is raised and the camera that is tied to the sensor just shows a black screen with 'No cameras rendering'. Additionally, the corresponding ML-Agent Python scripts halts when connecting.

This appears to be a duplicate of issue #5961, but that was closed without resolution.

To Reproduce
Steps to reproduce the behavior:

  1. Open new project with Unity 'Universal 3D'
  2. Install the ML-Agents v4.1.0 package
  3. Add some game object, e.g. Cube
  4. Give the game object an Agent script
  5. Give the game object a CameraSensor component with default settings
  6. Assign the main camera to the CameraSensor
  7. Play the scene

Screenshots
Image

Image

Console logs / stack traces

In my main project where I was getting this problem, the console log I was getting was:

NullReferenceException: Object reference not set to an instance of an object
Unity.MLAgents.Sensors.CameraSensor.ObservationToTexture (UnityEngine.Camera obsCamera, UnityEngine.Texture2D texture2D, System.Int32 width, System.Int32 height) (at ./Library/PackageCache/com.unity.ml-agents@51fac9654958/Runtime/Sensors/CameraSensor.cs:159)
Unity.MLAgents.Sensors.CameraSensor.Update () (at ./Library/PackageCache/com.unity.ml-agents@51fac9654958/Runtime/Sensors/CameraSensor.cs:115)
Unity.MLAgents.Agent.UpdateSensors () (at ./Library/PackageCache/com.unity.ml-agents@51fac9654958/Runtime/Agent.cs:1154)
Unity.MLAgents.Agent.SendInfoToBrain () (at ./Library/PackageCache/com.unity.ml-agents@51fac9654958/Runtime/Agent.cs:1114)
Unity.MLAgents.Agent.SendInfo () (at ./Library/PackageCache/com.unity.ml-agents@51fac9654958/Runtime/Agent.cs:1377)
Unity.MLAgents.Academy.EnvironmentStep () (at ./Library/PackageCache/com.unity.ml-agents@51fac9654958/Runtime/Academy.cs:586)
Unity.MLAgents.AcademyFixedUpdateStepper.FixedUpdate () (at ./Library/PackageCache/com.unity.ml-agents@51fac9654958/Runtime/Academy.cs:43)

But in the reproduction environment described earlier, I was unable to produce any raised exception.

Environment (please complete the following information):

  • Unity Version: 6000.3.6f1
  • OS + version: Windows 10 & Debian 13
  • ML-Agents version: ML-Agents v4.1.0
  • Environment: Unity Universal 3D

NOTE: We are unable to help reproduce bugs with custom environments. Please attempt to reproduce your issue with one of the example environments, or provide a minimal patch to one of the environments needed to reproduce the issue.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproduction steps in a Unity Universal 3D project, then inspect Runtime/Sensors/CameraSensor.cs at the reported ObservationToTexture and Update lines, along with the Agent.cs sensor update path. Done means reproducing and explaining the NullReferenceException, black camera output, and Python connection halt, or documenting why the supplied environment does not reproduce them.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, machine-learning, python, unity
Domain
game-dev, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.