Implementing AR Mode

Goal: Set the background Environment to a webcam Texture to create the illusion the user is wearing an AR headset. It should be toggleable by a keyboard shortcut.

Attempting to set the background to BG_CAMERA_FEED

The Environment type supports BG_CAMERA_FEED:

However, setting it in SimulaServer.hs

G.set_background environment 6 -- BG_CAMERA_FEED

Just causes the background sky to turn black.

Godot doesn't support webcams for Linux

If we inspect the number of detected cameras from godot via CameraServer

      cameraServer <- unsafeInstance GodotCameraServer "CameraServer"
      numCameras <- G.get_feed_count cameraServer
      putStrLn $ "CameraServer get_feed_count: " ++ (show numCameras)

...we just get "0", since it turns out that Godot doesn't even support webcams for Windows, let alone Linux. You can tell this also by file inspection:

george@UbuntuBox:~/Simula/submodules/godot/modules/camera$ ls
camera_ios.h  camera_ios.mm  camera_osx.h  camera_osx.mm  camera_win.cpp  camera_win.h  config.py  __pycache__  register_types.cpp  register_types.h  SCsub

E.g. there's no camera_x11.* here. So maybe we should make one?

godot-webcam doesn't work out of the box

There's godot-webcam, which I got to compile but which doesn't actually get webcam textures to show:

Glimmer of hope: running this demo does get my webcam's green light to turn on.

The CameraFeed machinery seems to already be in Godot, even though camera detection/texture grabbing isn't working

I think our preferred approach should be working with pre-existing CameraFeed machinery within Godot, since the steps are already in place for it to render webcam textures (even though webcams aren't working). E.g. taking a look at RasterizerSceneGLES3::render_scene() from SimulaVR/godot:

void RasterizerSceneGLES3::render_scene(const Transform &p_cam_transform, const CameraMatrix &p_cam_projection, bool p_cam_ortogonal, InstanceBase **p_cull_result, int p_cull_count, RID *p_light_cull_result, int p_light_cull_count, RID *p_reflection_probe_cull_result, int p_reflection_probe_cull_count, RID p_environment, RID p_shadow_atlas, RID p_reflection_atlas, RID p_reflection_probe, int p_reflection_probe_pass) {
//..
	Ref<CameraFeed> feed;
//..
} else if (env->bg_mode == VS::ENV_BG_CAMERA_FEED) {
		feed = CameraServer::get_singleton()->get_feed_by_id(env->camera_feed_id);
		storage->frame.clear_request = false;
	} else {
//..
case VS::ENV_BG_CAMERA_FEED:
				if (feed.is_valid() && (feed->get_base_width() > 0) && (feed->get_base_height() > 0)) {
					// copy our camera feed to our background

					glDisable(GL_BLEND);
					glDepthMask(GL_FALSE);
					glDisable(GL_DEPTH_TEST);
					glDisable(GL_CULL_FACE);

					storage->shaders.copy.set_conditional(CopyShaderGLES3::USE_DISPLAY_TRANSFORM, true);
					storage->shaders.copy.set_conditional(CopyShaderGLES3::DISABLE_ALPHA, true);
					storage->shaders.copy.set_conditional(CopyShaderGLES3::SRGB_TO_LINEAR, true);

					if (feed->get_datatype() == CameraFeed::FEED_RGB) {
						RID camera_RGBA = feed->get_texture(CameraServer::FEED_RGBA_IMAGE);

						VS::get_singleton()->texture_bind(camera_RGBA, 0);
					} else if (feed->get_datatype() == CameraFeed::FEED_YCBCR) {
						RID camera_YCbCr = feed->get_texture(CameraServer::FEED_YCBCR_IMAGE);

						VS::get_singleton()->texture_bind(camera_YCbCr, 0);

						storage->shaders.copy.set_conditional(CopyShaderGLES3::YCBCR_TO_SRGB, true);

					} else if (feed->get_datatype() == CameraFeed::FEED_YCBCR_SEP) {
						RID camera_Y = feed->get_texture(CameraServer::FEED_Y_IMAGE);
						RID camera_CbCr = feed->get_texture(CameraServer::FEED_CBCR_IMAGE);

						VS::get_singleton()->texture_bind(camera_Y, 0);
						VS::get_singleton()->texture_bind(camera_CbCr, 1);

						storage->shaders.copy.set_conditional(CopyShaderGLES3::SEP_CBCR_TEXTURE, true);
						storage->shaders.copy.set_conditional(CopyShaderGLES3::YCBCR_TO_SRGB, true);
					};

					storage->shaders.copy.bind();
					storage->shaders.copy.set_uniform(CopyShaderGLES3::DISPLAY_TRANSFORM, feed->get_transform());

					_copy_screen(true, true);

					//turn off everything used
					storage->shaders.copy.set_conditional(CopyShaderGLES3::USE_DISPLAY_TRANSFORM, false);
					storage->shaders.copy.set_conditional(CopyShaderGLES3::DISABLE_ALPHA, false);
					storage->shaders.copy.set_conditional(CopyShaderGLES3::SRGB_TO_LINEAR, false);
					storage->shaders.copy.set_conditional(CopyShaderGLES3::SEP_CBCR_TEXTURE, false);
					storage->shaders.copy.set_conditional(CopyShaderGLES3::YCBCR_TO_SRGB, false);

					//restore
					glEnable(GL_BLEND);
					glDepthMask(GL_TRUE);
					glEnable(GL_DEPTH_TEST);
					glEnable(GL_CULL_FACE);
				} else {
					// don't have a feed, just show greenscreen :)
					clear_color = Color(0.0, 1.0, 0.0, 1.0);
				}
				break;
			default: {
//...
//..
}

If I'm right here, this means we should implement our own .submodules/godot/modules/camera/camera_x11.cpp (probably by borrowing some snippets from godot-webcam). @kanetw thoughts?