intel / intel/ros2_openvino_toolkit

LLM usecase support

Open
#314 1 comment 0 reactions 0 assignees View on GitHub
hackathon
Dominant language
C++
Stars
187
Forks
89
PR merge metrics
No merged PRs in 30d

Description

We need a LLM feature integrated into this project:
- Input: A ROS String Topic (see [topic template](https://docs.ros.org/en/ros2_packages/rolling/api/std_msgs/interfaces/msg/String.html)) containing an instruction to AMR or manipulator arm.
- Output: A ROS topic ([template](https://docs.ros.org/en/noetic/api/geometry_msgs/html/msg/PoseStamped.html))containing a navigation goal for mobile robot or a grasp pose for manipulator arm.

This repo should be added new code and logic to fulfill this Input->Inference->Output process: 1) Subscribe ROS topic of related audio common message. 2) Adopt LLM agent for the inference body. 3) Publish the corresponding topic(s).

See below picture as the pipeline:

![Image](https://github.com/user-attachments/assets/b8d5307b-4251-42ed-b3ec-a84b900fc851)

LLM like Llama support can leverage OpenVINO-GenAI, please refer to [OpenVINO™ GenAI Library](https://github.com/openvinotoolkit/openvino.genai/blob/master/src/README.md)

Development Scope:
- Wrap the audio to ROS [topic](https://docs.ros.org/en/noetic/api/audio_common_msgs/html/index-msg.html).
- Use STT model convert audio to text [**optional**]
- Subscribe a ROS string topic and convert to input type of LLM agent.
- Prompt Intelligent agent to adapt usage (like navigation)
- Wrap the decode results to ROS topic, like [PoseStamped](https://docs.ros.org/en/noetic/api/geometry_msgs/html/msg/PoseStamped.html).

You have the freedom to choose the intelligent agent.
Notes: Maybe here need a end-to-end LLM model like some VLM(Visual Language Model, for example [VLM](https://github.com/TRI-ML/prismatic-vlms)) models. Anyway, this is not easy to enable such a VLM model on Intel HW platform since there is no reference sample in OpenVINO currently.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.