intel / intel/ros2_openvino_toolkit
LLM usecase support
- Dominant language
- C++
- Stars
- 187
- Forks
- 89
- PR merge metrics
- No merged PRs in 30d
Description
We need a LLM feature integrated into this project:
- Input: A ROS String Topic (see [topic template](https://docs.ros.org/en/ros2_packages/rolling/api/std_msgs/interfaces/msg/String.html)) containing an instruction to AMR or manipulator arm.
- Output: A ROS topic ([template](https://docs.ros.org/en/noetic/api/geometry_msgs/html/msg/PoseStamped.html))containing a navigation goal for mobile robot or a grasp pose for manipulator arm.
This repo should be added new code and logic to fulfill this Input->Inference->Output process: 1) Subscribe ROS topic of related audio common message. 2) Adopt LLM agent for the inference body. 3) Publish the corresponding topic(s).
See below picture as the pipeline:

LLM like Llama support can leverage OpenVINO-GenAI, please refer to [OpenVINO™ GenAI Library](https://github.com/openvinotoolkit/openvino.genai/blob/master/src/README.md)
Development Scope:
- Wrap the audio to ROS [topic](https://docs.ros.org/en/noetic/api/audio_common_msgs/html/index-msg.html).
- Use STT model convert audio to text [**optional**]
- Subscribe a ROS string topic and convert to input type of LLM agent.
- Prompt Intelligent agent to adapt usage (like navigation)
- Wrap the decode results to ROS topic, like [PoseStamped](https://docs.ros.org/en/noetic/api/geometry_msgs/html/msg/PoseStamped.html).
You have the freedom to choose the intelligent agent.
Notes: Maybe here need a end-to-end LLM model like some VLM(Visual Language Model, for example [VLM](https://github.com/TRI-ML/prismatic-vlms)) models. Anyway, this is not easy to enable such a VLM model on Intel HW platform since there is no reference sample in OpenVINO currently.
Contributor guide
Assessment
This issue has not been assessed yet.