Connecting the Dots: From Data to Knowledge in Human–Robot Interaction

Robots can recognise people, detect movements, and analyse their surroundings. But how can they bring all this information together to understand what is happening around them?

In environments where humans and robots work alongside one another, information comes from many different sources. Visual analysis can identify gestures, body poses, and facial expressions, while other foundational model-based solutions provide information about speech, human intentions, and the surrounding environment. Each source contributes a different perspective on human behaviour and the environment, but understanding a situation requires more than processing these signals independently. It requires connecting them, identifying their relationships, and interpreting them within a shared context. Addressing this challenge is a key objective of the Human State Analyzer (HSA) developed within the TORNADO project.

At the heart of this effort, the Foundation for Research and Technology – Hellas (FORTH) is developing a novel Human–Robot Interaction (HRI) ontology and a dynamic Knowledge Graph to bring heterogeneous information into a common representation. The ontology establishes a structured view of people, robots, objects, activities, and their relationships, providing a consistent way to represent information across different environments, from industrial facilities to healthcare settings. Building on this foundation, FORTH’s Knowledge Graph Generator processes incoming data and transforms it into an interconnected network of entities and relationships. The approach combines Large Language Models (LLMs) for extracting information, ontology-based semantic matching, and vector embeddings for identifying connections between semantically related entities. Rather than simply storing observations, the resulting graph organises them into a structured representation of the situation that can be continuously updated as new information becomes available.

But organising knowledge is only part of the challenge. The next step is making use of it. FORTH is also developing a Knowledge Graph Reasoner, designed to retrieve and interpret relevant information from the graph in response to requests for human or social context. The system combines graph-based retrieval techniques with LLM-driven processing to identify information relevant to a query, explore connections between entities, and formulate context-aware responses. By grounding these responses in information retrieved from the Knowledge Graph, the approach aims to support more informed reasoning about the people, activities, and circumstances represented within it.

Together, these developments form an important part of TORNADO’s approach to human-aware robotics. By transforming diverse observations into connected knowledge and making that knowledge available to other robotic modules, FORTH’s work aims to support a richer understanding of human behaviour and the surrounding environment. As development and integration progress, these capabilities will contribute to robotic systems that can better interpret the context in which they operate and adapt their behaviour to the people and situations around them.

Leave a Reply