Skip to content

Code Documentation

This document provides an overview of the code of the helloric_ui_com package.

HelloRICMgr

The HelloRICMgr class is a ROS2-enabled WebSocket connection manager that acts as a bridge between the ROS2 ecosystem and web-based user interfaces. It extends the WebSocketConnectionManager to provide real-time communication capabilities between ROS2 nodes and WebSocket clients for the HelloRIC robot interface.

Note: The manager leverages FastAPI's WebSocket support and integrates with ROS2 services to provide bidirectional communication for audio processing, tour management, and emotion display.

Attributes

The manager maintains the following key attributes:

Attribute Type Description
ros_node ROSCom The ROS2 communication node for service calls
tour_name List[str] List of available tour names
tour Optional[Dict[str, object]] Current tour data
success bool Status flag for operation success

Main Loop

The manager runs a continuous main loop that processes ROS2 messages:

async def main_loop(self):
    while rclpy.ok():
        if self.ros_node:
            rclpy.spin_once(self.ros_node, timeout_sec=0.1)
        await asyncio.sleep(0.01)

This loop ensures that ROS2 callbacks are processed while maintaining WebSocket responsiveness.

Client Management

client_defaults(user_id: str)

  • Description: Sends initial default state to a newly connected client.
  • Parameters:
  • user_id (str): Unique identifier for the WebSocket client
  • Default State:
  • microphoneActive: True
  • emotion: Emotion.CALM.value

set_microphone_state(active: bool, user_id: str | None = None)

  • Description: Updates the microphone state for clients.
  • Parameters:
  • active (bool): Whether the microphone should be active
  • user_id (str | None): Specific client ID, or None to broadcast to all clients
  • Behavior: Sends JSON message with microphoneActive field

Audio Message Handling

send_audio_msg(audio_bytes, text, language, emotion, name=None)

  • Description: Sends audio messages to the frontend with synchronized metadata and audio data.
  • Parameters:
  • audio_bytes (array[int]): Raw audio data
  • text (str): Text content of the audio message
  • language (str): Language identifier
  • emotion (str): Emotion identifier for the message
  • name (str | None): Target client name, or None to broadcast
  • Protocol:
  • Generates unique message ID based on timestamp
  • Sends JSON metadata with emotion, text, and message ID
  • Sends binary audio data with 4-byte message ID header
  • Message Format:
  • JSON: {"emotion": str, "text": str, "messageId": int}
  • Binary: [4-byte message ID][audio data]

How it Works

  1. Initialization: The manager initializes the WebSocket connection manager and sets up ROS2 integration.

  2. Client Connection: When a client connects via WebSocket, a unique user ID is generated and default state is sent.

  3. Audio Processing Pipeline:

  4. Client sends audio data via WebSocket
  5. Manager forwards audio to ROS2 /chat service
  6. Receives processed audio response with text, language, and emotion

  7. Sends synchronized audio and metadata back to client

  8. Service Commands: Clients can send JSON commands (e.g., "clear_history") that are forwarded to appropriate ROS2 services.

  9. Real-time Communication: The main loop ensures continuous processing of ROS2 messages while maintaining WebSocket connections.

Tour Data Models

TourStepModel

  • Description: Pydantic model representing a single step in a tour.
  • Fields:
  • description (str): Human-readable description of the step
  • destination (Optional[str]): Target location for the step
  • dialogue (Optional[str]): Speech content for the step

TourModel

  • Description: Pydantic model representing a complete tour.
  • Fields:
  • tour_name (str): Unique identifier for the tour
  • map_name (str): Associated map identifier
  • steps (List[TourStepModel]): Ordered list of tour steps

Tour Conversion Functions

convert_tour_to_ros(tour: TourModel) -> Tour

  • Description: Converts a Pydantic tour model to a ROS2 Tour message.
  • Parameters:
  • tour (TourModel): Input tour model
  • Returns: Tour - ROS2 message format
  • Process: Maps Pydantic fields to ROS2 message fields and creates Step messages for each tour step

convert_ros_to_tour(ros_tour: Tour) -> Dict

  • Description: Converts a ROS2 Tour message to a dictionary format.
  • Parameters:
  • ros_tour (Tour): Input ROS2 tour message
  • Returns: Dict - Dictionary representation suitable for JSON serialization
  • Process: Extracts tour metadata and converts step messages to dictionary format

WebSocket Initialization (init_websocket)

The init_websocket function sets up the WebSocket endpoint and message processing pipeline.

WebSocket Endpoint (/ws)

  • Description: Main WebSocket endpoint for real-time communication with clients.
  • Features:
  • Automatic client registration and cleanup
  • Task management for concurrent audio processing
  • Error handling and disconnection management
  • Support for both binary (audio) and text (JSON) messages

Message Processing

process_audio(audio_bytes, user_id)

  • Description: Processes incoming audio data from clients.
  • Parameters:
  • audio_bytes (array[int]): Raw audio data from client
  • user_id (str): Client identifier
  • Process:
  • Disables client microphone during processing
  • Creates /chat service request with audio data
  • Calls ROS2 chat service asynchronously
  • Sends processed audio response back to client
  • Re-enables client microphone
  • Error Handling: Resets client to default state on errors

process_json(data, user_id)

  • Description: Processes JSON commands from clients.
  • Parameters:
  • data (dict): JSON command data
  • user_id (str): Client identifier
  • Supported Commands:
  • clear_history: Calls ROS2 /clear_history service

HTTP Endpoints (add_http_endpoints)

The system provides RESTful HTTP endpoints for tour management.

/tour (GET)

  • Description: Retrieves list of available tours.
  • Response: List[str] - Array of tour names
  • ROS2 Service: /tour_list

/tour/{tour_name} (GET)

  • Description: Retrieves specific tour details.
  • Parameters: tour_name (str) - Name of the tour to retrieve
  • Response: Dict - Tour data with steps and metadata
  • ROS2 Service: /tour_get
  • Error Handling: Returns 404 if tour not found

/tour (POST)

  • Description: Creates a new tour.
  • Request Body: TourModel - Complete tour definition
  • Response: {"success": bool} - Creation status
  • ROS2 Service: /tour_create

/tour (PUT)

  • Description: Updates an existing tour.
  • Request Body: TourModel - Updated tour definition
  • Response: {"success": bool} - Update status
  • ROS2 Service: /tour_update

/tour/{tour_name} (DELETE)

  • Description: Deletes a specific tour.
  • Parameters: tour_name (str) - Name of the tour to delete
  • Response: {"success": bool} - Deletion status
  • ROS2 Service: /tour_delete

FastAPI Initialization (init_fastapi)

Function Signature

def init_fastapi(logger, loop, host="0.0.0.0", port=7000):
  • Description: Initializes the complete FastAPI application with WebSocket and HTTP endpoints.
  • Parameters:
  • logger: Logging instance for application events
  • loop: Asyncio event loop for async operations
  • host (str): Server host address (default: "0.0.0.0")
  • port (int): Server port number (default: 7000)
  • Returns: Tuple[HelloRICMgr, Server, FastAPI] - Manager, server, and app instances

Initialization Process

  1. FastAPI App Creation: Creates FastAPI application instance
  2. Manager Setup: Initializes HelloRICMgr with ROS2 node
  3. WebSocket Integration: Sets up WebSocket endpoints and message handlers
  4. HTTP Endpoints: Adds RESTful API endpoints for tour management
  5. Server Configuration: Configures Uvicorn server with specified host and port

Key Features

  • Async/Await Support: Full async support for concurrent operations
  • Real-time Communication: WebSocket support for low-latency interactions
  • RESTful API: Standard HTTP endpoints for tour management
  • ROS2 Integration: Seamless bridge between web clients and ROS2 ecosystem
  • Error Handling: Comprehensive error handling and logging
  • Client Management: Automatic client registration, tracking, and cleanup

ROS Communication Module (ros_com.py)

The ROS communication module provides the bridge between the FastAPI WebSocket server and the ROS2 ecosystem.

Emotion Module (emotion.py)

Emotion Enum

  • Description: Enumeration of supported emotional states for the robot interface.
  • Values: AMUSED, BORED, CALM, EXCITED, FRUSTRATED, HAPPY, SAD, WORRIED, THINKING
  • Usage: Used to standardize emotion representation across the system

to_emotion(value: str) -> Emotion

  • Description: Converts string emotion values to Emotion enum.
  • Parameters: value (str) - String representation of emotion
  • Returns: Emotion - Corresponding enum value, defaults to CALM if not found
  • Process: Normalizes input by stripping whitespace and converting to lowercase

WebSocket Connection Manager (utils.py)

The utility module provides the base WebSocket connection management functionality that HelloRICMgr extends.

WebSocketConnectionManager

  • Description: Base class for managing multiple WebSocket connections with named identification.
  • Features:
  • Named connection tracking
  • Broadcast and unicast messaging
  • JSON and binary message support
  • Automatic UUID generation for unnamed connections
  • Connection lifecycle management