Mixed Reality Robot Control

This project implements a robot control for a Fischertechnik TXT robot by using a Mixed Reality simulation on the Microsoft HoloLens 2 and the HP Reverb G2. The AR/VR simulation for these headsets is implemented with the Mixed Reality Toolkit (MRTK) for Unity by Microsoft.

Installation

This guide explains how to setup the devices to get started with the Mixed Reality robot control.

Setting up the Raspberry Pi

The following instructions explain how to setup the Raspberry Pi for the robot control. If the Raspberry Pi is used that was already setup for this project, only steps 7 and 8 are necessary.

1. Turn on the Raspberry Pi and login with user pi and password Mrtk2021.

2. Install the MQTT broker Mosquitto:

   $ sudo apt-get install -y mosquitto
   

3. Install Python3 and pip3:

   $ sudo apt-get install -y python3 pip3
   

4. Install the required Python packages for the control:

   $ pip3 install ftrobopy paho-mqtt
   

5. Restart the Raspberry Pi with sudo reboot to start Mosquitto in order to accept commands.

6. Clone the repository and change into the HighBayStorageRack directory:

   $ git clone https://github.com/mortenterhart/mixed-reality-robot-control
   $ cd mixed-reality-robot-control/HighBayStorageRack
   

   This directory contains the Python script hbsr.py that receives the commands and redirects them to the robot.

7. Connect the Raspberry Pi to the robot's Wifi ft-txt_5528. Instructions for this can be found here. The password is displayed at the robot.

8. Run hbsr.py to start the robot control:

   $ python3 hbsr.py
   

Now the Raspberry Pi is ready to accept commands from the headsets and redirect them to the robot.

Setting up the Headsets

The following instructions explain how to transfer the AR and VR simulation of the control from the PC to the Microsoft HoloLens 2 and the HP Reverb G2. To execute the control, install Visual Studio 2019 with the workloads Development with C++ and Development for Universal Windows Platform (UWP). More information on the required software can be found here.

1. Download the Unity builds for AR and VR from https://github.com/mortenterhart/mixed-reality-robot-control/releases/latest and extract the ZIP files to your disk.

For Microsoft HoloLens 2:

2. Turn on the headset and connect it to the PC with USB.

3. Enable the developer mode on the HoloLens. Instructions for this can be found here.

4. Connect the HoloLens to the robot's Wifi ft-txt_5528.

5. Open the solution file mr-robot-control.sln from the AR build in Visual Studio.

6. Select the run configuration Master, the platform ARM64 and the deployment target Device:

   

7. Start the simulation on the HoloLens using the menu Debug 🡒 Start without Debugging or press Ctrl+F5.

For HP Reverb G2:

2. Connect the headset to the PC over USB and plug in the power cable.

3. Connect the PC to the robot's Wifi ft-txt_5528.

4. Open the solution file mr-robot-control.sln from the VR build in Visual Studio.

5. Select the run configuration Master, the platform x86 and the deployment target Local Computer:

   

6. Start the simulation on the HP Reverb G2 using the menu Debug 🡒 Start without Debugging or press Ctrl+F5.

The robot control is now executed on the headsets and can submit commands to the robot over the Raspberry Pi. An important requirement for this is that all devices are properly connected to the robot's Wifi.

Helpful Resources

Installation

• Fischertechnik TXT Community Firmware
• Installing the tools for MRTK

Getting Started

• Choosing Unity Version and XR Plugin
• Creating an MRTK project in Unity for HoloLens 2
• Using the Mixed Reality OpenXR Plugin

Development

• Unity Development for HoloLens
• Unity Development for VR and Windows Mixed Reality

Deployment

• Using Visual Studio to deploy and debug
• Using the HoloLens Emulator
• Using the Windows Mixed Reality Simulator for immersive headsets
• Recommended Unity Settings

Examples

• Examples for HoloLens 2
• Sample Unity projects for OpenXR
• Periodic Table of the Elements