1. Introduction

Augmented Reality for Assembly (AR system) is aimed at increasing the value of an operator on the shop floor as well as to lower the difficulty of assembly for new operators. This is done by projecting the work instructions for subsequent assembly steps in front of the operator such that the operator does not have to search for information. By this method,  the operator is relieved of mental strain and can focus on the manipulations needed for assembly. The digital system takes care of the process and the human is used in its strength which is flexibility.

  1. System delineation

a. Context

The AR system resides in a factory where assembly of complex assemblies is required. Here it works together with the MPMS, HORSE middleware and other HORSE components. The operator working with the AR system receives visual instructions about how to perform certain assembly steps.

b. Scope

The AR system provides the instructions to the operator for assembly. It does not provide the physical tools required for assembly nor the parts that have to be assembled.

c. Operational overview

A task for the AR system is instantiated from the MPMS and provided via the HORSE middleware to the local task manager and AR system. Upon receiving this information, it is queued in the local task manager that takes care of the synchronisation between actors in the work cell. When the work cell is ready for the new task, the AR system retrieves the required working instructions corresponding to the task from the database. AR guidance for assembly begins and when the operator has finished the task, this information is transmitted again via the HORSE middleware to the MPMS.

  1. Requirements and functions


Figure 1 The conceptual framework of HORSE requirements (figure 41 D2.1)


System Functions

Stakeholder expectation

System Function

Functional requirement

The HORSE system should make it possible for humans to work in close proximity to pre-determined automated actors.

SF-01: Work cell situation awareness


The HORSE System should be able to perform synchronised activities with human actors.

Rotate actors for manufacturing tasks, based on cumulative workload, capabilities and availability of actors and system status

SF-04: Task allocation between human and automated actors based on workload




SF-05: Ergonomic work station supporting user friendly interaction


The HORSE System should present them instructions of their tasks
The HORSE System should provide detailed work lists and operating procedures to improve output quality.

SF-09: Augmented reality manufacturing tasks


The HORSE System should provide the operator with necessary information to visually inspect pre-defined critical product parameters.

SF-10: Augmented reality to assist visual inspection


The HORSE System should allocate tasks between robots and workers according to their capabilities.

SF-04: Task allocation based on resource characteristics



The HORSE system should visually display the current status of the manufacturing system (status of production orders, occupation of actors, process performance), based on information received from other systems and actors.

SF-14: Monitoring and control of work progress of all actors in the manufacturing process


Functional requirements


Functional requirement



The HORSE System shall be able to monitor for every Robotic Actor:

a) its capabilities, including at least a.1) maximum load, and a.2) availability,

b) its performance , including at least b.1) task actual completion time, b.2) task estimated completion time, b.3) task successful execution estimation


The Robotic Actor shall be able to accept commands, (including task orders and configuration) via a portable device.


The Robotic Actor shall be able to present the state of production task to the Production human actor remotely


The HORSE System shall be able to present information about the production process to the Production human actor via a portable or wearable device, including at least a) alerts and b) task guidelines.


The HORSE System shall be able to present to the Operator visual captures of a predefined object.


The HORSE System shall be able to select and allocate appropriate actors to tasks based on their specific capabilities.


The HORSE System shall be able to manage operational mode and the task state, for all actors.


The HORSE System shall be configurable


The HORSE system shall not harm humans.


  1. System description and demonstration scenario

Another setup of HORSE Framework has been assembled for validation and demonstration in the official HORSE test site in the Thomas Regout International factory in Maastricht, Netherland. An Augmented Reality solution that gives instructions for assembling the tool, superimposed on real tool parts was tested in “P1” area. This includes an advanced planning system (MPMS) to help with planning change overs with minimal reconfiguration of tooling, as well as a mobile platform to fetch tooling blocks.


Figure 2 Schematic overview of the actors and their tasks in this scenario


Figure 3: Overview photo of the on site testing at Thomas Regout International


Figure 4: Work desk with AR for TRI P1

  1. Download information

Customer specific implementation is very high, for that reason, an example implementation of the framework is provided that allows the integrator to have a software stack from MPMS to the local component running. The actions of the local component, which was AR for assembly in this case, can be easily defined in Python.


1. Configuration 


XML-file with the basic configuration for the Horse middle ware.

2. MPMS example process


A simple MPMS example process is included in the package that can interface with the other parts in the package.

Figure 5 Simple MPMS program (human_robot_simple.bpmn)

3. Python script to start MPMS task


This simple script allows to start an MPMS task remotely without using the GUI of the MPMS itself. It is especially valuable during debugging and integration stages where the whole software stack needs to be tested.

Python local component

Script that simulates a local component (e.g. AR) and has the required interfaces to the middleware to communicate with the MPMS. This script listens for new tasks. Upon receiving a task it automatically completes it and sends back the result. Connection to the horse middleware is configured via the configuration script (

Other examples

The package contains more files that we created during the project to test various functionality.

Download link