COTORRA: COntext-aware Testbed fOR Robotic Applications
Edge & Fog computing have received considerable attention as promising candidates for the evolution of robotic systems. In this letter, we propose COTORRA, an Edge & Fog driven robotic testbed that combines context information with robot sensor data to validate innovative concepts for robotic systems prior to being applied in a production environment. In lab/university, we established COTORRA as an easy applicable and modular testbed on top of heterogeneous network infrastructure. COTORRA is open for pluggable robotic applications. To verify its feasibility and assess its performance, we ran set of experiments that show how autonomous navigation applications can achieve target latencies bellow 15ms or perform an inter-domain (DLT) federation within 19 seconds.
Networked robotics combines robotics with communication networks to enhance capabilities of robots. A clear example is Cloud robotics , which aims to integrate Cloud computing resources in robotics systems to increase re-configurability as well as to decrease the complexity and cost of robots. Thanks to Edge  & Fog  computing, real-time applications as autonomous navigation will run closer to the robots, and the robot-to-cloud communication will be enhanced thanks to improved latency and reliability. Since the robot-to-cloud communication will traverse fewer network links, higher bandwidth rates, and bounded jitter are achievable. Moreover, real-time context information about the robots is expected to be available in the Edge, allowing dynamic adaptation of application’s logic to the actual status of the communication (e.g., radio channel) . Therefore, executing robotics applications in the Edge of the network is considered as a potential evolution of robotics system.
Most of the existing work focuses on Cloud robotics testbeds or platforms [7, 10, 5, 11, 8, 13]. In , the authors indicate the feasibility and challenges of a real world Cloud robotics implementation over Cloud testbed infrastructure. Besides,  suggests a Cloud robotics testbed that can be effective for developing not only robot-related experiments but also network applications. By providing a Cloud robotics testbed for mobile robots, VC-bots  tries to address some important factors such as mobility, traffic scenario, protocol, cloud resources and localization awareness. Furthermore, due to the set of constrains (e.g., cost, time, safety) when performing physical testing of Unamanned Areal Vehicles,  develops an Cloud-enabled, open-source simulation testbed for UAVs, thus reducing the entry barrier of UAV development and research. Finally,  and  propose Cloud robotics platforms that address number of implementation-related issues.
However, it is relevant to note that all the existing robotics testbeds or platforms depend on the use of Cloud computing. Unfortunately, no Edge & Fog robotics testbeds are explicitly mentioned in the existing literature, despite the need for an analysis on implementation issues, standards and validation for Edge robotics applications. There is a lack of practical experience on context-aware real-time environments in robotics systems. That leads to limited applicability in production systems and final products.
In this work we propose COTORRA, a real-time context-aware testbed architecture that we implement and validate. This testbed is particularly suitable for time-sensitive robotic applications and mobile robots in Edge & Fog environment. Additionally, to consider the time-sensitivity in computational offloading decisions, the COTORRA testbed offers network emulation capabilities. To validate COTORRA, we implemented two applications on top of it: an orchestration and a federation solution for robotic systems. The experiments and results illustrate how COTORRA can be used to enhance robotic applications.
Ii COTORRA system design
COTORRA system model follows a modular design as
shown in Fig. 1.
The hardware layer (bottom) is a weighted graph
with graph embeddings
Ii-a COTORRA core layer building blocks
This section presents the functionality of each COTORRA core layer building block, and explains the exchange of information among layers and COTORRA core layer building blocks.
The Robot Middleware (a) collects, (b) stores and (c) exposes real-time robots’ data; and (d) sends instructions to the robots, e.g., robots’ movements. The Robot Middleware periodically collects the robots’ Radio Unit (RU) attachment , and a vector of sensor data holding values such as the robot position , with being the space dimension.
4Using the aforementioned information, the Robot Middleware creates a contextual embedding 5 (1)
that is stored (via the Measurements API) in the Measurements building block, and exposed (via its own API) to the plug-in layer (see Fig. 1). Latter, a plug-in uses the contextual embedding to elaborate real-time navigation (e.g., robot movement) and RU attachment instructions , and forwards them to the Robot Middleware API. Finally, Robot Middleware sends the plug-in instructions to the robots.
The Network Control (a) collects, (b) exposes, and (c) stores network and RU context information. One of its main functionalities is to (d) emulate network conditions. As well, it (e) maps Virtual Network Functions (VNF) and Virtual Links (VL) in the hardware graph, (f) stores, and (g) exposes the VNF/VL mappings. The Network Control collects a vector with RUs’ context information , so as the links’ throughput , and delay between every robot, RU, switch, and server in the hardware graph . In addition, the Network Control exposes the real-time RU context information, links’ delay, and throughput via its API; and stores the prior data in the Measurements entity using the Measurements API. To emulate network conditions, Network Control (i) retrieves throughput, and delay embeddings from the Measurements entity; and (ii) introduces artificial queuing delay so as packet drop rate . As a result, the links in the hardware graph offer a delay of , and a throughput of . The Network Control API exposes operations to request both, queueing delay, and packet drop.
Additionally, the Network Control acts as a Virtual Infrastructure Manager (VIM)  that (i) maps VNFs to hardware nodes ; and (ii) maps VLs in the hardware links . Both VNFs, and VLs mappings are periodically (h) stored and (i) exposed by the Measurements component (using the Measurements API); and plug-ins can issue new mappings over the exposed Network Control API.
The Measurements building block (a) stores, and (b) exposes a history of every embedding. The embeddings history is represented as a set
with denoting the contextual embedding of robot at time . As shown in (2), the embeddings history stores the VNF/VL mappings, network delay, and throughput information. The embeddings history is exposed via the Measurements API to the plug-in layer (see Fig. 1).
Ii-B COTORRA applicability: an example use case
COTORRA is designed following the principles and definitions of some of the most known implementations of Edge and Fog computing such as Multi-access Edge Computing (MEC)  and OpenFog working group in Industrial Internet Consortium (IIC) . COTORRA building blocks, namely, Robot Control, Network Control, Measurements and Plug-ins can be represented as autonomous applications and context-aware services running on top of a virtualization infrastructure. This implies that by using COTORRA we are contributing to the experimental evaluation of MEC and IIC compliant applications.
Lets consider an Edge robotic autonomous navigation scenario where the robot driver VNF runs on the robot , and an autonomous navigation VNF runs on an Edge server . A plug-in can use the Network Control and Measurements APIs to obtain RUs’ context information (e.g., RSSI, throughput, etc.) and Robot Middleware to obtain robot sensor data (e.g., odomety , LIDAR, and camera data). Based on this information plug-ins can be designed, developed and tested with autonomous navigation algorithms that through the Robot Middleware APIs can adapt the robot speed (present inside ) based on the radio link status . Additionally, packet drop and queuing delay can be emulated using the Network Control to simulate congested network environments.
Iii COTORRA implementation
Fig. 2 shows an implementation of COTORRA testbed based on a mobile robot. This implementation is used to verify the effectiveness of COTORRA in testing innovative mobility features. The COTORRA testbed consists of: (i) x5 ASUS WL500G Premium v1 Access Points (APs) running OpenWrt 18.06.2; (ii) x5 MiniPC, with x4 vCPUs and 8GB of RAM each. Two MiniPCs are used as APs and the rest as servers. Two of the server MiniPCs simulate Edge and Cloud nodes by introducing an artificial queuing delay of 5 and 10 seconds, respectively. The third MiniPC server acts as a local Fog node. All APs and MiniPCs are deployed along a corridor of the Universidad Carlos III de Madrid, interconnected with 10Gbps Ethernet connectivity. For the mobile robot, we used a ROS-compatible Kobuki Turtlebot S2 robot equipped with a laptop with 8-GB RAM and 2 CPUs, and a RPLIDAR A2 lidar for 360-degree omnidirectional laser range scanning.
The COTORRA core layer is implemented using VNFs (shown as red rounded boxes on Fig. 2). The Robot Middleware is implemented using Robotic Operating System (ROS) to (i) collect and expose the lidar and odometry sensor data to the plug-ins layer, and (ii) send to the robots the navigation instructions received from the plug-ins. In addition, as part of the Robot Middleware, we use a Localization service to provide probabilistic robot localization
based on the lidar data. The Network Control is implemented using a WLAN Access Information Service (WAIS) to provide real-time context information (e.g., signal level , transmission and reception bit rates , etc.) through a REST API. In addition, Network Control emulation is implemented with NetEm, so as to introduce artificial latency and packet loss.
And for the Network Control VIM functionalities, we have used Fog05
Iv Experimental validation
In this section we validate COTORRA testbed by an evaluation of (an orchestration and a federation) applications on top of the COTORRA implementation described in Section III.
Iv-a Service orchestration that meets key performance indicators
OKpi  is a service orchestration heuristic that meets latency, and mobility requirements. OKpi was implemented as a plug-in interacting with COTORRA to orchestrate an autonomous navigation service as described in Section II-B. In the experiment, OKpi used the Localization service to obtain the real-time robot position , and MySQL for network links delays’ . Based on that, it sent (i) to fog05 the autonomous navigation VNF mapping , and (ii) 802.11r handovers to the robot as it traveled along a corridor (see Fig. 2 dashed trajectory).
The experiment goal was to compare
a State Of the Art (SoA) autonomous driving with
periodic probing for RU attachment,
against the theoretical
OKpi-t and empirical OKpi-e performance
of OKpi; and if the latter achieved a service time
Iv-B DLT federation
In , we designed a federation of a robotic service using a Distributed Ledger Technology (e.g., Blockchain). Service federation is a set of procedures that enable orchestration of services across multiple administrative domains. We used COTORRA testbed to deploy the DLT solution and perform experimental evaluation of the multi-domain federation. The experiment consisted of a moving robot towards out-of-coverage area of Domain 1 (on Fig. 2) and extending the connectivity range, on-the-fly, in Domain 2. An orchestrator plug-in was deployed in each domain, and a DLT plug-in was used for the inter-domain interaction.
The mobile robot moved from a starting position (similar as on Fig. 2) towards the coverage area of Domain 2. The Robot brain, analyzed the location data from the Localization service, and triggered a federation procedure through the Domain 1 orchestrator. Both orchestrators negotiated and established deployment of a vAP through the DLT plug-in, and the deployment decision was sent to fog05. The mobile robot attached to the newly deployed vAP in Domain 2 without any movement or robotic service interruption. As shown on Fig. 4, it took around 19 seconds to complete the described federation procedure.
In this letter, we introduce COTORRA, a modular tested built to enable and validate innovative mechanisms or applications in robotic system. Additionally, we showed that COTTORA can be easily implemented over commodity hardware that offers support for a rapid prototyping and validation. The network emulation feature in COTORRA can simulate unpredictable network conditions that enables near production robotic environment where new pluggable applications can be tested. We validated this by running experiments for orchestration algorithms that interact with both the network infrastructure, and mobile robots. Finally, the flexibly and scalability of COTORRA enabled us to showcase an application of DLT for federation in an Edge robotic service, guaranteeing service footprint extension.
- todo: Milan: edit the figure to: (i) show real time data consumption and historical data consumption, (ii) add link between the robot COTORRA mgmt and the measurements, (iii) point out the close loop feature in system. Both the robot closed loop and the infrastructure one
- COTORRA uses node and edge embeddings of the hardware graph .
- todo: KIRIL: After the first sentence, everything is confusing. Can we just say that the plugins are applications that can be used for robotic services e.g., that send instructions to the robots and receive feedback from the sensors via the COTORRA core. And to note that the core layer is crucial inter-working layer that makes it easier to extend the hardware or plug different plugins as well as emulate different conditions to test the dedicated plugins. And then explain the building blocks of the core layer that do so. Otherwise I get lost. Milan: we remove the fist paragraph of this subsection
- Wheeled robots move on a space, and UAVs on a space.
- todo: KIRIL: Maybe is me, but I don’t understand the meaning of ”elaborates a contextual embedding”. Does it mean like gathers and stores in to a json-like structure? Or it is something else? It confuses me. Can it be: ”creates a contextual structure” JORGE: an embedding is a function in the graph domain, e.g., in this case it is a function taking a node (robot) as input, and giving as output a vector. This function is the contextual embedding.
- Fog05 project: https://fog05.io/ [Accessed: 26 November 2020]
- Elapsed time between the emission of robot position packets, and the reception of robot navigation packets.
- (2020) DLT federation for edge robotics. In 2020 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN), pp. 71–76. Cited by: §IV-B.
- (2014) Networks and devices for the 5g era. IEEE Communications Magazine 52 (2), pp. 90–96. Cited by: §I.
- (2019) Network Transformation; (Orchestration, Network and Service Management Framework). Note: V1 Cited by: 2nd item.
- (2018) IEEE approved draft standard for adoption of openfog reference architecture for fog computing. IEEE P1934/D2.0, April 2018 (), pp. 1–175. Cited by: §II-B.
- (2016) VC-bots: a vehicular cloud computing testbed with mobile robots. In Proceedings of the First International Workshop on Internet of Vehicles and Vehicles of Internet, IoV-VoI ’16, New York, NY, USA, pp. 31â36. External Links: Cited by: §I.
- (2020) OKpi: All-KPI Network Slicing Through Efficient Resource Allocation. In IEEE INFOCOM 2020 - IEEE Conference on Computer Communications, Vol. , pp. 804–813. External Links: Cited by: §IV-A.
- (2019) Cloud robotics experimentation testbeds: a cloud-based navigation case study. In 2019 IEEE 4th Colombian Conference on Automatic Control (CCAC), Vol. , pp. 1–6. External Links: Cited by: §I.
- (2015) Rapyuta: a cloud robotics platform. IEEE Transactions on Automation Science and Engineering 12 (2), pp. 481–493. External Links: Cited by: §I.
- (2018) A comprehensive survey on fog computing: state-of-the-art and research challenges. IEEE Communications Surveys Tutorials 20 (1), pp. 416–464. Cited by: §I.
- (2019) Open cloro: an open testbed for cloud robotics. In 2019 IEEE 24th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD), Vol. , pp. 1–5. External Links: Cited by: §I.
- (2018) OpenUAV: a uav testbed for the cps and robotics community. In 2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS), Vol. , pp. 130–139. External Links: Cited by: §I.
- (2016) Edge computing: vision and challenges. IEEE Internet of Things Journal 3 (5), pp. 637–646. Cited by: §I.
- (2015) Rospeex: a cloud robotics platform for human-robot spoken dialogues. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. , pp. 6155–6160. External Links: Cited by: §I.
- (2017) On multi-access edge computing: a survey of the emerging 5g network edge cloud architecture and orchestration. IEEE Communications Surveys Tutorials 19 (3), pp. 1657–1681. Cited by: §II-B.
- (2016-01) Cloud robotics: current status and open issues. IEEE Access 4, pp. 1–1. External Links: Cited by: §I.