Whole-Body Bilateral Teleoperation of a Redundant Aerial Manipulator
Attaching a robotic manipulator to a flying base allows for significant improvements in the reachability and versatility of manipulation tasks. In order to explore such systems while taking advantage of human capabilities in terms of perception and cognition, bilateral teleoperation arises as a reasonable solution. However, since most telemanipulation tasks require visual feedback in addition to the haptic one, real-time (task-dependent) positioning of a video camera, which is usually attached to the flying base, becomes an additional objective to be fulfilled. Since the flying base is part of the kinematic structure of the robot, if proper care is not taken, moving the video camera could undesirably disturb the end-effector motion. For that reason, the necessity of controlling the base position in the null space of the manipulation task arises. In order to provide the operator with meaningful information about the limits of the allowed motions in the null space, this paper presents a novel haptic concept called Null-Space Wall. In addition, a framework to allow stable bilateral teleoperation of both tasks is presented. Numerical simulation data confirm that the proposed framework is able to keep the system passive while allowing the operator to perform time-delayed telemanipulation and command the base to a task-dependent optimal pose.
The use of unmanned aerial vehicles (UAV) as a flying base for robotic manipulators has been object of intensive research in the recent years [19, 24, 11, 20]. One of the main goals of such systems is to replace and assist humans in tasks as inspection and repairing of bridges, high-voltage electric lines, and wind-turbine blades [19, 24]. The use of such systems help overcome a series of limitations of fixed-based robots, e.g., limited workspace. Some recent works focused on performing aerial manipulation tasks autonomously [28, 12, 10]. However, in order to more deeply explore the manipulation capacity of such systems, telemanipulation arises as an alternative [7, 17, 9, 14]. Adding a human operator to the loop allows for the simplification of many of the system features, especially in what concerns perception and reasoning.
Nevertheless, during field experiments performed in the scope of the AEROARMS project [19, 14], an additional condition for aerial telemanipulation has been found. Since tasks are usually performed without direct eye contact, the use of video cameras to stream images to the distant operator is a necessary add-on to haptic feedback. Adding to that, in order to facilitate the operator’s reasoning, a first-person or eye-to-hand view, i.e., a view of the entire arm, has been found beneficial. In the case of aerial manipulators, like the DLR Suspended Aerial Manipulator (SAM) [27, 26, 14], a camera is usually attached to the flying base in order to provide an eye-to-hand view (see Fig. 1). If correctly positioned, the camera image allows the operator to have a view of the arm and the area of interest for manipulation.
In case the aerial manipulator is redundant, there might be some freedom to move the base and, consequently, the video camera without affecting the end-effector. For such systems, defining the camera pose autonomously could allow for a better view to the operator. However, since its optimal location is rather subjective and task-dependent, it would be reasonable to provide the operator with some authority to control that task without disturbing the end-effector pose. Adding to that, since the set of possible base poses for a given end-effector pose is usually a smaller subset of the Cartesian-space, the master device can be used in order to inform the operator if the pose to which the base is being commanded is approaching the limits of that subset.
In light of that, this paper proposes a framework to allow a human operator to teleoperate the flying base of a redundant aerial manipulator in order to achieve a desired view of the task being performed while receiving meaningful haptic feedback about the reachability of the commanded pose without disturbing the end-effector. For that purpose, a previously presented whole-body control framework is applied in order to define a task hierarchy, where the positioning of the base is performed in the null space  of the main telemanipulation task. Adding to that, a novel haptic concept called Null-Space Wall is introduced in order to inform the operator in an intuitive way whether the pose to where the base is being commanded is reachable. This concept is defined in such a way that no additional fixtures have to be added in order to create haptic feedback. Elseways, the already computed forces from the whole-body controller are explored. The use of haptic feedback in order to constrain the commanded motion has been extensively explored in the literature (e.g., [3, 16, 15]). However, to the best of the authors’ knowledge this is the first time that such a technique is applied in order to allow hierarchical whole-body telemanipulation of redundant robots.
In order to evaluate the proposed framework, bilateral teleoperation of the SAM (see Fig. 1) is simulated with round-trip time-delays up to 300 ms. The stability and efficacy of the approach in providing the user with meaningful haptic information about the limits of the null space can be verified in the data.
Ii Time Domain Passivity Approach
Time Domain Passivity Approach (TDPA) [8, 25, 6] was developed in order to enforce stability of both haptic and bilateral teleoperation setups where velocity and force signals are exchanged. In teleoperation systems, time-delay and package losses introduced by the channel might compromise the overall stability of the system, whereas in haptics instability can be a result of sampling.
In bilateral teleoperation setups, the communication channel is usually represented by one or more Time Delay Power Networks (TDPNs, ), which are two-port networks that exchange velocities and forces. In addition to constant or variable time-delays, TDPNs can also model package losses in the signals being transmitted. Fig. 2 shows the signal flow of the TDPN. and are the energies computed on the master and slave sides, respectively. The and subscripts are used to represent the direction of flow, namely into or out of the channel.
The pairs / and / from Fig. 2 are the flow-effort pairs on each side of the TDPN, such that in coordinates
where is the sampling time and is the discrete-time index.
A sufficient condition for passivity of a TDPN is that
where and are the observed left-to-right and right-to-left energy flows observed on the right and left-hand sides of the TDPN. and are the forward and backward delays, respectively.
One of the most common teleoperation schemes is the P-F architecture , where the master velocity is sent through the channel and serves as desired velocity to the slave. In turn, the force produced by the slave-side controller is sent back to the master. Following the framework presented by Artigas et al. , using a hybrid of circuit and network representation, the slave side of the P-F architecture can be represented as shown in Fig. 3. There, the communication channel is represented by a TDPN; and are the velocities of the master and slave devices; is the force exerted by the slave-side controller; is the delayed master velocity; and is the delayed slave control force applied to the master device. Adding to that, and are the the drift compensation velocity source [6, 5] and the admittance-type passivity controller, which will be addressed subsequently.
In order to take into account the energy removed by the passivity controllers up to the previous time steps ( and ), the energy flow on each side of the TDPN is computed as
The passivity controller (PC) acts as an adaptive damping in order to guarantee the passivity of the channel. It can be applied in impedance or admittance configuration, according to the architecture requirements. In Fig. 3 the PC () is being applied in admittance configuration in order to modify the velocity coming out of the channel.
In order to fulfill the passivity conditions from (3) and (4) in a P-F architecture, the passivity controller is applied in admittance configuration on the slave side and in impedance configuration on the master side.
The admittance-type PC can be applied as 
and is a symmetric positive-definite weighting matrix.
The velocity removed by the PC from the delayed master velocity in order to keep the system passive will be
and the resulting velocity used as a reference by the slave will be
assuming all velocities are represented in the same frame.
Likewise, the impedance-type passivity controller can be applied as 
and is a symmetric positive-definite weighting matrix.
The force removed by the passivity controller from the delayed slave force in order to keep the system passive will be
and the resulting velocity used as a reference by the slave will be
Iii Whole-Body Bilateral Teleoperation of Redundant Aerial Manipulators
Iii-a Notations and Definitions
The Special Euclidean group and its Lie algebra
The pose of a rigid body in space can be represented by elements of the special Euclidean group , whose matrix form is
where describes the position vector and is an element of the Special Orthogonal group relative to the rotation of the body. Elements of can also be identified as . Furthermore, the velocity of a rigid body can be expressed by elements of the Lie algebra of , namely , as
where indicates the skew-symmetric operator applied to a vector in and are angular and linear velocities, respectively. Adding to that, due to the isomorphism between and , it is useful to define the operator , such that the velocities of rigid bodies can be expressed as elements of or . Such velocities can be represented in body (), or in spatial frame () .
Dynamical systems in
Iii-B Decoupling Control of Redundant Aerial Manipulators
The dynamics of a fully actuated aerial manipulator with joints can be written as
where is a set of generalized coordinates, is the inertia matrix, is a vector of Coriolis and centrifugal forces, and is the gravitational generalized torque vector. and are the generalized control and external torque vectors, respectively.
Following the framework presented by Ott et al.  for hierarchical whole-body control of kinematically redundant robots, where the number of minimal end-effector pose coordinates, , is less than the number of joint generalized coordinates, i.e., , the set of task coordinates can be extended to
where and are the set of Cartesian and null space velocities, respectively. is the Jacobian matrix that maps generalized to end-effector velocities. In this paper, it is assumed that is the body velocity of the end-effector with respect to some fixed inertial frame. Therefore is the body Jacobian  of the Cartesian task. Under the assumption of full row rank of , the matrix can be constructed as 
where is a full row rank null space base matrix, such that .
Such formulation allows the manipulator dynamics to be written as
where and are the elements of a block-diagonal Cartesian inertia matrix . In addition, , , , and are the elements of the Cartesian Coriolis-centrifugal matrix .
In order to control the pose of the flying base as the secondary task and the end-effector as the primary one, the following hierarchically-decoupling control law can be applied 
where is the Jacobian that maps the joint velocities to the body velocities of the flying base . Moreover, is a Cartesian-space wrench applied to the end-effector and is the control wrench of the secondary task, before being projected into the null space of the primary one. Both wrenches are assumed to be body wrenches .
In order to control the pose of the end-effector and of the flying base (as a secondary task), the following error elements can be defined
where , , , and are homogeneous transformation matrices in , which describe the desired and current poses of the end-effector (subscript ) and of the flying base (subscript ) with respect to a fixed inertial frame. The body velocities relative to these elements are , , , and , respectively, which relate to their time derivative according to (19).
where the matrices are positive definite gain matrices. Furthermore, and are the scalar and vector parts of a quaternion representation of , respectively, and . Moreover, and are the body velocities of and , respectively.
It is important to note in (27) that the control force goes through a projection before being commanded to the actuators. In this projection, the components of that conflict with the primary task are projected into zero and will not be commanded to the actuators via .
Iii-C Null-Space Wall Concept
Although is not always entirely applied to the base, it carries out important information about the limits of the null space of the primary task. In case the base is teleoperated, the operator directly commands and . However, in case a position which is not reachable without affecting the end-effector position is commanded, the resulting force will be projected to zero. If the operator keeps commanding towards that direction, the errors between desired and current pose in (31) will increase. As a consequence, an increase in will be observed. This behavior resembles what happens when the robot being teleoperated hits a wall. Therefore, by sending as haptic feedback, the operator will have the feeling as if the robot were trying to penetrate a wall when an unfeasible pose is commanded to the flying base. The stiffness of that wall will be initially defined by and , but could be scaled according to the task requirements.
On the other hand, if the base pose commanded by the operator is reachable without disturbing the end-effector, the operator will perceive lower forces, which are due to the imperfect tracking capabilities of the controller, as is usually the case for a P-F architecture in teleoperation. Therefore, for each end-effector pose there would be a region where the operator will perceive the robot in free-space motion and another where a wall penetration feeling will be perceived, which is the case when the commanded position is not feasible. An illustration of this behavior can be seen in Fig. 4.
It is important to remark that, without the aforementioned haptic feedback the operator would have to rely on visual feedback in order to know if the flying base is following the commands. This could be a complex task since (1) the camera is usually attached to the base, which, therefore, does not appear in the frame, (2) it is hard to distinguish motion of the base from that of the end-effector in the camera images. On the other hand, the wall penetration feeling perceived by the operator would indicate that the desired direction of motion is not feasible.
Iii-D Bilateral Null-Space Teleoperation
In order to allow bilateral teleoperation, the slave side of the P-F architecture (see Fig. 3) can be modified to implement a logic where the operator is able to switch between the primary task and the secondary one. For that purpose, the whole-body architecture depicted in Fig. 5 is proposed. In that architecture, the operator has the possibility of switching between the primary and the secondary task.
Initially, the initial values of and have to be set in the integrators represented by the summation blocks, which implement (20). When the switch is set for the primary task, the velocity commanded by the operator will be integrated into while will hold its value (, ). In addition, the force will be fed back through the communication channel while will act locally in order to keep the base at its last commanded pose. On the other hand, when the operator switches to the secondary task, the commanded velocity will be integrated into (, ) and the force will be sent back through the channel before going through the change of coordinates defined by and the projection defined by . In addition, will be applied at all moments in order to compensate for gravity and diagonalize the matrix (see Section III-B).
It is also important to note that, if the communication channel introduces time-delay and/or package loss, the teleoperation system is likely to become unstable. Therefore, in order to keep the communication channel passive, four PO-PC pairs can be implemented, two on the master and two on the slave side ( and in Fig. 5). These controllers will be responsible for monitoring and correcting the value of four potential functions, namely , , , and , which can be defined as
The directions and of the energies are defined by the sign of the power variables and , which can be calculated as
where and are the velocity of the master device and the force applied to it, as defined in Section II, while and will be the desired velocity and the computed force of the primary or the secondary task, depending on which one is being teleoperated (see Fig. 5). Furthermore, the energy values can then be defined as
where is the flag for null-space teleoperation.
The use of separate energy functions and their dissipation only in the respective hierarchical level was chosen in order to prevent the loss of the energetic decoupling between them. However, the cross-coupled dissipation law presented in  can also be applied without compromising the passivity of the system.
Iv Numerical Validation
In order to validate the proposed whole-body teleoperation approach, three sets of numerical simulation were performed. In all of them, the master device was modeled as a non-redundant robot with dynamics as in (21) and the slave was the simulated SAM. Initially, the linear axes of the base were teleoperated in the null space of the end-effector, which was locally commanded to keep its initial pose. In that task whose purpose was to clearly show the behavior of the null-space wall, no time-delay was simulated. Fig. 6 shows the positions and forces of the flying base, which shows that, between and seconds, the position commanded by the master , , and could not be followed by the slave because it reached the limits of the null-space motion. From that moment on, the interaction forces were increased as the master kept penetrating the null-space wall. The force in the opposite direction, which is fed to the master device, in this case, would inform the operator that that the commanded position is not reachable.
The second task involved teleoperating the end-effector under 300 ms round-trip delays and, at seconds (dashed vertical line), switching to the base teleoperation. For the sake of simplicity, the weighting matrices and described in Section II were set to be identity matrices. Additionally, the drift compensator from  was applied in order to avoid drift caused by TDPA. Fig. 8 displays the master and slave positions of both the end-effector and the flying base, and the force transmitted through the communication channel, which before 5 seconds is relative to the free motion of the end-effector and, after that, is due to the motion of the flying base, which hits the null-space wall twice. It can be noted that the end-effector motion also required the flying base to slightly move, which is due to its higher level of priority in the control structure. Adding to that, since the end-effector was teleoperated in free-space motion, the force feedback provided to the user in that period is much smaller that the one generated by the reaching of the null-space limits, which resembles wall-contact forces. It can also be seen that, since there is a dependency between the wall position in all three axes, the wall is not completely flat. However, since the end-effector is steady, the curvature of the wall is reasonably smooth. Fig. 8 presents the two energy flow of the channel. It can be noted that the TDPA was able to keep the channel passive at all time steps.
The last task aimed at analyzing the behavior of the wall when the end-effector is moving. For that purpose, after switching to null-space teleoperation, the end-effector was commanded to move autonomously in open loop. It can be seen from Fig. 10 that the limits of the null-space wall changed according to the end-effector motion, making the slave oscillate when in contact with the wall. It is also important to remark that the haptic forces displayed a high frequency sinusoidal wave, which is due to the end-effector motion. This feature could also be beneficial to inform the operator in case the end-effector is externally disturbed during teleoperation of the flying base. In can be noted that, despite not being included in the theoretical analysis, a moving null-space wall seems not to destabilize the system.
V Conclusion and Future Work
This paper presented a framework for performing stable bilateral teleoperation of both the end-effector and the base of redundant aerial manipulators. This feature is especially beneficial when a video camera is attached to the flying base whose position has to be changed without disturbing the end-effector. It has been shown that applying a hierarchically-decoupling controller and feeding back the secondary control force before being multiplied by the projection term provides the operator with a wall-contact feeling whenever the base is commanded to move to an unreachable position. This characteristic was confirmed by numerical simulation data, which also showed that the proposed framework can render the system passive in the presence of communication delays. Future work will involve applying the proposed framework to teleoperation of arbitrary hierarchy levels . In addition, an extension to a four-channel architecture  to include measured-force feedback will be studied.
- (2016) Kontur-2: force-feedback teleoperation from the international space station. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 1166–1173. Cited by: §V.
- (2011) Network representation and passivity of delayed teleoperation systems. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 177–183. Cited by: §II.
- (2012) A task-specific analysis of the benefit of haptic shared control during telemanipulation. IEEE Transactions on Haptics 6 (1), pp. 2–12. Cited by: §I.
- (1995) Proportional derivative (pd) control on the euclidean group. In European Control Conference, Vol. 2, pp. 1091–1097. Cited by: §III-A2.
- (2019) Multi-dof time domain passivity approach based drift compensation for telemanipulation. In 19th International Conference on Advanced Robotics (ICAR), pp. 701–707. Cited by: §II-2, §II, §III-A2, §IV.
- (2018) Smoother position-drift compensation for time domain passivity approach based teleoperation. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5525–5532. Cited by: §II, §II.
- (2014) The flying hand: a formation of uavs for cooperative aerial tele-manipulation. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 4335–4341. Cited by: §I.
- (2002) Time-domain passivity control of haptic interfaces. IEEE Transactions on Robotics and Automation 18 (1), pp. 1–10. Cited by: §II.
- (2019) Artificial and virtual impedance interaction force reflection-based bilateral shared control for miniature unmanned aerial vehicle. IEEE Transactions on Industrial Electronics 66 (1), pp. 329–337. Cited by: §I.
- (2019) Contact-based bridge inspection multirotors: design, modeling, and control considering the ceiling effect. IEEE Robotics and Automation Letters 4 (4), pp. 3561–3568. Cited by: §I.
- (2018) Aerial manipulationâa literature survey. Robotics and Autonomous Systems 107, pp. 221–235. Cited by: §I.
- (2014) Aerial manipulation robot composed of an autonomous helicopter and a 7 degrees of freedom industrial manipulator. In 2014 IEEE international conference on robotics and automation (ICRA), pp. 2107–2112. Cited by: §I.
- (1993) Stability and transparency in bilateral teleoperation. IEEE transactions on robotics and automation 9 (5), pp. 624–637. Cited by: §II.
- (2020) Visual-inertial telepresence for aerial manipulation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), Cited by: §I, §I.
- (2019-10) Dynamic active constraints for surgical robots using vector-field inequalities. IEEE Transactions on Robotics 35 (5), pp. 1166–1185. Cited by: §I.
- (2018) Shared planning and control for mobile robots with integral haptic feedback. The International Journal of Robotics Research 37 (11), pp. 1395–1420. Cited by: §I.
- (2016) Cooperative aerial tele-manipulation with haptic feedback. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5092–5098. Cited by: §I.
- (1994) A mathematical introduction to robotic manipulation. CRC press. Cited by: §I, §III-A1, §III-B, §III-B.
- (2018-12) The aeroarms project: aerial robots with advanced manipulation capabilities for inspection and maintenance. IEEE Robotics Automation Magazine 25 (4), pp. 12–23. Cited by: §I, §I.
- (2018) Aerial manipulation. Springer. Cited by: §I.
- (2011-10) Posture and balance control for biped robots based on contact force optimization. In 2011 11th IEEE-RAS International Conference on Humanoid Robots, Vol. , pp. 26–33. Cited by: §III-B.
- (2011) Subspace-oriented energy distribution for the time domain passivity approach. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 665–671. Cited by: §II-2, §III-D.
- (2015) Prioritized multi-task compliance control of redundant manipulators. Automatica 53, pp. 416–423. Cited by: §III-B, §III-B, §V.
- (2018) Aerial manipulation: a literature review. IEEE Robotics and Automation Letters 3 (3), pp. 1957–1964. Cited by: §I.
- (2010) A passive bilateral control scheme for a teleoperator with time-varying communication delay. Mechatronics 20 (7), pp. 812–823. Cited by: §II, §II, §III-D.
- (2020) Optimal oscillation damping control of cable-suspended aerial manipulator with a single imu sensor. In 2020 IEEE International Conference on Robotics and Automation (ICRA), Cited by: §I.
- (2019) Development of sam: cable-suspended aerial manipulator. In IEEE International Conference on Robotics and Automation (ICRA), Cited by: §I.
- (2013) Avian-inspired grasping for quadrotor micro uavs. In ASME 2013 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Cited by: §I.
- (2000) Spatial compliance modeling using a quaternion-based potential function method. Multibody System Dynamics 4 (1), pp. 75–101. Cited by: §III-B.