Comparing Alternate Modes of Teleoperation for Constrained Tasks

Comparing Alternate Modes of Teleoperation for Constrained Tasks

Christopher E. Mower, Wolfgang Merkt, Aled Davies, and Sethu Vijayakumar School of Informatics, University of Edinburgh, UK. Email: division, Costain PLC, UK.This research is supported by the Engineering and Physical Sciences Research Council (EPSRC, grant reference EP/L016834/1) and EU H2020 project Memory of Motion (MEMMO, project ID: 780684). Christopher E. Mower is partially supported by The Costain Group PLC.

Teleoperation of heavy machinery in industry often requires operators to be in close proximity to the plant and issue commands on a per-actuator level using joystick input devices. However, this is non-intuitive and makes achieving desired job properties a challenging task requiring operators to complete extensive and costly training. Despite this, operator fatigue is common with implications for personal safety, project timeliness, cost, and quality. While full automation is not yet achievable due to unpredictability and the dynamic nature of the environment and task, shared control paradigms allow operators to issue high-level commands in an intuitive, task-informed control space while having the robot optimize for achieving desired job properties.

In this paper, we compare a number of modes of teleoperation, exploring both the number of dimensions of the control input as well as the most intuitive control spaces. Our experimental evaluations of the performance metrics were based on quantifying the difficulty of tasks based on the well known Fitts’ law as well as a measure of how well constraints affecting the task performance were met. Our experiments show that higher performance is achieved when humans submit commands in low-dimensional task spaces as opposed to joint space manipulations.



I Introduction

Teleoperation of industrial manipulators is generally repetitive and requires high levels of concentration and manual dexterity. Excessive cognitive loads invariably lead to fatigue that can become dangerous. This danger is prevalent in the construction sector seen by having one of the highest levels of incidents involving fatalities and serious injuries per annum in the United Kingdom [1].

Fig. 1: Shotcrete application in a freshly excavated tunnel using a 5-DoF concrete spraying unit. Image provided by Costain Laing O’Rourke Joint Venture.

An example of such a task is concrete spraying as shown in Fig. 1. Here, a skilled human operator manipulates the device via some interface to spray a lining of wet concrete onto a excavated tunnel surface. The tunnel surface is often unstructured, due to excavation, and the operators visibility is restricted by high amounts of dust. Despite these restrictions, operators are required to manipulate these devices so that they (1) ensure job-site safety, (2) achieve high task performance, and (3) minimize costs [2]. Simultaneously accounting for safety and task performance in teleoperation tasks is required in a multitude of industries: robotic surgery [3], nuclear waste disposal [4], space robotics [5], assembly [6], and subsea [7].

Devices, such as the concrete sprayer systems, are generally controlled on a per-actuator level and consequently these control architectures do not lend themselves to easy operation since they force the operator to submit commands directly in the joint space. Since humans typically model tasks in the three dimensional Cartesian space they must learn naturally an inverse kinematic mapping imposing high costs in terms of monetary costs and time for specialized training. In the literature, methods such as the inverse-kinematics method [8] and optimization techniques [9, 10] have been developed that can allow control commands to be submitted in alternative control spaces. A number of works leveraging these advancements have developed assistive techniques for teleoperation, for example, virtual fixtures [11], shared control frameworks that merge human input and autonomy [4], and human supervisory capabilities implementing sliding autonomy [12]. However, for assistive techniques in general the issue of sub-task allocation between human and autonomy remains one of the main challenges [13].

In this work, we investigate the allocation of sub-tasks in the context of control spaces for target acquisition tasks. That is, we ask which control spaces should a human operator submit commands in order to achieve high task performance? Our main source of influence comes from the concrete spraying example [14] however our investigation can be generalized to other tasks such as paint spraying [15], semi-autonomous grasping [16], wiping [17], and robotic surgery [18]. We have designed an experiment incorporating the well-known Fitts’ law, that quantifies task difficulty, in order to compare and contrast different control modes. This paper provides knowledge of intuitive and effective control spaces for teleoperation and shared control that will allow for more grounded formulations. Our main contributions are listed as follows:

  1. Analysis into the sub-task allocation for target acquisition tasks through an empirically driven investigation.

  2. An extensive data set of results that contains objective and subjective metrics.

  3. A comparison between two subsets of our participants based on the participants personal habits that we identified to effect performance.

  4. Two generalized performance metrics for Fitts’ law relating to teleoperated target acquisition tasks.

The resulting data support several conclusions about control, user preferences, and how habitual traits may effect task performance. Our analysis indicates that there is a relation between control space dimensionality and performance. It has been seen in our experiments that lower dimensional task spaces in general elicit highest performance and these are generally preferred by users. Users who were identified to play video games on a regular basis were generally able to complete tasks quicker than those who do not. However, those who do not play video games were seen to maintain other performance metrics to a higher standard. An accompanying video is available at

Ii Related Work

Prior work has explored techniques that guide robots motions via synthesized constraints, i.e., potential fields and virtual fixtures. Potential fields, originally developed by Khatib [19], guide a user towards or away from a goal or obstacle [20]. Virtual fixtures create virtual barriers or forbidden regions in the task space that provide assistance in protecting sensitive areas. Rosenberg [11] provided early work in the development of virtual fixtures. A number of guidance and/or preventative virtual fixture designs were compared using a Fitts’ law paradigm for a remote teleoperation peg placement task. Later work [21] has developed on-the-fly forbidden region generation techniques using real-time sensor data.

The work of Dragan and Srinivasa [22, 23] formalizes assistive teleoperation under a framework of policy blending. The system, grounded in inverse reinforcement learning, attempts to predict the intentions of the human operator in order to arbitrate the operator and robot control policies. Other learning-based methods have also been developed by Abi-Farraj et al. [24] and attempt to refine unskilled operator input based on skilled operator input learned by exploiting learning from demonstration techniques.

From the concrete spraying literature, a task space control framework was developed by Honegger and Codourey [25] based on the inverse-kinematics method. The system allows the operator to control the end-effector of the concrete spraying unit using a 6-DoF space mouse. Later work by Girmscheid and Moser [14] developed a sliding autonomy approach for a high-DoF concrete spraying unit. They define three levels of autonomy: (1) A manual mode, consisting of no autonomy and implementing common joint-level control, (2) A semi-automated mode that, using a pre-collected laser scan of a tunnel cavity and computed geometry, allows the operator to command only the position on the wall to spray whilst the system accounts for motion constraints, and (3) An automated mode that plans and executes an entire trajectory. The latter mode, they note, is only in development for tunnel-boring machine projects where the excavated tunnel surface is regular and very smooth. Modes of teleoperation compared in this work are inspired by these control techniques developed for concrete spraying applications.

The above works represent a number of developments in shared autonomy and collaborative manipulation. There exists in the literature a number of studies that compare various methods for different levels of task complexity. A number of strategies were compared by You and Hauser [26] for a reach-to-target task that must deal with collision avoidance, dynamic constraints, and erroneous input. A human was given the task of controlling a robot arm to reach to a given target. The robot was positioned in a simulated environment of various different complexities. The strategies compared in their study each have varying levels of autonomy: direct joint control, inverse kinematics, inverse kinematics with predictive safety filter, reactive potential field, and sampling-based motion planner. Kim et al. [27] compared manual and autonomous control modes for a pick-and-place task. A study by Leeper et al. [28] compares a number of strategies for assisted and non-assisted remote robot grasping. These studies compare various methods for varying levels of autonomy and/or interface designs. To the best of our knowledge, there has not been work in educing control spaces that are intuitive and elicit high task performance for unskilled operators. In this work, we use the well-known Fitts’ law to quantify task complexity in order to systematically compare the performance of several modes of teleoperation.

Iii Methodology

Iii-a General formulation

We assume a teleoperation setting akin to Fig. 1, i.e., a human operates a manipulation system via some interface. Let us model the process of teleoperation by the functional


where represents joint configuration for an -DoF manipulator, are inputs from the interface, and denotes time. Note, we assume and to be bounded, with control inputs scaled to the range . Assuming a known initial state , all future state can thus be computed by (1). It is also assumed that the function can be computed fast enough for real-time actuation.

The chosen value for coupled with the space in which commands are submitted categorizes what we define as the mode of teleoperation. In Section III-C, we describe four modes developed for our experiments to test our hypotheses detailed in Section III-B. To simplify our notation, the remainder of the paper will neglect sub-script .

Iii-B Hypotheses

Below we state our hypotheses; these represent alternative hypotheses. Hypotheses HIII-B and HIII-B relate to the objective results (Section V-A) whilst HIII-B and HIII-B relate to the subjective results (Section V-C). As mentioned above, humans typically model tasks in three-dimensional Cartesian spaces as opposed to the manipulator joint configuration space and thus intuitively we expect that modes of teleoperation that allow control commands to be submitted in these task spaces will achieve higher performance. Therefore, we base our alternative hypotheses on this intuition.

Hypothesis (HIii-B):

Modes of teleoperation with lower dimensions will see higher performance.

Hypothesis (HIii-B):

Modes of teleoperation that submit commands in the manipulator task space will see higher performance.

Hypothesis (HIii-B):

Participants will rate higher those modes of teleoperation with fewer dimensions.

Hypothesis (HIii-B):

Participants will rate higher those modes of teleoperation that submit commands in the manipulator task space.

Iii-C Modes of teleoperation

Iii-C1 Full joint mode (FJ)

Industrial manipulators are operated on a per-actuator level. That is, there exists a one-to-one mapping between each joystick axis and manipulator actuator. Thus, the mode of teleoperation can be modeled using


where is a diagonal matrix such that the entries are maximum joint velocities, and is the control loop time step. Joint limits are handled by checking (2) prior to sending actuation signals.

Iii-C2 Reduced joint mode (RJ)

Here, we relieve a number of joints from the human and assign these as a task constraint. A split in the joints of the manipulator must be made; we split the manipulator in the middle relieving joints near the end-effector from the human control. Let where denote the joints under human control, and denote the autonomous joints. Inspired by the spraying a wall task, joints are assigned the task of end-effector alignment with the surface normal. We model the mode using


where represents the next joint state for joints and similarly for . The human controlled joints are computed in the same way as in the full joint control mode and thus, takes the same form as (2). The functional form for is found using an unconstrained optimization expressed by


where is the mapping from the configuration space to the angular task space, denotes the Euclidean norm, and represents the two-dimensional (pitch and roll angles) task space goal.

Iii-C3 Full task mode (FT)

An alternative class of modes is defined when the human instead submits commands to the manipulator task space rather than the joint space. For spraying, the manipulator task space comprises three translational and two rotational dimensions of the end-effector. This mode is thus expressed by


where is the mapping from the configuration space to the translational and angular task space.

Iii-C4 Reduced task mode (RT)

Humans typically model manipulation tasks in the task space. Spraying for an intuitive point of view is a two-dimensional task, i.e., the position on the wall to spray. Assuming some model of the environment geometry , a two-dimensional point on the surface defined by the human can be used to generate task space goal. We model this mode by


where is some five-dimensional task space goal computed using the two-dimensional input from the human. The computed value of encodes ideals such as perpendicularity to the spraying plane and a given standoff distance from the surface. For example, as in Fig. (d)d, this amounts to minimizing and maintaining a constant value, say, for .

Iv Experiments

Iv-a Participant selection

We have obtained results in this study by evaluating the performance of 21 participants (16 male, 5 female). The age distribution of the participants were 7 (21-25), 11 (26-30), 2 (31-35), and 1 (36+). During preliminary investigations a difference in performance was noticed for participants who regularly played video games. During the final experiments, we asked participants to provide a rating on how often they play video games222Note, those who participated in the preliminary investigation were barred from the final experiments.. They were asked “How often do you play video games (e.g., Xbox, PS4, PC)? [1 Never], [2 Bi-monthly], [3 Monthly], [4 Bi-weekly], [5 Weekly], [6 Regularly (but not daily)], [7 Daily]”. The responses are shown in Fig. 2. A participant was considered a gamer if they gave a rating greater than or equal to 4, and a non-gamer otherwise.

Fig. 2: Participant selection gaming regularity.

Iv-B Experimental design

Fitts’ law is widely considered to be a robust measure of performance for target acquisition (pointing) tasks that makes the analogy between movement time and transmission of information [29]. A common usage of Fitts’ law from the human-computer interaction literature is to compare the usability of computer input devices (e.g. a mouse, trackball, and a stylus with a tablet) [30]. The law makes two underlying assumptions: (1) task difficulty is linearly correlated with performance, and (2) a complete move is performed through a number of iterations of feedback-guided corrective sub-movements, i.e. the deterministic iterative-corrections model [31]. We use Fitts’ law as a method to specify task difficulty.

Fitts’ established the information capacity of the human motor system by deriving a model for the the index of performance (in ) expressed by


where is the index of difficulty (in ) and (in seconds) is the average movement time. The index of performance is a metric that quantifies task performance; higher values for indicate better performance. The index of difficulty is a metric that defines the task difficulty; higher values for imply the task is more difficult. Under the deterministic iterative-corrections model and by analogy with Shannons Theorem 17 [32] a formula for is derived, see [31] for details, given by


where is the distance to a target and is the width of the target, see Fig. (b)b. Define a condition by the tuple . Intuitively, a difficult task is when is large, thus is small and is large, and an easy task is when is small thus is large and is small.

Operators in industry must maintain a number of motion constraints to achieve high task performance. Inspired from the concrete spraying task, we define two additional performance metrics that generalize Fitts’ law.

Define the angular length as the total change in for a complete move and expressed by


where as in Fig. (d)d. This quantity describes how well a user is able to maintain perpendicularity to the wall during transitioning between one target and the next.

Define the delta length denoted as the least absolute deviation in the standoff distance with respect to a given ideal over the duration of a complete move and expressed by


where is the standoff distance as in Fig. (d)d and is an ideal standoff distance.

For each mode of teleoperation participants completed 17 conditions: 16 have been generated by scaling the condition values used in Fitts’ original work [29] while one has been hand-tuned to add spread to the values. The conditions used in this study are shown in I representing the range .

(m) (m)
0.0125 0.0469, 0.0938, 0.1875, 0.3750
0.0500 0.0938, 0.3750, 0.7500
0.0250 0.0469, 0.0938, 0.1875, 0.7500
0.1000 0.1875, 0.3750, 0.7500
0.1500 0.1875, 0.3750, 0.7500
TABLE I: Condition values used in experiments.

Iv-C System description

(a) An operator controlling the KUKA LWR arm.
(b) Multi-directional task.
(c) Target order of appearance for a single condition.
(d) Angle to the plane and standoff distance used to compute performance metrics.
Fig. 3: Experimental setup. Whilst condition order of appearance is randomized, target order was always kept the same as in (c). The black line indicates an example path of the focus point during manipulation; the path was not shown to participants, only the focus point indicated by the green dot.
(a) Full joint space control.
(b) Reduced joint space control.
(c) Full task space control.
(d) Reduced task space control.
Fig. 4: Gamepad mappings for the modes of teleoperation implemented in experiments. Markings indicate where the human interacts with on the gamepad and where that interaction is perceived to be on the robot model.

We have implemented each mode of teleoperation described in Sec. III-C using a position control open-loop framework on a 7-DoF KUKA LWR robot arm. The operator interfaced with the system using an F710 Logitech gamepad, see Fig. 4 for the mappings between the interface and robot for each mode. Each experiment was initiated by the user clicking a button on the gamepad. Inter-process communication is handled by the Robot Operating System (ROS) [33] and modes requiring numerical optimization were computed using the Extensible Optimization Toolkit (EXOTica) [34]. Goal joint positions were streamed at . Targets and the focus point were displayed on a cross-diagonal display placed specifically such that the transform between the robot base and the television screen was known. Targets were scaled to the real world and the position of the focus point was found using the robot forward kinematics.

An ideal standoff distance of was chosen, justified as follows: In industry, the MEYCO Oruga (Fig. 1) has an approximate maximum reach of and the ideal standoff distance is . The KUKA LWR arm has an approximate maximum reach of . We thus find by balancing the ratios and adding a ten percent safety distance.

Iv-D Experimental protocol

Participants were tasked with teleoperating a KUKA LWR robot arm to reach-and-point to a number of circular targets, indicated in red, arranged in a circle as in Fig. (b)b. Targets were displayed on a screen in a known position and orientation with respect to the robot base, see Fig. (a)a. For each condition, targets were presented in the order as in Fig. (c)c. Conditions were randomized for every participant. The participants were instructed to use as many controllable dimensions available to them in order to manipulate the robot in such a way that completes the task as fast as possible while maintaining the angular and delta length constraints; i.e. simultaneously minimize , , and to the best of their ability.

Participants were allowed to move around the laboratory during the experiment, akin to concrete sprayer operators. For safety, they were not allowed within 1.5m of the robot.

Targets and conditions were presented to the participant in succession. As one target was deemed acquired the next immediately followed and as one condition was completed (i.e., one full cycle of targets in the order shown in Fig. (c)c) the next immediately followed. Target acquisition is when the focus point (i.e., the point on the display screen the robot is pointing) comes into contact with the target.

Experiments were completed for each mode of teleoperation. The order in which modes were presented to each participant was randomized to minimize skill-transfer. At the start of every mode the robot was reset to the same starting configuration. Following each mode participants were asked to fill out a questionnaire (Sec. IV-E).

Iv-E Questionnaire

Participants were asked to provide a rating on their speed perception, accuracy perception, fatigue, and the mental capacity for each mode. The questionnaire shown in Table III was devised from the ISO 9241 standard and work by Douglas et al. [35]. Questions 1-5 use a 7-point scale and 6-7 are open-ended.

Questions: Rating: 1 7
1) The mental effort required for operation was high low
2) Accurate pointing was difficult easy
3) Operation speed was fast slow
4) Finger fatigue high none
5) Overall, the mode of teleoperation was difficult easy
6) Did you have any trouble with this mode of teleoperation?
7) Do you have any comments in general about using this mode of teleoperation?
TABLE II: Questionnaire used in investigation.

V Results

Fig. 5: Log-Index of performance distribution results. Values in red indicate D’Agostino’s normality test . By means of a one-sided paired-sample t-test, pairs accepted under HIII-B and HIII-B () are indicated above.
Fig. 6: Throughput results that estimate the index of performance. We report the results for the full data set and the two subsets: gamer and non-gamer. We also provide the Pearson correlation coefficient above each bar.

V-a Low dimensional task space elicits high performance

Data for 21 participants was collected for 17 conditions with 9 targets for each condition. In order to compute our performance metrics, we collected target acquisition timestamps and joint states.

We have filtered the data based on the following. We ignore data for the first target of every condition since data collected when transitioning between conditions is not representative of skill in either condition. We also remove outliers that are outside 1.5 times the standard deviation of the task performance metrics. We have collected an extensive database of results constituting, in total, 8882 data points. Using each metric (time, angular length, and delta length) and the index of difficulty we compute an index of performance for every metric and for each point in our data set. The distributions of is shown in Fig. 5. Using this data we have been able to perform a one-sided paired-sampled t-test to test our hypotheses.

Precisely the same conclusions for time and delta length metrics have been observed. We see HIII-B is accepted as expected for reducing control space dimensionality, except the FJ-RJ comparison. For the angular length HIII-B is accepted; this is unsurprising since RJ supports end-effector alignment. For cross-modes, e.g. FJ-RT, we see that HIII-B is accepted.

When comparing between task space modes and joint space modes we see that HIII-B has been accepted. An interesting point to note that in the time metric and delta length metric HIII-B is accepted when comparing the RJ and FT modes. This suggests that low dimensional joint spaces in some cases elicit worse performance than higher dimensional task spaces. We postulate that the RJ mode may fight intuition resulting in the machine performing motions that do not match the operators innate model of the system dynamics.

V-B Habitual traits effect performance

During preliminary investigations, it had been noted that participants who were known to play video games regularly seemed to have higher performance than those that did not. In our final experimental design we decided to include this comparison as part of our multivariate analysis. As described in Sec. IV-A we asked each participant to indicate their gaming regularity; a participant is considered a gamer if they gave a rating greater than or equal to 4, and a non-gamer otherwise. Using this division we split our data set into two subsets.

By the assumption that difficulty and performance are linearly correlated we are able to compare across the various modes of teleoperation using linear regression. In order to compare the general performance for each trait we use an estimate of throughput [29]. The throughput is another indicator of overall performance. For the time metric, the throughput is estimated by the reciprocal of a scalar where are regression parameters such that ; so . For the angular length and delta length, throughput for each has been estimated in the same fashion. Throughput estimates for our full data set and the gamer/non-gamer subsets are shown in Fig. 6.

Regarding time performance, in general, excluding the result for the RJ mode, a similar pattern is seen when compared to the results in Fig. 5; the RT mode has highest throughput in general and task space modes have higher throughput than joint space. For each mode of teleoperation we observe a higher throughput for gamers than the non-gamers. We noted that gamers generally seemed more familiar using two analog joysticks at once as opposed to the non-gamers. We posit, due to this ability, gamers were able to achieve faster times.

Regarding angular length performance, we see the general trend in performance as discussed in the previous section. Gamers have higher throughput than non-gamers for each mode except the FT mode. It should be noted that the correlation coefficients for these are generally low which renders these results as potentially spurious. Drawing conclusions from these results may be unreliable. The throughput results for the delta length unfortunately suffer from similar issues and so reliable conclusions cannot be made for these either.

V-C Participants approve low dimensional task spaces

Question FJ RJ FT RT
1 \FPeval\resultround(7-5.19,2)\result  1.33 \FPeval\resultround(7-3.71,2)\result  1.59 \FPeval\resultround(7-3.43,2)\result  1.57 \FPeval\resultround(7-1.95,2)\result  1.36
2 \FPeval\resultround(7-4.43,2)\result  1.66 \FPeval\resultround(7-4.10,2)\result  1.73 \FPeval\resultround(7-3.86,2)\result  1.39 \FPeval\resultround(2.95,2)\result  1.69
3* 4.62 0.92 4.71 1.06 3.81 1.08 4.29 1.15
4 \FPeval\resultround(7-3.71,2)\result  2.00 \FPeval\resultround(7-3.38,2)\result  1.88 \FPeval\resultround(7-2.33,2)\result  1.43 \FPeval\resultround(7-2.10,2)\result  1.37
5 3.10 1.55 4.24 1.26 4.86 1.28 5.95 1.07
Question RJ FT RT
1 FJ 0.0048 (3) 0.0009 (3) 5.2e-7 (3/4)
RJ - 0.5867 0.0003 (3/4)
FT - - 0.0011 (3)
2 FJ 0.5538 0.1435 0.0281 (3/4)
RJ - 0.6339 0.0104 (3/4)
FT - - 0.0727
3* FJ 0.7245 0.0202 0.2596
RJ - 0.0068 0.1428
FT - - 0.1158
4 FJ 0.4907 0.0120 (3) 0.0044 (3/4)
RJ - 0.0100 (4) 0.0005 (3/4)
FT - - 0.3086
5 FJ 0.0244 (3) 0.0009 (3) 3.3e-6 (3/4)
RJ - 0.1198 1.0e-5 (3/4)
FT - - 0.0047 (3)
TABLE III: Results of the questionnaire and paired-sampled -test (). Bold indicates the one-sided hypothesis HIII-B/HIII-B is accepted. Note, since Q3 does not evaluate the users preference (highlighted in bold) indicates the result of a two-sided significance test.

The results of the questionnaire are shown in Tab. III. The mean and standard deviation of the responses on questions 1-6 are shown above a paired-sampled -test to determine the responses’ statistical significance.

The results for question 1 show that participants felt the FJ mode required the highest amount of mental effort. There seems to be no statistical difference between the RJ and FT modes. Participants indicated the RT required the least amount of mental effort.

Overall, participants indicated the task space modes were the easiest to point accurately. There is no statistical difference between FJ and RJ modes both having ratings indicating that participants felt these were the most difficult to accurately point. The results of the -test indicate we reject HIII-B/HIII-B when comparing FJ, RJ, and FT modes. However, whilst HIII-B/HIII-B are accepted when comparing between RT and the joint space modes, HIII-B is rejected when comparing the task space modes. The p-value for the RJ-FT comparison is reasonably high and the FJ-FT comparison is higher than the p-value for the FT-RT comparison. Comparing the results of this question to the responses given for questions 6 and 7 (shown below), we suggest that, despite the computed p-value, accepting HIII-B for the FT-RT comparison has potential grounding as a conclusion.

During our experiments, maximum joint velocities were reduced to a conservative range for safety. Question 3 attempted to ascertain whether the participants felt the robot motion was too slow or indeed too fast. Mean values indicate participants felt the operation speed was neither too fast nor too slow. There are not significant differences between the results apart from the FT mode compared to the joint space modes.

The results for question 4 indicate the participants experienced the least finger fatigue for task space control modes with no statistical difference between the two. The joint space modes caused the most fatigue.

For question 5, participants rated the RT mode as the easiest to use and the FJ the most difficult. There is no significant difference between the FT and RJ modes.

We summarize the responses to questions 6 and 7. Selected are common statements made by participants.

Full joint mode:

6) Wrist joints felt slower than base joint.
Joint mappings felt inverted and was easy to get into strange configurations.
7) Hard to maintain constraints.
Only used two joints at a time.

Reduced joint mode:

6) Attention was directed to focus point, not robot, making operation very difficult.
Often requiring to re-adjust.
7) Slightly better than full joint mode.
Easy for small values.

Full task mode:

6) Unexpected motions at times.
Maintaining delta length was effectively impossible. Perhaps with more practice this mode would be more efficient.
7) Felt it was possible to go faster.
Favored over both joint space control modes; intuitive, mostly easy to use.

Reduced task control mode:

6) Some unexpected motions.
Sometimes felt too slow.
7) Very intuitive and easy to use.

The responses generally correlate with the answers to questions (1)-(5). The mappings for both joint modes were identified as inverted by many of the participants. The direction the joint moved under these modes was matched with the joystick direction. For example, for the FJ mode, pushing forward on the left joystick moved joint 2 in the direction such that the end-effector and focus point moved downwards. Participants felt the robot end-effector and focus point should instead move upwards. An issue for some participants for the task space modes is that the robot would make unexpected motions in certain configurations. The source of this issue is that at times a target configuration was computed requiring a joint velocity surpassing the maximum; an unfortunate consequence of unconstrained optimization. Whilst these issues were observed at times they did not render the task impossible to complete.

Vi Conclusions

In this paper, we developed, implemented, and carried out a study to determine which modes of teleoperation elicit high task performance for unskilled human operators on a task inspired by concrete spraying in industry. A Fitts’ law paradigm was used to quantify difficulty and educe the performance of each mode. We have generalized Fitts’ law for two additional performance metrics. An extensive data set was collected from an experiment consisting of 21 participants. The results and analysis performed support several conclusions regarding control, sub-task allocation between human and autonomy, and how habitual traits can effect performance.

A reduced task space control mode with has been shown to outperform all three other control modes with regards to the index of performance over three performance metrics; time, angular length, and delta length. The results of the questionnaire support this conclusion as the RT mode was generally favored the highest. The RT mode is the closest model that directly regulates the performance parameters. Given that our hypotheses were were accepted this opens new avenues for shared control design. Based on these results, we posit the contention that there is a positive correlation between cognitive load and number of human controllable dimensions for task space control modes. Both joint space modes were not favored by participants in this study with some noting that the reduced joint space control felt like having the “worst of both worlds” with regards to joint space and task space control.

This work assumed unskilled participants and so the long-term learning effects on performance are not considered here. In future work, we intend to study the effect of learning curve in order to predict average learning time. The ability of the participants classed as gamers to achieve higher performance than non-gamers may be a consequence of their familiarity with the game pad controller used as the interface in our experiments. Future investigations will take this into account the effect the interface used by comparing other interfaces such as a joystick, 6-DoF space mouse, a combination of computer mouse and keyboard. Metrics quantifying cognitive load will be explored and compared against the number of controllable dimensions in-order to investigate our proposition of a positive correlation between the two quantities.

This study indicates the mode specification highly impacts the design and performance of shared control systems. We intend to use the knowledge acquired here to inform the development of new formulations for shared autonomous and collaborative methods that adapt dynamic motion constraints on-the-fly using multi-modal sensory data.


  • [1] Health and Safety Executive (HSE), “Fatal injuries arising from accidents at work in Great Britain 2017,”, 2017.
  • [2] M. Ballou, “Shotcrete rebound – how much is enough?” Shotcrete magazine, American Shotcrete Association, 2003.
  • [3] G. T. Sung and I. S. Gill, “Robotic laparoscopic surgery: a comparison of the da vinci and zeus systems,” Urology, vol. 58, no. 6, pp. 893 – 898, 2001.
  • [4] F. Abi-Farraj, N. Pedemonte, and P. R. Giordano, “A visual-based shared control architecture for remote telemanipulation,” in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Oct 2016, pp. 4266–4273.
  • [5] N. Y. Lii, Z. Chen et al., “Toward understanding the effects of visual- and force-feedback on robotic hand grasping performance for space teleoperation,” in 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, Oct 2010, pp. 3745–3752.
  • [6] M. Sagardia, T. Hulin et al., “A platform for bimanual virtual assembly training with haptic feedback in large multi-object environments,” in Proceedings of the 22Nd ACM Conference on Virtual Reality Software and Technology, ser. VRST ’16.   New York, NY, USA: ACM, 2016, pp. 153–162.
  • [7] R. R. Murphy, K. L. Dreger et al., “Use of remotely operated marine vehicles at minamisanriku and rikuzentakata japan for disaster recovery,” in 2011 IEEE International Symposium on Safety, Security, and Rescue Robotics, Nov 2011, pp. 19–25.
  • [8] O. Khatib, “A unified approach for motion and force control of robot manipulators: The operational space formulation,” IEEE Journal on Robotics and Automation, vol. 3, no. 1, pp. 43–53, February 1987.
  • [9] A. Escande, N. Mansard, and P.-B. Wieber, “Hierarchical quadratic programming: Fast online humanoid-robot motion generation,” The International Journal of Robotics Research, vol. 33, no. 7, pp. 1006–1028, 2014.
  • [10] M. Zucker, N. Ratliff et al., “Chomp: Covariant hamiltonian optimization for motion planning,” The International Journal of Robotics Research, vol. 32, no. 9-10, pp. 1164–1193, 2013.
  • [11] L. B. Rosenberg, “Virtual fixtures: Perceptual tools for telerobotic manipulation,” in Proceedings of the 1993 IEEE Virtual Reality Annual International Symposium, ser. VRAIS ’93.   Washington, DC, USA: IEEE Computer Society, 1993, pp. 76–82.
  • [12] W. Merkt, Y. Yang et al., “Robust shared autonomy for mobile manipulation with continuous scene monitoring,” in 2017 13th IEEE Conference on Automation Science and Engineering (CASE), Aug 2017, pp. 130–137.
  • [13] T. Inagaki, “Adaptive automation: Sharing and trading of control,” in Handbook of cognitive task design.   CRC Press, 2003, pp. 171–194.
  • [14] G. Girmscheid and S. Moser, “Fully automated shotcrete robot for rock support,” Computer-Aided Civil and Infrastructure Engineering, vol. 16, no. 3, pp. 200–215, 2001.
  • [15] H. Chen, W. Sheng et al., “Automated robot trajectory planning for spray painting of free-form surfaces in automotive manufacturing,” in Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292), vol. 1.   IEEE, 2002, pp. 450–455.
  • [16] J. Vogel, K. Hertkorn et al., “Flexible, semi-autonomous grasping for assistive robotics,” in 2016 IEEE International Conference on Robotics and Automation (ICRA), May 2016, pp. 4872–4879.
  • [17] L. Armesto, J. Moura et al., “Constraint-aware learning of policies by demonstration,” The International Journal of Robotics Research, vol. 37, no. 13-14, pp. 1673–1689, 2018.
  • [18] F. Rydén and H. J. Chizeck, “Forbidden-region virtual fixtures from streaming point clouds: Remotely touching and protecting a beating heart,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.   IEEE, 2012, pp. 3308–3313.
  • [19] O. Khatib, “Real-time obstacle avoidance for manipulators and mobile robots,” in Proceedings. 1985 IEEE International Conference on Robotics and Automation, vol. 2, March 1985, pp. 500–505.
  • [20] J. W. Crandall and M. A. Goodrich, “Characterizing efficiency of human robot interaction: a case study of shared-control teleoperation,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 2, Sept 2002, pp. 1290–1295 vol.2.
  • [21] S. Nia Kosari, F. Rydén et al., “Forbidden region virtual fixtures from streaming point clouds,” Advanced Robotics, vol. 28, no. 22, pp. 1507–1518, 2014.
  • [22] A. Dragan and S. Srinivasa, “Formalizing assistive teleoperation,” in Robotics: Science and Systems, July 2012.
  • [23] A. D. Dragan and S. S. Srinivasa, “A policy-blending formalism for shared control,” The International Journal of Robotics Research, vol. 32, no. 7, pp. 790–805, 2013.
  • [24] F. Abi-Farraj, T. Osa et al., “A learning-based shared control architecture for interactive task execution,” in 2017 IEEE International Conference on Robotics and Automation (ICRA), May 2017, pp. 329–335.
  • [25] M. Honegger and A. Codourey, “Redundancy resolution of a cartesian space operated heavy industrial manipulator,” in Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146), vol. 3, May 1998, pp. 2094–2098 vol.3.
  • [26] E. You and K. Hauser, “Assisted teleoperation strategies for aggressively controlling a robot arm with 2d input,” in Proceedings of Robotics: Science and Systems, Los Angeles, CA, USA, June 2011.
  • [27] D. Kim, R. Hazlett-Knudsen et al., “How autonomy impacts performance and satisfaction: Results from a study with spinal cord injured subjects using an assistive robot,” IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, vol. 42, no. 1, pp. 2–14, Jan 2012.
  • [28] A. Leeper, K. Hsiao et al., “Strategies for human-in-the-loop robotic grasping,” in 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI), March 2012, pp. 1–8.
  • [29] P. M. Fitts, “The information capacity of the human motor system in controlling the amplitude of movement.” Journal of experimental psychology, vol. 47, no. 6, p. 381, 1954.
  • [30] I. S. MacKenzie, A. Sellen, and W. A. S. Buxton, “A comparison of input devices in element pointing and dragging tasks,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI ’91.   New York, NY, USA: ACM, 1991, pp. 161–166.
  • [31] E. Crossman and P. Goodeve, “Feedback control of hand-movement and fitts’ law,” The Quarterly Journal of Experimental Psychology Section A, vol. 35, no. 2, pp. 251–278, 1983.
  • [32] C. E. Shannon and W. Weaver, A Mathematical Theory of Communication.   Champaign, IL, USA: University of Illinois Press, 1963.
  • [33] M. Quigley, K. Conley et al., “Ros: an open-source robot operating system,” in ICRA workshop on open source software, vol. 3, no. 3.2.   Kobe, Japan, 2009, p. 5.
  • [34] V. Ivan, Y. Yang et al., EXOTica: An Extensible Optimization Toolset for Prototyping and Benchmarking Motion Planning and Control.   Cham: Springer International Publishing, 2019, pp. 211–240.
  • [35] S. A. Douglas, A. E. Kirkpatrick, and I. S. MacKenzie, “Testing pointing device performance and user assessment with the iso 9241, part 9 standard,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI ’99.   New York, NY, USA: ACM, 1999, pp. 215–222.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description