Unsupervised Trajectory Segmentation and Promoting
of Multi-Modal Surgical Demonstrations
To improve the efficiency of surgical trajectory segmentation for robot learning in robot-assisted minimally invasive surgery, this paper presents a fast unsupervised method using video and kinematic data, followed by a promoting procedure to address the over-segmentation issue. Unsupervised deep learning network, stacking convolutional auto-encoder, is employed to extract more discriminative features from videos in an effective way. To further improve the accuracy of segmentation, on one hand, wavelet transform is used to filter out the noises existed in the features from video and kinematic data. On the other hand, the segmentation result is promoted by identifying the adjacent segments with no state transition based on the predefined similarity measurements. Extensive experiments on a public dataset JIGSAWS show that our method achieves much higher accuracy of segmentation than state-of-the-art methods in the shorter time.
Surgical trajectory segmentation is a fundamental problem in the field of robot-assisted minimally invasive surgery (RMIS). It can be applied to several applications, such as demonstration learning , skill assessment , complex task automation  and so forth. Each surgical procedure is usually represented by synchronized video and kinematic recordings, and can be decomposed into several meaningful sub-trajectories. Since the segments are atomic with less complexity, lower variance and easier to eliminate outliers, the capability of further robot learning and assessment can be improved. However, it is a challenging task to segment the surgical trajectory accurately and rapidly. Even an identical surgical procedure can vary remarkably in the spatial and temporal domains due to the skill difference among surgeons. Moreover, the trajectory is susceptible to the random noise.
Traditional solutions usually transfer the surgical trajectory segmentation to a clustering problem, and are mainly divided into two categories: supervised and unsupervised methods. As the supervised methods, Linear Discriminate Analysis (LDA) , Hidden Markov Models (HMMs) , Descriptive Curve Coding (DCC) , and Conditional Random Field (CRF)  are proposed. However, the supervised method is time-consuming because of the manual annotations of experts for training dataset. Thus, unsupervised methods have drawn more attention in recent years. Some unsupervised methods based on Gaussian Mixture Model (GMM) and Dirichlet Processes (DP) are proposed [7, 8]. Although GMM and DP based methods can get rid of the manual annotations, the room to improve the accuracy of surgical trajectory segmentation remains since only the kinematic data is taken into account. Recently, video data are involved by using a deep learning based method, since traditional pattern recognition based feature extraction methods can’t model the variations among surgeon’s videos well. A. Murali et al.  employ VGGNet to extract features from video followed by Transition State Clustering (TSC) for task-level segmentation using both kinematic and video data. Although the involvement of video source enables the higher accuracy of segmentation, the feature extraction from videos is time-consuming and easily leads to over-segmentation.
This paper focuses on the unsupervised surgical trajectory segmentation by means of both video and kinematic data in this paper. There are challenges to find consistent segments from the varying and noising recordings from surgeons with different skills for a specific task. First, although the video is capable of improving the performance of segmentation, it is challenging to extract the distinguishing features in an efficient way. In addition, random noise has to be considered due to the difference of surgeons’ skill. Second, state-of-the-art methods generally suffer from the over-segmentation issue. We need to provide an effective way to identify the adjacent segments with no state transition.
As shown in Fig. 1, a fast unsupervised method for surgical trajectory segmentation is proposed using the video and kinematic data. In particular, a promoting procedure is presented to alleviate the over-segmentation issue. First, a compact but effective unsupervised learning network called stacking convolutional auto-encoder (SCAE) is employed to speed up the feature extraction of video. Wavelet transform is then used to filter the features from videos and kinematic data for the further clustering based on TSC. We refer the proposed segmentation method as TSC-SCAE for abbreviation. Finally, the segmentation result is promoted by merging the clusters according to four similarity measurements called PMDD based on principal component analysis, mutual information, data average and dynamic time warping, respectively.
Ii UNSUPERVISED TRAJECTORY SEGMENTATION BASED on TSC-SCAE
Ii-a Visual Feature Extraction Using SCAE
Stacked Convolutional Auto-Encoder (SCAE)  is an unsupervised feature extractor which is well compatible to high-dimensional input. It is much faster than other methods such as TSC-VGG and TSC-SIFT because of the simple neural network and unsupervised method. SCAE has more advantages in image processing as it can preserve the spatial relationship between pixels. The SCAE network for visual feature extraction is shown in Fig. 2, and the corresponding configuration is summarized in TABLE I.
Fig. 2 illustrates that the basic structure of encoder consists of convolutional layer and pooling layer. The input feature maps (for the first layer, it is the original image ) are convolved with a convolution layer to transfer the information to subsequent layers with the spatial relationship between pixels preserved. These feature maps then pass through a max-pooling layer to reduce the feature map size. After several above conv-pooling layers, a low dimension feature map can get from the encoder.
As shown in Fig. 2, the task of the decoder with the similar topology with the encoder is to reconstruct the encoding result to get the implied image information. Therefore, we need to up-sample the encoding result to recover the feature maps. To prevent the checkerboard effect caused by traditional transposed convolution, we use bilinear interpolation to do up-sampling before each convolutional layer. For further reduction of feature dimension, we employ two convolutional layers with the kernel size of after the last layer of the encoder and before the first layer of decoder respectively.
Adam optimization algorithm  is employed to minimize the MSE (mean-square error) based loss function, which can estimate the similarity between the reconstructed image of decoder output and the original image input to encoder. After the network training, a model (i.e., the weights of each layer) for image encoding and reconstructing can be got from the network. In the phase of feature extraction, we exclusively load the model’s encoder part to extract the features of each frame in the surgical video.
|Type||Patch Size||Stride||Output Size|
Ii-B Denoising Based on Wavelet Transform
After the feature extraction from the demonstration video, the visual and kinematic features are then feed to nonparametric mixture model for clustering. However, we find that these features usually suffer from the random noise. To get rid of it, wavelet transform based filter is employed due to its ability of multi-scale filtering and a low-pass filter is designed.
In this paper, we process the kinematic data and visual features with db10 wavelet, and a 5-level wavelet decomposition for denoising is performed. Fig. 3 and Fig. 4 demonstrate the comparison of kinematic and visual features before and after filtering based on wavelet transform.
After the filtering, visual and kinematic features then feed to a nonparametric mixture model to segment surgical trajectory. Considering the clustering performance, Transition State Clustering (TSC)  is adopted in this paper.
Iii SEGMENTATION PROMOTING BASED on PMDD
Most unsupervised trajectory segmentation methods usually have the problem of over-segmentation. To correct the wrongly segmented sub-trajectories that belong to the same cluster, a criterion is required to evaluate the similarity between segments. Taking a deep insight into the same sub-trajectory, they have a few implicit and explicit associations. Besides the similarity in spatial and temporal space, inner structure, variation node and moving trend are also the important factors. Taking these factors into consideration, we proposed a promoting algorithm based on PMDD consisting of four similarity measurements based on Principal Component Analysis (PCA), Mutual Information (MI), Data Average (DA) and Dynamic Time Warping (DTW).
Similarity measurement based on PCA: W. Krzanowski et al.  show that the PCA can be used to measure the similarity between segments. PCA mainly determine the internal link and structure between the segments. Considering two segments and , PCA could find several principle components of and , which make up a subspace representing the main information of and . The smaller subspace angle between and means the greater internal consistency between them. Thus, Similarity measurement based on PCA is defined by the angles between their subspaces comprised of principle components:
where is the number of principle components.
Similarity measurement based on MI: The surgery is a continuous process, the data change of the segments in same surgery sub-process is similar. Entropy can be interpreted as a measurement of the uncertainty of the particular variables. Therefore, MI is a good similarity measurement for variation degree between two segments, which is obtained by subtracting the joint entropy from the entropy and of both segments:
Similarity measurement based on DA: DA mainly reflects the spatial characteristic. During a sub-process of surgery, the trajectory in a short time interval is similar in the spatial space. Therefore, the distance between the centers of segments in spatial space is taken into account, as written as follows:
where and are mean vectors of segments and .
Similarity measurement based on DTW: Due to the difference of surgeons’ skill, the same action may show different sub-trajectories. One typical is the same behavior of different performance in temporal domain. The key issue of DTW is warping curve. Here, we take cumulative distance to calculate the best warping path while measure DTW similarity .
where is the element of warping path, is the compensation parameter that can be identified by cumulative distance.
where is the Euclidean distance between point and .
All above four similarity measurements are in different dimensions. Thus, the normalization is required to obtain the final measure. For , , , the smaller the value is, the more similar the two segments are. We perform the normalization of them using Eq. (6), and the normalization for is perform using Eq. (7). After that, the final similarity can be calculated by Eq. (8).
Then, according to final similarity measurements, segments that have high similarity can be merged iteratively. Considering the segmentation results , the final similarity of each pair of two adjacent segments will be calculated by Eq. (8) in each iteration, and then we obtain a set of results . Merge the pairs with the highest final similarity, and update comprehensive similarity , merge the most similar segments in the next iteration, until overall final similarity smaller than threshold . The segmentation promoting algorithm is summarized in Algorithm 1.
Iv Experimental Results
In this section, two sets of experiments are conducted to verify the performance of proposed unsupervised segmentation algorithm for surgical trajectory. In the first experiment, TSC-SCAE is evaluated with respect to the accuracy and overall running time, compared with the classic clustering methods including GMM and TSC. The effects of different data sources and wavelet transform based filtering are analyzed quantitatively. Second, the promoting method of segmentation is verified by following different methods using the kinematic data alone and the combination of video and kinematic data, respectively.
The dataset JIGSAWS  from Johns Hopkins University is used in the experiments, including data recordings and manual annotations. Data recordings consist of surgical video and kinematic data collected from Da Vinci Surgical System. The sampling frequency for both video and kinematics sources is 30Hz. The dataset contains three surgical tasks: Suturing (SU), Needle-Passing (NP) and Knot-Tying (KT), which are performed and annotated by 8 surgeons with different skill levels. The suturing and needle passing task are commonly used in literatures. In this paper, we adopt 11 demonstrations of these two tasks in the experiments, including the videos and kinematic data from 5 experts (E), 3 intermediates (I) and 3 novice (N). The kinematic data are in 38 dimensions, including position, angle velocity, angle of grasper, etc. All 11 videos of each task are used for SCAE model training and features extraction. The computational configuration used in the experiments is summarized in TABLE II.
|CPU||32 Intel Xeon E5-2620 v4 @ 2.10GHz|
|GPU||NVidia Tesla K40|
|CUDA Compute Capability||3.5|
Iv-a Quantitative Analysis of TSC-SCAE
Iv-A1 Accuracy Comparison
In this section, the accuracy of TSC-SCAE is compared using Normalize Mutual Information (NMI), which indicates the transfer status similarity between a predictive clustering result and the ground truth (manual annotations), it can be calculated by
where and are the information entropies of and , respectively. is mutual information. The range of NMI is [0,1], where 0 means that there is no correlation between two clustering results, while 1 represents the results are completely related.
We compare the proposed method TSC-SCAE with state-of-the-art methods, including TSC, GMM, TSC-VGG, TSC-SIFT and TSC-SCAE on the selected surgical demonstrations. According to the data source in the different methods, the experiments are divided into two categories: one use kinematics data alone and the other use both video and kinematic data. TABLE III shows NMI measurements of segmentation. We can see that our method TSC-SCAE achieves the best NMI among all trajectory segmentation tasks, it thanks to the use of video data and wavelet transform. Especially, using both video and kinematic data, the accuracy is improved by more than 2.6 times at most, compared with TSC-SIFT.
Overall, methods with both video and kinematic data are generally better than the ones using kinematics data alone. It is consistent with the results reported in literatures. The NMI of methods using kinematics data alone has a trend of decreasing with the growing proportion of non-expert (I N) demonstrations. This phenomenon is very significant in the suturing task, it is mainly because of the complexity and non-regularity of suturing task. What’s more, demonstrations from experts are usually smoother and rapider than non-experts do. However, when considering both kinematics and video data, the phenomenon is obviously weakened. It proves that video data can help eliminate the influence of irregular trajectory from intermediates and novices and is an effective compensation to achieve the better surgical trajectory segmentation.
As aforementioned, random noise may cause the potential interference to the result of segmentation. To solve this problem, we perform a multi-scale smoothing processing to the dataset by using db10 wavelet to filter out the small-scale noise, which indirectly improve the segmentation accuracy. Compared with the experiments without filtering in needle-passing task, the NMI is increased by 3.5-6.5, the improvement is 1.2-3.4 in suturing task.
Iv-A2 Overall Running Time Comparison
Another key indicator is overall running time, although surgery segmentation is not in strong real-time, the task also needs to be as fast as possible. Methods based on kinematics data alone, the running time is the cost of clustering and segmentation, while we need to add the time cost of video feature extraction for methods using visual and kinematic data (TSC-VGG, TSC-SIFT, etc.). For our method TSC-SCAE, the time cost is calculated in three parts, including visual feature extraction, wavelet transform based filtering and clustering segmentation.
The running time in different steps is summarized in TABLE IV. The segmentation methods based on both visual and kinematic features are about 10 times slower than the ones using kinematic data alone. It is mainly because of the time-consuming visual feature extraction. However, for the methods using both data sources, our method TSC-SCAE is almost 10 times faster than TSC-VGG and TSC-SIFT. The improvements of time efficiency is due to the high-efficiency unsupervised model for feature extraction of video data we employed.
Iv-B Evaluation of Segmentation Promoting
Over-segmentation is a common problem of clustering segmentation algorithm. To prove the validity of the proposed promoting approach as the post-processing step, we apply it to the mainstream clustering segmentation algorithms, including GMM, TSC based methods. NMI is used to measure the similarity of transition status in the segmentation clustering method. But it is not based on transfer state to merge in the promoting stage. Therefore, we choose segmentation accuracy (seg-acc) as the evaluation matrix, which can measure the similarity between the segmentation result and ground truth intuitively and accurately.
The calculation of seg-acc can divided in two steps. In the first step, we match resultant segments to the ground truth by maximizing the number of overlap frames between predicted segments and ground truth . In second step, it is true positive if the IOU (Intersection over Union) between the ground-truth segment and its corresponding resultant segments is more than a default threshold 40. We calculate the accuracy of each segment separately and then sum up them. Fig. 5 illustrates the calculation process and the seg-acc can be obtained using
where ‘’ and ‘’ represent start and end frame of segment and .
||Before Promoting||After Promoting|
|Needle Passing||Suturing||Needle Passing||Suturing|
As shown in TABLE V, the seg-acc of each method has been improved obviously for most cases. TSC-K is the biggest beneficiary with the improvement of seg-acc by 15.2 on average, while the accuracy is improved less for TSC-SIFT and TSC-VGG. In the experiment, we notice that it is difficult to refine the segmentation if the clustering results is far away from the ground truth. As shown in Fig. 11, each color represents a surgical activity segment, while the white segment indicates incorrect segment or over-segmented segment. Among all methods, the seg-acc of GMM based method even declines after the promoting. Because GMM needs to specify the number of merged class, so over-segmentation in GMM is not very common instead is wrong segmentation. For our method TSC-SCAE, the segmentation promoting yields up to 16.7 improvement with respect to seg-acc. In most cases, the resultant segmentation after the promoting is significantly improved. From the view of TABLE V, we notice that the improvement of non-expert demonstration is more outstanding than the expert do, because the non-expert demonstration produces more over-segmentation fragments.
In all experiments, TSC-SCAE obtains the best result of segmentation, it is proved that the proposed promoting method is very effective for the surgical trajectory segmentation. In general, it can be extended to most clustering segmentation algorithms.
This paper proposed a fast unsupervised method for surgical trajectory segmentation based on a compact stacking convolutional auto-encoder model and wavelet transform based filtering using multi-modal surgical demonstrations. The improvement with respect to the efficiency of segmentation is three-fold. First, new involved model can generate more discriminative visual features faster. Second, the short-range noises in the visual and kinematic features are filtered based on wavelet transform. Last but not least, a promoting approach is proposed to handle the over-segmentation problem. Compared with the state-of-the-art methods, experimental results demonstrate that the proposed algorithm can improve the accuracy of segmentation in an more efficient way.
This work was supported by the Project of Beijing Municipal Commission of Education (KM201710028017), National Natural Science Foundation of China (61702348, 61772351, 61602324), National Key R & D Program of China (2017YFB1303000, 2017YFB1302800), the Project of the Beijing Municipal Science & Technology Commission (LJ201607), Capacity Building for Sci-Tech Innovation - Fundamental Scientific Research Funds (025185305000), and Youth Innovative Research Team of Capital Normal University.
-  A. Guha, Y. Yang, C. Fermuuller, and Y. Aloimonos, “Minimalist plans for interpreting manipulation actions,” in Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on. IEEE, 2013, pp. 5908–5914.
-  C. E. Reiley, H. C. Lin, B. Varadarajan, B. Vagvolgyi, S. Khudanpur, D. Yuh, and G. Hager, “Automatic recognition of surgical motions using statistical modeling for capturing variability,” Studies in health technology and informatics, vol. 132, p. 396, 2008.
-  K. Shamaei, Y. Che, A. Murali, S. Sen, S. Patil, K. Goldberg, and A. M. Okamura, “A paced shared-control teleoperated architecture for supervised automation of multilateral surgical tasks,” in Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on. IEEE, 2015, pp. 1434–1439.
-  H. C. Lin, I. Shafran, T. E. Murphy, A. M. Okamura, D. D. Yuh, and G. D. Hager, “Automatic detection and segmentation of robot-assisted surgical motions,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2005, pp. 802–810.
-  N. Ahmidi, Y. Gao, B. Béjar, S. S. Vedula, S. Khudanpur, R. Vidal, and G. D. Hager, “String motif-based description of tool motion for detecting skill and gestures in robotic surgery,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2013, pp. 26–33.
-  L. Tao, L. Zappella, G. D. Hager, and R. Vidal, “Surgical gesture segmentation and recognition,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2013, pp. 339–346.
-  S. H. Lee, I. H. Suh, S. Calinon, and R. Johansson, “Autonomous framework for segmenting robot trajectories of manipulation task,” Autonomous robots, vol. 38, no. 2, pp. 107–141, 2015.
-  S. Krishnan, A. Garg, S. Patil, C. Lea, G. Hager, P. Abbeel, and K. Goldberg, “Transition state clustering: Unsupervised surgical trajectory segmentation for robot learning,” The International Journal of Robotics Research, vol. 36, no. 13-14, pp. 1595–1618, 2017.
-  A. Murali, A. Garg, S. Krishnan, F. T. Pokorny, P. Abbeel, T. Darrell, and K. Goldberg, “Tsc-dl: Unsupervised trajectory segmentation of multi-modal surgical demonstrations with deep learning,” in Robotics and Automation (ICRA), 2016 IEEE International Conference on, 2016, pp. 4150–4157.
-  J. Masci, U. Meier, D. Cireşan, and J. Schmidhuber, “Stacked convolutional auto-encoders for hierarchical feature extraction,” in International Conference on Artificial Neural Networks. Springer, 2011, pp. 52–59.
-  D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
-  W. Krzanowski, “Between-groups comparison of principal components,” Journal of the American Statistical Association, vol. 74, no. 367, pp. 703–707, 1979.
-  D. J. Berndt, “Finding patterns in time series: a dynamic programming approach,” Advances in knowledge discovery and data mining, pp. 229–248, 1996.
-  Y. Gao, S. S. Vedula, C. E. Reiley, N. Ahmidi, B. Varadarajan, H. C. Lin, L. Tao, L. Zappella, B. Béjar, D. D. Yuh, et al., “Jhu-isi gesture and skill assessment working set (jigsaws): A surgical activity dataset for human motion modeling,” in MICCAI Workshop: M2CAI, vol. 3, 2014, p. 3.
-  C. Wu, J. Zhang, S. Savarese, and A. Saxena, “Watch-n-patch: Unsupervised understanding of actions and relations,” in Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on. IEEE, 2015, pp. 4362–4370.