Characterization of the Handwriting Skills as a Biomarker for Parkinson’s Disease

Characterization of the Handwriting Skills
as a Biomarker for Parkinson’s Disease

R. Castrillón, A. Acien, J.R. Orozco-Arroyave, A. Morales, J.F. Vargas, R. Vera-Rodríguez, J. Fierrez, J. Ortega-Garcia, A. Villegas Universidad Católica de Oriente, Rionegro, Colombia. Universidad de Antioquia, Medellín, Colombia Universidad Autónoma de Madrid, Madrid, España This work was financed by CODI from University of Antioquia grant PRG2015-7683 and the Neurometrics project funded by UAM-Banco Santander.    Anonymous FG 2019 submission
Paper ID 163978
Abstract

In this paper we evaluate the suitability of handwriting patterns as potential biomarkers to model Parkinson’s disease (PD). Although the study of PD is attracting the interest of many researchers around the world, databases to evaluate handwriting patterns are scarce and knowledge about patterns associated to PD is limited and biased to the existing datasets. This paper introduces a database with a total of 935 handwriting tasks collected from 55 PD patients and 94 healthy controls (45 young and 49 old). Three feature sets are extracted from the signals: neuromotor, kinematic, and non-linear dynamic. Different classifiers are used to discriminate between PD and healthy subjects: support vector machines, k-nearest neighbors, and a multilayer perceptron. The proposed features and classifiers enable to detect PD with accuracies between 81% and 97%. Additionally, new insights are presented on the utility of the studied features for monitoring and detecting PD.

\FGfinalcopy\pubid

978-1-7281-0089-0/19/$31.00 ©2019 IEEE

I Introduction

Parkinson’s disease (PD) is a neurodegenerative disorder that occurs due to loss of dopamine, a neurotransmitter that helps in regulating muscle movements. The disease is chronic and progressive, and affects multiple areas of the central nervous system. PD is characterized by alterations of the motor system such as bradykinesia, resting tremor, muscular rigidity, and posture [1]. There is no known cure for PD [2] and its early diagnosis is crucial for more effective treatments [3]. The process to diagnose and evaluate the progression of PD is purely subjective [4]. To evaluate the disease progression neurologists administer different clinical tests such as the MDS-UPDRS and H&Y scales to the patients. These scales are usually limited to evaluate upper limb motor skills [5]. Also, small changes in the progression are not detectable through those scales. Besides, the probability of incorrect diagnosis based on such scales is around 25% [6]. A more accurate assessment of motor activities would allow medical doctors and researchers to measure the disease progression and make timely decisions about the therapy.

Handwriting analysis offers the possibility to assess and monitor those motor skills of PD patients. Different abnormal behaviors in handwriting are observed in PD patients. For instance, micrographia occurs in 5% of the patients before other motor symptoms appear, and about 30% of the handwriting worsening cases are reported after the medical diagnosis [7]. Handwriting tasks have significant advantages: they are simple, less intrusive, natural, do not need specialized infrastructure and can be administered remotely. There are several studies in the literature which are focused on the automatic evaluation of handwriting of PD patients considering different tasks and machine learning algorithms. Drotar et al. use kinematic and pressure analysis to classify between PD patients and healthy subjects. Using a population of 37 PD and 38 Healthy Controls (HC), they report classification accuracies of up to 82%. The studies are carried out considering different tasks including spirals, sentences and characters [6]. Mucha et al. proposed a new approach named “Fractional Derivative” to improve the classification considering kinematic handwriting signals extracted from drawings of a spiral. They use a database of 30 PD/36 HC and report a classification accuracy of 72.39% [8]. Using repetitive cursive loops and kinematic features for evaluation, Haremans et al. found a correlation of between the handwriting measurements and the medical scales considering a corpus with 30 PD/15 HC [9]. From a population of 24 PD/20 HC, Kotsavasiloglou et al. report classification results of 91% using kinematic features and entropy analysis from drawings of horizontal lines [10]. Finally, Taleb et al. report results of 96.87% when classifying between PD patients and HC subjects. Seven tasks are considered from a corpus with 16 PD/16 HC [11]. All these efforts and others have been summarized in recent surveys [12] [13].
          To the best of our knowledge, this paper introduces the largest database for the analysis of online handwriting of PD patients and control users including both young healthy and elderly people with similar age and educational level with respect to the PD patients. As we will see later, both groups of controls are important in order to characterize neuromotor diseases against typical neuromotor degradation caused by the age. Additionally, we propose to evaluate different tasks (individually and merged) modeled considering several feature sets including kinematics, nonlinear dynamics, and neuromotor. The automatic discrimination between PD patients and HC subjects is evaluated considering three different classifiers: k-nearest neighbors (KNN), support vector machines (SVM), and a multilayer perceptron (MLP).

PD Patients EHC (Old Controls) YHC (Young Controls)
Male Female Male Female Male Female
Number ..20 35 27 22 27 18
Age …..6410.3 5812.9 6611.1 5910.6 254.93 233.9
Range of Age ..41–80 29–83  49–85  43–83  17–42  19–32
UPDRS 3919 3322
H&Y 2.30.5 2.40.8
TABLE I: Demographic and clinical data of patients and controls

Ii Data

Ii-1 Subjects

A total of 55 non-demented PD patients (35 female) with an average age of 60 years (standard deviation 11.9) were enrolled for this study. Two sets of healthy participants are also considered, one is formed with 49 elderly people (22 female), namely EHC, who are matched in age and education level with respect to the patients, and the other one includes 45 young healthy controls (18 female), namely YHC, with ages between 17 and 42 years. We consider that it is important to include both groups of controls to differentiate between patterns associated to the PD disease and patterns associated to the natural degradation of the neuromotor abilities with the age. During the recording sessions all of the patients were under the effect of their medication (i.e., ON-state). Further details about the participants are presented in Table I.

Ii-2 Tasks

The participants were asked to complete 17 different handwriting tasks following a template. The first tasks consisted of writing the letters l and m in a continuous and long trace. Other tasks include the digits (0 to 9), the ID, name and signature of the participant, a free sentence, and the alphabet. The other nine tasks consist of geometrical figures including an Archimedean spiral, a circle with and without a template, a house, two concentric rectangles, a rhombus, a cube, and the Rey-Osterrieth complex figure.

Ii-3 Acquisition system

The handwriting signals were recorded using a commercial tablet Wacom Cintiq (13HD Touch, 180 Hz of sampling frequency), which captures different signals including x-position, y-position, pressure, in-air movement, azimuth and altitude of the pen over the tablet, and writing time. Although all of these signals were captured with the tablet, this paper only considers experiments with the first three signals.

Iii Methodology

Iii-a Feature Extraction

Kinematic, non-linear dynamics and neuromotor features are extracted from the signals. Since there are tasks composed by several strokes, the features are extracted globally, i.e., per task, and also per stroke within each task. Strokes are segmented according to the pen-down and pen-up movements. A total of eleven statistical functionals are computed from the features: mean value, median, standard deviation, 1st percentile, 99th percentile, difference between the 99th and 1st percentiles, maximum, minimum, kurtosis and skewness. This procedure results in a 921-dimensional feature vector per task containing a total of 452 kinematic features, 354 nonlinear dynamics features and 115 neuromotor features. Although we have not included all features proposed in the literature, we consider that this feature set is representative of the state-of-the-art [13].

Iii-A1 Kinematic features

Global features refer to features extracted from the whole handwriting task. Mean velocity, max acceleration, distance between adjacent points or total duration are examples of these features. Global feature sets have been used to characterize handwriting signatures for many years with good performance [14] and, more recently, to characterize swipe patterns [15]. But they have never been studied to characterize PD, so in this paper we will analyze if they are suitable for this purpose. Many global feature sets have been proposed in the online handwriting recognition literature [16, 17, 18]. In this paper we use the extended set proposed in [17] which comprises 100 of the best performing global features adapted for online signatures.

Iii-A2 Nonlinear Dynamics features

These are computed with the aim of modeling stability, non-stationarity in muscular movements that cannot be accurately modeled with classical approaches like those based on kinematics. There are studies that show the relationship between motor activities of handwriting and chaotic processes [19]. The nonlinear approach has been successfully applied to model other bio-signals like voice [20] and gait [21]. The first step in the analysis of non-linear dynamics is the reconstruction of the phase space. It allows the study of the dynamic behavior of a time series. Periodic signals exhibit closed trajectories in the phase space while non periodic signals show irregular and chaotic patterns. In this paper, different nonlinear features are extracted from the reconstructed attractor such as correlation dimension, Lempel-Ziv complexity, largest Lyapunov exponent, Hurst exponent, empirical mode decomposition, and entropy. Other non linear features typically extracted to model handwriting signals are considered including the Shannon entropy, 2nd and 3rd order Renyi entropy, and the signal-to-noise ratio calculated using the conventional energy definition and the Teager-Kaiser energy [22].

Iii-A3 Neuromotor features

The Sigma-Lognormal model first introduced by [23] decomposes the velocity profile of human handwriting into stroke velocity signals with lognormal shape. The Sigma-Lognormal theory states that lognormal functions describe the rapid changes in the velocity of writing movements produced by neuromotor signals [24]. According to this, the velocity signal of each of these strokes, , can be described with lognormal shape. The velocity profile can be used as a marker of neuronological disorders. Healthy patients tend to show velocity signals with less number of lognormals and stable bandwidths while the PD patients velocity signals show a large number of lognormals (due to poor motor control ability) and variable bandwidths. Up to 28 features are calculated and used to depict the neuromotor ability of the user according to space-based and time-based features from the lognormal parameters (see [25] for all details). The neuromotor feature set is computed averaging the parameters of all lognormals from the handwriting task to obtain a single value for task/feature.

Iii-B Classification and parameter optimization

Three different classifiers are tested in this study, KNN, SVM and MLP. The KNN consists of assigning certain class to a test sample according to the number of nearest neighbors () that such a sample has in the feature space. The second classifier considered is a radial basis function SVM (RBF-SVM). In this case two meta-parameters (the margin parameter and the bandwidth of the Gaussian kernel ) need to be optimized in the training process with the aim of finding the optimal hyperplane that better separates PD patients and HC subjects. The third classifier is a fully connected feed-forward neural network with an input layer, several hidden layers, and an output layer. A single neuron in the MLP is able to separate the input space into two subspaces by a hyperplane which is defined by the weights and the threshold. The MLP classifier uses the back-propagation algorithm for the adaptation of the weights [26].

Meta-parameter optimization

The meta-parameters of the three classifiers are optimized following a similar strategy. The process consists of a leave-one-out cross-validation strategy in a grid-search over a set with different candidate values for the meta-parameters. The optimal values are found considering only one of the 17 tasks that the participants performed during the recording process. The other 16 tasks are considered as the test set. In the case of the KNN the number of neighbors is optimized in the set ; the meta-parameters of the RBF-SVM, and , are optimized also in a grid search where and similarly . The number of layers in the MLP is also optimized in a grid search within the set .

Experiments and feature sets
YHC vs PD EHC vs PD YHC vs EHC
Kinematic Non linear Neuromotor All Kinematic Non linear Neuromotor All Kinematic Non linear Neuromotor All
Alphabet 91.8% 87.6% —- 90.7% 73.3% 71.3% —- 71.3% 88.3% 74.5% —- 77.7%
Circletemplate 70.3% 73.0% 59.5% 71.6% 55.7% 65.7% 55.7% 65.7% 68.4% 73.7% 65.8% 75.0%
Cube 84.2% 72.3% 70.3% 76.2% 68.6% 65.7% 61.0% 68.6% 76.6% 69.1% 62.8% 60.6%
Freewriting 83.7% 80.6% —- 81.6% 65.7% 63.7% —- 68.6% 79.3% 75.0% —- 72.8%
House 81.2% 77.2% 72.3% 82.2% 76.2% 72.4% 65.7% 72.4% 67.0% 60.6% 62.8% 72.3%
ID 78.8% 84.8% 69.7% 83.8% 68.0% 66.0% 55.3% 67.0% 86.2% 78.7% 73.4% 75.5%
Name 91.0% 81.0% 69.0% 82.7% 63.5% 63.5% 62.5% 60.6% 84.0% 76.6% 59.6% 80.9%
Numbers 83.8% 84.8% 68.7% 72.0% 65.0% 61.2% 61.2% 65.0% 73.4% 73.4% 64.9% 75.5%
Line1 86.7% 72.0% 77.3% 90.0% 62.0% 62.0% 47.9% 69.0% 78.9% 69.7% 73.7% 76.3%
Line2 76.0% 77.3% 54.7% 85.9% 64.8% 56.3% 53.5% 62.0% 80.3% 80.3% 71.1% 80.3%
Rectangles 83.2% 79.2% 68.3% 71.3% 61.0% 65.7% 50.5% 73.3% 62.8% 67.0% 61.7% 63.8%
Rey 87.5% 77.1% —- 90.6% 68.7% 64.6% —- 73.7% 80.6% 77.4% —- 78.5%
Rhombus 72.3% 78.2% 65.3% 74.3% 54.3% 61.0% 54.3% 59.0% 72.3% 64.9% 64.9% 70.2%
Signature 87.9% 81.8% 70.7% 93.9% 71.8% 69.9% 65.0% 75.7% 78.7% 62.8% 58.5% 76.6%
Spiral 71.3% 73.3% 68.3% 77.2% 61.0% 61.9% 59.0% 63.8% 58.5% 73.4% 55.3% 61.7%
Spiraltemplate 78.2% 77.2% 79.2% 80.2% 60.6% 62.5% 53.8% 72.1% 69.9% 64.5% 63.4% 66.7%
Circle* 78.2% 77.2% 73.3% 73.3% 61.9% 58.1% 55.2% 67.6% 78.7% 66.0% 72.3% 75.5%
Opt. params. C=1000 C= 10 C=1 C=2000 C=2000 C=100 C=1 C=100 C=10 C=1 C=2000 C=10
* This result corresponds to the training process for optimizing the meta-parameters.
TABLE II: Classification results per task using the RBF-SVM

Iv Experimental Results

Table II shows the best results obtained for the different classification experiments: YHC vs PD; EHC vs PD; and YHC vs EHC. All of them correspond to the RBF-SVM, and similar results were found with the MLP. The results obtained with the KNN were not satisfactory. Note that similar results are obtained with the kinematic features and with the combination of all of the features. Alphabet and Signature tasks present the best classification accuracy: over 90% for YHC vs PD and over 70% for EHC vs PD. On the other hand, Circle with template and Rectangles present performances below 60% in some cases. In the tasks of greatest complexity (Alphabet, Freewriting and Rey) the neuromotor characteristics were not calculated because these features are designed for tasks of simple shapes and short strokes. In addition to the tests on individual tasks, all of the tasks are combined to create a generalized model. This combination is performed in late-fusion strategy, i.e., the scores of the classifier (RBF-SVM in all of the cases) are combined according to the mean rule to obtain a new score [27]. The results are reported in Table III.

Set of features Class. YHC vs PD EHC vs PD YHC vs EHC
Kinematics SVM 96.9% 78.3% 94.4%
KNN 93.7% 74.9% 87.5%
MLP 96.9% 73.4% 94.4%
Non linear SVM 95.6% 78.3% 91.6%
KNN 94.3% 61.4% 86.2%
MLP 96.9% 78.3% 94.4%
Neuromotor SVM 88.0% 51.6% 81.9%
KNN 89.3% 65.2% 79.1%
MLP 81.8% 59.9% 81.9%
All SVM 96.9% 81.7% 97.2%
KNN 96.9% 73.4% 87.5%
MLP 96.9% 78.3% 94.4%
Note: the Circle task was not considered, as it was used before for training.
TABLE III: Results of the classification combining all tasks

The results obtained with the fusion strategy are better than those obtained with individual models for each task. Further research in this topic may help to clarify which are the most important tasks to discriminate between PD and healthy subjects, and what is the best way to combine them.

Figure 1 shows the receiver operating characteristics curves (ROC) that result from the analysis of the classification between PD and HC subjects. The four curves correspond to the results obtained with the RBF-SVM considering three different feature sets (kinematics, nonlinear dynamics, and neuromotor) and the combination of all of them following the late-fusion strategy. It can be observed that the experiment classifying between YHC and PD patients presents the best results in most of the cases. When nonlinear dynamics features are used, the results of YHC vs PD are similar to those obtained in YHC vs EHC. The combination of all of the models presents the best results which confirms that complimentary information can be obtained from each feature set. Note also that the most difficult experiment is always the classification between EHC and PD patients, which confirms other works in the literature where the effect of aging in handwriting is reported.

(a) Kinematics.
(b) Dynamics Non Linear.
(c) Neuromotor.
(d) All features.
Fig. 1: ROC curves YHC vs PD, EHC vs PD, and EHC vs YHC (SVM-RBF classifier). Area Under the Curve AUC=1 for perfect classification.

V Conclusions

In this work we have presented a large database of online handwriting for research in Parkinson Disease (PD) diagnosis. The richness of the database is not only in the number of PD patients and control subjects, but also in the quantity and diversity of the tasks performed. The techniques used both for the extraction of features and for the optimization and classification process show the potential of online handwriting as a valid biomarker for the study of PD. We report values of 96.9% accuracy in the classification of PD vs YHC (Young Healthy Controls), 81.7% in the classification of PD vs EHC (Elderly Healthy Controls), and 97.2% in the classification of EHC vs YHC.

Handwriting is a complex task that involves different dimensions: cognitive, intelligibility, visual and motor, where different muscle groups intervene, degrees of freedom, movement of the arms, turns of the wrist, extension of the fingers, etc. Within motor activity, bradykinesia, tremor, involuntary movements, and muscle stiffness are all distinctive signs of PD that can be evaluated from handwriting tasks. Such handwriting tasks can help to identify specific characteristics of the disease in early stages for the opportune diagnosis and monitoring of the disease.

The analysis of kinematic features showed an outstanding accuracy in the identification of patients with PD. Those features were also expanded including non-linear dynamics and neuromotor features. With the neuromotor features we evaluated the correlation existing between the order issued in the central nervous system and the action performed directly by the motor system. This synergistic behavior is strongly affected by the alteration of the dopaminergic system typical of PD. In a complementary way, the analysis of nonlinear dynamics measured atypical handwriting produced by chaotic behaviors at muscular level. In practice, the considered neuromotor features did not yield satisfactory results. With other processing methods we may be more successful in generating relevant information related to neuromotor actions from the handwriting signals considered.

On the other hand, motor skills deteriorate naturally with aging, and it is crucial to differentiate such degradation with respect to the motor degradation caused by PD. In this work we have also used the different tasks to study aspects related to aging by considering both young and elderly control subjects. We advocate to use both types of controls in this kind of studies to distinguish between the damage caused by the disease and age-specific effects.

As future work we propose to carry out a taxonomic study of each of the tasks based on the experience of medical specialists, to complement the proposed analyses with offline writing analysis from historical samples of patients, to expand the database periodically to perform analysis of the progression of the disease, to explore other techniques of classification and feature extraction, to include techniques of dimensionality reduction and feature selection.

References

  • [1] J. Jankovic, ”Parkinson’s disease: clinical features and diagnosis”, in Journal of Neurology, Neurosurgery & Psychiatry, vol. 79, no 4, 2008, pp 368-376.
  • [2] B.S Connolly and A.E. Lang, ”Pharmacological treatment of Parkinson disease: a review”, in Jama, vol. 311, no 16, 2014, pp 1670-1683.
  • [3] G. Becker et al, ”Early diagnosis of Parkinson’s disease”, in Journal of Neurology, vol. 249, no 3, 2002, pp iii40-iii48.
  • [4] S. Rosenblum et al, ”Handwriting as an objective tool for Parkinson’s disease diagnosis” in Journal of Neurology, vol. 260, no 9, 2013, pp 2357-2361.
  • [5] E.J. Smits et al, ”Standardized handwriting to assess bradykinesia, micrographia and tremor in Parkinson’s disease” in PloS ONE, vol. 9, no 5, 2014, pp e97614.
  • [6] P. Drotár et al, ”Evaluation of handwriting kinematics and pressure for differential diagnosis of Parkinson’s disease” in Artificial intelligence in Medicine, vol. 67, 2016, pp 39-46.
  • [7] M. Thomas and A. Lenka and P. Kumar Pal, ”Handwriting analysis in Parkinson’s disease: Current status and future directions”, in Movement Disorders Clinical Practice, vol. 4, no 6, 2017, pp 806-818.
  • [8] J. Mucha et al, ”Fractional Derivatives of Online Handwriting: A New Approach of Parkinsonic Dysgraphia Analysis”, in Intl. Conf. on Telecommunications and Signal Processing, IEEE, 2018.
  • [9] E. Heremans et al, ”Handwriting Impairments in People With Parkinson’s Disease and Freezing of Gait”, in Neurorehabilitation and neural repair, vol. 30, no 10, 2016, pp 911-919.
  • [10] C. Kotsavasiloglou et al, ”Machine learning-based classification of simple drawing movements in Parkinson’s disease”. Biomedical Signal Processing and Control,vol. 31, 2017, pp 174-180.
  • [11] C. Taleb et al, ”Feature selection for an improved Parkinson’s disease identification based on handwriting”, in Intl. Workshop on Arabic Script Analysis and Recognition, IEEE, 2017. pp 52-56.
  • [12] D. Impedovo and G. Pirlo, ”Dynamic handwriting analysis for the assessment of neurodegenerative diseases: a pattern recognition perspective”, in IEEE Reviews in Biomedical Engineering, vol. 3, 2018.
  • [13] C De Stefano and F. Fontanella and D. Impedovo and G. Pirlo and AS. di Freca, ”Handwriting analysis to support neurodegenerative diseases diagnosis: A review”, in Pattern Recognition Letters, 2018.
  • [14] J. Fierrez and J. Ortega-Garcia, ”On-line signature verification”, in Handbook of Biometrics, Springer, 2008, pp 189-209.
  • [15] A. Acien and A. Morales and J. Fierrez and R. Vera-Rodriguez and J. Hernandez-Ortega, ”Active detection of age groups based on touch interaction”, in IET Biometrics, vol. 8, no 1, 2019, pp 101-108.
  • [16] A. Serwadda and V.V. Phoha and Z. Wang, ”Which verifiers work?: A benchmark evaluation of touch-based authentication algorithms”, in IEEE BTAS, 2013.
  • [17] M. Martinez-Diaz and J. Fierrez and R. P. Krish and J. Galbally, ”Mobile signature verification: Feature robustness and performance comparison”, in IET Biometrics, vol. 3, no 4, 2014, pp 267-277.
  • [18] J. Fierrez and A. Pozo and M. Martinez-Diaz and J. Galbally and A. Morales, ”Benchmarking touchscreen biometrics for mobile authentication”, in IEEE Trans. on Information Forensics and Security, vol. 13, no 11, 2018, pp 2720-2733.
  • [19] M.G. Longstaff and A. Heath, ”A nonlinear analysis of the temporal characteristics of handwriting”, in Human Movement Science, vol. 18, no 4, 1999, pp 485-524.
  • [20] C.M. Travieso and J.B. Alonso-Hernández and J.R. Orozco-Arroyave et al, ”Detection of different voice diseases based on the nonlinear characterization of speech signals”, in Expert Systems with Applications, vol. 82, no 1, 2017, pp 184-195.
  • [21] P.A. Pérez-Toro, J.C. Vásquez-Correa, T. Arias-Vergara et al, ”A non-linear dynamics approach to classify gait signals of patients with Parkinson’s disease”, in Communication in Computers and Information Science, vol. 916, 2018, pp 268-278.
  • [22] P. Drotar et al, ”Decision support framework for Parkinson’s disease based on novel handwriting markers”, in IEEE Trans. on Neural Systems and Rehabilitation Engineering, vol. 23, no 3, 2015, pp 508-516.
  • [23] C. O’Reilly and R. Plamondon, ”Development of a sigma-lognormal representation for on-line signatures”, in Pattern Recognition, vol. 42, no 12, 2009, pp 3324-333.
  • [24] T. Duval and C. Remi and R. Plamondon and J. Vaillant and C. O’Reilly, ”Combining sigma-lognormal modeling and classical features for analyzing graphomotor performances in kindergarten children”, in Human Movement Science, vol. 43, 2015, pp 183-200.
  • [25] A. Fischer and A. Plamondon, ”A dissimilarity measure for on-line signature verification based on the sigmalognormal model”, in Biennial Conf. of the International Graphonomics Society, 2015.
  • [26] D. Gil and D.J. Manuel, ”Diagnosing parkinson by using artificial neural networks and support vector machines”, in Global Journal of Computer Science and Technology, vol. 9, no 4, 2009, pp 63-71.
  • [27] J. Fierrez and A. Morales and R. Vera-Rodriguez and D. Camacho, ”Multiple classifiers in biometrics. Part 1: Fundamentals and review”, in Information Fusion, vol. 44, 2018, pp 57-64.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...

Supplementary Materials

346959
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description