A complex network approach to stylometry
Statistical methods have been widely employed to study the fundamental properties of language. In recent years, methods from complex and dynamical systems proved useful to create several language models. Despite the large amount of studies devoted to represent texts with physical models, only a limited number of studies have shown how the properties of the underlying physical systems can be employed to improve the performance of natural language processing tasks. In this paper, I address this problem by devising complex networks methods that are able to improve the performance of current statistical methods. Using a fuzzy classification strategy, I show that the topological properties extracted from texts complement the traditional textual description. In several cases, the performance obtained with hybrid approaches outperformed the results obtained when only traditional or networked methods were used. Because the proposed model is generic, the framework devised here could be straightforwardly used to study similar textual applications where the topology plays a pivotal role in the description of the interacting agents.
Keywords: complex networks, network topology, pattern recognition
The human language represents a major factor responsible for the success of our species; and its written form is one of the main expression used to convey and share information. Owing to the ubiquity of language in several contexts, many linguistic aspects have been studied via the application of methods and tools borrowed from diverse scientific fields. As a consequence, several findings related to the origins, organization and structure of the language have been unveiled. One of the most fundamental patterns arising from statistical analysis of huge amounts of text is the Zipf’s law, which states that the frequency of the words decreases inversely to their rank [1, 2, 3]. Other fundamental recurrent pattern is the Heap’s law, which states that the vocabulary size grows slowly with the number of tokens of the document [4, 5, 6]. More recently, concepts from Physics have been applied to model several language features. For example, with regard to long-scale properties, concepts from dynamical systems have successfully been employed to compare the burstiness of the spatial disbribution of words in documents [7, 8]. In a similar fashion, the spatial distribution of words in documents and analogous systems have also been studied in terms of level statistics , entropy  and intermittency measurements 
A well-known approach to study written texts is the word adjacency network model [12, 13, 14], which considers short-scale textual properties to form the networks. In this representation, relevant words conveying meaning are modeled as nodes and adjacency relationships are used to establish links. Using this model, several characteristics of texts and languages have been inferred from statistical analyses performed in the structure of networks . Language networks have been increasingly employed to understand theoretical linguistic aspects, such as the origins of fundamental properties  and the underlying mechanisms behind language acquisition in early years . In practical terms, networks have been applied in the context of machine translations , autommatic summarization , sense disambiguation [21, 20], complexity/quality analysis [22, 23] and document classification . Despite the relative success of applying networks concepts to better understand language phenomena, in many real-world applications attributes extracted from networked models have not contributed to the advancement of the state of art. For example, when one considers the document classification task, a strong dependency of network features on textual characteristics have been observed. However, the best performance are still achieved with traditional statistical natural language processing features. In this context, the present study address this problem by devising methods that effectively take advantage of network properties to boost the performance of the textual classification task. Here I focus on the text classification based on stylistic features, where a text is classified according to stylistic marks left by specific authors  or literary genres . Upon introducing a hybrid classifier relying on the fuzzy definition of supervised pattern recognition methods , I show that the performance of style-based classifications can be significantly improved when topological information is included in traditional models. Given the generality of the proposed method, the framework devised here could be applied to improve the characterization and description of many related textual applications.
2 Representing texts as networks
There are several ways to map texts into networks [12, 15, 28, 29, 19, 30]. The most suitable form depends on the context of the application. In occasions where the semantics is relevant, the words sharing some semantical relationship are linked [20, 31, 32, 33, 22]. In a similar fashion, other semantic-based models link the words appearing in a given context [34, 35] (e.g. in the same sentence or paragraph). A general model for establishing significant “semantical” links between co-occurring elements was devised in . More specifically, the model considers the existence of a set of elements , which may occur in one or more sets of a given collection . Given two elements and , the model computes the probability that more than sets in contain both elements and as
where is the probability that and co-occur exactly times in the same set. Given a confidence level , the strength of the link between and is .
In applications where the style (or structure) plays an important role, the links among words are established according to syntactical relationships [19, 30]. A well-known approach for grasping stylistic features of texts is the word adjacency model [13, 37, 38], which basically connect adjacent words in the text. Differently from the model devised in , the word adjacency model captures the stylistic features of texts . It has been shown that the adjacency model is able to capture most of the syntactical links with the benefit of being language independent. Despite being a simplification of the syntactical analysis, the adjacency model has been employed in several contexts because the topological properties of word adjacency and syntactical networks are similar. Such high degree of similarity can be explained by the fact that most of the syntactical links occurs among neighbouring words . In the current paper, the traditional word adjacency representation was adopted.
Before mapping the text into a word adjacency network, some pre-processsing steps are usually applied. First, words conveying low semantic content, such as articles and prepositions, are removed from the text. These words, referred to as stopwords, are disregarded because they just serve to link content words. As a consequence, they can be straightforwardly replaced by network edges. The remaining words are then lemmatized, i.e. they are transformed to their canonical forms. To assist the lemmatization process, all words are labeled with their part-of-speech tags. Particularly, in this study, I used a model based on maximum entropy . To exemplify the construction of a word adjacency network, I show in table 1 the pre-processing steps performed in a short text. The corresponding network is shown in figure 1.
|Original text||In the middle of the road there was a stone / there was a stone|
|in the middle of the road there was a stone in the middle of the|
|road there was a stone. Never should I forget this event / in the|
|lifetime of my fatigued retinas / Never should I forget that in|
|the middle of the road / there was a stone / there was a stone|
|in the middle of the road / in the middle of the road there was|
|middle road stone stone middle road stone middle road stone never|
|Stopwords removal||forget event lifetime fatigued retinas never forget middle road stone|
|stone middle road middle road stone|
|middle road stone stone middle road stone middle road stone never|
|Lemmatization||forget event lifetime fatigue retina never forget middle road stone|
|stone middle road middle road stone|
3 Topological characterization of networks
A network can be defined as , where denotes the set of nodes and denotes the set of edges, which serve to link nodes. An unweighted network can be represented by an adjacency matrix , where each element stores the information concerning the connectivity of nodes and . If and are connected, then . Otherwise, . Note that, in a undirected network devoid of self loops, . Currently, there are several network measurements available to characterize the topology of complex networks . Here, I describe the main measurements applied for text analysis:
Degrees: the degree () is the number of edges connected to the node, i.e. . A relevant feature related to the degree that has been useful for text analysis are the are the average node degree and the standard deviation of the neighbors, which are given by
In text networks, both and have been useful to quantify the structural organization of texts .
Accessibility: the accessibility measurement is a extension of the node degree centrality . It is defined as a normalization of the diversity measurement , which quantifies the irregularity of a accessing neighbors through self-avoiding random walks . To define the accessibility, let be the probability of a random walker starting at node to reach node in exactly steps. The heterogeneity of access to neighbors can be quantified with the diversity measurement:
Given eq. 4, the accessibility () is computed as
It can be shown that the accessibility is bounded according to the relation , where is the number of nodes at the -th concentric level . An example of the computation of the accessibility in a small network is shown in figure 2. In the example, nodes 2, 3, 4 and 5 belong to the first concentric level and nodes 6, 7, 8, 9 and 10 belong to the second concentric level. When one considers a regular access to the second level (red configuration), the accessibility corresponds to the total number of nodes located at the -th concentric level. When some nodes are more accessed than others, the accessibility decreases because less nodes are effectively accessed.
In textual networks, it has been shown that the accessibility is more advantageous than other traditional centrality measurements as it is able to capture more information at further hierarchical levels . This measurement has been successfully applied to detect core concepts in texts . Furthermore, it has also been employed to generate informative summaries . A dependency of the distribution of this measurement with stylistic features of text was observed in .
Betweenness: the betweenness is a centrality measurement that considers that a node is relevant if it is highly accessed via shortest paths. If is the number of geodesic paths between and passing through node ; and is the total number of shortest paths linking and , then the betweenness is defined as:
In word adjacency networks, high frequent words usually take high values of betweenness. However, some words may act as articulation points whenever they link two semantical contexts or communities . It has been shown that the betweenness is able to identify the generality of contexts in which a word appears . More specifically, domain-specific words tend to assume lower values of betweenness when compared with more generic words.
Assortativity: several real-world networks are formed of nodes with a specific type of classification. For example, in social networks, individuals may be classified by considering their age, sex or race. When analyzing the connectivity patterns of networks, it might be relevant to study how distinct classes connect to each other. This type of analysis is usually performed with the so-called assortativity measurement . The assortativity can be computed as the Pearson correlation coefficient ():
where is the total number of edges. In word adjacency networks, the assortativity quantifies how words with distinct frequency appear as neighbors.
Clustering coefficient: the clustering coefficient () quantifies the local density of neighbors of a given node. The local definition of the clustering coefficient is given by the fraction of the number of triangles among all possible connected sets of three nodes:
Similarly to the betweenness, the clustering coefficient is useful to detect words appearing in generic contexts . However, differently from the betweenness, the clustering coefficient analyzes only the local neighborhood of nodes.
Average shortest path length: the average shortest path length () is the typical distance between any two nodes in the network. This measurement was used in this paper because it has been useful in stylistic-based applications . In texts, the average shortest path length quantifies words relevance. More specifically, according to this measurement, the most relevant words are those that are close to the hubs.
Most of the measurements described in this section are local measurements, i.e. each node possesses a value , where . For the purposes of this paper, it is necessary to sum up the local measurements. The most natural choice is to characterize the documents by computing the average , where stands for the average computed over the distinct words of the text. A disadvantage associated with this type of summing procedure is that all words receive the same weight, regardless of their number of occurrences in the text. To avoid this potential problem, I also computed the average value obtained when only the most frequent words are considered. The standard deviations and the skewness were also used to characterize the documents.
A drawback associated to the computation of local topological measurements in word adjacency networks is the high correlation found between these measurements and the node degree (i.e. the word frequency). To minimize this correlation, the following procedure was adopted. Each of the measurements was normalized by the average obtained over texts produced using a word shuffling technique, where the frequencies of words are preserved. If and are the average and deviation obtained over the random realizations, the normalized measurement and the error related to are
4 Traditional stylistic features
Frequency of words and characters
Traditional methods usually perform statistical analysis using specific textual features . An important contribution to the stylometry was introduced by Mosteller and Wallace , which showed that the frequency of function words (such as any, of, a and on) is useful to characterize the style in texts. Frequent words have also been used in strategies devised by physicists where the distance of frequency ranks was used to compute the similarity between texts [51, 52, 53]. Another relevant feature for characterizing styles in texts is the frequency of character bigrams (i.e. a sequence of two adjacent characters). These attributes have proven useful e.g. to detect the stylistic marks of specific authors .
The uneven spatial distribution of words along texts is a feature useful for characterizing the style of texts . The quantification of the homogeneity of the distribution of words along texts can be performed by using recurrence times, a standard measure employed to study time series . In texts, the concept of time is represented in terms of the number of words occuring in a given interval. For each word , the recurrence time is defined as the number of words appearing between two successive occurrences of plus one. For example, the recurrence times of the word “stone” in the lemmatized text shown in table 1 are , , , , and . If a word occurs times in a text comprising words, it generates a sequence of recurrence times . In order to consider the time until the first occurrence of and the time , the number of words between the last occurrence of and the last word of the text, the recurrence time is added to the set of recurrence times of word . As such, , where is the average over distinct ’s. Note that the average recurrence time does not provide additional information, since it only depends on the frequency. The intermittence (or burstiness) of the word is obtained from the coefficient of the variation of the recurrence times:
It has been shown that the intermittence has been useful to identify core concepts in texts, even when a large corpus is not available. Relevant words usually take high values of intermittency, i.e. . Stopwords, on the other hand, are evenly distributed along the text . To illustrate the properties of the intermittency measurement, figure 3 shows the spatial distribution of the words long and Hobson in the book “Adventures of Sally”, by P.G. Wodehouse. The frequency of these words are similar, yet their values of intermittency are quite different. Note that, while the distribution of long is relatively homogeneous, Hobson is unevenly distributed along the text. Because bursty concepts are the most relevant words , in the example of figure 3, Hobson would be considered a relevant character in the plot.
5 Supervised classification
In this section, I describe the two methods employed to combine distinct strategies of classification. The objective here is to combine evidences from both traditional and CN-based methods. Before introducing the methods for combining classifiers, the main concepts concerning the supervised learning task are presented.
In a typical supervised learning task, two datasets are employed: the training and the test dataset. The training dataset is the set of instances whose classes are known beforehand. The first component of the tuples represents the values of attributes used to describe the -th instance. The second element represents the class label of the -th training instance. In the supervised learning task, the objective is to obtain the map . The quality of the map obtained is evaluated with the test dataset . The technique used to evaluate the performance of the classifiers used in this study is the well-known 10-fold cross-validation .
An usual procedure in classification tasks is the quantification of the relevance of each attribute for the task. To quantify the relevance of attributes to discriminate the data, several indices have been proposed . A well-known index, the information gain, quantifies the homogeneity of the set of instances in when the value of the attribute is specified . Whenever a single class prevails when the value of an attribute is specified, the information gain associated to takes high values. Mathematically, the information gain is defined as
where is the entropy of the training dataset and is the entropy of the training dataset when the -th attribute is specified. The quantity is computed as
where is the set comprising all values taken by in .
To define this classifier, consider the following definitions. Let be the strength associating the -th instance to the -th class, where . The term , henceforth referred to as membership strength, can be interpret as the likelihood of instance to belong to class . Note that the quantification of depends on the classifier being used. A detailed description of methods for computing the membership strength can be found in [27, 58, 59, 60, 61]. Let be the membership strength obtained when topological features of complex networks are used and the membership strength obtained when traditional statistical features are analyzed. According to the hybrid strategy, the combination of both topological and traditional methods is achieved according to the following convex combination
where accounts for the weight associated to the topological strategy. Note that the combination of evidences performed in equation 14 yields a membership strength ranging in the interval . The final decision is then made according to the rule
where denotes the correct class label associated to the -th test instance. In practical applications, a screening on the can be useful to find its best value. However, this process might be computationally unaffordable for very large datasets. In this case, the process of finding the adequate value of the parameter can be made via application of optimization heuristics [62, 63].
To illustrate how the decision is performed with the hybrid algorithm, figure 4 shows the classification of a text modeled as a network. In the top panel, the central network is the text whose class is to be inferred. The left and right networks represent two networks in the training dataset. The node color denotes the node label, i.e. the word associated to the node. In this example, two texts are semantically similar if they share the same words. In the central panel, the scenario where the hybrid classified is set with is considered. Because is close to zero, the classification is mostly based on the number of shared words. In fact, the decision boundary in this case is created so that the test instance is classified in the same class as the right training instance. The bottom panel illustrates the decision made with . Now, the topological features of the text is the most prevalent feature used for the classification. As a consequence, the boundary decision moves to the right side so that the test instance is classified as the same class as the left training instance. Note that, the test instance as belonging to class because the and are topologically similar.
Similarly to the hybrid classifier, this classification scheme uses as attributes both traditional and topological features of texts modelled as networks. The objective of this approach is to use the topological attributes only when the classification performed with traditional features is not reliable, as revealed by the values of membership strength. Consider that the two most likely classes to which the unknown instance belongs are and , according to traditional features. As a consequence, , for each class . Let be the difference between the probabilities associated to the two most likely classes, i.e. . If such difference surpasses a given threshold , the tiebreaker performs the classification by using only traditional attributes. Conversely, topological attributes are employed to infer the class of the unknown instance. Equivalently,
Note that the threshold ultimately decides which attributes (traditional or topological) are used to perform the classification. An example of classification using this approach is shown in figure 5.
6 Results and discussion
The effectiveness of the combination of traditional textual features and topological measurements of networks was evaluated in the context of two natural language processing tasks. Both tasks rely upon the characterization of stylistic marks of texts. The studied tasks were the authorship attribution and genre detection problems. The classifiers chosen to compound the combining techniques were: kNN, Support Vector Machines (SVM), Random Forest (RFO) and Multilayer Perceptron (MLP). These classifiers were chosen because they usually yield good accuracy rates with default parameters .
6.1 Authorship recognition
In the authorship attribution task, the objective is to identify the authors of texts whose identity is lacking . The dataset employed here comprises books written by eight authors: Arthur Conan Doyle (ACD), Bram Stoker (BRS), Charles Dickens (CHD), Thomas Hardy (THH), Pelham Grenville Wodehouse (PGW), Hector Hugh Munro (HHM) and Herman Melville (HME). To show how the topological properties of the networks modelling books can be useful to improve the characterization of authors’ styles in texts, the following combination of attributes were considered:
INT+CN: the intermittence of stopwords were considered along with the topological measurements of the network modeling the text. As stopwords, I considered all the words that appeared at least once in all books of the dataset. According to the dataset, there is a total of stopwords.
FR+CN: the frequency of stopwords were considered along with the topological attributes of the networks.
BG+CN: the frequency of character bigrams were considered along with the topological attributes of the networks. All the possibilities of character bigrams () were considered in this analysis.
For each combination of attributes, the gain in performance when the topology is considered as an additional feature is represented as:
where denotes the accuracy rate obtained with either the hybrid or tiebreaker classifier and is the corresponding parameter employed, i.e. for the hybrid classifier and for the tiebreaker classifier. Figure 6 shows the values of when the kNN is used to compose the hybrid classifier. The results obtained for additional values of the parameter are shown in table 2. Figure 6 also shows the maximum accuracy rate obtained with the variation of :
With regard to the INT+CN combination, the addition of topological features of texts yielded accuracy rates much higher than the ones obtained with intermittency features alone, since for . Note that, in this case, the network approach alone performed better than the method based solely on intermittency features, because . This observation might explain the high gain obtained when networks are included as an additional feature. As for the FR+CN combination, the highest gain in performance was , which was obtained for . The lowest gain in accuracy occurred for the BIG+CN combination. In this case, the maximum gain was for . When other classifiers were used to compose the hybrid classifier, similar results have been found (see table 3): the highest and lowest improvement in performance occurred for the INT+CN and BIG+CN combinations, respectively. In general, the hybrid classification that combined traditional statistical features and topological measurements of networks improved the classification performance in the authorship recognition task. Interestingly, in several cases, the combination outperformed the accuracy rates obtained when the compounding strategies were analyzes separately, i.e. and .
Traditional and topological attributes of texts were also combined using the tiebreaker classifier. The results obtained with the kNN classifier for are shown in figure 7. The results obtained for other values of the parameter are displayed in table 4. The tiebreaker classifier built using the INT+CN combination provided an increase in performance of up to 31%. The maximum gain, obtained with the combinations FR+CN and BG+CN, were respectively 34.2% and 4.8%. With regard to the other classifiers compounding the tiebreaker technique, the results obtained are shown in table 5. Interestingly, there was no improvement when the SVM was used (). Differently, when the RFO classifier was used, an improvement of performance was observed in all three combinations of attributes. Finally, the tiebreaker classifier built with the MLP only provided a gain in accuracy for the combination INT+CN. While providing a gain in performance in several scenarios, the tiebreaker technique usually performed worst than the hybrid classifications performed with the same compound classifiers and parameters.
The results in tables 2-5 confirms that the topology of networks modeling texts is able to improve the characterization of texts because such textual description grasps relevant patterns that are mostly disregarded by traditional statistical approaches. This can be confirmed by optimized results obtained for and . To better understand the factors behind the discriminability power of the network model, the relevance of each topological for discriminating authors was computed with the information gain criterion. The best topological features were the standard deviation of the accessibility at the third level (), the standard deviation of the average neighboorhood degree () and the average clustering coefficient (). An example of discriminability provided by the topological strategy is shown in figure 8.
6.2 Style identification
In this section, I investigate if the topological features of complex networks are useful to complement the traditional textual characterization for the style identification task . The dataset used was the Brown corpus , which comprises documents that are classified either as informative prose (e.g. press reportage, popular folklore and scientific manuscripts) or imaginative prose (e.g general fiction, romance and love stories). Differently from the authorship attribution task, the objective here is to cluster together documents written using the same style, regardless of their authorship.
The results obtained for the style identification in the Brown corpus using the hybrid technique are shown in tables 6 and 7. Considering the INT+CN combination, the highest gain in performance were 29.0% and 25.8% for the kNN () and SVM methods. Minor improvements in accuracy were observed for other combinations, which might be explained by the high discriminability rates observed for the traditional techniques based on the frequency of stopwords () and character bigrams (). The results obtained with the tiebreaker technique were inferior to those obtained with the hybrid technique (results not shown). All in all, the results confirm that the inclusion of topological attributes might also improve the characterization of documents in the context of stylistic-based classification tasks. In this task, the most relevant topological features were the skewness and deviation of the clustering coefficient ( and ). A visualization of the discriminability provided by topological features is shown in figure 9.
In this study, I have shown that the styles of texts can be captured by measuring the topological properties of the corresponding network representation. More important than just to note a dependency between the structure of networks and stylistic features of texts, this study devised techniques to combine traditional and topological features, which were found to be of paramount importance to enhance the quality of current classification strategies. Two traditional stylometry tasks were studied: the authorship attribution and genre identification problems. In both tasks, the addition of topological features provided an improvement in classification performance. The highest improvement occurred with the hybrid classifier, which uses a linear convex combination of features from distinct textual evidences. The different nature of the quantities used to characterize texts suggests a complementary role in capturing distinct aspects of written texts. Because the adequate choice of parameters for the proposed technique is far from being a trivial task, I intend, as a future work, to devise a method that automatically assigns a value of and for each test instance.
I acknowledge financial support from São Paulo Research Foundation (FAPESP-Brazil) (grant number 14/20830-0).
-  Zipf GK (1949) Human Behavior and the Principle of Least Effort. Addison-Wesley
-  Kali R (2003) The city as a giant component: a random graph approach to Zipf’s law. Appl. Econ. Lett. 10 (4) 717–720.
-  Baixeries J, Elvevag B, Ferrer-i-Cancho R (2013) The evolution of the exponent of Zipf’s law in language ontogeny. PLoS ONE 8 e53227.
-  Bochkarev VV, Lerner EY, Shevlyakova AV (2014) Deviations in the zipf and heaps laws in natural languages. J. Phys.: Conf. Ser. 490 012009.
-  Font-Clos F, Boleda G, Corral A (2013) A scaling law beyond Zipf’s law and its relation to Heaps’ law. New J. Phys. 15 093033.
-  Lu L, Zhang Z, Zhou T (2013) Deviation of Zipf’s and Heaps’ laws in human languages with limited dictionary sizes. Sci. Rep. 3 1082.
-  Katz SM (1996) Distribution of content words and phrases in text and language modelling. Nat. Lang. Eng. 2 15–59.
-  Amancio DR (2015) Authorship recognition via fluctuation analysis of network topology and word intermittency. J. Stat. Mech. P03005.
-  Carpena P, Bernaola-Galvan P, Hackenberg M, Coronado AV, Oliver JL (2009) Level statistics of words: Finding keywords in literary texts and symbolic sequences. Phys. Rev. E 79 3.
-  Herrera JP, Pury PA (2008) Statistical keyword detection in literary corpora. EPJ B 63 135–146.
-  Allegrini P, Grigolini P, Palatella L (2004) Intermittency and scale-free networks: a dynamical model for human language complexity. Chaos Soliton Fract. 20 95.
-  Kulig A, Drozdz S, Kwapien J, Oswiecimka P (2015) Modeling the average shortest-path length in growth of word-adjacency networks. Phys. Rev. E 91 032810.
-  Roxas RM, Tapang G (2010) Prose and poetry classification and boundary detection using word adjacency network analysis. Int. J. Mod. Phys. C 21 503–512.
-  Masucci AP, Rodgers GJ (2006) Network properties of written human language. Phys. Rev. E 74 026102.
-  Cong J, Liu H (2014) Approaching human language with complex networks. Phys. Life Rev. 11 4.
-  Ferrer i Cancho R, Sole RV (2003) Least effort and the origins of scaling in human language. Proc. Natl. Acad. Sci. USA 100 788–91.
-  Corominas-Murtra B, Balverde S, Solé R (2009) The ontogeny of scale-free syntax networks: phase transitions in early language acquisition. Advs. Complex Syst. 12(3) 371–392.
-  Amancio DR, Antiqueira L, Pardo TAS, Oliveira Jr. ON, Nunes MGV (2008) Complex networks analysis of manual and machine translations. Int. J. Mod. Phys. C 19 583.
-  Amancio DR, Nunes MGV, Oliveira Jr. ON, Costa LF (2012) Extractive summarization using complex networks and syntactic dependency. Physica A 391 1855–1864.
-  Amancio DR, Oliveira Jr. ON, Costa LF (2012) Unveiling the relationship between complex networks metrics and word senses. Europhys. Lett. 98 18002.
-  Navigli R (2009) Word sense disambiguation: a survey. ACM Comput. Surv. 41 2.
-  Amancio DR, Aluisio SM, Oliveira Jr. ON, Costa LF (2012) Complex networks analysis of language complexity. Europhys. Lett. 100 58002.
-  Antiqueira L, Nunes MGV, Oliveira Jr ON, Costa LF (2007) Strong correlations between text quality and complex networks features. Physica A 373 811–820.
-  Mihalcea R, Radev D (2012) Graph-based natural language processing and information retrieval. Cambridge University Press, isbn 978-0-521-89613-9.
-  Stamatatos E (2009) A survey of modern authorship attribution methods. J. Am. Soc. Inf. Sci. Technol. 60 3 538–556.
-  Yu B (2006) An Evaluation of Text Classification Methods for Literary Study. Ph.D. Dissertation. University of Illinois at Urbana-Champaign, Champaign, IL, USA.
-  Keller JM, Gray MR, Givens Jr. JA (1985) A fuzzy k-nearest neighbor algorithm. IEEE Trans. Syst. Man, Cybern. 15 4 580–585.
-  Baronchelli A, Ferrer-i-Cancho R, Pastor-Satorras R, Chater N, Christiansen MH (2013) Networks in cognitive science. Trends Cogn. Sci. 17 348–60.
-  Antiqueira L, Oliveira Jr. ON, Costa LF, Nunes MGV (2009) A complex network approach to text summarization, Inform. Sci. 179 584–599.
-  Ferrer i Cancho R, Solé R, Kohler R (2004) Patterns in syntactic dependency networks. Phys. Rev. E 69 51915.
-  Capitan JA, Borge-Holthoefer J, Gomez S, Martinez-Romo J, Araujo L, Cuesta JA, Arenas A (2012) Local-based semantic navigation on a networked representation of information. PLoS ONE 7 8 e43694.
-  Gravino P, Servedio VDP, Barrat A, Loreto V (2012) Complex structures and semantics in free word association. Adv. Complex Syst. 15 1250054–1.
-  Silva TC, Amancio DR (2012) Word sense disambiguation via high order of learning in complex networks. Europhys. Lett. 98 58001.
-  Veronis J (2004) Hyperlex: Lexical cartography for information retrieval. Comput. Speech Lang. 18 3 223–252.
-  Widdows D, Dorow B (2002) A graph model for unsupervised lexical acquisition. Proceedings of the 19th International Conference on Computational Linguistics 1–7.
-  Martinez-Romo J, Araujo L, Borge-Holthoefer J, Arenas A, Capitan JA, Cuesta JA (2011) Disentangling categorical relationships through a graph of co-occurrences. Phys. Rev. E 84 046108.
-  Amancio DR, Oliveira Jr. ON, Costa LF (2012) Identification of literary movements using complex networks to represent texts. New J. Phys. 14 043029.
-  Amancio DR, Nunes MGV, Oliveira Jr. ON, Pardo TAS, Antiqueira L, Costa LF (2011) Using metrics from complex networks to evaluate machine translation. Physica A 390 131–142.
-  Amancio DR, Altmann EG, Rybski D, Oliveira Jr. ON, Costa LdF (2013) Probing the statistical properties of unknown texts: application to the Voynich manuscript. PLoS ONE 8 e67310.
-  Berger AL, Della Pietra VJ, Della Pietra SA (1996) A maximum entropy approach to natural language processing. Comput. Linguist. 22 1 39–71.
-  Costa LF, Rodrigues FA, Travieso G, Villas Boas PRV (2007) Characterization of complex networks: a survey of measurements. Adv. Phys. 56 167–242.
-  Antiqueira L, Pardo TAS, Nunes MGV, Oliveira Jr. ON (2007) Some issues on complex networks for author characterization. Inteligencia Artificial 11 51–58.
-  Travencolo B, Costa LF (2008) Accessibility in complex networks. Phys. Lett. A 373 89–95.
-  Costa LF, Tognetti MAR, Silva FN (2008) Concentric characterization and classification of complex network nodes: application to an institutional collaboration network. Physica A 387 6201–6214.
-  Amancio DR (2015) Probing the topological properties of complex networks modeling short written texts. PLoS ONE 10 e0118394.
-  Fortunato S (2010) Community detection in graphs. Phys. Rep. 486 75–174.
-  Amancio DR, Altmann EG, Oliveira Jr. ON, Costa LF (2011) Comparing intermittency and network measurements of words and their dependence on authorship. New J. Phys. 13 123024.
-  Newman MEJ (2003) Mixing patterns in networks. Phys. Rev. E 67 2 026126.
-  Manning CD, Schutze H (1999) Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA, USA.
-  Mosteller F, Wallace D (1964). Inference and Disputed Authorship: The Federalist. Reading, MA: Addison-Wesley.
-  Havlin S (1995) The distance between zipf plots. Physica A 216 148–50.
-  Vilensky B (1996) Can analysis of word frequency distinguish between writings of different authors? Physica A 231 705–11.
-  Yang ACC, Peng CK, Yien WK, Goldberger AL (2003) Information categorization approach to literary authorship disputes. Physica A 329 473–83.
-  Jankowska M, Keselij V, Milios E (2012) Relative n-gram signatures: document visualization at the level of character n-grams. IEEE Conference on Visual Analytics Science and Technology, Seattle, WA, USA.
-  Goh KI, Barabási AL (2008) Burstiness and memory in complex systems. Europhys. Lett. 81 48002.
-  Witten IH, Frank E (2005) Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, San Francisco.
-  Mitchell TM (1997) Machine Learning. The Mc-Graw-Hill Companies, Inc. ISBN 0070428077.
-  Lin C-F abd Wang S-D (2002) Fuzzy support vector machines. IEEE Trans. Neural Netw. Learn. Syst. 13(2) 464–471.
-  Olaru C and Wehenkel L (2003) A complete fuzzy decision tree technique. Fuzzy Set. Syst. 138(2) 221–254.
-  Bonissone P, Cadenas JM, Garrido MC and DÃaz-Valladares RA (2010) A fuzzy random forest. Int. J. Approx. Reason. 51(7) 729–747.
-  Pal SK and Mitra S (1992) Multilayer perceptron, fuzzy sets, and classification. IEEE Trans. Neural Netw. Learn. Syst. 3(5) 683–697.
-  Bergstra J and Bengio Y (2012) Random search for hyper-parameter optimization. J. Machine Learning Research 13: 281–305.
-  Bergstra J, Bardenet R, Bengio Y and Kegl B (2011) Algorithms for hyper-parameter optimization. Adv. Neural Inform. Process. Syst.
-  Amancio DR, Comin CH, Casanova D, Travieso G, Bruno OM, Rodrigues FA and Costa LF (2014) A systematic comparison of supervised classifiers. PLoS ONE 9 e94137.
-  Oakes M (2004) Ant colony optimisation for stylometry: the federalist papers. Proc. 5th Int. Conf. on Recent Advances in Soft Computing.
-  Patton JM, Can F (2004) A stylometric analysis of Yasar Kemal’s Ince Memed tetralogy. Computers and the Humanities 38 (4) 457–467.
-  Available from http://clu.uni.no/icame/manuals/BROWN/INDEX.HTM
-  McLachlan GJ (2004) Discriminant Analysis and Statistical Pattern Recognition. Wiley Interscience. ISBN 0-471-69115-1.