Generalized entropies and anomalous diffusion
Many physical, biological or social systems are governed by history-dependent dynamics or are composed of strongly interacting units, showing an extreme diversity of microscopic behavior. Macroscopically, however, they can be efficiently modeled by generalizing concepts of the theory of Markovian, ergodic and weakly interacting stochastic processes. In this paper, we model stochastic processes by a family of generalized Fokker-Planck equations whose stationary solutions are equivalent to the maximum entropy distributions according to generalized entropies. We show that at asymptotically large times and volumes, the scaling exponent of the anomalous diffusion process described by the generalized Fokker-Planck equation and the phase space volume scaling exponent of the generalized entropy bijectively determine each other via a simple algebraic relation. This implies that these basic measures characterizing the transient and the stationary behavior of the processes provide the same information regarding the asymptotic regime, and consequently, the classification of the processes given by these two exponents coincide.
Real world processes are often characterized by the presence of a large number of interacting phenomena at multiple time- or length scales , and thus, they are usually described by stochastic models that are strongly interacting or history-dependent [2, 3, 4, 5]. A way to understand and classify these processes in terms of stationary and non-stationary probability densities is to generalize the concepts of statistical mechanics that already proved to be very powerful for describing weakly interacting, ergodic and Markovian systems [6, 7]. One such concept is entropy, which assigns a likelihood to macrostates, that is, to stationary distributions over microstates . Maximizing this likelihood, possibly in the presence of external constraints, yields the most probable stationary distribution characterizing the system, called the Maximum Entropy (MaxEnt) distribution, which plays a key role in describing the stationary behavior of stochastic systems.
For example, the Boltzmann-Gibbs entropy form, , where runs over the microstates, follows from the assumption that the system realizations are independent and distinguishable: based on prior microstate probabilities , the likelihood that the fraction of N realizations is in microstate , is given by a multinomial distribution as
In the expression above, the likelihood can be factorized to a -independent multiplicity and a -dependent factor corresponding to the prior distribution over the microstates. Taking the logarithm of both sides and rescaling by yields
When considering the thermodynamic limit, becomes the relative entropy between the prior distribution and the realized histogram ; is corresponding to the Boltzmann-Gibbs entropy, and is the so-called cross-entropy. Finding the most likely histogram, taking into account the possible non-uniform prior distribution , is in fact equivalent to the minimization of the relative entropy, not to the maximization of the entropy. Nevertheless, with uniform prior , where is corresponding to the phase space volume, the resulting distributions coincide.
In general, however, the realizations are not independent. Instead, their interaction can be macroscopically modeled by a corresponding entropy functional, which in principle can take infinitely many different forms. Similarly to the theory of renormalization group describing critical phenomena, an apprehensive characterization of these entropies can be made by observing what are the relevant and irrelevant parameters as we approach infinite system size . Both axiomatic and combinatorial considerations suggest that the asymptotic scaling of the generalized entropy forms with provides a meaningful classification of the entropies.
This classification is based on the fundamental result by Hanel and Thurner about the entropy functionals that can be written as a sum of a pointwise function over microstates, . As they showed, the first three Shannon-Khinchin (SK1-SK3) axioms [11, 12]
is continuous in
is maximal for the uniform distribution,
is invariant under adding a zero-probability state to the system,
permit only the following asymptotic scaling relation for any entropic forms:
or, equivalently, in terms of ,
with and .
Hence, the scaling exponent can be used to parametrize the equivalence classes of the generalized entropy forms. For example, the Boltzmann-Gibbs entropy, where , is corresponding to , whereas the Tsallis entropy , with is corresponding to . Consequently, each such equivalence class can be represented by a Tsallis entropy.
Using combinatorial arguments, Hanel, Thurner and Gell-Mann showed that generalizing the MaxEnt principle (2) to non-multinomial processes yields SK1-SK3 compatible entropies with asymptotic form 3, hence, such entropies can also be classified in the same manner by means of the exponent .
Note that the fourth Shannon-Khinchin axiom is not considered in this analysis, therefore, the entropy of a joint distribution is not always decomposable to the entropy of the marginal and the entropy of the conditional distribution , averaged over .
In this paper we consider continuous entropy forms, and our main focus is on the relation between the MaxEnt distribution and the solution of the partial differential equation describing the macroscopic non-stationary dynamics. According to that, any entropy we discuss here, determined by a function , is assumed to be written as
where is asymptotically characterized by (4) and is a time-independent scalar function of the phase space coordinate. The partial differential equation describing the dynamics is a generalized Fokker-Planck equation, governing the time-evolution of a probability density , written in the form of
where and are constants, describes some external potential, and is analogous to an effective density [13, 14, 15, 16]. Such non-stationary processes can be classified phenomenologically by the asymptotic scaling of the spread of over time, termed as anomalous diffusion [17, 18]:
where the scaling factor of the space coordinate keeps the probability density invariant when the timescale is changed as . In general, (7) is satisfied only in the asymptotic limit, i.e., when . Nevertheless, this scaling relation, parametrized by , classifies the governing dynamics described by (6). For example, the diffusion equation falls into the equivalence class .
A key relation between the stationary behavior of the process, determined by the corresponding generalized entropy given in the form of (5), and the non-stationary behavior governed by a generalized Fokker-Planck equation is that the stationary solution of (6) agrees with the MaxEnt distribution [14, 19]. E.g., in case of the Boltzmann-Gibbs entropy, where , the MaxEnt distribution restricted by a constraint on the expected value of takes the form of . However, this is also equivalent to the stationary solution of the Fokker-Planck equation describing ordinary diffusion in the presence of some external potential ,
which is a special case of 6 with . Setting zero net flux at the boundaries yields .
Based on the above, the Boltzmann-Gibbs entropy, belonging to the entropy class , is corresponding to the Fokker-Planck equation describing simple diffusion, which in turn is a member of the anomalous diffusion scaling class . A natural question arising based on this observation is the following: Does every entropy belonging to the universality class correspond to a generalized Fokker-Planck equation from the anomalous diffusion class ? And does every generalized Fokker-Planck equation belonging to the class correspond to an entropy belonging to the class? In other words, does and give rise to the same equivalence class, therefore, bijectively determine each other? In this paper we show that this is true not only for and , but for every and , where the exponents and are connected by a simple algebraic relation. This implies that the asymptotic scaling of generalized entropies with phase space volume and the asymptotic anomalous diffusion scaling of the corresponding generalized Fokker-Planck equation classify the processes in the same way, and consequently, they provide the same information about their asymptotic behavior. In Fig.1. we show a schematic illustration of the above concept.
In order to derive a relationship between the asymptotic exponents and , let us first consider the MaxEnt distribution corresponding to entropies given in the form of (5). By following a variational principle approach and taking into account the normalization and expected value constraints we can write
where the constants are omitted for simplicity and the and Lagrange multipliers are introduced for fixing the zeroth and first moment, respectively. From (9) we obtain
where is the inverse of the MaxEnt distribution corresponding to the entropy defined by . In case of the Boltzmann-Gibbs entropy the inverse of the MaxEnt distribution is given by the (appropriately shifted and rescaled) logarithm function, . Therefore, is usually referred to as the generalized logarithm for any entropy in general [6, 20].
Based on a given entropy and the corresponding generalized logarithm , our next step is to find the related generalized Fokker-Planck equation in the form of (6), where is chosen such that the stationary solution of the equation becomes equivalent to the MaxEnt distribution of the entropy. By replacing with in (6) according to (10) and using the chain rule we obtain that the stationarity condition is fulfilled if
Thus, defining the generalized Fokker-Planck equation via (6) can be given as
In particular, for the Boltzmann-Gibbs entropy and , yielding , which results in . Since is formulated based on in (12), we call the resulting equation
the -Fokker-Planck equation in order to distinguish it from the many other possible generalizations.
In the following, let us consider the -Fokker-Planck equation with no external potential,
In the asymptotic limit when the scaling rule (7) applies to . Thus, if we change to the rescaled variables and , the derivatives according to the new variables can be written as
where we denoted by for simplicity. The term on the right hand side can be further transformed based on the scaling of given in (4), where by a change of variable we obtain (being valid for ). By substituting this into (12) we obtain
According to that, the -Fokker-Planck equation (16) in the rescaled variables yields
This equation is satisfied if and only if
resulting in the general relation
between the exponent related to the anomalous diffusion, characterizing the scaling of in the limit and the exponent , describing the scaling of the generalized entropy with the phase space volume.
In order to demonstrate this general result, in Table 1. we list a few different generalized entropy forms from the literature together with the corresponding -Fokker-Planck equations and the related and exponents. Although the actual algebraic form of the entropies along with their phase space volume scaling, their MaxEnt distributions and the corresponding generalized Fokker-Planck equations are different for any , their asymptotic anomalous diffusion scaling is completely determined by via (20). This exemplifies the fact that although the mapping between entropies and Fokker-Planck equations are defined at any (phase space volume or time) scale, any entropy, characterized by asymptotic exponent , can only be mapped to a Fokker-Planck equation describing anomalous diffusion with asymptotic exponent given by (20). In close connection to Table 1., Fig.2. shows the finite scale phase space volume scaling of some generalized entropies, illustrating the numerous possible ways of convergence to the asymptotic value .
In conclusion, the starting point of our study was that the classification of strongly interacting or non-Markovian systems is highly non-trivial due to the extreme diversity of possible underlying microscopic rules. Macroscopically, however, they collapse to asymptotic equivalence classes according to their stationary and non-stationary behavior, quantified by the corresponding generalized entropy’s phase space volume scaling and the anomalous diffusion scaling, respectively. Here we have shown that these two classifications coincide, suggesting that either of these asymptotic exponents is indeed providing a useful characterization of the systems themselves, and not just describing their behavior in the stationary or non-stationary regime. Furthermore, the surprising versatility of the theoretical framework behind Tsallis statistics to model various aspects of strongly interacting systems might be explained by the fact that the family of Tsallis entropies, characterized by their deformation index , provides an algebraically simple representative of each such asymptotic equivalence class.
The research was partially supported by the European Union through project RED-Alert (grant no.: 740688-RED-Alert-H2020-SEC-2016-2017/H2020-SEC-2016-2017-1) and by the Novo Nordisk Foundation (grant no.: Novo-Nordisk-2017-CY-78167/459-35)
-  Y. Bar-Yam and M. Bialik. Beyond big data: Identifying important information for real world challenges. Cambridge, NECSI, 2013.
-  F. Shibata, Y. Takahashi, and N. Hashitsume. A generalized stochastic liouville equation. non-markovian versus memoryless master equations. J. Stat. Phys., 17(4):171–187, 1977.
-  T.D. Frank. A note on the markov property of stochastic processes described by nonlinear fokker–planck equations. Physica A, 320:204–210, 2003.
-  R. Hanel and S. Thurner. Generalized (c, d)-entropy and aging random walks. Entropy, 15(12):5324–5337, 2013.
-  B. Corominas-Murtra, R. Hanel, and S. Thurner. Understanding scaling through history-dependent processes with collapsing sample space. Proc. Natl. Acad. Sci. U.S.A., 112(17):5348–5353, 2015.
-  C. Tsallis. Introduction to nonextensive statistical mechanics. Springer, 2009.
-  C. Beck. Generalised information and entropy measures in physics. Contemp. Phys., 50(4):495–510, 2009.
-  R. K. Pathria. Statistical mechanics. 1972.
-  H. E. Stanley. Scaling, universality, and renormalization: Three pillars of modern critical phenomena. Rev. Mod. Phys., 71(2):S358, 1999.
-  R. Hanel and S. Thurner. A comprehensive classification of complex statistical systems and an axiomatic derivation of their entropy and distribution functions. Europhys. Lett., 93(2):20006, 2011.
-  C. E. Shannon. A mathematical theory of communication. Bell Sys. Tech. J., 27(3):379, 1948.
-  A. I. Khinchin. Mathematical foundations of information theory. 1957.
-  R. Toral and P. Colet. Stochastic numerical methods: an introduction for students and scientists. Wiley, 2014.
-  A. R. Plastino and A. Plastino. Non-extensive statistical mechanics and generalized fokker-planck equation. Physica A, 222(1-4):347–354, 1995.
-  L. Borland. Microscopic dynamics of the nonlinear fokker-planck equation: A phenomenological model. Phys. Rev. E, 57(6):6634, 1998.
-  T. D. Frank. Nonlinear Fokker-Planck equations: fundamentals and applications. Springer Science & Business Media, 2005.
-  J.-P. Bouchaud and A. Georges. Anomalous diffusion in disordered media: statistical mechanisms, models and physical applications. Phys. Rep., 195(4-5):127–293, 1990.
-  A. A. Dubkov, B. Spagnolo, and V. V. Uchaikin. Lévy flight superdiffusion: an introduction. Int J Bifurcat Chaos, 18(09):2649–2672, 2008.
-  T.D. Frank and A. Daffertshofer. Nonlinear fokker–planck equations whose stationary solutions make entropy-like functionals stationary. Physica A, 272(3):497–508, 1999.
-  R. Hanel, S. Thurner, and M. Gell-Mann. Generalized entropies and logarithms and their duality relations. Proc. Natl. Acad. Sci. U.S.A., 109(47):19151–19154, 2012.
-  G.-A. Tsekouras and C. Tsallis. Generalized entropy arising from a distribution of q indices. Phys. Rev. E, 71(4):046144, 2005.
-  E. M. F. Curado and F. D. Nobre. On the stability of analytic entropic forms. Physica A, 335(1):94–106, 2004.
-  C. Tsallis. Possible generalization of boltzmann-gibbs statistics. J. Stat. Phys., 52(1):479–487, 1988.
-  G. Kaniadakis. Statistical mechanics in the context of special relativity. Phys. Rev. E, 66(5):056125, 2002.
-  F. Shafee. Lambert function and a new non-extensive form of entropy. IMA J. Appl. Math., 72(6):785–800, 2007.