Entropies in case of continuous time

# Entropies in case of continuous time

Detlef Holstein Max Planck Institute for the Physics of Complex Systems, Nöthnitzer Str. 38, 01187 Dresden, Germany
July 9, 2019
###### Abstract

Information theory on a time-discrete setting in the framework of time series analysis is generalized to the time-continuous case. Considerations of the Roessler and Lorenz dynamics as well as the Ornstein-Uhlenbeck process yield for time-continuous entropies a new possibility for the distinction of chaos and noise. In the deterministic case an upper threshold of the joint uncertainty in the limit of infinitely high sampling rate can be found and the entropy rate can be calculated as a usual time derivative of the entropy. In a three-dimensional representation the dependence of the joint entropy on space resolution, discretization time step length and uncertainty-assessed time is shown in a unified manner. Hence the dimension and the Kolmogorov-Sinai entropy rate of any dynamics can be read out as limit cases from one single graph.

Information theory, time-continuous limit, KS entropy, dimension, Renyi entropy
###### pacs:
05.45.Ac, 05.10.Gg, 05.45.Tp, 89.70.Cf

## I Introduction

Uncertainty of the outcome of random variables is usually evaluated by Shannon entropies cover91 ()

 H(X)=−N∑i=1P(xi)lnP(xi), (1)

where are the possible realizations of . Relaxing the restrictions in the underlying Khinchin or Fadeev axioms it is possible to introduce the family of Renyi entropies renyi61 ()

 H(q)(X)=11−qlnN∑i=1P(xi)q, (2)

which in the case of Renyi order are quite often used for estimation of entropies via the correlation sum grassproc83 (). In the case of time series analysis it is a common task to evaluate the uncertainty of random variables belonging to successive time steps kantzschreiber04 (). The corresponding joint entropy in the suitable notational representation reads

 H(q)m(ϵ,τ)=11−qln⎡⎣M1(ϵ)∑i1=1...Mm(ϵ)∑im=1(Pi1,...,im(ϵ,τ))q⎤⎦. (3)

In this expression is the space resolution and is the step length of the time discretization. After having decided for a suitable Renyi order, it is often omitted. Now the finite-m-entropy rate can be defined as

 hm(ϵ,τ):=H1|m(ϵ,τ)τ:=Hm+1(ϵ,τ)−Hm(ϵ,τ)τ. (4)

This quantity is a special conditional entropy per time step length. Since obviously this entropy rate is obtained from a quotient of differences, a time-continuous formulation of the relationships of entropic quantities should naturally also be at hand. The idea of connection of the entropic quantities by derivatives is also supported in crutch03 (), where however, the time discretization step length is still kept finite. The questions of consequences of variable time step length and especially the limit of for entropic quantities are addressed in this paper and it is intended to give theoretical insights into the structure of the relationships behind such entropic quantities. With those efforts this paper tries to contribute to a symmetrization of the treatment of entropies concerning space, where continuity is already realized in formulas, and time.

First, the line of thought from joint Shannon entropies to the Kolmogorov-Sinai entropy rate is outlined in sec. II, staying quite close to the paper of Gaspard and Wang gaspard93 (), the central paper behind this work. The limit of infinitesimal time step length is performed after having performed the limit of infinite time . Theoretically, sec. III contains the central aim of this paper. It is the generalization of sec. II with omission of the limit of infinite uncertainty-assessed time , concentrating on the limit of infinitesimal time discretization step length . In sec. IV the presented ideas are tested numerically for the examples of the Roessler, Lorenz and Ornstein-Uhlenbeck dynamics with the detection of qualitative discrepancies of deterministic and stochastic dynamics. Sec. V concludes the results of this paper.

## Ii From Shannon entropy to Kolmogorov-Sinai entropy rate

The starting point as given in gaspard93 () is the joint Shannon entropy

 H(A,τ,t)=Hm:=t/τ(A,τ)=−∑ω0,...,ωm−1P(ω0,...,ωm−1)lnP(ω0,...,ωm−1). (5)

The symbol , being the time step length, appears on the right side only implicitly as the time between successive realizations and the partition appears on the right side implicitly in the range of values , which corresponds to in eq. (1), can take. In the following it is assumed that the partition for is consistent with the partition for . It is possible to define the rate

 ~h(A,τ,t):=H(A,τ,t)t. (6)

The entropy per unit time with respect to partition is then defined as the limit

 ~h(A,τ):=limt→∞~h(A,τ,t). (7)

Starting from eq. (5) it is also possible to define another rate

 h(A,τ,t):=H(A,τ,t+τ)−H(A,τ,t)τ, (8)

and the corresponding limit

 h(A,τ):=limt→∞h(A,τ,t). (9)

In general it holds

 ~h(A,τ,t)≠h(A,τ,t),but~h(A,τ)=h(A,τ). (10)

The -entropy rate is derived by

 h(ϵ,τ)=infA:diam(Ai)≤ϵh(A,τ). (11)

On the other hand, having obtained from , e.g., again via infimum

 Hm(ϵ,τ)=infA:diam(Ai)≤ϵHm(A,τ), (12)

by the limit of infinite time from the conditional entropy rate also -entropy rates can be obtained cencini00 ():

 h∞(ϵ,τ)=limm→∞hm(ϵ,τ)=limm→∞1τ[Hm+1(ϵ,τ)−Hm(ϵ,τ)]. (13)

Depending on the state space resolution and time resolution , and both are the uncertainty per time step of the immediate future time step ahead if infinite time conditioning is imposed. The Kolmogorov-Sinai entropy rate for processes in continuous time is obtained from

 hKS=limϵ→0,τ→0h∞(ϵ,τ)=limϵ→0,τ→0h(ϵ,τ). (14)

## Iii Time-continuous information theory

In the former section the succession of limit procedures for accessing the KS entropy rate was redisplayed. Starting with the same formula (5), it is a naturally arising question what happens if the limits of and are not performed, but instead the limit is investigated holstdiss07 (). This does not lead to dynamical invariants, but instead to prediction-relevant quantities of information theory in the time-continuous case, because prediction deals with finite-time conditioning.

The partition dependence of eq. (5) or a similar resolution dependence is suppressed in notation in this section, since the time aspects should be pointed out, and it remains the entropy . Since is the uncertainty of the realization of one random variable, it should be independent of . This results in

 H(τ,t=τ)=constforτ>0, (15)

where a double - dependence on the left side of the equation leads surprisingly to -independence of the whole expression.

For fixed now the limit has to be carried out. If the limit exists, then

 H(τ=0,t)≡limτ→0H(τ,t) (16)

is the time-continuous joint entropy. Otherwise a time-continuous treatment of the joint uncertainty is not possible for the process at hand. Taking on the other hand the other involved variable equal to zero it holds

 H(τ,t=0)≡0. (17)

Hence it is inferred

 H(τ=0,t=0)=0. (18)

From being (not necessarily strong) monotonously increasing in it is inferred that also in the limit of the entropy is (not necessarily strong) monotonously increasing in t. An interesting question is the finiteness of for finite uncertainty-assessed time t for various process classes, which will be answered numerically by treating examples in sec. IV shown in fig.’s 3, 4, 7 and 8. For fixed the performed limit causes

 limτ→0m=limτ→0tτ=∞. (19)

From eqs. (5) and (19) it has to be inferred that as the uncertainty of the whole path has finally to be understood in terms of path integral-type quantities, where the initial and final states are not fixed.

Eq. (4) in notation suitable for the purpose of a time-continuous formulation or eq. (8) without partition dependence read

 h(τ,t)=H(τ,t+τ)−H(τ,t)τ. (20)

If the following limit exists, then

 h(τ=0,t)≡limτ→0h(τ,t) (21)

is the finite time entropy rate in the time-continuous case. The discrepancy from the usual differentiation is that the function to be differentiated depends explicitly on the parameter of the differentiation. This is unusual, but not untreatable. In case of existence of in eq. (16) for the corresponding derivative it holds that

 h(τ=0,t)=limΔt→0H(0,t+Δt)−H(0,t)Δt, (22)

i.e., from eq. (21) is obtained as a usual derivative from . is found to be a perturbation of the usual quotient of differences , but the limit of is the same in both cases. From eqs. (20), (17) and (15) it is derived

 τ⋅h(τ,t=0)=H(τ,t=τ)=const. (23)

One concludes trivially that for arbitrary dynamics

 limτ→0h(τ,t=0)=∞ifH(τ,t=τ)>0. (24)

For deterministic dynamics the behaviour of the entropy rate will be shown in fig. 5. It should be mentioned that the same treatment as here for the first derivative can of course also be carried through for higher derivatives holstdiss07 ().

Eq. (20) inverted and iterated leads with

 H(τ,t=mτ) =(h(τ,t=(m−1)τ)+...+h(τ,0))⋅τ (25)

in the time-continuous limit to a representation of the joint entropy via integral:

 H(τ=0,[0,t]) ≡H(τ=0,t) =limτ→0[m−1∑k=0τ⋅h(τ,t=kτ)] \lx@stackrelmτ=m′τ′=limτ→0limτ′→0[m′−1∑k=0τ′⋅h(τ,t=kτ′)] =limτ→0∫t0duh(τ,u) =∫t0duh(τ=0,u). (26)

Interchangeability of the limit procedures differentiation and integration is needed in the final step.

## Iv Numerical calculations for time-continuous entropies

### iv.1 Roessler system (deterministic chaotic dynamics)

The Roessler system is given by

 ˙x =−y−z, ˙y =x+ay, ˙z =b+z(x−c). (27)

The parameters are chosen as and . The quadratic term ’’ is the only nonlinearity. The largest Lyapunov exponent of the Roessler attractor is and the fractal dimension is .

In fig. 1 the joint entropy of the x-coordinate of the Roessler system is given as a function of resolution , time step length and uncertainty-assessed time . Convergence of the joint entropy for decreasing time step length can be seen. The joint entropy depends logarithmically on the resolution , from which the dimension is obtained as a slope according to

 D(q)m=−limϵ→0H(q)m(ϵ)lnϵ (28)

(schusterjust05 (), p.106) with a value as predicted.

It is possible to see in fig. 2 (in comparison with fig. 1 one smaller decade of resolutions is shown) that the z-coordinate of the Roessler attractor carries a different uncertainty for the same values of () compared with the x-coordinate. Nevertheless the dimension of the attractor is captured also by the joint entropy of the z-coordinate.

In fig. 3 the slice of fig. 1 for resolution with extended time is shown. It is found that asymptotically in the joint uncertainty of the Roessler system increases linearly. The measured slope for smallest in between and is 0.08. It estimates the KS entropy rate

 hKS=limϵ→0limτ→0limt→∞h(ϵ,τ,t). (29)

The deviation from the expected value 0.07 can be found in the fact that still too small or too large are used for the estimation. It can be concluded from the small slope that the uncertainty of the first few points is larger than the uncertainty of a rather long motion in the attractor given the first few points.

In fig. 4 the slice of fig. 2 for resolution with extended time is shown. Compared to fig. 3 a much smaller initial uncertainty is observed for the z-coordinate. This is understood from the high probability of the z-coordinate to stay at zero. Furthermore in contrast to the x-coordinate a wave-like structure of the joint entropy for smaller time is observed for the z-coordinate. A non-monotonous entropy rate as a function of has to be inferred. This unintuitive signature was robustly reproduced under various parameter values and confidence in this result is caused by the fact that asymptotically for smallest available the slope of 0.08 is found, which serves as a correct estimation of the KS entropy also from the z-coordinate of the Roessler system. Since also (from fig. 2) the dimension of the Roessler dynamics is correctly extractable the result should not be a numerical artefact. Nevertheless it is not expected that this should be true physics in the sense of increasing uncertainty by enlarged conditioning. According to Shannon entropies, for which additional conditioning cannot increase uncertainty, this behaviour is forbidden. A possible explanation of this behaviour can be assigned to effects of Renyi order (cmp. eq. (2)), because in this case it seems unproven that increasing conditioning necessarily reduces the resulting entropy. With this reasoning the example carries the potential of tearing apart the interpretation of uncertainty from - entropies. Another possible explanation that the wave-like structure corresponds to a finite sample effect being responsible for non-ergodicity has to be treated as rather improbable, because also for smallest resolution (largest ) it was not possible to detect a qualitative change in the sense of smoothing of the wave-like structure of the joint entropy under drastical enlargement of the length of the dataset.

With consideration of the fact that the values of and were estimated for finite large and finite small instead of in the true limit it can be concluded that the same values are obtained from the x- and the z-coordinate of the chaotic Roessler system in the limit cases, in principle in accordance with the theorem of Takens takens81 (). On the other hand, it can be clearly seen that for finite time and finite resolution the joint uncertainty is allowed to behave qualitatively different for different observables as the x- and the z-coordinate of the chaotic Roessler system.

In fig. 5 the entropy rate of the Roessler system is shown. For every the entropy rate as a function of and is represented by a surface. Those surfaces are interleaved for different . Depending on the ranges shown along the axes different results become apperent: With the example of the Roessler system even for deterministic dynamics a true divergence of can be observed for sufficiently small seen in particular in the upper panel of fig. 5. The trivial argument of eq. (24) already gave a hint for such behaviour. It is possible to see that for finite increases logarithmically in with a slope depending on . In the lower panel for larger the entropy rate indicates a small finite value in the front right corner of the plot, which for sufficiently large approximates the KS entropy rate.

### iv.2 Lorenz system (deterministic chaotic dynamics)

The implemented discretized equations for the Lorenz dynamics are

 xn+1 =xn+σ(−xn+yn)Δt, yn+1 =yn+(−xnzn+rxn−yn)Δt, zn+1 =zn+(xnyn−bzn)Δt (30)

with the usual parameters

 r=28.0,σ=10.0,b=83. (31)

The fractal dimension of the Lorenz attractor is about and the largest Lyapunov exponent is .

In fig. 6 the joint entropy of the Lorenz system is shown as a function of the resolution , time step length and uncertainty-assessed time . As for the Roessler system it is possible to see the convergence of the joint entropy for decreasing time step length and a logarithmic dependence of the joint entropy on the resolution with a slope giving the expected dimension in the limit of infinitesimal small .

In fig. 7 a slice of fig. 6 for fixed resolution is shown. An asymptotically linear behaviour in time can be found. The fact, that the slope is steeper than that of the chaotic Roessler attractor of fig. 3 is in accordance with the larger number of nonlinear terms and the known largest Lyapunov exponents. The slightly larger slope compared to the known KS entropy of the Lorenz system can be explained with estimation at rather large . The appearance of finite sample fluctuations in entropy estimation in fig. 6 for small , small and large under the same estimation conditions sooner in the Lorenz-case than in the Roessler-case is in accordance with the enhanced uncertainty of the Lorenz dynamics.

Furthermore fig. 7 indicates that the joint entropies for different finite -values do not coincide for asymptotically large times even though this cannot finally be proven in a plot for finite . The slopes seem to reach the same value in all cases as already observed in fig. 3 for the x-coordinate of the Roessler dynamics, and hence the estimation of the KS entropy rate is rather independent of the choice of , i.e., in the case of deterministic dynamics the limit with respect to in eq. (14) is not so important. On the other hand, the behaviour of for finite rather small is better resolved for smaller , and this in general is rather important with respect to prediction.

### iv.3 Ornstein-Uhlenbeck process (linear stochastic dynamics)

The Ornstein-Uhlenbeck process

 ˙Xt=−αXt+√D˙Wt (32)

can be numerically implemented by its discretisation

 Xn+1=(1−αΔt)Xn+√D√Δtηn+1,ηn+1∼N(0,1)iid. (33)

In fig. 8 the numerical analysis of the Ornstein-Uhlenbeck process indicates the non-existence of the limit for sufficiently small and finite except at for time- and amplitude-continuous stochastic dynamics with continuous trajectories. Whereas in the deterministic case, e.g. for the Lorenz dynamics, the value of the KS entropy rate was approximately seen as a slope at sufficiently small finite also for finite , in the time- and amplitude-continuous stochastic case the slope of with respect to asymptotically in for sufficiently small finite is seen in fig. 8 to be a function of . Furthermore it was possible to show numerically that for sufficiently small and non-zero finite fixed time the entropy rate diverges logarithmically with decreasing in the example of Ornstein-Uhlenbeck dynamics. Consistently, fig. 8 supports for this stochastic dynamics, interestingly already from the -behaviour. It should be mentioned here that for eq. (20) can be seen as a naturally given regularization of the unavailable eq. (22) in the stochastic case.

The author is aware of the fact that the result concerning the -dependence contradicts (gaspard93 (), p.322) according to which processes with continuous realizations are said to have -independent -entropies per unit time, which corresponds to the suggestion of finite . The argument for this behaviour was essentially that for finite a finite crossing time of underlying boxes of the partition leads to finite uncertainty for continuous trajectories (gaspard93 (), p.317). On the other hand, an explanation for the numerically found behaviour in fig. 8 could be that infinite uncertainty arises from the trajectories close to the border of boxes of partitioning, where in the limit of infinitely often moving back and forth between boxes in finite time is in principle possible for the Ornstein-Uhlenbeck process. This would create an infinite and hence dominant contribution for an averaged uncertainty calculation.

From fig. 9 it is obtained that in the transition regime of large the behaviour of non-existing is suppressed. The plots do not allow for the decision of the question if the transition from finite to infinite occurs at the resolution of the size of the system or at a smaller value for . According to eq. (28) fig. 9 yields finite dimensions for finite increasing in for stochastic dynamics, even though the limit can of course not be seen. In the limit of , what is effectively like the limit of , fig. 9 suggests an infinite dimension for stochastic dynamics.

The results of this section are qualitatively robust against changes of the parameter of the Ornstein-Uhlenbeck process and hence qualitatively robust against changes of the autocorrelation of the process.

## V Conclusion

The central objects of investigation in this paper are the entropy and the entropy rate as a function of the resolution , the time discretization step length and the uncertainty-assessed time with numerical access to the case of Renyi order . Special focus is laid on the analysis of the behaviour of these quantities for varying and especially on the time-continuous limit while is kept finite. In case of existence, in the time-continuous limit the entropy rates can be understood as usual time-derivatives of entropies. However, for finite , in consequence of the explicit -dependence of entropies, discrepancies from the usual quotient of differences are present.

Numerically the analysis of time- and space-continuous dynamics was carried out with the Roessler and Lorenz system for the deterministic case and with the Ornstein-Uhlenbeck process for the stochastic case. Qualitative discrepancies of deterministic and stochastic dynamics were found. In the deterministic case the uncertainty is finite for all finite and . The entropy rate becomes constant for large and sufficiently small . In the stochastic case the uncertainty is infinite for all if is below some threshold. Only for above the threshold it is finite and then effectively like in the deterministic case. The stochastic result is qualitatively independent of the value of the width of the input noise as long as the width is not exactly zero, corresponding to determinism and introducing the qualitative change. This behaviour of entropies in the limit seems to offer a new possibility for distinction of chaos and noise (see also cencini00 ()).

The convergence of in the limit in the deterministic case can be interpreted as a saturation of the gain from higher sampling rates, such that a threshold sampling rate can be postulated, above which almost nothing more can be learned, i.e., in the deterministic case the continuous limit can be well approximated by discrete sampling with sufficiently high sampling rate. A criterion for an optimal sampling rate could be formulated.

In the limit case the partial derivative of the joint entropy with respect to yields (minus) the dimension and if furthermore the limit is carried out, the partial derivative of with respect to yields the KS entropy rate. All information concerning entropy rates (including the KS entropy rate) and dimensions of the dynamics can be extracted from one single plot of . In the deterministic case known values were verified in examples. For time- and amplitude-continuous stochastic dynamics it was seen that the partial derivative of with respect to minus becomes infinite for or and the partial derivative of H with respect to becomes infinite for or .

According to usual rules the total differential of the joint entropy can be written as

 dH=∂H∂tdt+∂H∂lnϵdlnϵ. (34)

For sufficiently small and sufficiently large this becomes

 dH=hKSdt−Ddlnϵ. (35)

It is possible to see that directional derivatives of in general mix the properties of dimension and entropy rate. For integral quantities   was suggested in (grassberger91 (), p.529), from which eq. (35) can be derived, but a prescription for the calculation of the constant is not given. Since eq. (35) avoids the offset problems it is a slightly reduced and hence preferable representation of the relationship of the involved quantities.

Concerning the limit cases and with the determination of dynamical invariants, which is discussed in the literature, e.g. see (farmer82 (), p.1321), the time-continuous case treated in this paper gives a new unified representation of singly known results. On the other hand, at least of the same importance, the case of finite time is of strong interest, if questions concerning optimal finite time conditioning with respect to prediction are addressed. A finite resolution is needed for optimality of local prediction methods. With finite and , the limit is of interest for the determination of the informational characteristic of the dynamics.

A non-monotonous entropy rate for Renyi order , i.e., a conditional entropy, which does not monotonously decrease in the length of conditioning, is numerically found for the z-coordinate of the Roessler system in fig. 4. This could be interpreted as a hint for problems of the interpretation of uncertainty for entropies with in maximal generality.

## References

• (1) T.M. Cover and J.A. Thomas, Elements of Information Theory, Wiley, New York (1991).
• (2) A. Renyi, On measures of entropy and information, Proc. IV Berkeley Symposium Math. Statist. and Prob., University California Press, 547-561 (1961).
• (3) P. Grassberger and I. Procaccia, Estimation of the Kolmogorov entropy from a chaotic signal, Phys. Rev. A 28 (4) 2591-2593 (1983).
• (4) H. Kantz and T. Schreiber, Nonlinear Time Series Analysis, Cambridge University Press, second edition (2004).
• (5) J.P. Crutchfield and D.P. Feldman; Regularities unseen, randomness observed: Levels of entropy convergence, Chaos 13 (1) 25-54 (2003).
• (6) P. Gaspard and X.-J. Wang; Noise, chaos and (, )-entropy per unit time, Phys. Rep. 235 (6) 291-343 (1993).
• (7) M. Cencini, M. Falcioni, E. Olbrich, H. Kantz and A. Vulpiani; Chaos or noise: Difficulties of a distinction, Phys. Rev. E 62 (1) 427-437 (2000).
• (8) D. Holstein, Generalized Markov Approximations from information theory and consequences for prediction, Dissertation, Wuppertal (2007).
• (9) H.G. Schuster and W. Just; Deterministic Chaos: An Introduction, Wiley, Weinheim, fourth edition (2005).
• (10) F. Takens; Detecting strange attractors in turbulence, Lecture Notes in Mathematics 898, Springer, New York (1981).
• (11) P. Grassberger, T. Schreiber, and C. Schaffrath; Nonlinear time sequence analysis, Int. Journ. of Bifurcation and Chaos 1 (3) 521-547 (1991).
• (12) J.D. Farmer; Information Dimension and the Probabilistic Structure of Chaos, Z. Naturforsch. 37a 1304-1325 (1982).
You are adding the first comment!
How to quickly get a good reply:
• Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
• Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
• Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters