Stochastic thermodynamics, fluctuation theorems, and molecular machines
Abstract
Stochastic thermodynamics as reviewed here systematically provides a framework for extending the notions of classical thermodynamics like work, heat and entropy production to the level of individual trajectories of welldefined nonequilibrium ensembles. It applies whenever a nonequilibrium process is still coupled to one (or several) heat bath(s) of constant temperature. Paradigmatic systems are single colloidal particles in timedependent laser traps, polymers in external flow, enzymes and molecular motors in single molecule assays, small biochemical networks and thermoelectric devices involving single electron transport. For such systems, a firstlaw like energy balance can be identified along fluctuating trajectories. For a basic Markovian dynamics implemented either on the continuum level with Langevin equations or on a discrete set of states as a master equation, thermodynamic consistency imposes a localdetailed balance constraint on noise and rates, respectively. Various integral and detailed fluctuation theorems, which are derived here in a unifying approach from one master theorem, constrain the probability distributions for work, heat and entropy production depending on the nature of the system and the choice of nonequilibrium conditions. For nonequilibrium steady states, particularly strong results hold like a generalized fluctuationdissipation theorem involving entropy production. Ramifications and applications of these concepts include optimal driving between specified states in finite time, the role of measurementbased feedback processes and the relation between dissipation and irreversibility. Efficiency and, in particular, efficiency at maximum power, can be discussed systematically beyond the linear response regime for two classes of molecular machines, isothermal ones like molecular motors, and heat engines like thermoelectric devices, using a common framework based on a cycle decomposition of entropy production.
pacs:
05.40.a: Fluctuation phenomena, random processes, noise, and Brownian motion; 05.70.Ln: Nonequilibrium and irreversible thermodynamics; 82.37.j: Single molecule kinetics; 87.16.Uv, Active transport processesContents
 1 Introduction
 2 Colloidal particle as paradigm
 3 Fluctuation theorems (FTs)
 4 Unification of FTs
 5 Experimental, analytical and numerical work for specific systems with continuous degrees of freedom
 6 Dynamics on a discrete set of states

7 Optimization, irreversibility, information and feedback
 7.1 Optimal protocols
 7.2 Quantifying time’s arrow

7.3 Measurement and feedback
 7.3.1 Feedback and the second law
 7.3.2 Measurement and information
 7.3.3 SagawaUeda equality (SUE) and a generalization
 7.3.4 Efficiency of Brownian information machines
 7.3.5 Further theoretical work and case studies
 7.3.6 Experimental illustrations
 7.3.7 Hamiltonian dynamics for microcanonical initial conditions
 8 Fluctuationdissipation theorem (FDT) in a NESS

9 Biomolecular systems
 9.1 Overview
 9.2 Biopolymers with continuous degrees of freedom
 9.3 Free energy recovery from nonequilibrium data

9.4 Enzymes and molecular motors with discrete states
 9.4.1 Thermodynamic states
 9.4.2 First law
 9.4.3 Role of chemiostats: Genuine NESS conditions
 9.4.4 Stochastic trajectory and ensemble
 9.4.5 Rates and local detailed balance
 9.4.6 Entropy production and FTs
 9.4.7 Timedependent rates and work
 9.4.8 Experiments: F1ATPase
 9.4.9 Biochemical reaction networks

10 Autonomous isothermal machines
 10.1 General aspects
 10.2 General framework for autonomous machines: Cycle representation and entropy production
 10.3 Power and efficiency
 10.4 Linear response: Relation to phenomenological irreversible thermodynamics
 10.5 Unicyclic machines
 10.6 Multicyclic machines: Strong vs weak coupling
 10.7 Efficiency and EMP of molecular motors
 11 Efficiency of stochastic heat engines
 12 Concluding perspective
1 Introduction
1.1 From classical to stochastic thermodynamics
Classical thermodynamics, at its heart, deals with general laws governing the transformations of a system, in particular, those involving the exchange of heat, work and matter with an environment. As a central result, total entropy production is identified that in any such process can never decrease, leading, inter alia, to fundamental limits on the efficiency of heat engines and refrigerators. The thermodynamic characterization of systems in equilibrium got its microscopic justification from equilibrium statistical mechanics which states that for a system in contact with a heat bath the probability to find it in any specific microstate is given by the Boltzmann factor. For small deviations from equilibrium, linear response theory allows to express transport properties caused by small external fields through equilibrium correlation functions. On a more phenomenological level, linear irreversible thermodynamics provides a relation between such transport coefficients and entropy production in terms of forces and fluxes. Beyond this linear response regime, for a long time, no universal exact results were available.
During the last 20 years fresh approaches have revealed general laws applicable to nonequilibrium system thus pushing the range of validity of exact thermodynamic statements beyond the realm of linear response deep into the genuine nonequilibrium region. These exact results, which become particularly relevant for small systems with appreciable (typically nonGaussian) fluctuations, generically refer to distribution functions of thermodynamic quantities like exchanged heat, applied work or entropy production.
First, for a thermostatted sheardriven fluid in contact with a heat bath, a remarkable symmetry of the probability distribution of entropy production in the steady state was discovered numerically and justified heuristically by Evans et al. [1]. Now known as the (steady state) fluctuation theorem (FT), it was first proven for a large class of systems using concepts from chaotic dynamics by Gallavotti and Cohen [2], later for driven Langevin dynamics by Kurchan [3] and for driven diffusive dynamics by Lebowitz and Spohn [4]. As a variant, a transient fluctuation theorem valid for relaxation towards the steady state was found by Evans and Searles [5].
Second, Jarzynski proved a remarkable relation which allows to express the free energy difference between two equilibrium states by a nonlinear average over the work required to drive the system in a nonequilibrium process from one state to the other [6, 7]. By comparing probability distributions for the work spent in the original process with the timereversed one, Crooks found a “refinement” of the Jarzynski relation (JR), now called the Crooks fluctuation theorem [8, 9]. Both, this relation and another refinement of the JR, the HummerSzabo relation [10] became particularly useful for determining free energy differences and landscapes of biomolecules. These relations are the most prominent ones within a class of exact results (some of which found even earlier [11, 12] and then rediscovered) valid for nonequilibrium systems driven by timedependent forces. A close analogy to the JR, which relates different equilibrium states, is the HatanoSasa relation that applies to transitions between two different nonequilibrium steady states [13].
Third, for driven Brownian motion, Sekimoto realized that two central concepts of classical thermodynamics, namely the exchanged heat and the applied work, can be meaningfully defined on the level of individual trajectories [14, 15]. These quantities entering the first law become fluctuating ones giving birth to what he called stochastic energetics as described in his monograph [16]. Fourth, Maes emphasized that entropy production in the medium is related to that part of the stochastic action, which determines the weight of trajectories that is odd under timereversal [17, 18].
Finally, building systematically on a concept briefly noticed previously [8, 19], a unifying perspective on these developments emerged by realizing that besides the fluctuations of the entropy production in the heat bath one should similarly assign a fluctuating, or ”stochastic”, entropy to the system proper [20]. Once this is done, the key quantities known from classical thermodynamics are defined along individual trajectories where they become accessible to experimental or numerical measurements. This approach of taking both energy conservation, i.e., the first law, and entropy production seriously on this mesoscopic level has been called stochastic thermodynamics [21], thus revitalizing a notion originally introduced by the Brussels school in the mideighties where it was used on the ensemble level for chemical nonequilibrium systems [22, 23].
1.2 Main features of stochastic thermodynamics
Stochastic thermodynamics as here understood applies to (small) systems like colloidal particles, (bio)polymers (like DNA, RNA, and proteins), enzymes and molecular motors. All these systems are embedded in an aqueous solution. Three types of nonequilibrium situations can be distinguished for these systems. First, one could prepare the system in a nonequilibrium initial state and study the relaxation towards equilibrium. Second, genuine driving can be caused by the action of timedependent external forces, fields, flows or unbalanced chemical reactions. Third, if the external driving is timeindependent the system will reach a nonequilibrium steady state (NESS). For this latter class, particularly strong exact results exist. In all cases, even under such nonequililibrium conditions, the temperature of the system, which is the one of the embedding solution, remains welldefined. This property together with the related necessary timescale separation between the observable, typically slow, degrees of freedom of the system and the unobservable fast ones made up by the thermal bath (and, in the case of biopolymers, by fast internal ones of the system) allows for a consistent thermodynamic description.
The collection of the relevant slow degrees of freedom makes up the state of the system. Since this state changes either due to the driving or due to the ever present fluctuations, it leads to a trajectory of the system. Such trajectories belong to an ensemble which is fully characterized by the distribution of the initial state, by the properties of the thermal noise acting on the system and by specifying the (possibily timedependent) external driving. The thermodynamic quantities defined along the trajectory like applied work and exchanged heat thus follow a distribution which can be measured experimentally or be determined in numerical simulations.
Theoretically, the timescale separation implies that the dynamics becomes Markovian, i.e., the future state of the system depends only on the present one with no memory of the past. If the states are made up by continuous variables (like position), the dynamics follows a Langevin equation for an individual system and a FokkerPlanck equation for the whole ensemble. Sometimes it is more convenient to identify discrete states with transition rates governing the dynamics which, on the ensemble level, leads to a master equation.
Within such a stochastic dynamics, the exact results quoted above for the distribution functions of certain thermodynamic quantities follow universally for any system from rather unsophisticated mathematics. It is sufficient to invoke a “conjugate” dynamics, typically, but not exclusively, timereversal, to derive these theorems in a few lines. Essentially, they lead to universal constraints on these distributions. One inevitable consequence of these theorems is the occurrence of trajectories with negative total entropy production. Such events have occasionally been called (transient) violations of the second law. In fairness to classical thermodynamics, however, one should emphasize that this classical theory ignores fluctuations. If the second law is understood as refering to the mean entropy production, it is indeed confirmed by these more recent exact relations. Moreover, they show that the probability for such events become typically exponentially small in the relevant system size which means that one has to sample exponentially many trajectories in order to observe these “violations”.
Since these constraints on the distributions are so universal, one might suspect that they are useless for uncovering system specific properties. Quite to the contrary, some of them offer a surprising relation between equilibrium and nonequilibrium properties with the JR as the most prominent and useful example. Moreover, such constraints can be used as an obvious check whether the assumptions of the model apply to any particular system. Finally, studying nonuniversal features of these distribution functions and trying to find further common aspects in these has become an important part of the activities in this field.
Going beyond the thermodynamic framework, it turns out that many of the FTs hold formally true for any kind of Markovian stochastic dynamics. The thermodynamic interpretation of the involved quantities as heat and work is not mandatory to derive such a priori surprising relationships between functionals defined along dynamic trajectories.
1.3 Hamiltonian, thermostatted and open quantum dynamics
Even though I will focus in the main part of this review on systems described by a stochastic dynamics, it is appropriate to mention briefly alternative approaches as some of the FTs have originally been derived using a deterministic framework.
Hamiltonian dynamics works, in principle, if the external driving is modelled by a timedependent potential arising, e.g., from a movable piston, tip of an atomic force microscope, or optical tweezer. Conceptually, one typically requires thermalized initial conditions, then imagines to cut off the system from the heat bath leading to the deterministic motion and finally one has to reconnect the heat bath again. In a second variant, the heat bath is considered to be part of the system but one then has to follow all degrees of freedom. One disadvantage of Hamiltonian dynamics is that it cannot deal with a genuine NESS, which is driven by a timeindepedent external field or flow, since such a setting inevitably heats up the system.
Thermostatted dynamics can deal with NESSs. Here, one keeps deterministic equations of motion and introduces a friction term making sure that on average the relevant energy (kinetic or total, depending on the scheme) does not change [24].
Even though a deterministic dynamics is sometimes considered to be more fundamental than a stochastic one, the latter has at least three advantanges from the perspective held in this review. First, from a practical point of view, in soft matter and biophysics a description focussing on the relevant (and measurable) degrees of freedom and ignoring water molecules from the outset has a certain economical appeal. Second, stochastic dynamics can describe transitions between discrete states like in (bio)chemical reactions with essentially the same conceptual framework used for systems with continuous degrees of freedom. Third, the mathematics required for deriving the exact relations and for stating their range of validity is surpringly simple compared to what is required for dealing with NESSs in the deterministic setting.
Open quantum systems will not be discussed explicitly in this review. Some of the FTs can indeed be formulated for these systems, sometimes at the cost of requiring somewhat unrealistic measurements at the beginning and end of a process, as reviewed in [25, 26]. The results derived and discussed in the following, however, are directly applicable to open quantum systems whenever coherences, i.e. the role of nondiagonal elements in the density matrix, can be ignored. The dynamics of the driven or open quantum system is then equivalent to a classical stochastic one. For the validity of the exact relations in these cases, the quantummechanical origin of the transition rates is inconsequential.
1.4 Scope and organization of this review
By writing this review, apart from focussing entirely on systems governed by Markovian stochastic dynamics, I have been guided by the following principles concerning format and content.
First, I have tried to present the field in a systematic order (and notation) rather than to follow the historical development which has been briefly alluded to in the introductory section above. Such an approach leads to a more concise and coherent presentation. Moreover, I have tried to keep most of the more technical parts (some of which are original) still selfcontained. Both features should help those using this material as a basis for courses which I have given several times at the University of Stuttgart and at summer schools in Beijing, Boulder and Jülich.
Second, as a consequence of the more systematic presentation, experimental, analytical and numerical case studies of specific systems are mostly grouped together and typically placed after the general theory where they fit best.
Third, for the exact results the notions “theorem”, “equality” and “relation” are used here in no particular hierarchy. I rather try to follow the practice established in the field so far. In particular, it is not implied that a result called here “theorem” is in any sense deeper that another one called “relation”.
This review starts in Sect. 2 by introducing a paradigm for this field which is a colloidal particle driven by a timedependent force as it has been realized in several experiments. Using this system, the main concepts of stochastic thermodynamics, like work, heat and entropy changes along individual trajectories, will be introduced. At the end of this section, simple generalizations of driven onedimensional motion such as threedimensional motion, coupled degrees of freedom and motion in external flow are discussed.
A general classification and a physical discussion of the major fluctuation theorems dealing with work and the various contributions to entropy production follows in Sect. 3. In Sect. 4, I present a unifying perspective on basically all known FTs for stochastic dynamics using the concept of a conjugate dynamics. It is shown explicitly, how these FTs follow from one master theorem. Sect. 5 contains an overview of experimental, analytical or numerical studies of Langevintype dynamics in specific systems. Sect. 6 deals with Markovian dynamics on a discrete set of states for which FTs hold even without assuming a thermodynamic structure.
The second part of the review deals with ramifications, consequences and applications of these concepts. In Sect. 7, the optimal driving of such processes is discussed and the relation between time’s arrow and the amount of dissipation derived. Both concepts can then be used to discuss the role of measurements and (optimal) feedback in these systems. Sect. 8 deals with generalizations of the wellknown fluctuationdissipation theorem to NESSs where it is shown that stochastic entropy plays a crucial role.
Biomolecular systems are discussed in Sect. 9 where special emphasis is given to the role of timescale separation between the fast (unobservable) degrees of freedom making up a welldefined heat bath for the nonequilibrium processes of the slow variables caused by mechanical or chemical imbalances. From a conceptual point of view the second essential new aspect of these systems is that each of the states is composed of many microstates which leads to the crucial notion of intrinsic entropy that enters some of the exact relations in a nontrivial way.
Coming back to the issues that stood at the origin of thermodynamics, the final two sections discuss the efficiency and optimization of nano and micro engines and devices where it is useful to distinguish isothermal engines like molecular motors discussed in Sect. 10 from heat engines like thermoelectric devices treated in Sect. 11. A brief summary and a few perspectives are sketched in Sect. 12.
1.5 Complementary reviews
A selection of further reviews dealing with the topics discussed in the first part of this article can roughly be grouped as follows.^{1}^{1}1Relevant reviews for the more specific topics treated in the second part of this review will be mentioned in the respective sections.
The influential essay [27] had an introductory character. More recent nontechnical accounts have been given by Jarzynski [28] and van den Broeck [29] who both emphasize the relation of the fluctuation theorems with irreversibility and time’s arrow. Other brief reviews by some of the main proponents include [17, 30, 31, 32] and the contributions in the collection [33]. Ritort has written a review on the role of nonequilibrium fluctuations in small systems with special emphasis on the applications to biomolecular systems [34]. A review focussing on experimental work by one of the main groups working on fluctuation theorems is [35].
For stochastic dynamics based on the master equation, a comprehensive derivation of FT’s has been given by Harris and Schütz [36]. The fluctuation theorem in the context of thermostatted dynamics has been systematically reviewed by Evans and Searles [37]. From the perspective of chaotic dynamics it is treated in Gallavotti’s monograph [38]. The links between different approaches and rigorous mathematical statements are surveyed in [39, 40, 41].
Stochastic thermodynamics focusses on a description of individual trajectories as does an alternative approach by Attard introducing a “second entropy” [42]. On a more coarsegrained level, phenomenological thermodynamic theories of nonequilibrium systems have been developed inter alia under the label of “extended irreversible thermodynamics” [43], ”GENERIC” [44], “mesoscopic dynamics of thermodynamic systems” [45] and “steady state thermodynamics” [46, 47].
2 Colloidal particle as paradigm
The main concepts of stochastic thermodynamics can be introduced using as a simple model system a colloidal particle confined to one spatial dimension, which can arguably serve as the paradigm in the field.
2.1 Stochastic dynamics
The overdamped motion of a colloidal particle (or any other system with a single continuous degree of freedom) can be described using three equivalent but complementary descriptions of stochastic dynamics, the Langevin equation, the path integral, and the FokkerPlanck equation.
The Langevin equation reads
(1) 
The systematic force can arise from a conservative potential and/or be applied to the particle directly as . Distinguishing these two sources is relevant only in the case of periodic boundary conditions on a ring of finite length. Both sources may be timedependent through an external control parameter varied from to according to some prescribed protocol.
The thermal noise has correlations
(2) 
where is the diffusion constant. In equilibrium, and the mobility are related by the Einstein relation
(3) 
where is the temperature of the surrounding medium with Boltzmann’s constant set to unity throughout this review to make entropy dimensionless. In stochastic thermodynamics, one assumes that the strength of the noise is not affected by the presence of a timedependent force. The range of validity of this crucial assumption can be tested experimentally or in simulations by comparing with theoretical results derived on the basis of this assumption.
The Langevin dynamics generates trajectories starting at with a weight
(4) 
where
(5) 
is the “action” associated with the trajectory. The last term arises from the Stratonovich convention for the discretization in the Jacobian when the weight for a noise history is expressed by . This symmetric discretization is used implicitly throughout this review. Path dependent observables can then be averaged using this weight in a path integral which requires a pathindependent normalization such that summing the weight (4,5) over all paths is 1. Throughout the review averages using this weight and a given initial distribution will be denoted by as in
(6) 
Equivalently, the FokkerPlanck equation for the probability to find the particle at at time is
(7)  
where is the probability current. This partial differential equation must be augmented by a normalized initial distribution . For further calculations, it is useful to define a mean local velocity
(8) 
2.2 Nonequilibrium steady states (NESSs)
For timeindependent control parameter , any initial distribution will finally reach a stationary state . For , this stationary state is the thermal equilibrium,
(9) 
with the free energy
(10) 
A nonconservative force acting on a ring as shown in Fig. 1 generates a paradigm for a genuine nonequilibrium steady state (NESS) with a stationary distribution
(11) 
where is the “nonequilibrium” potential. In one dimension, can be obtained explicitly by quadratures [51] or by an intriguing mapping to an equilibrium problem [54]. Characteristic for such a NESS is a steady current
(12) 
with the mean local velocity . Even for timedependent driving, one can express the total force
(13) 
through quantities refering to the corresponding stationary state which is sometimes helpful.
Occasionally, we will use and to emphasize when averages or correlation functions are taken in genuine equilibrium and in a NESS, respectively.
2.3 Stochastic energetics
2.3.1 The first law
Sekimoto suggested to endow the Langevin dynamics with a thermodynamic interpretation by applying the notions appearing in the first law
(14) 
to an individual fluctuating trajectory [14, 15]. Throughout the article, we use the convention that work applied to the particle (or more generally system) is positive as is heat transferred or dissipated into the environment.
It is instructive first to identify the first law for a particle in equilibrium, i.e. for and constant . In this case, no work is applied to the system and hence an increase in internal energy, defined by the position in the potential, , must be associated with heat taken up from the reservoir.
Applying work to the particle either requires a timedependent potential and (or) an external force . The increment in work applied to the particle then reads
(15) 
where the first term arises from changing the potential at fixed particle position. Consequently, the heat dissipated into the medium must be identified with
(16) 
This relation makes physical sense since in an overdamped system the total force times the displacement corresponds to dissipation. Integrated over a time interval , one obtains the expressions
(17) 
and
(18) 
and the integrated first law
(19) 
on the level of an individual trajectory.
The expression for the heat requires a prescription of how to evaluate . As above in the path integral, one has to use the midpoint , i.e., Stratonovitch rule for which the ordinary rules of calculus for differentials and integrals apply.
The expression for the heat dissipated along the trajectory can also be written in the form
(20) 
as a ratio involving the weight (5) for this trajectory given its initial point compared to the weight of the timereversed trajectory under the reversed protocol for . This formulation points to the deep relation between dissipation and time reversal which repeatedly shows up in this field.
2.3.2 Housekeeping and “excess” heat
Motivated by steady state thermodynamics, it will be convenient to split the dissipated heat into two contributions [46, 13]
(21) 
The housekeeping heat is the heat inevitably dissipated in maintaining the corresponding NESS. For a Langevin dynamics, it reads
(22) 
The “excess” heat
(23)  
is the heat associated with changing the external control parameter where we have used (13,18).
2.3.3 Heat and strong coupling
This interpretation of the first law and, in particular, of heat relies on the implicit assumption that the unavoidable coupling between particle (or, more generally, system) described by the slow variable and the degrees of freedom making up the heat bath does neither depend crucially on nor on the control parameter . Such an idealization may well be obeyed for a colloidal particle in a laser trap but will certainly fail for more complex systems like biomolecules. In the following, we first continue with this simple assumption. In Sect. 9.2, we discuss the general case and point out which of the results derived in the following will require a ramification. Roughly speaking, most of the fluctuation theorems hold true with minor modifications whereas infering heat correctly indeed requires one more term compared to (16).
2.3.4 Alternative identification of work
The definition of work (15) has been criticised for supposedly being in conflict with a more conventional view that work should be given by force times displacement, see [55] and, for rebuttals, [56, 57, 58]. In principle, such a view could be integrated into the present scheme by splitting the potential into two contributions,
(24) 
the first being an intrinsic timeindependent potential, and the second one a timedependent external potential used to transmit the external force. If one defines work as
(25) 
it is trivial to check that the first law then holds in the form
(26) 
with the corresponding change in internal energy and the identification of heat (16) unchanged. Clearly, within such a framework, it would be appropriate to identify the internal energy with changes in the intrinsic potential only. Integrated over a trajectory, this definition of work differs from the previous one by a boundary term, .
It is crucial to appreciate that exchanged heat as a physical concept is, and should be, independent of the convention how it is split into work and changes in internal energy. The latter freedom is inconsequential as long as one stays within one scheme. A clear disadvantage of this alternative scheme, however, is that changes in the free energy of a system are no longer given by the quasistatic work relating two states. In this review, we will keep the definitions as introduced in Sect. 2.3.1 and only occasionally quote results for the alternative expression for work introduced in this section.
2.4 Stochastic entropy
Having expressed the first law along an individual trajectory, it seems natural to ask whether entropy can be identified on this level as well. For a simple colloidal particle, the corresponding quantity turns out to have two contributions. First, the heat dissipated into the environment should obviously be identified with an increase in entropy of the medium
(27) 
Second, one identifies as a stochastic or trajectory dependent entropy of the system the quantity [20]
(28) 
where the probability obtained by first solving the FokkerPlanck equation is evaluated along the stochastic trajectory . Thus, the stochastic entropy depends not only on the individual trajectory but also on the ensemble. If the same trajectory is taken from an ensemble generated by another initial condition , it will lead to a different value for .
In equilibrium, i.e., for and constant , the stochastic entropy just defined obeys the wellknown thermodynamic relation, , between entropy, internal energy and free energy in the form
(29) 
now along the fluctuating trajectory at any time with the free energy defined in (10) above.
Using the FokkerPlanck equation the rate of change of the entropy of the system (28) follows as [20]
(30) 
Since the very last term can be related to the rate of heat dissipation in the medium (18), using , one obtains a balance equation for the trajectorydependent total entropy production as
(31) 
The first term on the right hand side signifies a change in which can be due to a timedependent or, even for a constant , due to relaxation from a nonstationary initial state .
As a variant on the trajectory level, occasionally has been suggested as a definition of system entropy. Such a choice is physically questionable as the following example shows. Consider diffusive relaxation of a localized initial distribution in a finite region . Since , will not change during this process. On the other hand, such a diffusive relaxation should clearly lead to an entropy increase. Only in cases where one starts in a NESS and waits for final relaxation, the change in system entropy can also be expressed by a change in the nonequilibrium potential according to .
2.5 Ensemble averages
Upon averaging, the expressions for the thermodynamic quantities along the individual trajectory should become the ensemble quantities of nonequilibrium thermodynamics derived previously for such FokkerPlanck systems, see, e.g. [19].
Averages for quantities involving the position of the particle are most easily performed using the probability . Somewhat more delicate are averages over quantities like the heat that involve products of the velocity and a function . These can be performed in two steps. First, one can evaluate the average conditioned on the position in the spirit of the Stratonovitch discretization as
(32)  
The averages in the bracket on the rhs can be evaluated by discretizing the path integral (4, 5) for one step. The first term straightforwardly yields . In the second one, the endpoint conditioning (best implemented by Bayes’ theorem) is crucial which leads to an additional contribution if the distribution is not uniform. The final result is [20]
(33) 
Any subsequent average over position is now trivial leading to
(34) 
With these relations, one obtains, e.g., for the averaged total entropy production rate from (31) the expression
(35) 
where equality holds in equilibrium only. In a NESS, which thus determines the mean dissipation rate. Averaging the increase in entropy of the medium along similar lines leads to
(36) 
Hence upon averaging, the increase in entropy of the system proper becomes . On the ensemble level, this balance equation for the averaged quantities can also be derived directly from the ensemble definition of the system entropy
(37) 
by using the FokkerPlanck equation (7).
2.6 Simple generalizations
2.6.1 Underdamped motion
For some systems, it is necessary to keep the inertial term which leads with mass and damping constant to the Langevin equation
(38) 
with the noise correlations .
The internal energy now must include the kinetic energy, , with . Since the identification of work (15) remains valid, the first law becomes
(39) 
Evaluating the stochastic entropy
(40) 
now requires a solution of the corresponding FokkerPlank equation
(41) 
for with an appropriate initial condition .
2.6.2 Interacting degrees of freedom
The framework introduced for a single degree of freedom can easily be generalized to several degrees of freedom obeying the coupled Langevin equations
(42) 
where is a potential and a nonconservative force. The noise correlations
(43) 
involve the mobility tensor . Simple examples of such system comprise a colloidal particle in three dimensions, several interacting colloidal particles, or a polymer where labels the positions of monomers. If hydrodynamic interactions are relevant, the mobility tensor will depend on the coordinates .
The corresponding FokkerPlanck equation becomes
(44) 
with the local (probability) current .
In particular for a NESS, there are two major formal differences compared to the onedimensional case. First, the stationary current becomes dependent and, second, the stationary distribution or, equivalently, the nonequilibrium potential are not known analytically except for the trivial case that the total forces are linear in .
On a formal level, all expressions discussed above for the simple colloidal particle can easily be generalized to this multidimensional case by replacing scalar operations by the corresponding vector or matrix ones.
2.6.3 Systems in external flow
So far, we have assumed that there is no overall hydrodynamic flow imposed on the system. For colloids, however, external flow is a common situation. Likewise, as we will see, colloids in moving traps can also be described in a comoving frame as being subject to some flow. We therefor recall the modifications required for the basic notions of stochastic thermodynamics in the presence of external flow field [59].
The Langevin equations for coupled particles at positions reads
(45) 
with the usual noise correlations
(46) 
In such a system, the increments in external work and dissipated heat are given by
(47) 
and
(48) 
respectively. Compared to the case withour flow, the two modifications involve replacing the partial derivative by the convective one and measuring the velocity relative to the external flow velocity. These expressions guarantee frame invariance of stochastic thermodynamics [59].
For the experimentally studied case of onedimensional colloid motion in a flow of constant velocity discussed in Sect. 5.2.2 below, the Langevin equation simplifies to
(49) 
and the ingredients of the first law become
(50) 
and
(51) 
3 Fluctuation theorems (FTs)
Fluctuation theorems express universal properties of the probability distribution for functionals , like work, heat or entropy change, evaluated along the fluctuating trajectories taken from ensembles with wellspecified initial distributions . In this section, we give a phenomenogical classification into three classes according to their mathematical appearance and point out some general mathematical consequences. The most prominent ones will then be discussed in physical terms with references to their original derivation. For proofs of these relations within stochastic dynamics from the present perspective, we provide in Sect. 4 the unifying one for all FTs that also shows that there is essentially an infinity of such relations.
3.1 Phenomenological classification
3.1.1 Integral fluctuation theorems (IFTs)
A nondimensionalized functional with probability distribution function obeys an IFT if
(52) 
The convexity of the exponential functions then implies the inequality
(53) 
which often represents a wellknown thermodynamic law. With the exception of the degenerate case, , the IFT implies that there are trajectories for which is negative. Such events have sometimes then been characterized as “violating” the corresponding thermodynamic law. Such a formulation is controversial since classical thermodynamics, which ignores fluctuations from the very beginning, is silent on issues beyond its range of applicability. The probability of such events quickly diminishes for negative . Using (52), it is easy to derive for [60]
(54) 
This estimate shows that relevant “violations” occur for of order 1. Restoring the dimensions in a system with relevant degrees of freedom, will typically be of order which implies that in a large system such events are exponentially small, i.e., occur exponentially rarely. This observation essentially reconciles the effective validity of thermodynamics on the macroscale with the still correct mathematical statements that even for large systems, in principle, such events must occur.
An IFT represents one constraint on the probability distribution . If it is somehow known that is a Gaussian, the IFT implies the relation
(55) 
between variance and mean of .
3.1.2 Detailed fluctuation theorems (DFTs)
A detailed fluctuation theorem corresponds to the stronger relation
(56) 
for the pdf . Such a symmetry constrains “one half” of the pdf which means, e.g., that the even moments of can be expressed by the odd ones and vice versa. A DFT implies the corresponding IFT trivially. Further statistical properties of following from the validity of the DFT (and some from the IFT) are derived in [61].
Depending on the physical situation, a variable obeying the DFT has often been called to obey either a transient FT (TFT) or a steady state FT (SSFT). These notions will be explained below for the specific cases.
3.1.3 (Generalized) Crooks fluctuation theorems (CFTs)
These relations compare the pdf of the original process one is interested in with the pdf of the same physical quantity for a “conjugate” (mostly the timereversed) process. The general statement then is that
(57) 
which implies the IFT (but not the DFT) for since is normalized.
3.2 Nonequilibrium work theorems
These relations deal with the probability distribution for work spent in driving the system from a (mostly equilibrium) initial state to another (not necessarily equilibrium) state. They require only a notion of work defined along the trajectory but not yet the concept of stochastic entropy.
3.2.1 Jarzynski relation (JR)
In 1997, Jarzynski showed that the work spent in driving the system from an initial equilibrium state at via a timedependent potential for a time obeys [6]
(58) 
where is the free energy difference between the equilibrium state corresponding to the final value of the control parameter and the initial state. In the classification scheme proposed here, it can technically be viewn as the IFT for the (scaled) dissipated work
(59) 
The paramount relevance of this relation – and its originally so surprising feature – is that it allows to determine the free energy difference, which is a genuine equilibrium property, from nonequilibrium measurements (or simulations). It represents a strengthening of the familiar second law which follows as the corresponding inequality. It has orginally been derived using a Hamiltonian dynamics (which requires decoupling from and coupling to a heat bath at the beginning and end of the process, respectively) but soon been shown to hold for stochastic dynamics as well [7, 8, 9]. Its validity requires that one starts in the equilibrium distribution but not that the system has relaxed at time into the new equilibrium. In fact, the actual distribution at the end will be but any further relaxation at constant would not contribute to the work anyways.
Within stochastic dynamics, the validity of the JR (as of any other FT with a thermodynamic interpretation) essentially rests on assuming that the noise in the Langevin equation (1) is not affected by the driving. A related issue arises in the Hamiltonian derivation of the JR which requires some care in identifying the proper role of the heat bath during the process [62, 63].
The JR has been studied for many systems analytically, numerically, and experimentally. Specific case studies for stochastic dynamics will be classified and quoted below in Sect. 5. As an important application, based on a generalization introduced by Hummer and Szabo [10], the Jarzynski relation can be used to reconstruct the free energy landscape of a biomolecule as discussed in Sect. 9.3 below.
3.2.2 BochkovKuzolev relation (BKR)
The Jarzynski relation should be distinguished from an earlier relation derived by Bochkov and Kuzolev [11, 12]. For a system initially in equilibrium in a timeindependent potential and for subject to an additional space and timedependent force (possibly arising from an additional potential), the work (25) integrated over a trajectory obeys the BochkovKuzolev relation (BKR)
(60) 
Contrary to some claims, the BKR is different from the Jarzynski relation since they apply a priori to somewhat different situations [21, 64, 65]. The JR as discussed above applies to processes in a timedependent potential, whereas the BKR applies to a process in a constant potential with some additional force. If, however, in the latter case, this explicit force arises from a potential as well, both the BKR and the JR (58) hold for the respective forms of work.
3.2.3 Crooks fluctuation theorem (CFT)
In the Crooks relation, the pdf for work spent in the original (the “forward”) process is related to the pdf for work applied in the reversed process where the control parameter is driven according to and one starts in the equilibrium distribution corresponding to . These two pdfs obey [8, 9]
(61) 
Hence, can be obtained by locating the crossing of the two pdfs which for biomolecular applications turned out to be a more reliable method than using the JR. Clearly, the Crooks relation implies the JR since is normalized. Technically, the Crooks relation is of the type (57) for with the conjugate process being the reversed one.
3.2.4 Further general results on
Beyond the JR and the CFT, further exact results on are scarce. For systems with linear equation of motion, the pdf for work (but not for heat) is a Gaussian for arbitrary timedependent driving [66, 67]. For slow driving, i.e., for where is the typical relaxation time of the system at fixed and the duration of the process, an expansion based on this timescale separation yields a Gaussian for any potential [68]. Such a result has previously been expected [69, 70] or justified by invoking arguments based on the central limit theorem [71]. Two observations show, however, that such an expansion is somewhat delicate. First, even in simple examples there occur terms that are nonanalytic in [68]. Second, for the special case of a “breathing parabola”, , any protocol with leads to for which is obviously violated by a Gaussian. How the latter effectively emerges in the limit of slow driving is investigated in [72].
From another perspective, Engel [73, 74] investigated the asymptotic behavior of for small using a saddle point analysis. The value of this approach is that it can provide exact results for the tail of the distribution. Specific examples show an exponential decay. Saha et al. [75] suggest that the work distribution for quite different systems can be mapped to a class of universal distributions.
3.3 FTs for entropy production
3.3.1 Ift
The total entropy production along a trajectory as given by
(62) 
with
(63) 
and defined in (27), obeys the IFT [20]
(64) 
for arbitrary initial distribution , arbitrary timedependent driving and an arbitrary length of the process.
Formally, this IFT can be considered as a refinement of the second law, , which is the corresponding inequality. Physically, however, it must be stressed that by using the Langevin equation a fundamental irreversibility has been implemented from the very beginning. Thus, this IFT should definitely not be considered to constitute a fundamental proof of the second law.
3.3.2 Steadystate fluctuation theorem (SSFT)
In a NESS with fixed , the total entropy production obeys the stronger SSFT
(65) 
again for arbitrary length . This relation corresponds to the genuine “fluctuation theorem”. It has first been found in simulations of twodimensional sheared fluids [1] and then been proven by Gallavotti and Cohen [2] using assumptions about chaotic dynamics. For stochastic diffusive dynamics as considered specifically in this review, it has been proven by Kurchan [3] and Lebowitz and Spohn [4]. Strictly speaking, in these early works the relation holds only asymptotically in the longtime limit since entropy production had been associated with what is here called entropy production in the medium. If one includes the entropy change of the system (28), the SSFT holds even for finite times in the steady state [20].
3.3.3 HatanoSasa relation
The HatanoSasa relation applies to systems with steady states . With the splitting of the dissipated heat into a housekeeping and excess one (21), the IFT [13]
(66) 
holds for any length of trajectory with . The corresponding inequality
(67) 
allows an interesting thermodynamic interpretation. The lefthand side can be seen as ensemble entropy change of the system in a transition from one steady state to another. Within the framework discussed in this review, this interpretation is literally true provided one waits for final relaxation at constant since then . A recent generalization of the HS relation leads to a variational scheme for approximating the stationary state [76].
With the interpretation of the left hand side as entropy change in the system, the inequality (67) provides for transitions between NESSs what the famous Clausius inequality does for transitions between equilibrium states. The entropy change in the system is at least as big as the excess heat flowing into the system. For transitions between NESSs, the inequality (67) is sharper than the Clausius one (which still applies in this case and becomes just ) since scales with the transition time whereas can remain bounded and can actually approach equality in (67) for quasistatic transitions.
Experimentally, the HatanoSasa relation has been verified for a colloidal particle pulled through a viscous liquid at different velocities which corresponds to different steady states [77].
3.3.4 IFT for housekeeping heat
Finally, it should be noted that the second contribution to heat, the housekeeping heat also obeys an IFT [78]
(68) 
for arbitrary initial state, driving and length of trajecories.
4 Unification of FTs
Originally, the FTs have been found and derived on a case by case approach. However, it has soon become clear that within stochastic dynamics a unifying strategy is to investigate the behaviour of the system under timereversal. Subsequently, it turned out that comparing the dynamics to its “dual” one [13, 79], eventually also in connection with timereversal, allows a further unification. In this section, we outline this general approach and show how the prominent FTs discussed above (and a few further ones mentioned below) fit into, or derive from, this framework. Even though this section is inevitably somewhat technical and dense, it is selfcontained. It could be skipped by readers not interested in the proofs or systematics of the FTs. For related mathematically rigorous approaches to derive FTs for diffusive dynamics, see [80, 81, 82, 83].
4.1 Conjugate dynamics
FTs for the original process with trajectories , , an initial distribution and a conditional weight are most generally derived by formally invoking a “conjugate” dynamics for trajectories . These are supposed to obey a Langevin equation
(69) 
with . The trajectories with weight run over a time and start with an initial distribution . Averages of the conjugate dynamics will be denoted by .
This conjugate dynamics is related to the original process by a onetoone mapping
(70) 
which allows to express all quantities occuring in the conjugate dynamics in terms of the orginal ones.
The crucial quantity leading to the FTs is a master functional given by the logratio of the unconditioned path weights
(71)  
that consists of a “boundary” term coming from the two initial distributions and a “bulk” term .
Three choices for the conjugate dynamics and the associated mapping have been considered so far. In all cases, neither the temperature nor the functional form of the mobilities have been changed for the conjugate dynamics, i.e., and .
(i) Reversed dynamics: This choice corresponds to “timereversal”. The mapping reads
(72) 
with no changes at the functional dependence of the force from its arguments, i.e., .
The weight of the conjugate trajectories is easily calculated using the mapping (72) in the weight (4,5) leading to
(73) 
which is the part of the action that is odd under timereversal.
This relation allows a deep physical interpretation. For given initial point and final point , the log ratio between the probability to observe a certain forward trajectory and the probability to observe the timereversed trajectory is given by the heat dissipated along the forward trajectory.
(ii) Dual dynamics: This choice alters the equations of motion for the trajectories such that (i) the stationary distribution remains the same for both processes and that (ii) the stationary current for the dual dynamics is minus the original one. Specifically, this mapping reads [79]
(74) 
which enters the conjugate Langevin equation (69) and no modification for and , i.e., and .
Calculating the action for the dual dynamics (69), the functional becomes
(75) 
(iii) Dualreversed dynamics: For this choice, the dual dynamics is driven with the timereversed protocol, i.e., the mapping of the force (74) is combined with the timereversal (72). In this case, the functional becomes [79]
(76) 
In summary, depending on the form of the conjugate dynamics, different parts of the dissipated heat form the functional . For later reference, we have introduced in the last two equations for the scaled contributions to the dissipated heat the corresponding entropies.
4.2 The master FT
4.2.1 Functionals with definite parity
The FT’s apply to functionals of the original dynamics that map with a definite parity to the conjugate dynamics according to
(77) 
such that represents the same physical quantity for the conjugate dynamics as does for the original one.
Examples for such functionals are work and heat that both are odd () for the reversed dynamics. For dual or dual reversed dynamics, however, these two functionals have no definite parity since both cases involve a different dynamics. Explicitly, the heat behaves under timereversal as . For dual dynamics, the heat transforms as which has, in general, no definite parity. On the other hand, the housekeeping heat is odd for the dual dynamics and even for both the reversed and the dualreversed dynamics.
The stochastic entropy , in general, has no definite parity under timereversal since is defined through the solution of the FokkerPlanck equation which is not odd under timereversal. In particular, does not solve the FokkerPlanck equation for the timereversed process even if one starts the reversed process with the final distribution of the original process. The change in the nonequilibrium potential , however, is odd under timereversal. This difference between and implies that occurs more frequently in FTs.
4.2.2 Proof
With these preparations, one can easily derive the master FT
(78)  
for any function depending on an arbitrary number of such functionals . For the second equality, we use the definitions (71) and the parity relation (77); for the third we recognize that summing over all daggered trajectories is equivalent to summing over all original ones both for and . With the choice , this FT leads to the most general IFT from which all known IFTlike relations follow as shown in Sect. 4.3.
By choosing for the characteristic function, one obtains a generalized FT for joint probabilities in the form
(79) 
that relates the pdf for the conjugate process to the pdf of the original one and a conditional average. All known DFTs for stochastic dynamics follow as special cases of this general theorem as shown in Sect. 4.4. The key point is to (i) select the appropriate conjugate process for which the quantity of interest has a unique parity, which is most often just the reversed dynamics, (ii) identify for the generally free initial distribution an appropriate function, and (iii) express the functional using physical quantities, preferentially the quantity of interest .
4.3 General IFTs
The simplest choice for the function in (78) is the identity, , leading to the IFT . Explicitly, one obtains for the three types of conjugate dynamics:
(i) By choosing the reversed dynamics (72) and with (73), the class of IFTs
(80) 
follows for any initial condition , any length of trajectories , and any normalized function [20]. By specializing the latter to the solution of the FokkerPlanck equation for one obtains the IFT for total entropy production (64).
For a system in a timedependent potential and by starting in an initial distribution given by the corresponding Boltzmann factor, one obtains the JR (58) for the choice corresponding to the Boltzmann distribution for the final value of the control parameter.
A variety of “endpoint” relations can be generated from (80) as follows. By choosing , one obtains
(81) 
for any function [84]. Likewise, for and , by choosing , one obtains
(82) 
which has been first derived by Crooks [9]. Here, the average on the right hand side is the equilibrium average at the final value of the control parameter. In the same fashion, one can derive
(83) 
by choosing for a timeindependent potential and arbitrary force which is the endpoint relation corresponding to the BKR (60). The latter follows trivially by choosing .
For processes with feedback control as discussed in Sect. 7.3 below, it will be convenient to exploit the endpoint conditioned average
(84) 
valid for any which follows from (80) by choosing . Equivalently, by choosing ,
(85) 
holds for summing over the initial point conditioned average.
(ii) By using the dual dynamics with , the IFT for the housekeeping heat [78]
(86) 
valid for any initial distribution follows.
(iii) For the dualreversed dynamics, one gets the class of IFTs from