Minimax representation of nonexpansive functions and application to zero-sum recursive games
We show that a real-valued function on a topological vector space is positively homogeneous of degree one and nonexpansive with respect to a weak Minkowski norm if and only if it can be written as a minimax of linear forms that are nonexpansive with respect to the same norm. We derive a representation of monotone, additively and positively homogeneous functions on spaces and on , which extends results of Kolokoltsov, Rubinov, Singer, and others. We apply this representation to nonconvex risk measures and to zero-sum games. We derive in particular results of representation and polyhedral approximation for the class of Shapley operators arising from games without instantaneous payments (Everett’s recursive games).
Key words and phrases:Nonexpansive maps, weak Minkowski norms, zero-sum games, recursive games, Shapley operators, risk measures, minimax representation
2010 Mathematics Subject Classification:49J35, 91A15, 26B25
Repeated zero-sum games can be studied by means of dynamic programming operators, also known as Shapley operators. For instance, in the case of a game with finite state space and perfect information, the Shapley operator is a self-map of given by
Here, and are the (possibly infinite) action spaces in state , is a stage payoff and gives the transition probability from state to state , when action is chosen by the first player and action is chosen by the second player. We have that and . This operator governs the evolution of the value function of the game. We refer the reader to [FV97, NS03, Sor04] for more background.
Shapley operators are characterized by the following two properties: they are monotone, i.e., order-preserving ( being endowed with the standard partial order), and they are additively homogeneous, i.e., they commute with the addition of a constant (a vector with all its entries equal). The importance of these properties in the theory of dynamic programming was recognized early on, in particular by Blackwell [Bla65] and Crandall and Tartar [CT80]. For more recent studies, we refer the reader to the work of Rosenberg and Sorin [RS01a, Sor04], providing a game-theoretic viewpoint, and of Martínez-Legaz, Rubinov and Singer [MLRS02], providing an abstract convexity viewpoint. Order preserving and additively homogeneous maps are known to be nonexpansive in the sup-norm, and also in certain weak Minkowski norms. Such weak norms have been studied in metric geometry, in particular by Papadopoulos and Troyanov [PT14]. They are substitutes of norms that are not necessarily symmetric or coercive, and arise naturally in the study of Shapley operators, see [GV12].
Conversely, Kolokoltsov showed that every operator from to that satisfies the two above properties (monotonicity and additive homogeneity) can be written as a Shapley operator [Kol92]. His proof is based on a minimax representation formula of Evans, which applies more generally to Lipschitz functions [Eva84]. Rubinov and Singer [RS01b] showed that the transition probabilities in (1) may even be chosen to be degenerate (deterministic), i.e., such that every stochastic vector may be required to have precisely one nonzero entry. Gunawardena and Sparrow obtained independently an equivalent result (see [Gun03, Prop. 2.3]). In the context of abstract convexity, a representation of functions with similar properties over a cone has been given in [DMLR04, DMLR08].
An important subclass of monotone and additively homogeneous operators arises by requiring every coordinate map to be convex. Operators in this class appear in stochastic control problems. Then, one can obtain a one-player type representation which has the same form as (1), but with no infimum, by exploiting the Fenchel-Legendre duality [AG03]. In infinite dimension, representation theorems for real functions with the same kind of properties have been proven in the setting of convex risk measures [FS02, FRG02].
Another important subclass is composed of positively homogeneous operators, i.e., those operators which commute with the product by a nonnegative constant. They appear in Perron-Frobenius theory [GG04], in the study of repeated zero-sum games [RS01a]. In infinite dimension, convex risk measures satisfying the positive homogeneity property are referred to as coherent risk measures [ADEH99, Del02].
In this paper, we first establish a general minimax representation theorem which applies to real functions on a topological vector space that are nonexpansive with respect to a weak Minkowski norm (Theorem 3.4). We also characterize the nonexpansive maps that are positively homogeneous of degree (Theorem 3.10). As a corollary, we arrive at our main application: a representation theorem for monotone, additively and positively homogeneous operators from to (Corollary 4.3) showing that they correspond precisely to the class of zero-sum games in which the payment occurs only at the last stage (Everett’s recursive games). This is motivated by the “operator approach” to zero-sum games [RS01a], in which properties of such games are derived from axiomatic properties of their dynamic programming operators: we show that the dynamic programming operators of the subclass of recursive games are characterized axiomatically as well.
This representation also leads to an approximation of such operators by polyhedral maps (involving a finite number of and operations). Such approximations can been used in the setting of “max-plus basis methods” for the numerical solution of Hamilton-Jacobi type PDE, see [McE06, AGL08] for background, and the recent work of McEneaney and Pandey [MP15] for an application of minimax approximations. We also arrive at a representation theorem for nonconvex risk measures.
2. Preliminary results
Throughout the paper, we denote by a real topological vector space (TVS). We denote by the dual space of (that is, the space of linear forms on ), by its topological dual space (that is, the space of continuous linear forms on ), and by the duality product.
We shall specially consider the situation in which is a vector space with an (Archimedean) order unit. I.e., we assume that is a real vector space with an order relation that is compatible with the algebraic structure of , that is, satisfying the following two axioms:
where is the set of nonnegative real numbers. We also assume that is equipped with a special vector (or simply if the context is clear), called an order unit, and such that, for every , there exists a scalar with . Finally, we assume that this order unit is Archimedean, i.e., such that, for any , we have if for all . Such a space will be endowed with the topology defined by the following norm:
see [PT09]. Note that if is the Euclidean space equipped with the standard partial order, then the standard unit vector of is an order unit, and the corresponding norm (2) is the usual sup-norm. Hence, we will abusively refer to (2) as the “sup-norm” even in general situations.
An important particular case is obtained when is an AM-space with unit, i.e., a Banach lattice equipped with an order unit, such that the norm satisfies (2), see [AB06]. By the Kakutani-Krein theorem, any AM-space with unit is isomorphic (lattice isometric) to a space of continuous functions over some compact Hausdorff set , equipped with the sup-norm, see [AB06, Th. 8.29] or [Sch74, Ch. II, Th. 7.4].
Given two vector spaces with an order unit, and , we will be interested in maps that satisfy some of the following properties:
The importance of the monotonicity and the additive homogeneity properties in optimal control and game theory is well known. Crandall and Tartar [CT80] showed that
when is a space. It is also known that
see, e.g., [AG03]. These relations are readily generalized to any vector space with an order unit.
The monotonicity and additive homogeneity properties turn out to be related with nonexpansiveness in weak Minkowski norms. By weak Minkowski norm on , we mean a function that is convex and positively homogeneous (property ((H))), but not necessarily symmetric (we do not require that ). Our definition is a variant of the one in [PT14], where is also required to be nonnegative and may take infinite values. We say that a real function on , , is nonexpansive with respect to if for every .
When , a useful example of weak Minkowski norm, arising in Hilbert geometry [PT14], is the “top” map defined by
or its variant,
When , we shall consider the following properties:
Gunawardena and Keane showed in [GK95] that
Again, these relations can be readily generalized to any vector space with an order unit, defining, for a vector space with an order unit , the function by
3. Representation theorems
In this section, we give a general minimax characterization of nonexpansive functions with respect to a weak Minkowski norm, which will lead to extension and refinements of the known minimax representations of Shapley operators.
3.1. Representation of nonexpansive functions with respect to weak Minkowski norms
We shall need the following lemma, which is a variation on Legendre-Fenchel duality.
Let be a real TVS and let be a weak Minkowski norm, continuous with respect to the topology of . Then, for every ,
where is a nonempty convex set of , compact for the weak* topology.
For we have, by definition of , . Let be the linear form defined on the vector subspace of generated by , such that . If , then , since is positively homogeneous. If , then , the inequality coming from the convexity of the weak norm: . Hence is dominated by on the vector subspace and according to the Hahn-Banach extension theorem [AB06, Th. 5.53], there is a linear extension of to that is dominated by on . Therefore, and , which shows that . This also proves that is nonempty.
Note that, since is continuous, every linear form dominated by is also continuous, hence . The convexity of is straightforward and since this set can also be written as , we deduce that is a weak* closed set. Furthermore, for every and every we have . Hence, is pointwise bounded. Applying the Tychonoff Product theorem [AB06, Th. 2.61], we deduce that is weak* compact. ∎
In Lemma 3.1, we did not assume that is Hausdorff and locally convex. When these properties hold, Lemma 3.1 becomes a direct application of [AB06, Th. 7.52], which applies more generally to dual pairs. Alternatively, this result can be obtained using the Legendre-Fenchel duality for convex proper lower semicontinuous functions [ET99, Prop. 4.1], still assuming that is locally convex and Hausdorff.
The following simple observation shows that the maximum in (4) is attained in the closure of the set of extreme points of .
We already showed in Lemma 3.1 that for every , the supremum
is attained at some point . It follows from the Krein-Milman theorem that is the closed convex hull of the set , the closure being understood with respect to the weak* topology. Hence, there exists a net of elements of the convex hull of that converges to in this topology. In particular, every can be written as a finite sum with , and , for some , and so, . We deduce that . The opposite inequality follows readily from (4). Using the weak* continuity of the map , we get . Finally, since , which is a weak* closed subset of the weak* compact set , is also weak* compact, we deduce that the latter supremum is attained. ∎
We deduce from the previous lemmas a minimax representation theorem that directly extends the result of Rubinov and Singer [RS01b, Th. 5.3].
Let be a real TVS and let be a weak Minkowski norm, continuous with respect to the topology of . Denote by . A function is nonexpansive with respect to if, and only if,
Moreover, the value of the latter expression does not change if the maximum is restricted to the points .
If is nonexpansive with respect to , then we have, by definition, for every . We readily deduce that , the minimum being attained in . Replacing by its expression given in (4), we get the minimax representation formula (6). The remaining part of the theorem follows directly from Lemma 3.3. Conversely, as a consequence of Lemma 3.1, any real function given by (6) is easily seen to be nonexpansive with respect to . ∎
The minimax representation in Theorem 3.4 has the following dual maximin version, the minimum and maximum being taken over the same sets.
Let , and be as in Theorem 3.4. A function is nonexpansive with respect to if, and only if,
Moreover, the value of the latter expression does not change if the minimun is restricted to the points .
Similarly, all the subsequent results admit dual versions.
If is also assumed to be convex, the minimization and maximization in (6) can be switched to obtain:
This new representation is equivalent to the property that is equal to , where denotes the Fenchel-Legendre transform of , together to the one that the subdifferential of is necessarily included in . This leads to a one-player type representation , which has the same form as (1), but with no infimum. Representation formulæ of this nature have appeared in the theory of convex risk measures [FS02, Del02] and also in the setting of Markov decision processes [AG03] when .
As remarked in the proof of Theorem 3.4, is nonexpansive with respect to if, and only if, , for all . The dual version also applies: is nonexpansive with respect to if, and only if, , for all . As noted in [AGK05, Prop. 6.7], these conditions can be interpreted in terms of Moreau conjugacies [Mor70], meaning that and , where is the Moreau conjugacy of with respect to the coupling function , and . Using these two conditions, one deduces that is nonexpansive with respect to if, and only if, . This leads to a minimax representation which does not have the form of (1).
Let be a vector space with an order unit , and the topology defined by the norm (2). Let
Then, a function is monotone and additively homogeneous if, and only if,
Moreover, the value of the latter expression is not changed if the minimum is restricted to those such that , or if the maximum is restricted to the points .
Let be a monotone and additively homogeneous real function on . As exposed in Section 2, this is equivalent to the function being nonexpansive with respect to the weak Minkowski norm . Let be defined as in Theorem 3.4 and let us show that . If , then, for all , we have , so that , hence . Moreover, and , so that and , which shows that . This shows that . Conversely, if , then for all such that , we have , so that . This implies that for all , hence . Applying Theorem 3.4, we get the first equality in the corollary. Now, remark that with , since and . Then, a change of variable leads to
The remaining part of the corollary follows from Theorem 3.4 and the converse is straightforward. ∎
3.2. Representation of positively homogeneous nonexpansive functions
We now consider nonexpansive functions that are positively homogeneous. The following theorem characterizes these functions as minimax of nonexpansive linear forms.
Let be a real TVS and let be a weak Minkowski norm, continuous with respect to the topology of . Denote by . A function is positively homogeneous and nonexpansive with respect to if, and only if,
where . Moreover, the value of the expression (8) does not change if the maximum is restricted to the points .
The sufficiency of the condition is straightforward, so we only prove that it is necessary. Let be a real function positively homogeneous and nonexpansive with respect to . As a direct consequence of nonexpansiveness, we get that . By a change of variable, we have , since is also positively homogeneous. There, the minimum in is attained at all with . Given , let be the function defined on by , so that . We next show that is a continuous weak Minkowski norm.
Firstly, is bounded above by (take ) and below by (since ). In particular, is bounded above on a neighborhood of each point of . Secondly, using the positive homogeneity of and , we check that also shares this property. Thirdly, is convex. Indeed, the function is convex because so is . Its perspective function, defined on by if and otherwise, is also convex [BC11, Prop. 8.23]. The conclusion follows from the fact that is the marginal function with respect to the first variable of this former function.
We have shown that is a positively homogeneous convex function, finite everywhere and bounded above on a neighborhood of each point of . Hence, it is also continuous on [AB06, Th. 5.43] and we deduce from Lemma 3.1 that it is the support function of the weak* compact convex set . To conclude, it remains to show that this set is .
Let . Then, for every and every , we have . Taking we deduce that , and taking and we deduce that . Hence, which shows that .
Consider now . For every and every we have
Taking the infimum for all in the right-hand side of the last inequality, we get that . Hence which shows that and consequently that . ∎
The following is an immediate corollary.
Let be a vector space with an order unit , and the topology defined by the norm (2). Then, a function is monotone, additively homogeneous, and positively homogeneous if, and only if,
Moreover, the value of the expression in (9) does not change if the minimum is restricted to those such that , or if the maximum is restricted to the points .
We now point out some applications of the present representation theorem to nonconvex risk measure and to the representation of recursive games.
4.1. Representation of nonconvex risk measures
Let be a probability space. Denote by the space of equivalence classes of a.s. bounded real-valued random variables, equipped with the usual norm, and the usual partial order, i.e., if any representative of is a random variable almost surely nonnegative. This is a special case of AM-space with unit, the unit being the random variable a.s. equal to . Its topological dual space is the space of finitely additive measures of bounded variation which are absolutely continuous with respect to , denoted by [DS88, Th. IV.8.16]. Following a standard notation, we will write instead of for and . We denote by the set of positive bounded finitely additive measures and by the set of finitely additive probability measures.
A risk measure is a function that satisfies the following conditions, for every :
see [FS11]. This is equivalent to the function being monotone and additively homogeneous.
where is a convex subset of the space of finitely additive probability measures on , closed with respect to the -topology. As a direct application of Corollary 3.8, we obtain a similar representation for general nonconvex risk measures.
Let be a risk measure on . Then,
The following corollary characterizes the nonconvex risk measures that are positively homogeneous. It is a direct application of Corollary 3.11.
Let be a positively homogeneous risk measure. Then,
4.2. Representation of payment-free Shapley operators
Here, we consider the vector space , denoting its usual scalar product. We call payment-free Shapley operator, an operator over that is monotone, additively and positively homogeneous. This terminology is justified by the following corollary, which shows that all operators of this kind precisely arise from zero-sum games in which the function representing the payment made at every stage is identically zero. Such games, in which the payment only occurs the “last day”, have been studied after Everett under the name of recursive games [Eve57]. The operators of these games also arise as recession functions of general Shapley operators. They allow one to study mean payoff and ergodicity problems, see [RS01a] and [AGH15].
Let be a payment-free Shapley operator, i.e. a monotone, additively and positively homogeneous operator . Then, can be represented as
where: ; for every and every , is a finite subset of ; for every and every , is a stochastic vector with at most two positive coordinates.
Any operator of the form (10) is a payment-free Shapley operator. Conversely, let be a payment-free Shapley operator. Each coordinate function is monotone, additively and positively homogeneous. Then, it follows from Corollary 3.11 that, for every and every ,
where is the standard simplex of .
Finally, a standard result of convex geometry shows that every extreme point of the intersection of a polytope with a half-space is either an extreme point of the polytope, or the convex combination of two extreme points of this polytope (see for instance Lemma 3 of [FP96]). It follows that every element in is either an extreme point of , either a convex combination of two extreme points of . ∎
4.3. Approximation of Shapley operators
We use the latter result to approximate payment-free Shapley operators over by minimax maps where the and operators are taken over finite sets. Such maps play an important role algorithmically, in max-plus finite element method [AGL08], and more generally in idempotent methods [McE11, MP15].
Let be a weak Minkowski norm. We say that a subset is an -net of the set with respect to (the symmetrization of) if
Note that here, since the dimension is finite, is continuous, and then it is always possible to find a finite -net of a compact set with respect to .
Let be a weak Minkowski norm and let be a map nonexpansive with respect to . Then, for every compact set , and for every finite -net of with respect to , the function
is such that
Let . Since is nonexpansive with respect to , we have for every . Thus, . We also know that there is an index such that . Hence,
from which the second inequality follows. ∎
Let be a payment-free Shapley operator. Then, for all , there exists a payment-free Shapley operator such that
for all and , which can be represented in the form
where and are finite sets, and every is a stochastic vector with at most two positive coordinates.
Let be an -net of the unit sphere of with respect to the sup-norm. Using the minimax representation of , steming from Corollary 4.3, we know that for all and , there exists an action for which the minimum is attained in formula (10) applied to . Let be the finite subset of containing all the latter optimal actions in state . Then, let be the payment-free Shapley operator the th coordinate map of which is given by
where the action spaces are the same as in the minimax representation (10) of . In particular, we know that they are finite and that all the vectors are stochastic, with at most two positive coordinates.
By construction, we have for all vectors , with equality for every . Now, given a vector in the unit sphere of , we can choose a vector such that . Since both and are nonexpansive with respect to the sup-norm, we deduce that for all ,
The conclusion follows from the positive homogeneity of and . ∎
- C. D. Aliprantis and K. C. Border, Infinite dimensional analysis, third ed., Springer, Berlin, 2006, A hitchhiker’s guide.
- P. Artzner, F. Delbaen, J.-M. Eber, and D. Heath, Coherent measures of risk, Math. Finance 9 (1999), no. 3, 203–228.
- M. Akian and S. Gaubert, Spectral theorem for convex monotone homogeneous maps, and ergodic control, Nonlinear Anal. 52 (2003), no. 2, 637–679.
- M. Akian, S. Gaubert, and A. Hochart, Ergodicity conditions for zero-sum games, Discrete Contin. Dyn. Syst. 35 (2015), no. 9, 3901–3931.
- M. Akian, S. Gaubert, and V. Kolokoltsov, Set coverings and invertibility of functional Galois connections, Idempotent mathematics and mathematical physics, Contemp. Math., vol. 377, Amer. Math. Soc., Providence, RI, 2005, pp. 19–51. MR 2148996
- M. Akian, S. Gaubert, and A. Lakhoua, The max-plus finite element method for solving deterministic optimal control problems: basic properties and convergence analysis, SIAM J. Control Optim. 47 (2008), no. 2, 817–848.
- H. H. Bauschke and P. L. Combettes, Convex analysis and monotone operator theory in Hilbert spaces, CMS Books in Mathematics/Ouvrages de Mathématiques de la SMC, Springer, New York, 2011.
- D. Blackwell, Discounted dynamic programming, Ann. Math. Statist. 36 (1965), 226–235.
- M. G. Crandall and L. Tartar, Some relations between nonexpansive and order preserving mappings, Proc. Amer. Math. Soc. 78 (1980), no. 3, 385–390.
- F. Delbaen, Coherent risk measures on general probability spaces, Advances in finance and stochastics, Springer, Berlin, 2002, pp. 1–37. MR 1929369
- J. Dutta, J. E. Martínez-Legaz, and A. M. Rubinov, Monotonic analysis over cones. I, Optimization 53 (2004), no. 2, 129–146.
- by same author, Monotonic analysis over cones. III, J. Convex Anal. 15 (2008), no. 3, 561–579.
- N. Dunford and J. T. Schwartz, Linear operators. Part I, Wiley Classics Library, John Wiley & Sons, Inc., New York, 1988, General theory, With the assistance of William G. Bade and Robert G. Bartle, Reprint of the 1958 original, A Wiley-Interscience Publication.
- I. Ekeland and R. Témam, Convex analysis and variational problems, english ed., Classics in Applied Mathematics, vol. 28, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 1999, Translated from the French.
- L. C. Evans, Some min-max methods for the Hamilton-Jacobi equation, Indiana Univ. Math. J. 33 (1984), no. 1, 31–50.
- H. Everett, Recursive games, Contributions to the theory of games, vol. 3, Annals of Mathematics Studies, no. 39, Princeton University Press, Princeton, N. J., 1957, pp. 47–78.
- K. Fukuda and A. Prodon, Double description method revisited, Combinatorics and computer science (Brest, 1995), Lecture Notes in Comput. Sci., vol. 1120, Springer, Berlin, 1996, pp. 91–111.
- M. Frittelli and E. Rosazza Gianin, Putting order in risk measures, Journal of Banking & Finance 26 (2002), no. 7, 1473–1486.
- H. Föllmer and A. Schied, Convex measures of risk and trading constraints, Finance and Stochastics 6 (2002), no. 4, 429–447.
- by same author, Stochastic finance, extended ed., Walter de Gruyter & Co., Berlin, 2011, An introduction in discrete time.
- J. Filar and K. Vrieze, Competitive Markov decision processes, Springer-Verlag, New York, 1997.
- S. Gaubert and J. Gunawardena, The Perron-Frobenius theorem for homogeneous, monotone functions, Trans. Amer. Math. Soc. 356 (2004), no. 12, 4931–4950 (electronic).
- J. Gunawardena and M. Keane, On the existence of cycle times for some nonexpansive maps, Tech. report, Technical Report HPL-BRIMS-95-003, Hewlett-Packard Labs, 1995.
- J. Gunawardena, From max-plus algebra to nonexpansive mappings: a nonlinear theory for discrete event systems, Theoret. Comput. Sci. 293 (2003), no. 1, 141–167.
- S. Gaubert and G. Vigeral, A maximin characterisation of the escape rate of non-expansive mappings in metrically convex spaces, Math. Proc. Cambridge Philos. Soc. 152 (2012), no. 2, 341–363.
- V. N. Kolokoltsov, On linear, additive, and homogeneous operators in idempotent analysis, Idempotent analysis, Adv. Soviet Math., vol. 13, Amer. Math. Soc., Providence, RI, 1992, pp. 87–101.
- W. M. McEneaney, Max-plus methods for nonlinear control and estimation, Systems & Control: Foundations & Applications, Birkhäuser Boston, Inc., Boston, MA, 2006.
- by same author, Distributed dynamic programming for discrete-time stochastic control, and idempotent algorithms, Automatica J. IFAC 47 (2011), no. 3, 443–451.
- J.-E. Martínez-Legaz, A. M. Rubinov, and I. Singer, Downward sets and their separation and approximation properties, J. Global Optim. 23 (2002), no. 2, 111–137.
- J.-J. Moreau, Inf-convolution, sous-additivité, convexité des fonctions numériques, J. Math. Pures Appl. (9) 49 (1970), 109–154.
- W. M. McEneaney and A. Pandey, Development of an idempotent algorithm for a network-delay game, 2015 Proceedings of the Conference on Control and its Applications, Paris, France, July 8-10, 2015, pp. 439–446.
- A. Neyman and S. Sorin, Stochastic games and applications, vol. 570, Springer, 2003.
- V. I. Paulsen and M. Tomforde, Vector spaces with an order unit, Indiana Univ. Math. J. 58 (2009), no. 3, 1319–1359.
- A. Papadopoulos and M. Troyanov, Weak Minkowski spaces, Handbook of Hilbert geometry, IRMA Lect. Math. Theor. Phys., vol. 22, Eur. Math. Soc., Zürich, 2014, pp. 11–32. MR 3329875
- D. Rosenberg and S. Sorin, An operator approach to zero-sum repeated games, Israel J. Math. 121 (2001), 221–246.
- A. M. Rubinov and I. Singer, Topical and sub-topical functions, downward sets and abstract convexity, Optimization 50 (2001), no. 5-6, 307–351.
- H. H. Schaefer, Banach lattices and positive operators, Springer-Verlag, New York-Heidelberg, 1974, Die Grundlehren der mathematischen Wissenschaften, Band 215.
- S. Sorin, Asymptotic properties of monotonic nonexpansive mappings, Discrete Event Dyn. Syst. 14 (2004), no. 1, 109–122.