Relative Interior Rule in BlockCoordinate Minimization
1 Introduction
(Block)coordinate minimization is an iterative optimization method which in every iteration finds a global minimum of the objective over a variable or a subset of variables, while keeping the remaining variables constant. For some problems, coordinate minimization converges to a global minimum. This class includes unconstrained problems with convex differentiable objective function [1, §2.7] or convex objective function whose nondifferentiable part is separable [11]. For general convex problems, the method need not converge to a global minimum but only to a local one, where ‘local’ is meant with respect to moves along (subsets of) coordinates.
For largescale nondifferentiable convex problems, (block)coordinate minimization can be an acceptable option despite its inability to converge to a global minimum. An example is a class of methods to solve the linear programming relaxation of the discrete energy minimization problem (also known as MAP inference in graphical models). These methods apply (block)coordinate minimization to various forms of the dual linear programming relaxation. Examples are maxsum diffusion [7, 9, 12], TRWS [5], MPLP [2], and SRMP [6]. For many problems from computer vision, it has been observed [10, 4] that TRWS converges faster than the competing methods and its fixed points are often not far from global minima, especially for large sparse instances.
When blockcoordinate minimization is applied to a general convex problem, in every iteration the minimizer over the current coordinate block need not be unique and therefore a single minimizer must be chosen. These choices can significantly affect the quality of the achieved local minima. We propose that this minimizer should always be chosen from the relative interior of the set of all minimizers over the current block. Indeed, it can be easily verified that maxsum diffusion satisfies this condition. We show that blockcoordinate minimization methods satisfying this condition are not worse, in a certain precise sense, than any other blockcoordinate minimization methods.
2 Main Results
For brevity, we will use
(1) 
to denote the set of all global minima of a function on a set .
Suppose we want to minimize a convex function on a closed convex set where is a finitedimensional vector space over . For that, we consider a coordinatefree generalization of blockcoordinate minimization. Let be a finite set of subspaces of , which represent search directions. Having an estimate of the minimum, the next estimate is always chosen such that
(2) 
for some . Clearly, . A point satisfying
(3) 
has the property that cannot be improved by moving from within along any single subspace from . We call such a point a local minimum of on with respect to . When and/or is clear from context, we will speak only about a local minimum of on or just a local minimum. Note that the term ‘local minimum’ is used here in a different meaning than is usual in optimization and calculus.
Coordinate minimization and blockcoordinate minimization are special cases of this formulation. In the former, we have and where denotes the th vector of the standard basis of . In the latter, we have and each element of is the span of a subset of the standard basis of .
Recall [8, 3] that the relative interior of a convex set , denoted by , is the topological interior of with respect to the affine hull of . We propose to modify condition (2) such that the minimum is always chosen from the relative interior of the current optimal set. Thus, (2) changes to
(4) 
A point always exists because the relative interior of any nonempty convex set is nonempty. We call a point that satisfies
(5) 
an interior local minimum of on with respect to . Clearly, every interior local minimum is a local minimum.
In our analysis, another type of local minimum will naturally appear: preinterior local minimum. It will be precisely defined later; informally, it is only a finite number of iterations (4) away from an interior local minimum.
Consider a sequence satisfying (2) resp. (4), where denotes the positive integers. To ensure that each search direction is always visited again after a finite number of iterations, we assume that the sequence contains each element of an infinite number of times. For brevity, we will often write only and instead of and . The following facts, proved in the sequel, show that methods satisfying (4) are not worse, in a precise sense, than methods satisfying (2):

For every sequence satisfying (4), if is an interior local minimum then is an interior local minimum for all .

For every sequence satisfying (4), if is a preinterior local minimum then is an interior local minimum for some .

For every sequence satisfying (2), if is a preinterior local minimum then for all .

For every sequence satisfying (4), if is not a preinterior local minimum then for some .
To illustrate this, consider an example of coordinate minimization applied on a simple linear program (see the picture below). Let , , (i.e., is constant vertically and decreases to the right), and . The set of global minima is the line segment , the set of local minima is , the set of interior local minima is , and the set of preinterior local minima is . The thick polyline shows the first few points of a sequence satisfying (4), where the sequence alternates between the two subspaces from . When starting from any point , every sequence satisfying (4) leaves any noninterior local minimum after a finite number of iterations, while improving the objective function. Informally, this is because when the objective cannot be decreased by moving along any single subspace from , condition (4) at least enforces the point to move to a face of of a higher dimension (if such a face exists), providing thus ‘more room’ to hopefully decrease the objective in future iterations. In contrast, condition (2) allows a sequence to stay in any (possibly noninterior) local minimum forever. Of course, when starting from , every sequence satisfying (2) will stay in forever. This just confirms the wellknown fact that for some nonsmooth convex problems, coordinate minimization can get stuck in a point that is not a global minimum.
Moreover, we prove the following convergence result: if the choices in (4) are fixed such that is a continuous function of , the elements of are visited in a cyclic order, and the sequence is bounded, then the distances of from the set of preinterior local minima converges to zero.
3 Global Minima Are Local Minima
As a warmup, we prove one expected property of local minima: every element of (global minimum) is a local minimum and every element of (which could be called interior global minimum) is an interior local minimum. Noting that global minima are local minima with respect to , we actually prove, in Theorem 2 below, a more general fact. For sets and of subspaces of , we say that dominates if for every there is such that .
Lemma 1.
Let and . Let . Then .
Proof.
To prove , we need to prove that implies . This is obvious because if holds for all , then it holds for all .
To prove , we need to prove that and imply . For that, it suffices to show that and imply that for all . This is true, because for some would imply . ∎
Theorem 2.
Let be a convex set and be a convex function. Let and be finite sets of subspaces of such that dominates .

Every local minimum with respect to is a local minimum with respect to .

Every interior local minimum with respect to is an interior local minimum with respect to .
4 Linear Objective Function
Using the epigraph form, the minimization of a convex function on a closed convex set can be transformed to the minimization of a linear function on a closed convex set. Therefore, further in §4 we assume that is closed convex and is linear. We will return to the case of nonlinear convex later in §5.
For , we denote
(7) 
For this is a line segment, for it is a singleton. It holds that
(8) 
For we have , for we have .
We recall basic facts about faces of a convex set [8, 3]. A face of a convex set is a convex set such that every line segment from whose relative interior intersects lies in , i.e.,
(9) 
The set of all faces of a closed convex set partially ordered by
inclusion is a complete lattice, in particular it is closed under
(possibly infinite) intersections.
For a point , let denote the intersection of all
faces (equivalently, the smallest face) of that contain . For
every ,
{IEEEeqnarray}rCrClCl
y&∈& F(X,x) & ⟺ & F(X,y)&⊆& F(X,x) , \IEEEyesnumber\IEEEyessubnumber*
y&∈& riF(X,x) & ⟺ & F(X,y)&=&F(X,x) .
y&∈& rbF(X,x) & ⟺ & F(X,y)&⊊& F(X,x) ,
where denotes the relative boundary of a
closed convex set . Equivalence (9) shows that is
in fact the unique face of having in its relative interior.
Note that (9) follows from (9) and (9).
The following simple lemmas will be used several times later:
Lemma 3.
Let be a convex set. We have iff for every there exists such that .
Proof.
The ‘onlyif’ direction is immediate from the definition of relative interior. For the ‘if’ direction see, e.g., [8, Theorem 6.4]. ∎
Lemma 4.
Let be closed convex sets such that . Let . Then
{IEEEeqnarray}rCrCrCr
y &∈& Y & ⟹ & y &∈& F(X,x) \IEEEyesnumber\IEEEyessubnumber*
y &∈& riY & ⟹ & y &∈& riF(X,x)
y &∈& rbY & ⟹ & y &∈& rbF(X,x)
Proof.
Lemma 5.
Let and . Then .
Proof.
Let be such that (note that if then is unique, otherwise we can choose any ). Let , hence . Subtracting the two equations yields , hence . ∎
The picture illustrates Lemma 5 for the points in a general position (i.e., not collinear):
4.1 Structure of the Set of Local Minima
It is wellknown that the set of global minima of a linear function on a closed convex set is an (exposed) face of . We show that local resp. interior local minima also cluster to faces of . Moreover, similarly as the set of all faces of , we show that the set of faces of containing local resp. interior local minima are closed under intersections.
In the theorems in the rest of this section, the letter will always denote a subspace of .
Theorem 6.
Let and . Then .
Proof.
Corollary 7.
If is a local minimum, then every point of is a local minimum.
But notice that if and are local minima such that , then we can have .
Lemma 8.
Let and . Then .
Proof.
Lemma 9.
Let . Then .
Proof.
Theorem 10.
Let . Let for all . Let . Then .
Corollary 11.
Let . If every point from is an interior local minimum, then every relative interior point of the face is an interior local minimum.
Corollary 12.
If is an interior local minimum, then every point of is an interior local minimum.
Proof.
This is Corollary 11 for . ∎
The results from this section lead to the following definitions and facts:

We call a face of a local minima face if all its points are local minima. Since the set of faces of is closed under intersection, it follows from Corollary 7 that the set of all local minima faces of (assuming fixed and ) is closed under intersections. Thus, it is a complete meetsemilattice (but not a lattice, because it need not have the greatest element).

We call a face of an interior local minima face if all its relative interior points are interior local minima. Corollary 11 shows that the set of all interior local minima faces of (assuming fixed and ) is closed under intersections. Thus, it again is a complete meetsemilattice.
We finally define one more type of local minimum: a point is a preinterior local minimum if for some interior local minimum . Motivation for introducing this concept will become clear later.
4.2 The Effect of Iterations
Here we prove properties of sequences satisfying conditions (2) resp. (4) under various assumptions.
Theorem 13.
Let be a sequence satisfying (4) such that is an interior local minimum. Then for all we have , , and is an interior local minimum.
Proof.
Theorem 14.
Let be a sequence satisfying (4) and for all . Then for all we have , there exists such that is an interior local minimum, and is a preinterior local minimum.
Proof.
Combining with (4) yields . Thus, for every there are two possibilities:

If then, by Lemma 4, we have .
In either case, we have . Moreover, if is not an interior local minimum for some , then after some finite number of iterations the second case occurs, therefore . But this implies . If were not an interior local minimum for any , for some we would have , which is impossible.
Since for all , the faces form a nondecreasing chain. In particular, for all . Since there is such that is an interior local minimum, is a preinterior local minimum. ∎
Theorem 15.
Let be a sequence satisfying (2) such that is a preinterior local minimum, i.e., for some interior local minimum . Then for all we have and .
Proof.
We will use induction on . The claim trivially holds for . We will show that for every , implies and .
Corollary 16.
Let be a sequence satisfying (4) such that is a preinterior local minimum. Then there exists such that is an interior local minimum.
Corollary 17.
Let be a sequence satisfying (4). Then is a preinterior local minimum iff for all .
4.3 Convergence
So far, we have not examined the convergence properties of sequences satisfying (4). For that, we impose some additional restrictions on the sequences and . Namely, we assume that the action of every iteration is continuous and the elements of are visited in a regular order.
Formally, we assume that for each a continuous map is given that satisfies
(10) 
for every . This map describes the action of one iteration. Let the map
(11) 
denote the action of one round of iterations, in which all elements of are visited (some possibly more than once) in the order given by a surjective map where .
In Theorem 14, the sequence is assumed to contain every element of an infinite number of times. The form of iterations given by gives a stronger property: each element of is always visited again after at most iterations. We adapt Theorem 14 to this situation. For that, we denote (i.e., is obtained by composing with itself times) where .
Theorem 18.
Let and . Then is an interior local minimum and is a preinterior local minimum.
Proof.
By similar arguments as in the proof of Theorem 14, for every it holds that:

If is an interior local minimum, then .

If is not an interior local minimum, then , hence .
Therefore, if and were not an interior local minimum, we would have , a contradiction. Since , is a preinterior local minimum. ∎
Starting from some , we will examine convergence properties of the sequence defined by . Recall that a limit point (also known as an accumulation point or cluster point) of a sequence is the limit point of its converging subsequence.
Theorem 19.
Let . Let the sequence be bounded. Then every limit point of the sequence satisfies .
Proof.
Let us denote . Let be a limit point of the sequence , i.e., for some strictly increasing function we have
(12) 
Since is a composition of a finite number of continuous maps, it is continuous. Applying to (12) yields
(13) 
We show that
(14) 
The first and last equality holds by applying the continuous function to equality (12) and (13), respectively. The second and third equality hold because the sequence is convergent (being bounded and nonincreasing), hence every its subsequence converges to the same point. ∎
Corollary 20.
Let . Let the sequence be bounded. Then every limit point of the sequence is a preinterior local minimum.
Let be a metric on . Denote the distance of a point from a set as
(15) 
Lemma 21.
For any , the function is Lipschitz, hence continuous.
Proof.
For all and we have . Taking over on the right gives . Swapping and gives . ∎
Lemma 22.
Let be closed, bounded, and continuous. Then is bounded.
Proof.
By monotonicity of closure, . The set is compact (closed and bounded), therefore is also compact. Hence is bounded. ∎
Lemma 23.
A sequence in a metric space is convergent iff it is bounded and has a unique limit point.
Proof.
The ‘onlyif’ direction is obvious. To see the ‘if’ direction, let be a limit point of a bounded sequence . For contradiction, suppose does not converge to . Then for some , for every there is such that . So there is a subsequence such that for all . As is bounded, by BolzanoWeierstrass it has a convergent subsequence, . But clearly cannot converge to . ∎
Theorem 24.
Let be a bounded sequence from a closed set . Let be such that every limit point of is in . Then .
Proof.
By Lemmas 21 and 22, the sequence is bounded. Thus it has a convergent subsequence, where is a subsequence of . By Lemma 23, it suffices to show that .
Being a subsequence of , the sequence is bounded. Therefore, it has a convergent subsequence, . Thus, is a limit point of . Therefore, . Applying the continuous function to this limit yields . Since the sequence is convergent, every its convergent subsequence converges to the same number. Since is one such subsequence, we have . ∎
Corollary 25.
Let . Let the sequence be bounded. Let be the set of all preinterior local minima of on . Then .
For the sequence to be bounded, it clearly suffices that is bounded. But there is a weaker sufficient condition: as the sequence is nonincreasing, it suffices that the set is bounded (note that is the halfspace whose boundary is the contour of passing through the initial point ).
5 Nonlinear Objective Function
As we said, the minimization of a convex function on a convex set can be transformed to the epigraph form, which is the minimization of a linear function on a convex set. Here we show that this transformation allows us to generalize the results from §4 to nonlinear convex objective functions.
The epigraph of a function is the set
(16) 
If is closed convex and is convex, then is closed convex. We have
(17) 
where is the linear function defined by
, i.e., the projection on the coordinate. For every
we have , i.e., is the minimum
value of on . Moreover,
{IEEEeqnarray}rCr
M(X,f) ×{t} &=& M(epif,π) , \IEEEyesnumber\IEEEyessubnumber*
riM(X,f) ×{t} &=& riM(epif,π) ,
which can equivalently be written as
{IEEEeqnarray}rCrCrCr
x &∈& M(X,f) & ⟺ & (x,f(x)) &∈& M(epif,π) , \IEEEyesnumber\IEEEyessubnumber*
x &∈& riM(X,f) & ⟺ & (x,f(x)) &∈& riM(epif,π) .
The following lemma will allow us to show that the concepts of local minima and the updates (2) and (4) remain ‘the same’ if we pass to the epigraph form, provided that instead of a subspace we use the subspace . To illustrate this, consider the case and coordinate minimization. In every iteration, we minimize over a single variable . In the epigraph form, we would minimize subject to over the pair . Clearly, both forms are equivalent.
Lemma 26.
Let be convex, be convex. Let
be a subspace and . Let
and . Then
{IEEEeqnarray}rCrCrCr
y &∈& M(X∩(x+I),f) & ⟺ & (y,f(y)) &∈& M(epif∩(¯x+¯I),π) , \IEEEyesnumber\IEEEyessubnumber*
y &∈& riM(X∩(x+I),f) & ⟺ & (y,f(y)) &∈& riM(epif∩(¯x+¯I),π) .
Proof.
References
 [1] D. P. Bertsekas. Nonlinear Programming. Athena Scientific, Belmont, MA, 2nd edition, 1999.
 [2] A. Globerson and T. Jaakkola. Fixing maxproduct: Convergent message passing algorithms for MAP LPrelaxations. In Neural Information Processing Systems, pages 553–560, 2008.
 [3] J. HiriartUrruty and C. Lemaréchal. Fundamentals of Convex Analysis. Grundlehren Text Editions. Springer, 2004.
 [4] J. H. Kappes, B. Andres, F. A. Hamprecht, C. Schnörr, S. Nowozin, D. Batra, S. Kim, B. X. Kausler, T. Kröger, J. Lellmann, N. Komodakis, B. Savchynskyy, and C. Rother. A comparative study of modern inference techniques for structured discrete energy minimization problems. Intl. J. of Computer Vision, 115(2):155–184, 2015.
 [5] V. Kolmogorov. Convergent treereweighted message passing for energy minimization. IEEE Trans. Pattern Analysis and Machine Intelligence, 28(10):1568–1583, 2006.
 [6] V. Kolmogorov. A new look at reweighted message passing. IEEE Trans. on Pattern Analysis and Machine Intelligence, 37(5), May 2015.
 [7] V. A. Kovalevsky and V. K. Koval. A diffusion algorithm for decreasing the energy of the maxsum labeling problem. Glushkov Institute of Cybernetics, Kiev, USSR. Unpublished, approx. 1975.
 [8] R. T. Rockafellar. Convex analysis. Princeton Mathematical Series. Princeton University Press, 1970.
 [9] M. I. Schlesinger and K. Antoniuk. Diffusion algorithms and structural recognition optimization problems. Cybernetics and Systems Analysis, 47:175–192, 2011.
 [10] R. Szeliski, R. Zabih, D. Scharstein, O. Veksler, V. Kolmogorov, A. Agarwala, M. Tappen, and C. Rother. A comparative study of energy minimization methods for markov random fields with smoothnessbased priors. IEEE Trans. on Pattern Analysis and Machine Intelligence, 30(6):1068–1080, 2008.
 [11] P. Tseng. Convergence of a block coordinate descent method for nondifferentiable minimization. J. Optim. Theory Appl., 109(3):475–494, June 2001.
 [12] T. Werner. A linear programming approach to maxsum problem: A review. IEEE Trans. Pattern Analysis and Machine Intelligence, 29(7):1165–1179, July 2007.