Affine matrix rank minimization problem via p-thresholding function

# Affine matrix rank minimization problem via p-thresholding function

Angang Cui Corresponding author
Jigen Peng 22email: jgpengxjtu@126.com
1 School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, 710049, China
2 School of Science, Xi’an Polytechnic University, Xi’an, 710048, China
Jigen Peng Corresponding author
Jigen Peng 22email: jgpengxjtu@126.com
1 School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, 710049, China
2 School of Science, Xi’an Polytechnic University, Xi’an, 710048, China
Haiyang Li Corresponding author
Jigen Peng 22email: jgpengxjtu@126.com
1 School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, 710049, China
2 School of Science, Xi’an Polytechnic University, Xi’an, 710048, China
Qian Zhang Corresponding author
Jigen Peng 22email: jgpengxjtu@126.com
1 School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, 710049, China
2 School of Science, Xi’an Polytechnic University, Xi’an, 710048, China
###### Abstract

To pursuit a much more efficient algorithm, the -thresholding function is taken to solve affine matrix rank minimization problem. Numerical experiments on image inpainting problems show that our algorithm performs powerful in finding a low-rank matrix comparing with some state-of-art methods.

###### Keywords:
Affine matrix rank minimization problem-thresholding functionGeneralized thresholding algorithm
65K1090C2690C59

## 1 Introduction

In this paper, we study affine matrix rank minimization problem

 (ARMP)      minX∈Rm×n rank(X)  s.t.  A(X)=b (1)

where the linear map and the vector are given. ARMP has attracted much attention in many application such as collaborative filtering in recommender systems, minimum order system and low-dimensional Euclidean embedding in control theory and so on (see e.g., candes1 (),jannach2 (),faze3 (),faze4 (),ji5 ()). It is a challenging non-convex optimization problem and is known as NP-hard recht6 ().

Nuclear-norm is the most popular alternative (see e.g.,candes1 (),faze4 (),recht6 (),candes7 (),faze8 (),candes9 ()), and the minimization problem has the following form

 (NuARMP)      minX∈Rm×n ∥X∥∗  s.t.  A(X)=b (2)

for the constrained problem and

 (RNuARMP)      minX∈Rm×n{12∥A(X)−b∥22+λ∥X∥∗} (3)

for the regularization problem, where is the regularization parameter, is nuclear-norm of matrix , and presents the -th largest singular value of matrix arranged in descending order.

As compact convex relaxation, NuARMP may possesses many theoretical and algorithmic advantages (see e.g., liu10 (),cai11 (),toh12 (),ma13 ()). However, it may be suboptimal for recovering a real low-rank matrix and yield a matrix with much higher rank and need more observations to recover a real low-rank matrix (see e.g., candes1 (),cai11 ()). Moreover, RNuARMP tends to lead to biased estimation by shrinking all the singular values toward to zero simultaneously, and sometimes results in over-penalization as the -norm in compressed sensing daubechies14 ().

This brings our attention to the non-convex functions. We substitute rank function by a continuous promoting low-rank non-convex function. Through this transformation, ARMP can be translated into a transformed ARMP (TrARMP) which has the following form

 (TrARMP)          minX∈Rm×n P(X)  s.t.  A(X)=b (4)

for the constrained problem and

 (RTrARMP)      minX∈Rm×n{∥A(X)−b∥22+λP(X)} (5)

for the regularization problem, where the continuous promoting low rank non-convex function is in terms of singular values of matrix , e.g.,

 P(X)=m∑i=1ρ(σi(X)). (6)

In li15 (), we take

 P(X)=Pa(X)=m∑i=1ρa(σi(X)), (7)

where

 ρa(t)=a|t|a|t|+1 (8)

is the fraction function.

With the change of parameter , we have

 lima→+∞ρa(t)={0,  if t=0;1,  if t≠0. (9)

So, the non-convex function interpolates the rank of matrix

 lima→+∞Pa(X)=lima→+∞∑σi(X)>0ρa(σi(X))=rank(X). (10)

By this transformation, the minimization problem (ARMP) can be turned into the following transformed form

 minX∈Rm×n Pa(X)  s.t.  A(X)=b (11)

for the constrained problem and

 minX∈Rm×n{12∥A(X)−b∥22+λPa(X)} (12)

for the regularization problem.

In li15 (), the iterative singular value thresholding algorithm (ISVTA) is proposed to solve the minimization problem (12). Numerical experiments on completion of low-rank random matrices show that ISVTA performs powerful in finding a low-rank matrix comparing with some state-of-art methods. However, the thresholding function for non-convex fraction function are too complicated to computing, and convergent slowly.

In order to pursuit a much more efficient algorithm, the -thresholding function is used to solve the minimization problem (ARMP) for all , and the -thresholding function voro16 () can be defined as

 rλ,p(t))=sign(t)max{0,|t|−λ|t|p−1}. (13)

In sparse information processing, the -thresholding function performs better in numerical examples than some state-of-art methods. When we take , the thresholding function is equivalent to the classical soft thresholding. For values of below 1, the thresholding penalizes small coefficients over a wider range and applies less bias to the larger coefficients, much like the hard thresholding but without discontinuities (see e.g., voro16 ()).

Moreover, in char17 (), the authors demonstrated that the -thresholding function is the proximal mapping of a non-convex penalty function with several desirable properties.

As representation (6), we take

 P(X)=Fp(X)=m∑i=1fp(σi(X)). (14)

Then, TrARMP could be transformed into the following minimization problem

 (RFTrARMP)      minX∈Rm×n{12∥A(X)−b∥22+λFa(X)}, (15)

and the iterative -thresholding algorithm is proposed to solve the minimization problem (RFTrARMP) for all .

The rest of this paper is organized as follows. In Section 2, the iterative -thresholding algorithm is proposed to solve the minimization problem (RFTrARMP). In section 3, the convergence of the iterative -thresholding algorithm is established. In Section 4, we demonstrate some numerical experiments on some image inpainting problems. Some conclusion remarks are presented in Section 5.

## 2 p-thresholding function for solving (RFTrARMP)

In this section, based on the -thresholding function, the iterative -thresholding algorithm is proposed to solve the minimization problem (RFTrARMP) for all .

### 2.1 Iterative p-thresholding algorithm for solving (RFTrARMP)

Based on the -thresholding function, we now briefly derive the closed form representation of the optimal solution to the minimization problem (RFTrARMP) for all , which underlies the iterative -thresholding algorithm to be proposed. Before computing the generalized thresholding algorithm, we need some results which plays a key role in our later analysis.

###### Definition 1

The -thresholding operator is a diagonally nonlinear analytically expressive operator, and can be specified by

 Rλ,p(x)=Diag(rλ,p(x1),⋯,rλ,p(xn)) (16)

where the -thresholding function is defined in (13) for all .

###### Theorem 2.1

char17 () Suppose is continuous, satisfies for , is strictly increasing on , and . Then the threshold operator is the proximal mapping of the penalty function where is even, strictly increasing and continuous on , differentiable on , and non-differentiable at 0 if and only if (in which case ). If is non-increasing on , then is concave on and satisfies the triangle inequality.

Theorem 2.1 show that the -thresholding function is the proximal mapping of a non-convex penalty function with several desirable properties.

###### Lemma 1

(von Neumann’s trace inequality) For any matrices ,

 Tr(XTY)≤m∑i=1σi(X)σi(Y),

where

 σ(X):σ1(X)≥σ2(X)≥⋯≥σm(x)≥0

and

 σ(Y):σ1(Y)≥σ2(Y)≥⋯≥σm(Y)≥0

are the singular value of matrices and respectively. The equality holds if and only if there exist unitary matrices and that such and as the singular value decompositions of the matrices and simultaneously.

Define a function of the matrix as

 T(Y)=∥X−Y∥2F+λFp(X) (17)

and

 X∗=argminX∈Rm×n{∥X−Y∥2F+λFp(X)}. (18)
###### Theorem 2.2

Let be the singular value decomposition of matrix . Then the optimal matrix can be expressed as

 X∗=Rλ,p(Y)=URλ,p(σ(Y))VT=UDiag(rλ,p(σi(Y)))VT. (19)
###### Proof

Since are the singular values of matrix , the minimization problem

 (20)

can be rewritten as

 minτ:τ1≥τ2≥⋯≥τm≥0{minσ(X)=τ{∥X−Y∥2F+λm∑i=1fp(τi)}}.

By Lemma 1, we have

 ∥X−Y∥2F = Tr(XTX)−2Tr(XTY)+Tr(YTY) = m∑i=1σ2i(X)−2Tr(XTY)+m∑i=1σ2i(Y) ≥ m∑i=1σ2i(X)−2m∑i=1σi(X)σi(Y)+m∑i=1σi(Y) = m∑i=1(σi(X)−σi(Y))2.

Notice that above equality holds when admits the singular value decomposition

 X=UDiag(σ(X))VT=UDiag(σ1(X),σ2(X),⋯,σm(X))VT

where and are the left and right orthonormal matrices in the singular value decomposition of matrix .

In this case, the optimization problem (20) reduces to

 minσ(X)m∑i=1{(σi(X)−σi(Y))2+λfp(σi(X))}, (21)

which is consistent with Theorem (2.1). This completes the proof.

Nextly, we will show that the optimal solution to minimization problem (RFTrARMP) can be expressed as -thresholding function.

For any fixed , and , let

 C1(X)=12∥A(X)−b∥22+λFp(X) (22)

and its surrogate function

 C2(X,Y)=μ[C1(X)−∥A(X)−A(Y)∥22]+∥X−Y∥2F. (23)

Clearly, .

###### Theorem 2.3

For any fixed , and matrix , then is equivalent to

 minX∈Rm×n{∥X−Bμ(Y)∥2F+λμFp(X)}

where .

###### Proof

In accordance with the definition, can be rewritten as

 C2(X,Y) = ∥X−(Z−μA∗A(Y)+μA∗(b))∥2F+λμFp(X)+μ∥b∥22 +∥Y∥2F−μ∥A(Y)∥22−∥Y−μA∗A(Y)+μA∗(b)∥2F = ∥X−Bμ(Y)∥2F+λμFp(X)+μ∥b∥22+∥Y∥2F−μ∥A(Y)∥22 −∥Bμ(Y)∥2F,

which implies that for any fixed , and matrix is equivalent to

 minX∈Rm×n{∥X−Bμ(Y)∥2F+λμFp(X)}.

This completes the proof.

Combing Theorem 2.1, Theorem 2.3 and the proof of Theorem 2.2, we can immediately conclude the corollary as following:

###### Corollary 1

Let matrix be the optimal solution of , it can be expressed as

 X∗=Rλ,p(Bμ(X∗))=U∗Rλμ,p(ΣBμ(X∗))(V∗)T (24)

where is the singular value decomposition of matrix , and and are the corresponding left and right orthonormal matrices.

Moreover, if we take the parameter properly, we have

###### Theorem 2.4

For any fixed and . If is the optimal solution of , then is also the optimal solution of , that is

 C2(X∗,X∗)≤C2(X,X∗)

for any .

###### Proof

By definition of , we have

 C2(X,X∗) = μ[C1(X)−∥A(X)−A(X∗)∥22]+∥X−X∗∥2F = μ[∥A(X)−b∥22+λFp(X)]+∥X−X∗∥2F −μ∥A(X)−A(X∗)∥22 ≥ μ[∥A(X)−b∥22+λFp(X)] = μC1(X) ≥ μC1(X∗) = C2(X∗,X∗),

where the first inequality holds by the fact that

 ∥A(X)−A(X∗)∥22 = ∥Avec(X)−Avec(X∗)∥22 ≤ ∥A∥22⋅∥vec(X)−vec(X∗)∥22 ≤ ∥A∥22⋅∥X−X∗∥2F.

This completes the proof.

With the representation (24), the iterative -thresholding algorithm for minimization problem (RFTrARMP) can be naturally defined as

 Xk+1=Rλμ,P(B(Xk)),    k=0,1,⋯ (25)

where .

### 2.2 The choice of regularization parameter λ>0

It is well known that the quantity of the solution of a regularization problem depends seriously on the setting of the regularization parameter , and the selection of the proper regularization parameters is a very hard problem. In this paper, the cross-validation method is accepted to choose the regularization parameter .

To make it clear, we suppose that the matrix of rank is the optimal solution to the minimization problem (RFTrARMP), and the singular values of matrix are denoted as

 σ1(Bμ(X∗))≥σ2(Bμ(X∗))≥⋯≥σm(Bμ(X∗)).

By equality (15), the following inequalities hold

 σi(Bμ(X∗))>(λμ)12−p⇔i∈{1,2,⋯,r},
 σi(Bμ(X∗))≤(λμ)12−p⇔i∈{r+1,r+2,⋯,m}

which implies

 (σr+1(Bμ(X∗)))2−pμ≤λ<(σr(Bμ(X∗)))2−pμ. (26)

In practice, we approximate by in (26), and a choice of is

 λk∈[(σr+1(Bμ(Xk)))2−pμ,(σr(Bμ(Xk)))2−pμ) (27)

in applications.

When doing so, the ISVTA will be adaptive and free from the choice of regularization parameter .

## 3 Convergence of iterative p-thresholding algorithm

In this section, the convergence of iterative -thresholding algorithm is established under some certain conditions.

###### Theorem 3.1

Let be the sequence generated by Algorithm 1 with the step size satisfying . Then the sequence is decreasing.

###### Proof

By the proof of Theorem 2.4, we have

 C2(Xk+1,Xk)=minX∈Rm×nC2(X,Xk). (28)

Moreover, according to the definition of , we have

 C1(Xk+1)=1μ[C2(Xk+1,Xk)−∥Xk+1−Xk∥2F]+∥A(Xk+1)−A(Xk)∥22. (29)

Since , we can get that

 C1(Xk+1)=1μ[C2(Xk+1,Xk)−∥Xk+1−Xk∥2F]+∥A(Xk+1)−A(Xk)∥22≤1μ[C2(Xk,Xk)−∥Xk+1−Xk∥2F]+∥A(Xk+1)−A(Xk)∥22=C1(Xk)−1μ∥Xk+1−Xk∥2F+∥A(Xk+1)−A(Xk)∥22≤C1(Xk). (30)

That is, the sequence is a minimization sequence of function , and

 C1(Xk+1)≤C1(Xk)

for all . This completes the proof.

###### Theorem 3.2

Let be the sequence generated by Algorithm 1 and . Then the sequence is asymptotically regular, i.e.,

 limk→∞∥Xk+1−Xk∥2F=0.
###### Proof

Let . Then and

 μ∥A(Xk+1−Xk)∥22≤(1−θ)∥Xk+1−Xk∥2F. (31)

By (30), we have

 1μ∥Xk+1−Xk∥2F−∥A(Xk+1)−A(Xk)∥22≤Cλ(Xk)−Cλ(Xk+1). (32)

Combing (31) and (32), we get

 N∑k=1{∥Xk+1−Xk∥2F} ≤ 1θN∑k=1{∥Xk+1−Xk∥2F} −1θN∑k=1{μ∥A(Xk+1)−A(Xk)∥22} ≤ μθN∑k=1{Cλ(Xk)−Cλ(Xk+1)} = μθ(C1(X1)−C1(XN+1)) ≤ μθC1(X1).

Thus, the series is convergent, which implies that

 ∥Xk+1−Xk∥2F→0  as  k→∞.

This completes the proof.

###### Theorem 3.3

Let be the sequence generated by Algorithm 1 and . Then the sequence converges to a stationary point of the iteration (25).

###### Proof

Denote

 Tλ,μ(Z,X)=∥Z−Bμ(X)∥2F+λμP(Z)

and let

 Dλ,μ(X)=Tλ,μ(X,X)−minZ∈Rm×nTλ,μ(Z,X).

Then

 Dλ,μ(X)≥0

and by (24), we have

 Dλ,μ(X)=0 if and only if X=Rλμ,p(Bμ(X)).

Assume that is a limit point of , then there exists a subsequence of , which is denoted as such that as . Since the iterative scheme

 Xkj+1=Rλμ,p(Bμ(Xkj)),

we have

 Dλ,μ(Xkj) = Tλ,μ(Xkj,Xkj)−Tλ,μ(Xkj+1,Xkj) = λμ(Fp(Xkj)−Fp(Xkj+1))−∥Xkj+1−Xkj∥2F +2⟨μA∗(b−A(Xkj)),Xkj+1−Xkj⟩,

which implies that

 λFp(Xkj)−λFp(Xkj+1)=1μ∥Xkj+1−Xkj∥2F+1μDλ,μ(Xkj)−2⟨A∗(b−A(Xkj)),Xkj+1−Xkj⟩. (33)

By (33), it follows that

 C1(Xkj)−C1(Xkj+1) = ∥A(Xkj)−b∥22+λFp(Xkj)−∥A(Xkj+1)−b∥22 −λFp(Xkj+1)) = 1μDλ,μ(Xkj)+1μ∥Xkj+1−Xkj∥2F −∥A(Xkj−Xkj+1)∥22 ≥

Since , we get

 Dλ,μ(Xkj)≤μ(C1(Xkj)−C1(Xkj+1)).

Combining the following fact that

 C1(Xkj)−C1(Xkj+1)→0  as j→∞,

we have

 Dλ,μ(X∗)=0.

This implies that the limit point of the sequence satisfies the equation

 X∗=Rλμ,p(Bμ(X∗)).

This completes the proof.

## 4 Numerical experiments

In the section, we carry out a series of simulations to demonstrate the performance of iterative -thresholding algorithm on image inpainting problems. We first present numerical results of iterative -thresholding algorithm for image inpainting problems, and then compare it with some other methods (singular value thresholding algorithm (SVTA) cai18 () and iterative singular value thresholding algorithmin an19 ()).

We denote the following quantities and they help to quantify the difficulty of the low rank matrix recovery problems

 Sampling ratio:SR=s/mn

where is the cardinality of observation set whose entries is sampled randomly.

 Freedom ration:FR=s/r(m+n−r)

is the ratio between the number of sampled entries and the ’true dimensionality’ of a matrix of rank , and it is a good quantity as the information oversampling ratio.

The stopping criterion is usually as following

 ∥Xk−Xk−1∥F∥Xk∥F≤Tol

where and are numerical results from two continuous iterative steps and is a given small number. In addition, we measure the accuracy of the generated solution of our algorithms by the relative error () defined as following

 RE=∥Xopt−M∥F∥M∥F.

In all of the experiments, we set and .

Table 1, 2 report the numerical results of iterative -thresholding algorithm for image inpainting problems with respectively. The numerical results show that is the best strategy for image inpainting problems.

The numerical results of iterative -thresholding algorithm, ISVTA and SVTA compared in Table 3, 4 under the same circumstances show that the iterative -thresholding algorithm performs much better than ISTA and SVTA on image inpainting problems for .

## 5 Conclusions

In the paper, the -thresholding function is taken to solve affine matrix rank minimization problem. Numerical experiments on image inpainting problems show that our algorithm performs powerful in finding a low-rank matrix comparing with other methods.

###### Acknowledgements.
The work was supported by the National Natural Science Foundations of China (11131006, 11271297) and the Science Foundations of Shaanxi Province of China (2015JM1012).

## References

• (1) E. J. Candès, B. Recht, Exact matrix completion via convex optimization. Foundations of Computational Mathematics, 9, 717-772 (2009)
• (2) D. Jannach, M. Zanker, A. Felfernig and G. Friedrich, Recommender Systerm: An Introduction. Cambridge university press, New York (2012)
• (3) M. Fazel, H. Hindi and S. Boyd, A rank minimization heuristic with application to minimum order system approximation. In proceedings of American Control Conference, Arlington, VA, 6, 4734-4739 (2001)
• (4) M. Fazel, H. Hindi and S. Boyd, Log-det heuristic for matrix minimization with applications to Hankel and Euclidean distance matrices. In Proceedings of American Control Conference, Denever, Colorado, 3, 2156-2162 (2003)
• (5) S. Ji, K. F. Sze and Z. Zhou, Beyond Convex Relaxation: A polynomial-time nonconvex optimization approach to network localization. INFOCOM, 2013 Proceedings IEEE, 12, 2499-2507 (2013)
• (6) B. Recht, M. Fazel and P. A. Parrilo, Guaranteed minimum-rank solution of linear matrix equations via nuclear norm minimization. SIAM Review, 52, 471-501 (2010)
• (7) E. J. Candès, T. Tao, The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory, 56, 2053-2080 (2010)
• (8) M. Fazel, Matrix Rank Minimization with Applications. PhD thesis, Stanford University (2002)
• (9) E. J. Candès, Y. Plan, Matrix completion with noise. Proceedings of the IEEE, 98, 925-936 (2010)
• (10) Y. Liu, D. Sun and K. C. Toh, An implementable proximal point algorithmic framewprk for nuclear norm minimization. Mathematical Programming, 133, 399-436 (2012)
• (11) J. Cai, E. J. candès and Z. W. Shen, A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization, 20, 1956-1982 (2010)
• (12) K. C. Toh, S. Yun, An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pacific Journal of Optimization, 6, 615-640 (2012)
• (13) S. Ma, D. Goldfarb and L. Chen, Fixed point and Bregman iterative methods for matrix rank minimization. Mathematical Programming, 128, 321-353 (2011)
• (14) I. Daubechies, M. Defrise and D. M. Christine, An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics, 57(11), 1413-1457 (2004)
• (15) H. Li, Q. Zhang, A. Cui and J. Peng. Minimization of fraction function penalty in compressed sensing. arXiv:1705.06048v1 [math.OC] 17 May 2017
• (16) S. Voronin, R. Chartrand. A new generalized thresholding algorithm for inverse problems with sparsity constraints. IEEE International Conference on Acoustics, Speech and Signal Processing Acoustics, Speech and Signal Processing (ICASSP), 1636-1640, 2013
• (17) R. Chartrand. Shrinkage mappings and their induced penalty functions. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1026-1029, 2014
• (18) J. Cai, E. J. candès and Z. W. Shen, A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization, 20, 1956-1982 (2010)
• (19) A. Cui, J. Peng, H. Li, C. Zhang and Y. Yu. Affine matrix rank minimization problem via non-convex fraction function penalty. arXiv:1611.07777v3 [math.OC] 30 Apr 2017
You are adding the first comment!
How to quickly get a good reply:
• Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
• Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
• Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters