|
|
| Line 1: |
Line 1: |
| '''Semidefinite embedding''' (SDE) or '''maximum variance unfolding (MVU)''' is an [[algorithm]] in [[computer science]] that uses [[semidefinite programming]] to perform [[non-linear dimensionality reduction]] of high-dimensional [[coordinate vector|vector]]ial input data. MVU can be viewed as a non-linear generalization of [[Principal component analysis]].
| | Irwin Butts is what my spouse enjoys to call me though I don't really like being known as like that. To do aerobics is a factor that I'm completely addicted to. Years ago we moved to North Dakota. Since she was eighteen she's been operating as a meter reader but she's always wanted her own business.<br><br>Feel free to surf to my site - [http://www.sportsbusinessu.com/groups/how-you-can-cure-a-painful-candida-albicans/ home std test] |
| | |
| Non-linear dimensionality reduction algorithms attempt to map high-dimensional data onto a low-dimensional [[Euclidean space|Euclidean]] [[vector space]]. Maximum variance Unfolding is a member of the [[manifold learning]] family, which also include algorithms such as [[isomap]] and [[locally linear embedding]]. In manifold learning, the input data is assumed to be sampled from a low dimensional [[manifold]] that is embedded inside of a higher dimensional vector space. The main intuition behind MVU is to exploit the local linearity of manifolds and create a mapping that preserves local neighborhoods at every point of the underlying manifold.
| |
| | |
| MVU creates a mapping from the high dimensional input vectors to some low dimensional Euclidean vector space in the following steps:
| |
| | |
| A [[neighborhood (graph theory)|neighborhood]] graph is created. Each input is connected with its k-nearest input vectors (according to Euclidean distance metric) and all k-nearest neighbors are connected with each other. If the data is sampled well enough, the resulting graph is a discrete approximation of the underlying manifold.
| |
| | |
| The neighborhood graph is "unfolded" with the help of semidefinite programming. Instead of learning the output vectors directly, the semidefinite programming aims to find an inner product matrix that maximizes the pairwise distances between any two inputs that are not connected in the neighborhood graph while preserving the nearest neighbors distances.
| |
| | |
| The low-dimensional embedding is finally obtained by application of [[multidimensional scaling]] on the learned inner product matrix.
| |
| | |
| The steps of applying semidefinite programming followed by a linear dimensionality reduction step to recover a low-dimensional embedding into a Euclidean space were first proposed by Linial, London, and Rabinovich.
| |
| | |
| ==Optimization Formulation==
| |
| | |
| Let <math>X \,\!</math> be the original input and <math>Y\,\!</math> be the embedding. If <math>i,j\,\!</math> are two neighbors, then the local isometry constraint that needs to be satisfied is:
| |
| | |
| :<math>|X_{i}-X_{j}|^{2}=|Y_{i}-Y_{j}|^{2}\,\!</math>
| |
| | |
| Let <math>G, K\,\!</math> be the Gram matrices of <math> X \,\!</math> and <math> Y \,\!</math> (i.e.: <math>G_{ij}=X_i \cdot X_j,K_{ij}=Y_i \cdot Y_j \,\!</math>). We can express the above constraint for every neighbor points <math>i,j\,\!</math> in term of <math>G, K\,\!</math>:
| |
| | |
| :<math>G_{ii}+G_{jj}-G_{ij}-G_{ji}=K_{ii}+K_{jj}-K_{ij}-K_{ji}\,\!</math>
| |
| | |
| In addition, we also want to constraint the embedding <math> Y \,\!</math> to center at the origin:
| |
| | |
| <math>\sum_{i}Y_{i}=0\Rightarrow(\sum_{i}Y_{i})^{2}=0\Rightarrow\sum_{i,j}Y_{i}Y_{j}=0\Rightarrow\sum_{i,j}K_{ij}=0</math>
| |
| | |
| As described above, except the distances of neighbor points are preserved, the algorithm aims to maximize the pairwise distance of every pair of points. The objective function to be maximized is:
| |
| | |
| <math>T(Y)=\dfrac{1}{2N}\sum_{i,j}|Y_{i}-Y_{j}|^{2}</math>
| |
| | |
| Intuitively, maximizing the function above is equivalent to pulling the points as far away from each other as possible and therefore "unfold" the manifold. The local isometry constraint prevents the objective function from going to infinity. Proof:
| |
| | |
| Let <math>\tau = max \{\eta_{ij}|Y_{i}-Y_{j}|^2\} \,\!</math> where <math> \eta_{ij} = 1 \,\!</math> if i and j are neighbors and <math> \eta_{ij} = 0 \,\!</math> otherwise.
| |
| | |
| Since the graph has N points, the distance between any two points <math>|Y_{i}-Y_{j}|^2 \leq N \tau \,\!</math>. We can then bound the objective function as follow:
| |
| | |
| :<math>T(Y)=\dfrac{1}{2N}\sum_{i,j}|Y_{i}-Y_{j}|^{2} \leq \dfrac{1}{2N}\sum_{i,j}(N\tau)^2 = \dfrac{N^3\tau^2}{2} \,\!</math>
| |
| | |
| The objective function can be rewritten purely in the form of the Gram matrix:
| |
| | |
| :<math>
| |
| \begin{align}
| |
| T(Y) &{}= \dfrac{1}{2N}\sum_{i,j}|Y_{i}-Y_{j}|^{2} \\
| |
| &{}= \dfrac{1}{2N}\sum_{i,j}(Y_{i}^2+Y_{j}^2-Y_{i} \cdot Y_{j} - Y_{j} \cdot Y_{i})\\
| |
| &{}= \dfrac{1}{2N}(\sum_{i,j}Y_{i}^2+\sum_{i,j}Y_{j}^2-\sum_{i,j}Y_{i} \cdot Y_{j} -\sum_{i,j}Y_{j} \cdot Y_{i})\\
| |
| &{}= \dfrac{1}{2N}(\sum_{i,j}Y_{i}^2+\sum_{i,j}Y_{j}^2-0 -0)\\
| |
| &{}= \dfrac{1}{N}(\sum_{i}Y_{i}^2)=\dfrac{1}{N}(Tr(K))\\
| |
| \end{align}
| |
| \,\!</math>
| |
| | |
| Finally, the optimization can be formulated as:
| |
| | |
| '''Maximize''' <math> Tr(K) \,\!</math>
| |
| | |
| '''Subject to''' <math> K \succeq 0\,\!</math> and
| |
| <math> \forall i,j \,\!</math> where <math> \eta_{ij} =1, G_{ii}+G_{jj}-G_{ij}-G_{ji}=K_{ii}+K_{jj}-K_{ij}-K_{ji} \,\!</math>
| |
| | |
| After the Gram matrix <math>K \,\!</math> is learned by semidefinite programming, the output <math>Y \,\!</math> can be obtained via [[Cholesky decomposition]]. In particular, the Gram matrix can be written as <math> K_{ij}=\sum_{\alpha = 1}^{N}(\lambda_{\alpha } V_{\alpha i} V_{\alpha j}) \,\!</math> where <math> V_{\alpha i} \,\!</math> is the i-th element of eigenvector <math> V_{\alpha} \,\!</math> of the eigenvalue <math> \lambda_{\alpha } \,\!</math>.
| |
| | |
| It follows that the <math> \alpha \,\!</math>-th element of the output <math> Y_i \,\!</math> is <math> \sqrt{\lambda_{\alpha }} V_{\alpha i} \,\!</math>.
| |
| | |
| ==Comparison to other methods==
| |
| Semidefinite embedding is much better in revealing the underlying dimension of the data compared to LLE and Laplacian eigenmaps.{{Citation needed|reason=empirical or theoretical proof needed|date=December 2012}} It also guarantees that the nearest neighbors in the embedding is the same as the original nearest neighbor for each point while the other two methods do not. On the other hand, semidefinite embedding is much slower and harder to scale to large data.
| |
| | |
| Semidefinite embedding outperforms Isomap when the manifold is not a convex subset of the Euclidean space.
| |
| | |
| ==See also==
| |
| * [[Locally linear embedding]]
| |
| | |
| ==References==
| |
| *[http://repository.upenn.edu/cgi/viewcontent.cgi?article=1000&context=cis_papers Unsupervised learning of image manifolds by semidefinite programming] K. Q. Weinberger and L. K. Saul (2004). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR-04), Washington D.C.
| |
| *[http://www.springerlink.com/content/t21q747q278qx4x1/ Unsupervised learning of image manifolds by semidefinite programming] K. Q. Weinberger and L. K. Saul (2005), International Journal of Computer Vision - In Special Issue: Computer Vision and Pattern Recognition-CVPR 2005 Guest Editor(s): [[Aaron Bobick]], [[Rama Chellappa]], [[Larry Davis]], pages 77–90, Volume 70, Number 1, [[Springer Netherlands]]
| |
| *[http://citeseer.ist.psu.edu/170127.html The geometry of graphs and some of its algorithmic applications], [[Nathan Linial]], [[Eran London]], [[Yuri Rabinovich]], [[IEEE]] Symposium on Foundations of Computer Science.
| |
| | |
| ==External links==
| |
| *[http://www.cse.wustl.edu/~kilian/code/code.html MVU Matlab code online]
| |
| | |
| [[Category:Computational statistics]]
| |
| [[Category:Dimension reduction]]
| |
Irwin Butts is what my spouse enjoys to call me though I don't really like being known as like that. To do aerobics is a factor that I'm completely addicted to. Years ago we moved to North Dakota. Since she was eighteen she's been operating as a meter reader but she's always wanted her own business.
Feel free to surf to my site - home std test