Multilinear Principle Component Analysis

In previous blog (PCA), one dimension latent space was introduced. However, in practice, a latent space with more dimensions is more popular. To do this, a set of projections $\{w_1,w_2,...,w_n\}$ is needed. Assume that the latent space is $\{y_1,y_2,...,y_n\}, y_i\in R^d$ and original data set is $\{x_1,x_2,...,x_n\},x_i\in R^F$, then:
$$y_i=\begin{bmatrix}            y_{i1}\\            ...\\           y_{id}        \end{bmatrix}      =\begin{bmatrix}           w_1^Tx_i\\           ...\\          w_d^Tx_i        \end{bmatrix}      =W^Tx_i$$
where
$$W=[w_1,w_2,...,w_d]$$
Multilinear Principle Component Analysis could be done via maximising the variance in each dimension:
$$\begin{align*} W_o&=\underset{W}{\arg\max}\quad\frac{1}{N}\sum_{k=1}^d\sum_{i=1}^N(y_{ik}-u_{ik})^2\\ &=\underset{W}{\arg\max}\quad\frac{1}{N}\sum_{k=1}^d\sum_{i=1}^Nw_k^T(x_i-u_i)(x_i-u_i)^Tw_k^T\\ &=\underset{W}{\arg\max}\quad\sum_{k=1}^dw_k^TS_tw_k\\ &=\underset{W}{\arg\max}\quad tr[W^TS_tW]\\ \end{align*}$$

$$s.t.    W^TW=I$$

Formulate the Lagrangian：
$$\begin{align*} \mathcal{L}(W,\Lambda)&=tr[W^TS_tW] -tr[\Lambda(W^TW-I)]\\ \frac{\partial \mathcal{L}(W,\Lambda)}{\partial W}&=2S_tW - 2W\Lambda\\ \end{align*}$$

Let  $\frac{\partial \mathcal{L}(W,\Lambda)}{\partial W}=0$
$$\Rightarrow\quad S_tW=W\Lambda$$

Take the above formula back to the original optimisation equation:
$$\begin{align*} W_o&=\underset{W}{\arg\max}\quad tr[W^TS_tW]\\ &=\underset{W}{\arg\max}\quad tr[W^TW\Lambda]\\ &=\underset{W}{\arg\max}\quad tr[\Lambda]\\ \end{align*}$$
Therefore, one needs to maximise the eigenvalues of $S_t$, and the projection matrix $W=\{w_1,w_2,...,w_d\}$ corresponds to the eigenvectors of the largest $d$ eigenvalues.

Done.