Adaptive PCA and PSA

Włodzimierz Kasprzak. My Home

Adaptive PCA and PSA algorithms

ANN for Principal Component Analysis

Motivation

Despite efforts, up to now no neural network based method is able to compete with standard numerical approaches for PCA

PCA is a biologically inspired method for image compression and it builds the framework for autoassociative memory

The following image compression-reconstruction scheme may be applied for images (Fig.1):

Fig. 1 Successive compression and reconstruction steps on the basis of 1,2,...,16 principal components

Example:

after 1-st principal component

after 3-th principal component

after 4-th PC, after 5-th PC

Combination of first few PC's and intermediate range PC's are valuable features for image classification and categorization

The work of Prof. Cottrell and his co-workers (UCLA in San Diego) and recent work of Valentine, Abdi document the importance of different range principal components for classification of face images according to sex and view direction (pose).

The CRLS PCA method

We have developed a new fast learning algorithm for neural network based PCA, called CRLS (cascade recurrent least square) PCA

Consider our important paper published in "Cybernetics and Systems", 1996.

The main features of this learning algorithm are: sequential PC extraction with a Hebbian like learning rule, deflation of input signals, an RLS scheme for automatic adaptation of learning rate.

Deflation (sequential signal reduction):

Fig. 2 Cascade neural network (two layers are shown) realizing the signal reduction (deflation) idea.

Fig. 3 A block scheme of a possible hardware realization of the learning algorithm (a single unit is shown).

We found it impossible, without signal deflation properly to extract the intermediate and lower range PC's by other neural PCA algorithms (as well as by the neural PSA algorithm). On the drawing in Figure 4 the quality of image compression-reconstruction by PC-s learned with signal deflation (left drawing) and without it (right drawing) are shown.

Fig. 4

This difference is usually only quantitatively measurable but may not be visually detectable. In the two-image animation

MPEG (1.66 MB), which accompanies the table above (Fig. 4) it may not be possible visually to detect any difference between the left and right sequence, as the intermediate and lower range PC's capture relatively small signal energies only and the wrongly learned weight vectors are of short length.
Adjust your mpeg_player to high quality (e.g. set option -dither color if this is possible on your display).
RLS (recurrent least squares) adaptation of the learning rate:
The learning rate in every cascade of the neural network is automatically initialized and adaptively updated according to the energy of input signal and a RLS (recurrent least squares) scheme. This RLS scheme for learning rate adaptation allows a fast learning, usually in one epoch of the image data. This means, that no iteration of the input sample may be required (on--line learning) if the relation of one data sample size (image window) to complete image size is small enough (e.g. a 64-pixel window moving over an 512x512-pixel image). This number depends on the homogenity characteristics of every single image.
The number of learning samples itself can not be shortened without limitation. In the movie below results of image compression-reconstruction are shown for two cases: one-epoch learning on full image data of every single principal component (m epochs for m PC's) (left image) and learning in half epoch (half image data, no iteration) of every PC. This time we have additionally normalized the learned weight vectors to 1 (as they should be after a complete learning convergence) in order better to demonstrate the visual quality of learned PC's. The addition of badly approximated intermediate and lower range principal components leads to degradation of the reconstructed image.

Fig. 5

(MPEG, 2.04 MB)
Generalization power of extracted principal components:
The advantage of principal components is their generalization strength. PC-s that have been learned on one image from some class (e.g. natural images) can be applied very well for compression-reconstruction of other signals from the same class.
Take a look on the 4-image animation, a MPEG movie (375 kB), below

or grab its longer version (MPEG, 644 kB) (2 frames per principal component). The results for four images are shown, on the basis of the first 1,2,...,20 principal components (total number of PC's was 64), that have been learned on the leftmost image Lenna only.
Compare the sequence with the original 4 images given below as a high quality
JPEG image (203 kB).
Many image learning process (e.g. nonstationary signal case, generalization of some image class)
If the learning process is continued on new images the learning rate should be initialized again for every new image.

Thus CRLS PCA is a simple neural network learning method that outperforms other similar neural network methods in the speed-to-quality factor, and that properly extracts all components (not only the highest range but the intermediate range and minor components also). Although for experimental validity of the method an image compression-reconstruction cycle was usually used, an image compression application was definitely NOT our motivation, as there are already very well established compression standards. Our main motivation was to derive a biologically justified (i.e. a learning method) which is able to compete with well known numerical approaches to PCA.

RLS learning rate in matrix form for PSA and ordered PCA methods

We also have introduced a matrix form of the learning rate and its adaptation according to the RLS scheme for the PSA (principal subspace analysis) and ordered PCA (e.g. the Brockett algorithm). We obtained fast learning algorithms, that are able to learn an approximation of highest range components or subspaces with reasonable quality.

See our paper published at ESANN 1996.

As these methods are assumed to work in parallel manner, the signal deflation idea can not be applied straightforward to them. But without deflation these algorithms are not able properly to extract the whole range of PC's or PS's, and thus they are not competitive with our CRLS PCA method.

[go back to top],

W. Kasprzak
Apr. 1996