Composite Kernels for Multisensor Image Cassification

Duration of the Thesis: 6 months
Completion: January 2018
Supervisors: Prof. Dr.-Ing. Uwe Sörgel, Prof. Dr.-Ing. Xiaoxiang Zhu (DLR & TUM), Dr. Pedram Ghamisi (DLR)
Examiner: Prof. Dr.-Ing. Uwe Sörgel

Recently, extinction profiles (EP) for multisensory remote sensing data classification has obtained much attention, due to the capability of spatial-spectral information extraction. In this thesis, we propose a novel framework to fuse the EPs of hyperspectral (HSI) and LiDAR data based on convolutional neural networks (CNN) and composite kernels (CK). Our framework employs CNN for high-level deep feature extraction (FE) based on the EPs from HSI and LiDAR data. Next, Composite kernels method is applied for fusion of the heterogeneous features obtained by each CNN feature extractors. As for classification step, Support vector machine (SVM) and extreme learning machine (ELM) with their composite kernels versions are used  to produce the final classification result. The proposed framework is tested over  an urban dataset Houston, USA and a rural dataset over Trento, Italy. The result further confirm that the proposed fusion framework could produce competitive results in both urban and rural area, even in a ill-posed situation, when only very limited training samples are available.

1. Extinction profiles

 In [1] the concept of EP are firstly proposed. EP are base on extinction filters (EFs) which are extrema-oriented connected independent filters. Different with the AP, in which performance of profiles are highly dependent of the kind of attribute being used (like area,volume and height), EP are naturally based on the number of extrema so could be free from the burden of setting threshold values for attributes. In order to generalize the concept of extinction profiles from gray-scale images to hyperspectral data [2], the extended extinction profiles (EEP) are proposed by applying EP on the most informative features which generated by dimension reduction approach, such as PCA and ICA.
Moreover, multi-extinction profiles (MEP) could be generated by stacking different EPs (such as: area, height, volume, diagonal of bounding box and standard deviation) together into a single feature, and extended multi-EP (EMEP) then could be formulated as follows:

2. Convolutional neural netwroks feature extraction

(CNN) have gained great research interests because their advantages of using local connections to handle spatial dependence problems and sharing weights to reduce the number of training parameters. Moreover, due to the layers construction of CNN, it also makes hierarchical feature extraction to be possible.
In general, CNN architectures contain of three parts: convolutional layers, pooling layer and nonlinear transformations [3].When we consider the CNN for classification, a fully connected logistic regression (LR) layer can be considered at the end of the network. Here in this thesis work, we carefully design three similar CNN feature extractor for spectral, spatial and elevation data.

In this thesis, we proposed a novel framework for the fusion of extinction profiles from multisensor data, which are based on convolution neural networks and composite kernels. Based on the morphological-level features, CNN based feature extractors are carefully designed to effectively extract abstract and invariant features from different EPs, respectively. Then a flexible composite kernels fusion strategy is used to fuse spectral, spatial and elevation deep features for classification purpose. Results from both two dataset confirmed that the usefulness and applicability of our fusion framework in both urban and rural area.

In the further work, we intend to investigate the detail effect of the hyperparameter of the CNN with respect the classification performance, such as using a range of windows size for training process. Besides, we would also keep to exploit the possibility of modify our proposed fusion approach to other data sources such as: Multispectral image or SAR image. A generalized composite kernel frame could also be adopted for find the best tradeoff between different data sources.

[1] P. Ghamisi, R. Souza, J. A. Benediktsson, X. X. Zhu, L. Rittner, and R. A. Lotufo, “Extinction profiles for the classification of remote sensing data,” IEEE Transactions on Geoscience and Remote Sensing, vol. 54, no. 10, pp. 5631–5645, Oct 2016.

[2] P. Ghamisi, R. Souza, J. A. Benediktsson, L. Rittner, R. Lotufo, and X. X. Zhu, “Hyperspectral data classification using extended extinction profiles,” IEEE Geoscience and Remote Sensing Letters, vol. 13, no. 11, pp. 1641–1645, Nov 2016.

[3] Y. Chen, C. L. H. Jiang, X. Jia, and P. Ghamisi, “Deep feature extraction and classification of hyperspectral images based on convolutional neural networks,” IEEE Transaction on Geoscience and Remote Sensing, vol. 54, no. 10, pp. 6232–6250, 10 2016.