Systematic benchmarking of microarray data classification: assessing the role of nonlinearity and dimensionality reduction.

Systematic benchmarking of microarray data classification:
assessing the role of nonlinearity and dimensionality reduction.

Nathalie Pochet, Frank De Smet, Johan A.K. Suykens and Bart L.R. De Moor

K.U.Leuven, ESAT-SCD
Kasteelpark Arenberg 10
B-3001 Leuven (Heverlee), Belgium
Email: {nathalie.pochet,frank.desmet,johan.suykens,bart.demoor}@esat.kuleuven.ac.be

Abstract

Motivation: Microarrays are capable of determining the expression levels of thousands of genes simultaneously. In combination with classification methods, this technology can be useful to support clinical management decisions for individual patients in for example oncology. The objective of this paper is to systematically benchmark the role of nonlinear versus linear techniques and dimensionality reduction methods.
Results: A systematic benchmarking study is performed by comparing linear versions of standard classification and dimensionality reduction techniques with their nonlinear versions based on nonlinear kernel functions with a radial basis function (RBF) kernel. Nine binary cancer classification problems, derived from seven publicly available microarray data sets, and twenty randomizations of each problem are examined.
Conclusions: Three main conclusions can be formulated based on the performances on independent test sets. 1. When performing classification with least squares support vector machines (LS-SVM) (without dimensionality reduction), RBF kernels can be used without risking too much overfitting. The results obtained with well-tuned RBF kernels are never worse and sometimes even statistically significantly better compared to results obtained with a linear kernel in terms of test set ROC and test set accuracy performances. 2. Even for classification with linear classifiers like LS-SVM with linear kernel, using regularization is very important. 3. When performing kernel principal component analysis (kernel PCA) before classification, using an RBF kernel for kernel PCA tends to result in overfitting, especially when using supervised feature selection. It has been observed that an optimal selection of a large number of features is often an indication for overfitting. Kernel PCA with linear kernel gives better results.
Availability: Matlab scripts are available on request.
Contact: Nathalie.Pochet@esat.kuleuven.ac.be
Supplementary Information: http://www.esat.kuleuven.ac.be/~npochet/Bioinformatics/

Supplementary Information

Tables and figures of the paper:

Data sets:

Files of the data sets:

Format ready for loading the data sets into Matlab.

Methods:

Detailed description of the methods.

Results:

Boxplots:

For each data set, the results of all numerical experiments are represented in two figures. Figure 1 shows the boxplots (boxplot in Matlab) of the training set accuracy, the LOO-CV and the test set accuracy. Figure 2 shows the boxplots of the area under the ROC area of training and the test set.

Statistical significance tests:

A non-parametric paired test, namely the Wilcoxon signed rank test (signrank in Matlab), has been used to evaluate the results. A threshold of 0.05 is respected, which means that two results are statistically significantly different if the value of the Wilcoxon signed rank test applied to both of them is lower than 0.05.

Colon cancer data set (Alon et al., 1999).

Acute leukemia data set (Golub et al., 1999).

Breast cancer data set (Hedenfalk et al., 2001) - BRCA1 mutations versus BRCA2 and sporadic mutations.

Breast cancer data set (Hedenfalk et al., 2001) - BRCA2 mutations versus BRCA1 and sporadic mutations.

Breast cancer data set (Hedenfalk et al., 2001) - sporadic mutations versus BRCA1 and BRCA2 mutations.

Hepatocellular carcinoma data set (Iizuka et al., 2003).

High-grade glioma data set (Nutt et al., 2003).

Prostate cancer data set (Singh et al., 2002).

Breast cancer data set (Van 't Veer et al., 2002).