Machine Learning for Bioinformatics: A User's Guide

Josh has recently first-authored a paper in New Phytologist, which introduces SeedGerm - a machine learning platofrm that enables scientists and seed companies to automatically quantify seed germination traits through computer vision techniques.

Matt, among his various projects, has been applying his machine learning expertise to study how the microbiome affects gut health and breast cancer, while contributing to a global effort to understand the systemic effects of COVID-19 on the human body.

The pair previously introduced us to the dos and don’ts of machine learning in “10 things you need to know about getting into machine learning”, which is where we’d direct a novice user.

In this second installment, they introduce five key concepts that any bioinformatician should know when applying machine learning.

Machine learning (ML) algorithms can be broadly split into two main categories: supervised and unsupervised learning. The choice of these approaches depends on the task at hand.

The vast majority of ML algorithm applications are supervised learning, where ground truth (empirical evidence) is known, and the algorithm’s task is to learn and predict this ground truth from corresponding data. This can be further split into two main subcategories; regression and classification.

In a regression task the predicted variable is a continuous number, whereas the predicted variable is a category for classification tasks. There are many different algorithms that can be used for these tasks, from traditional methods such as Logistic Regression, Random Forest and Support Vector Machine to deep approaches such as Deep Neural Networks. An example application of supervised learning would be to predict which patients are healthy or have cancer from genomic data.

This differs from unsupervised learning, where the algorithm only sees the input data with no regard to the ground truth, which makes it ideal for tasks such as clusterings and association analysis. Again, there are many different algorithms that are used but some of the most common are PCA, K-means and hierarchical clustering. Deep approaches can also be applied using the autoencoder architecture.

Above: The two figures above demonstrate the difference between a supervised approach and an unsupervised approach for the same dataset. In the supervised approach, here a support vector machine creates a dividing line known as the hyperplane which best splits the data based on their known grown truth. In the unsupervised approach, however, we are asking the algorithm to find 4 clusters from the data which are most alike, with no regard to the ground truth. In this, a K-means clustering algorithm was used.

Feature engineering is one of the most important steps in the application of machine learning. The goal of feature engineering is to wrangle your data into a form that is more interpretable for the machine learning algorithm. This makes it easier for the algorithm to learn the structure of the data and achieve good model performance.

There are many different techniques you can use to transform your data. These include imputation, scaling, transformations, handling outliers, one-hot encoding, binning, and the list goes on. Here we will discuss scaling and binning in particular.

Scaling aims to bring two continuous features into the same range such that they are comparable. In some cases, this also increases the computation speed of the algorithm, due to faster convergence. Binning, on the other hand, is the process of assigning a subcategory to a feature based on a range or condition, and can be applied to both categorical and continuous features.

Feature engineering also plays a role in the shift to deep learning architectures; one advantage of the deep approach is that it incorporates feature engineering within the model.

Ultimately, the aim of feature engineering is to increase the robustness and performance of a model. That said, choosing the right features is important. If the features you transform are not representative of the problem then your results may be misleading.

Above: The above distribution represents a feature within a dataset. We can simplify this data by reducing the number of components into a predefined set of bins. Therefore, creating discrete values from a continuous distribution.

High dimensional datasets are commonly analysed in life sciences (e.g. gene expression) but we are typically concerned with only a subset of those features.

To reduce a high dimensional dataset into a more interpretable subset of features that contain the relationship between input and output we are interested in learning, we must undergo feature selection. Thie three forms of feature selection are filter, wrapper, and embedded methods. As embedded methods link closely to regularisation, which is discussed in the next section, filter and wrapper methods will be the focus here.

The filter method of feature selection involves using a metric to measure the strength of the relationship between a feature and the targetin order to distinguish between relevant and irrelevant features. Examples of these metrics would be Pearson’s correlation and mutual information. Every feature is scored using the chosen metric, and the best scoring features are taken forward for training purposes. An example of applying a filter method of feature selection would be using a metric to select statistically relevant genes in an expression dataset so that the model is not influenced by irrelevant features.

Wrapper methods focus on searching the feature space for the best performing combination of features based on the model’s validation error. Examples of wrapper methods would be recursive feature elimination (RFE) and sequential feature selection (SFS). Both examples are computationally expensive search algorithms that iteratively either add or remove features, calculating the error at each iteration until all features have been added/removed, allowing an estimate of the optimal subset to be selected.

Above: Filter methods used to select; a relevant feature selected by measuring its correlation with the target; two features whose distributions are effective at discriminating between two classes using mutual information

Sometimes in a machine learning project we encounter situations where our model performs significantly better on the training set compared to the test set. This difference is referred to as overfitting and is usually caused by fitting a model that is too complex for the training data.

The excessive complexity of the model enables it to learn an unrealistically accurate mapping from input to output. But, when tasked with making predictions based on unseen data, the mapping it has learned utilises noise that is specific to the training data and does not generalise, resulting in poor predictions. Techniques such as regularisation and cross-validation can be used to combat overfitting.

Regularisation, an embedded method of feature selection, comes in many forms but a shared goal of this technique is to reduce overfitting by penalising model complexity. When optimising a model, a regularisation term that is typically calculated as a function of either the weights or hyperparameters of the model can be added to the error, meaning the model that minimises the error is more likely to be one that has balanced predictive power and complexity.

Cross-validation is a method of estimating a model’s error on unseen data before making predictions on the test data. The method first involves partitioning the training dataset into training and validation subsets. The model is trained on the training subsets and predictions are made on the validation subsets, with each error on the validation subsets being one estimate of the error on the test data.

Different combinations of training and validation subsets are attempted and the average error across these combinations is an estimate of the error on the test data, allowing overfitting to be identified and rectified before exposure to the test data.

Above: Two trained models; the left displaying an overfitted classifier that has learned a complicated decision boundary that is unlikely to generalise, the right displaying a classifier that performs worse on the training data, but is likely to generalise better to unseen data.

Choosing the appropriate evaluation metric is a key part of the machine learning procedure, allowing you to interpret your model’s performance and communicate your results. When comparing similar models, it’s common practice to choose the model that minimises or maximises the evaluation metric that best quantifies its effectiveness at the learning task.

Selecting an inappropriate metric can lead to a dangerous misunderstanding of a model’s performance. An example of this would be selecting accuracy as the evaluation metric when working with an imbalanced target distribution. If the task is to classify samples as infected or uninfected, but 95% of samples belong to the uninfected class, a model that predicts every sample as being uninfected would score an accuracy of 95% despite being unable to discriminate between the two classes.

More appropriate classification metrics for this task are sensitivity and specificity, which consider the number of false negatives and false positives respectively. Using multiple evaluation metrics is likely to give a more detailed interpretation of the model but using a confusion matrix that displays true positives, true negatives, false positives, and false negatives is likely to be best.

Common evaluation metrics for regression tasks would be the mean squared error (MSE), mean absolute error (MAE), and coefficient of determination (R2). Choosing an appropriate metric for classification tasks is more difficult as it usually depends on the relative importance of false positives and false negatives.

Accuracy, F1, precision, recall, and AUROC are all effective evaluation metrics for a classification task but selecting the optimal metric, whether in a regression or classification setting, is dependent on both the dataset and how the success of predictions should be quantified.

Above: Two confusion matrices that include the number of true positives, true negatives, false positives, and false negatives. Without viewing the confusion matrices or using multiple evaluation metrics, one could mistake the models as performing similarly despite the large differences in predictions.