# OctaiPipe Models#

OctaiPipe provides a range of Machine Learning algorithms to solve classification, regression, and clustering problems. OctaiPipe has implementations for the most common and popular models in Scikit-learn, but also models from XGBoost, statsmodels, and LightGBM. However, we also provide you with a way to solve the Remaining Useful Life prediction of devices in an IoT environment as well as Anomaly Detection in Time Series Sensor data. With OctaiPipe, you can also build your own custom model which we will explain as well as part of this article.

Below we explain the available models in OctaiPipe:

## Supported Models#

### Linear Regression#

*In OctaiPipe configs: ‘ex_lin_reg’*
This class implements the ordinary least squares Linear Regression. It is built on top
of sklearn’s LinearRegression class. More details about the implementation can be found in this link.

### Ridge Regression#

*In OctaiPipe configs: ‘ridge_reg’*
Ridge regression class. In ridge regression, a penalty term is added to the cost function in training to penalise
large weight values that cause overfitting. More details about the implementation can be found in this link.

### Elastic Net Regressor#

*In OctaiPipe configs: ‘enet_reg’*
The Elastic Net Regression class builds on the sklearn-implementation found at this link.
Elastic Net is a linear regression with combined l1 and l2 regularizers.

### Random Forest Regression#

*In OctaiPipe configs: ‘random_forest_reg’*
Random forest regression class. A random forest is an ensemble of decision trees, whereby predictions are
taken as the average of each decision tree. More details about the implementation can be found in this link.

### Tweedie Regressor#

*In OctaiPipe configs: ‘tweedie_reg’*
The Tweedie Regression class builds on the sklearn-implementation found at this link.
It builds on Generalized Linear Regression, where the distribution of the output parameter is
defined by the power argument.

### Quantile Regressor#

*In OctaiPipe configs: ‘quantile_reg’*
The Quantile Regression class builds on the sklearn-implementation found at this link.
Quantile regression uses Mean Absolute Deviations to predict a quantile (commonly median) output, as opposed to
the mean output for linear regression. This can be useful for getting confidence intervals for
a continuous prediction if for example the 5th, 50th and 95th quantile is predicted.

### Support Vector Regressor#

*In OctaiPipe configs: ‘svr’*
The Support Vector Regression class builds on the sklearn-implementation found at this link.
The support vector regressor is based on Support Vector Machines. This implementation works
best for samples below 10,000. For larger samples, consider using SGD regression.

### Stochastic Gradient Descent Regressor#

*In OctaiPipe configs: ‘sgd_reg’*
The SGD Regression class builds on the sklearn-implementation found at this link.
The model tries to fit to the data by gradient descent where the model
is updated stepwise to find a local minimum.

### Bayesian Ridge Regressor#

*In OctaiPipe configs: ‘bayes_ridge_reg’*
The SGD Regression class builds on the sklearn-implementation found at this link.
Bayesian Ridge regression is a conditional model where the mean of the output variable is
described by the inputs and the model aims at finding the posterior probability of the regression coefficients.

### ARD Regressor#

*In OctaiPipe configs: ‘ard_reg’*
The Automatic Relevance Determination Regression class builds on the sklearn-implementation found at this link.
The ARD regressor is a Bayesian Ridge regression but with an ARD prior. The ARD regressor
is often stronger than Bayesian Ridge Regression out the box.

### Gradient Boosting Regressor#

*In OctaiPipe configs: ‘gbm_reg’*
The Gradient Boosting Regression class builds on the sklearn-implementation found at this link.
Gradient boosting regression is a tree based model that improves the fit in an iterative
fashion by combining the model at stage **n** with a penalized version of the previous models.

### Extreme Gradient Boosting Regressor#

*In OctaiPipe configs: ‘xgb_reg’*
The XGB Regression class builds on the sklearn-implementation found at this link.
XGBoost is an optimized gradient booster which is designed to be efficient and flexible.

### Skopt for LightGBM#

*In OctaiPipe configs: ‘skopt_lgbm_reg’*
Tuning the hyper-parameters of a machine learning model is often carried out using an exhaustive
exploration of the space of all hyper-parameter configurations for an estimator such as an GridSearch,
which often results in a very time consuming operation. scikit-optimize allows a sequential model-based
optimization using a Bayesian approach which we use to tune our LightGBM model. Refer to this link for more details.

### LightGBM Regressor#

*In OctaiPipe configs: ‘lgb_reg_sk’*
Refer to the following this link for more details about the implemented model.

### Logistic Regression#

*In OctaiPipe configs: ‘log_reg’*
Logistic regression is a linear model where the model is passed through a logit function,
bounding the output between 0 and 1. For the sklearn implementation, follow this link.

### k-Nearest Neighbors Classification#

*In OctaiPipe configs: ‘knn_class’*
k-Nearest Neighbors, or kNN, is an algorithm that classifies data points based on the majority class
of the *k* nearest neighbors of the data point. More details on the Sklearn implementation can be found here.

### Support Vector Classification#

*In OctaiPipe configs: ‘svc’*
The Support Vector Classification class builds on the sklearn-implementation found at this link.
The support vector classification is based on Support Vector Machines. This implementation works
best for samples below 10,000. Above that, other models are favoured.

### Random Forest Classification#

*In OctaiPipe configs: ‘random_forest_class’*
Random forest classification class. A random forest is an ensemble of decision trees, whereby predictions are
taken as the average of each decision tree. More details about the implementation can be found in this link.

### Extreme Gradient Boosting Classifier#

*In OctaiPipe configs: ‘xgb_class’*
The XGB Classification class builds on the sklearn-implementation found at this link.
XGBoost is an optimized gradient booster which is designed to be efficient and flexible.

### LightGBM Classification#

*In OctaiPipe configs: ‘lgb_class_sk’*
Refer to the following this link for more details about the implemented model.

### KMeans Clustering#

*In OctaiPipe configs: ‘kmeans’*
Checkout the sklearn documentation for more details.

### Exponential Smoothing#

*In OctaiPipe configs: ‘exp_smooth’*
Exponential smoothing is a time series model for univariate data that allows for handling trends
and seasonality. More information can be found here.

### Principal Component Analysis#

*In OctaiPipe configs: ‘pca’*
Principal Component Analysis is a dimensionality reduction technique used to reduce a dataset
of high dimensionality (large number of input features) to a smaller number of
orthogonal features explaining the highest proportion of variance. More information can
be found here.

## Custom Models#

With OctaiPipe, you can define your own ML model. To create custom model class, you should inherit from the base class `Model`

.
Currently custom models have to be defined within custom pipeline steps.
Each custom model has to implement a `_build_new`

method which assigns the model an estimator, e.g. this
implementation using `LinearRegression`

from `sklearn`

:

`self.estimator = LinearRegression(**kwargs)`

It also needs to implement a `_to_onnx`

method which converts `self.estimator`

to onnx, e.g. this implementation
converting an sklearn model using `convert_sklearn`

from `skl2onnx`

and the `data_schema`

attribute:

`self.onnx_estimator = convert_sklearn(self.estimator, initial_types=self.data_schema)`

## Model Management#

After training a model, it is desirable to have a way to persist the model for future use without having to retrain. The following sections give you some hints on how to persist an OctaiPipe model.

Each time we run the Model Training Step, the model file will be saved to both locally and to the cloud. The current Implementation of OctaiPipe provides two formats to save the trained model.

**ONNX** (Open Neural Network Exchange Format) which is a format designed to represent
any type of Machine Learning and Deep Learning model. Some examples of supported frameworks
are: PyTorch, TensorFlow, Keras and many more. In this way, ONNX can make it easier to convert
models from one framework to another.

**Serialized Python Objects** The model is saved using joblib.dump() and joblib.load() which
provide a replacement for pickle to work efficiently on arbitrary Python objects such as model
files.

## Model Versioning#

The ML model versioning convention is aimed to help the user identify, given models of the same
name, which of them are compatible with the data fed to it, be it data for re-training or for
inference. The model version is a string of the form “X.Y”, where X is the major version, and
the Y is the minor version, e.g. “1.2”. The major version characterises the following model
metadata: `type`

, `solutionType`

, `nClasses`

, `targetLabel`

, `columnNames`

. Models
with the same name and same major version have the same metadata for these fields. They are
differentiated by the minor version, which increments every time a new model is trained.

See Model Training Step for more detail on model specifications.