A curated list of awesome active Python machine learning frameworks, tools, and other related stuff in Python.

This is a living document, if you have any additions, please do not hesitate to make a pull-request with your additions or contact me.

In order to be an active library on the list, the framework must have a commit no older than a year.

For a list of machine learning frameworks in more languages please see the excellent list [https://github.com/josephmisiti/awesome-machine-learning]

• Scikit-Learn - A general purpose ML library. Most common algorithms and metrics implemented.
• Dask - Dask-ML provides scalable machine learning in Python using Dask alongside popular machine learning libraries like Scikit-Learn.
• XGBoost - XGBoost is an optimized distributed gradient boosting library designed to be highly efficient, flexible and portable.
• PyTorch - Tensors and Dynamic neural networks in Python with strong GPU acceleration.
• Metric learn - A ML library for learning metrics.
• TensorFlow - TensorFlow is an open source software library for numerical computation using data flow graphs.
• Keras - Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano.
• Imbalanced-learn - imbalanced-learn is a python package offering a number of re-sampling techniques commonly used in datasets showing strong between-class imbalance.
• Caffe - Caffe is a deep learning framework made with expression, speed, and modularity in mind.
• Annoy - Annoy (Approximate Nearest Neighbors Oh Yeah) is a C++ library with Python bindings to search for points in space that are close to a given query point.
• PySpark - Spark is a fast and general cluster computing system for Big Data.
• Orange - Orange is a component-based data mining software. It includes a range of data visualization, exploration, preprocessing and modeling techniques.
• TPOT - Consider TPOT your Data Science Assistant. TPOT is a Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.
• pgmpy - pgmpy is a python library for working with Probabilistic Graphical Models.
• Apache MXNET - Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
• Shogun - The SHOGUN machine learning toolbox. Unified and efficient Machine Learning since 1999.
• CNTK - The Microsoft Cognitive Toolkit (https://cntk.ai) is a unified deep learning toolkit that describes neural networks as a series of computational steps via a directed graph.
• PyOD - PyOD is a comprehensive and scalable Python toolkit for detecting outlying objects in multivariate data.
• LightGBM - LightGBM is a gradient boosting framework that uses tree based learning algorithms.
• CatBoost - CatBoost is a machine learning method based on gradient boosting over decision trees.
• auto_ml - Automated machine learning for production and analytics.
• Apache Singa - Distributed deep learning system.
• SimpleAI - This lib implements many of the artificial intelligence algorithms described on the book "Artificial Intelligence, a Modern Approach", from Stuart Russel and Peter Norvig.
• astroML - Machine learning, statistics, and data mining for astronomy and astrophysics.
• Turi Create - Turi Create simplifies the development of custom machine learning models. You don't have to be a machine learning expert to add recommendations, object detection, image classification, image similarity or activity classification to your app.
• NuPIC - The Numenta Platform for Intelligent Computing (NuPIC) is a machine intelligence platform that implements the HTM learning algorithms. HTM is a detailed computational theory of the neocortex.
• Lasagne - Lasagne is a lightweight library to build and train neural networks in Theano.
• Chainer - Chainer is a Python-based deep learning framework aiming at flexibility.
• Prophet - Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects.
• Surprise - Surprise is a Python scikit building and analyzing recommender systems.
• nilearn - Nilearn is a Python module for fast and easy statistical learning on NeuroImaging data.
• neuropredict - Easy and comprehensive assessment of predictive power, with support for neuroimaging features.
• pyhsmm - This is a Python library for approximate unsupervised inference in Bayesian Hidden Markov Models (HMMs) and explicit-duration Hidden semi-Markov Models (HSMMs), focusing on the Bayesian Nonparametric extensions, the HDP-HMM and HDP-HSMM, mostly with weak-limit approximations.
• SKLL - This Python package provides command-line utilities to make it easier to run machine learning experiments with scikit-learn.
• neurolab - Neurolab is a simple and powerful Neural Network Library for Python. Contains based neural networks, train algorithms and flexible framework to create and explore other neural network types.
• pomegranate - pomegranate is a package for probabilistic models in Python that is implemented in cython for speed.
• deap - DEAP is a novel evolutionary computation framework for rapid prototyping and testing of ideas.
• mlxtend - Mlxtend (machine learning extensions) is a Python library of useful tools for the day-to-day data science tasks.
• scikit-fuzzy - scikit-fuzzy is a fuzzy logic toolkit for SciPy.
• fylearn - FyLearn is a fuzzy machine learning library, built on top of SciKit-Learn.
• tflearn - TFlearn is a modular and transparent deep learning library built on top of Tensorflow.
• Regularized Greedy Forest - Regularized Greedy Forest (RGF) is a tree ensemble machine learning method
• fuku-ml - Simple machine learning library.
• Edward - Edward is a Python library for probabilistic modeling, inference, and criticism.
• stacked_generalization - Library for machine learning stacking generalization.
• modAL - modAL is an active learning framework for Python3, designed with modularity, flexibility and extensibility in mind.
• neonrvm - neonrvm is an experimental open source machine learning library for performing regression tasks using RVM technique.
• xLearn - xLearn is a high performance, easy-to-use, and scalable machine learning package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM), which can be used to solve large-scale machine learning problems.
• ml-ens - A Python library for high performance ensemble learning.
• mindsdb - MindsDB's goal is to make it very simple for developers to use the power of artificial neural networks in their projects.
• Mars - Mars is a tensor-based unified framework for large-scale data computation.
• Hyperopt-sklearn - Hyper-parameter optimization for sklearn.
• H2O - H2O is an in-memory platform for distributed, scalable machine learning.
• seglearn - Python module for machine learning time series.
• pycobra - python library implementing ensemble methods for regression, classification and visualisation tools including Voronoi tesselations.
• scikit-multilearn - A scikit-learn based module for multi-label et. al. classification.
• auto-sklearn - auto-sklearn is an automated machine learning toolkit and a drop-in replacement for a scikit-learn estimator.
• skits - A library for SciKit-learn-Inspired Time Series models.
• tsfresh - Automatic extraction of relevant features from time series.
• pyqlearning - pyqlearning is Python library to implement Reinforcement Learning and Deep Reinforcement Learning.
• keras-rl - Deep Reinforcement Learning for Keras.
• mushroom-rl - Python library for Reinforcement Learning experiments.
• chainerrl - ChainerRL is a deep reinforcement learning library built on top of Chainer.
• tensorforce - Tensorforce: a TensorFlow library for applied reinforcement learning.
• Determined - Deep learning training platform with integrated support for distributed training, hyperparameter tuning, smart GPU scheduling, experiment tracking, and a model registry.

• NumPy - NumPy is the fundamental package needed for scientific computing with Python.
• Pandas - pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with "relational" or "labeled" data both easy and intuitive.
• Modin - Modin: Speed up your Pandas workflows by changing a single line of code.
• dfply - The dfply package makes it possible to do R's dplyr-style data manipulation with pipes in python on pandas DataFrames.
• xlwings - xlwings is a BSD-licensed Python library that makes it easy to call Python from Excel and vice versa.
• pyflux - Open source time series library for Python.
• petl - Python Extract Transform and Load Tables of Data
• pypeln - Concurrent data pipelines made easy.
• botflow - Python Fast Dataflow programming framework for Data pipeline work.
• Great Expectations - Great Expectations is a framework that helps teams save time and promote analytic integrity with a new twist on automated testing: pipeline tests.
• pandera - Validating pandas data structures for people seeking correct things.
• pyjanitor - Clean APIs for data cleaning. Python implementation of R package Janitor.
• PandasSchema - A validation library for Pandas data frames using user-friendly schemas.
• engarde - A library for defensive data analysis.
• sklearn-pandas - Pandas integration with sklearn.
• Blaze - Blaze translates a subset of modified NumPy and Pandas-like syntax to databases and other computing systems.
• scikit-datasets - Scikit-learn-compatible datasets.

• SciPy - SciPy (pronounced "Sigh Pie") is open-source software for mathematics, science, and engineering. It includes modules for statistics, optimization, integration, linear algebra, Fourier transforms, signal and image processing, ODE solvers, and more.
• Statsmodels - Statsmodels is a Python package that provides a complement to scipy for statistical computations including descriptive statistics and estimation and inference for statistical models.
• pymc3 - PyMC3 is a Python package for Bayesian statistical modeling and Probabilistic Machine Learning focusing on advanced Markov chain Monte Carlo (MCMC) and variational inference (VI) algorithms.
• sympy - A Python library for symbolic mathematics.
• pmdarima - A package that brings R's beloved auto.arima to Python, making an even stronger case for why Python > R for data science.
• scikit-posthocs - Pairwise multiple comparisons (post hoc) tests in Python.

• Lime - Lime: Explaining the predictions of any machine learning classifier.
• eli5 - ELI5 is a Python package which helps to debug machine learning classifiers and explain their predictions.
• SHAP - SHAP (SHapley Additive exPlanations) is a unified approach to explain the output of any machine learning model.
• LOFO - LOFO (Leave One Feature Out) Importance calculates the importances of a set of features.

• Matplotlib - Matplotlib is a Python 2D plotting library which produces publication-quality figures in a variety of hardcopy formats and interactive environments across platforms.
• Seaborn - Seaborn is a Python visualization library based on matplotlib. It provides a high-level interface for drawing attractive statistical graphics.
• Bokeh - Bokeh is an interactive visualization library for Python that enables beautiful and meaningful visual presentation of data in modern web browsers. With Bokeh, you can quickly and easily create interactive plots, dashboards, and data applications.
• plotly.py - plotly.py is an interactive, open-source, and browser-based graphing library for Python.
• scikit-plot - An intuitive library to add plotting functionality to scikit-learn objects.
• plotnine - plotnine is an implementation of a grammar of graphics in Python, it is based on ggplot2.
• Cufflinks - This library binds the power of plotly with the flexibility of pandas for easy plotting.
• Chartpy - Easy to use Python API wrapper to plot charts with matplotlib, plotly, bokeh and more.
• Vispy - VisPy is a high-performance interactive 2D/3D data visualization library.
• pycm - Multi-class confusion matrix library in Python.
• Altair-Catplot - Utility to generate plots with categorical variables using Altair.
• pdvega - Interactive plotting for Pandas using Vega-Lite.
• folium - Python Data. Leaflet.js Maps.
• jmpy - Quick plotting and data visualization of pandas and numpy data.
• missingno - Missing data visualization module for Python.
• Yellowbrick - Visual analysis and diagnostic tools to facilitate machine learning model selection.
• netron - Netron is a viewer for neural network, deep learning and machine learning models.
• PrettyPandas - PrettyPandas is a Pandas DataFrame Styler class that helps you create report quality tables with a simple API.

• gensim - Gensim is a Python library for topic modelling, document indexing and similarity retrieval with large corpora. Target audience is the natural language processing (NLP) and information retrieval (IR) community.

• Numba - A Just-In-Time Compiler for Numerical Functions in Python.
• Jupyter Notebook - A rich explorative data analysis tool.
• boto3 - AWS SDK for Python.
• PennAI - PennAI is an easy-to-use data science assistant. It allows researchers without machine learning or coding expertise to run supervised machine learning analysis through a clean web interface.

• BigML Python Bindings - These BigML Python bindings allow you to interact with BigML.io, the API for BigML. You can use it to easily create, retrieve, list, update, and delete BigML resources (i.e., sources, datasets, models and, predictions).
• python-timbl - python-timbl is a Python extension module wrapping the full TiMBL C++ programming interface. With this module, all functionality exposed through the C++ interface is also available to Python scripts. Being able to access the API from Python greatly facilitates prototyping TiMBL-based applications.
• thampi - thampi creates a machine learning prediction server on AWS Lambda.
• MLPACK - mlpack: a scalable C++ machine learning library (with Python bindings)
• PyStan - PyStan provides a Python interface to Stan, a package for Bayesian inference using the No-U-Turn sampler, a variant of Hamiltonian Monte Carlo.

• pm4py - PM4Py is a python library that supports (state-of-the-art) process mining algorithms in python.
• Optimus - Optimus is the missing framework to profile, clean, process and do ML in a distributed fashion using Apache Spark(PySpark).
• impyute - Impyute is a library of missing data imputation algorithms.
• Stairs - Framework which helps you make parallel/distributed calculations using data pipelines.
• fastText - Library for fast text representation and classification.
• pendulum - Python datetimes made easy.
• loguru - Python logging made (stupidly) simple.

3 types of Machine Learning Algorithms

1. Supervised Learning How it works: This algorithm consist of a target / outcome variable (or dependent variable) which is to be predicted from a given set of predictors (independent variables). Using these set of variables, we generate a function that map inputs to desired outputs. The training process continues until the model achieves a desired level of accuracy on the training data. Examples of Supervised Learning: Regression, Decision Tree, Random Forest, KNN, Logistic Regression etc.

2. Unsupervised Learning How it works: In this algorithm, we do not have any target or outcome variable to predict / estimate. It is used for clustering population in different groups, which is widely used for segmenting customers in different groups for specific intervention. Examples of Unsupervised Learning: Apriori algorithm, K-means.

3. Reinforcement Learning: How it works: Using this algorithm, the machine is trained to make specific decisions. It works this way: the machine is exposed to an environment where it trains itself continually using trial and error. This machine learns from past experience and tries to capture the best possible knowledge to make accurate business decisions. Example of Reinforcement Learning: Markov Decision Process

Here is the list of commonly used machine learning algorithms. These algorithms can be applied to almost any data problem:

1.Linear Regression 2.Logistic Regression 3.Decision Tree 4.SVM 5.Naive Bayes 6.kNN 7.K-Means 8.Random Forest 9.Dimensionality Reduction Algorithms 10.Gradient Boosting algorithms 11.GBM 12.XGBoost 13.LightGBM 14.CatBoost

It is used to estimate real values (cost of houses, number of calls, total sales etc.) based on continuous variable(s). Here, we establish relationship between independent and dependent variables by fitting a best line. This best fit line is known as regression line and represented by a linear equation Y= a *X + b.

The best way to understand linear regression is to relive this experience of childhood. Let us say, you ask a child in fifth grade to arrange people in his class by increasing order of weight, without asking them their weights! What do you think the child will do? He / she would likely look (visually analyze) at the height and build of people and arrange them using a combination of these visible parameters. This is linear regression in real life! The child has actually figured out that height and build would be correlated to the weight by a relationship, which looks like the equation above.

In this equation:

Y – Dependent Variable a – Slope X – Independent variable b – Intercept These coefficients a and b are derived based on minimizing the sum of squared difference of distance between data points and regression line.

Linear Regression is mainly of two types: Simple Linear Regression and Multiple Linear Regression. Simple Linear Regression is characterized by one independent variable. And, Multiple Linear Regression(as the name suggests) is characterized by multiple (more than 1) independent variables. While finding the best fit line, you can fit a polynomial or curvilinear regression. And these are known as polynomial or curvilinear regression.

Don’t get confused by its name! It is a classification not a regression algorithm. It is used to estimate discrete values ( Binary values like 0/1, yes/no, true/false ) based on given set of independent variable(s). In simple words, it predicts the probability of occurrence of an event by fitting data to a logit function. Hence, it is also known as logit regression. Since, it predicts the probability, its output values lies between 0 and 1 (as expected).

Again, let us try and understand this through a simple example.

Let’s say your friend gives you a puzzle to solve. There are only 2 outcome scenarios – either you solve it or you don’t. Now imagine, that you are being given wide range of puzzles / quizzes in an attempt to understand which subjects you are good at. The outcome to this study would be something like this – if you are given a trignometry based tenth grade problem, you are 70% likely to solve it. On the other hand, if it is grade fifth history question, the probability of getting an answer is only 30%. This is what Logistic Regression provides you.

Coming to the math, the log odds of the outcome is modeled as a linear combination of the predictor variables.

odds= p/ (1-p) = probability of event occurrence / probability of not event occurrence ln(odds) = ln(p/(1-p)) logit(p) = ln(p/(1-p)) = b0+b1X1+b2X2+b3X3....+bkXk Above, p is the probability of presence of the characteristic of interest. It chooses parameters that maximize the likelihood of observing the sample values rather than that minimize the sum of squared errors (like in ordinary regression).

Now, you may ask, why take a log? For the sake of simplicity, let’s just say that this is one of the best mathematical way to replicate a step function. I can go in more details, but that will beat the purpose of this article.

This is one of my favorite algorithm and I use it quite frequently. It is a type of supervised learning algorithm that is mostly used for classification problems. Surprisingly, it works for both categorical and continuous dependent variables. In this algorithm, we split the population into two or more homogeneous sets. This is done based on most significant attributes/ independent variables to make as distinct groups as possible.

To split the population into different heterogeneous groups, it uses various techniques like Gini, Information Gain, Chi-square, entropy.

The best way to understand how decision tree works, is to play Jezzball – a classic game from Microsoft (image below). Essentially, you have a room with moving walls and you need to create walls such that maximum area gets cleared off with out the balls.

So, every time you split the room with a wall, you are trying to create 2 different populations with in the same room. Decision trees work in very similar fashion by dividing a population in as different groups as possible.

It is a classification method. In this algorithm, we plot each data item as a point in n-dimensional space (where n is number of features you have) with the value of each feature being the value of a particular coordinate.

For example, if we only had two features like Height and Hair length of an individual, we’d first plot these two variables in two dimensional space where each point has two co-ordinates (these co-ordinates are known as Support Vectors)

SVM1 : Now, we will find some line that splits the data between the two differently classified groups of data. This will be the line such that the distances from the closest point in each of the two groups will be farthest away.

SVM2 : In the example shown above, the line which splits the data into two differently classified groups is the black line, since the two closest points are the farthest apart from the line. This line is our classifier. Then, depending on where the testing data lands on either side of the line, that’s what class we can classify the new data as.

It is a classification technique based on Bayes’ theorem with an assumption of independence between predictors. In simple terms, a Naive Bayes classifier assumes that the presence of a particular feature in a class is unrelated to the presence of any other feature. For example, a fruit may be considered to be an apple if it is red, round, and about 3 inches in diameter. Even if these features depend on each other or upon the existence of the other features, a naive Bayes classifier would consider all of these properties to independently contribute to the probability that this fruit is an apple.

Naive Bayesian model is easy to build and particularly useful for very large data sets. Along with simplicity, Naive Bayes is known to outperform even highly sophisticated classification methods.

Bayes theorem provides a way of calculating posterior probability P(c|x) from P(c), P(x) and P(x|c). Look at the equation below: Bayes_rule

Here,

P(c|x) is the posterior probability of class (target) given predictor (attribute). P(c) is the prior probability of class. P(x|c) is the likelihood which is the probability of predictor given class. P(x) is the prior probability of predictor. Example: Let’s understand it using an example. Below I have a training data set of weather and corresponding target variable ‘Play’. Now, we need to classify whether players will play or not based on weather condition. Let’s follow the below steps to perform it.

Step 1: Convert the data set to frequency table

Step 2: Create Likelihood table by finding the probabilities like Overcast probability = 0.29 and probability of playing is 0.64.

Bayes_4

Step 3: Now, use Naive Bayesian equation to calculate the posterior probability for each class. The class with the highest posterior probability is the outcome of prediction.

Problem: Players will pay if weather is sunny, is this statement is correct?

We can solve it using above discussed method, so P(Yes | Sunny) = P( Sunny | Yes) * P(Yes) / P (Sunny)

Here we have P (Sunny |Yes) = 3/9 = 0.33, P(Sunny) = 5/14 = 0.36, P( Yes)= 9/14 = 0.64

Now, P (Yes | Sunny) = 0.33 * 0.64 / 0.36 = 0.60, which has higher probability.

Naive Bayes uses a similar method to predict the probability of different class based on various attributes. This algorithm is mostly used in text classification and with problems having multiple classes.

It can be used for both classification and regression problems. However, it is more widely used in classification problems in the industry. K nearest neighbors is a simple algorithm that stores all available cases and classifies new cases by a majority vote of its k neighbors. The case being assigned to the class is most common amongst its K nearest neighbors measured by a distance function.

These distance functions can be Euclidean, Manhattan, Minkowski and Hamming distance. First three functions are used for continuous function and fourth one (Hamming) for categorical variables. If K = 1, then the case is simply assigned to the class of its nearest neighbor. At times, choosing K turns out to be a challenge while performing kNN modeling.

KNN: KNN can easily be mapped to our real lives. If you want to learn about a person, of whom you have no information, you might like to find out about his close friends and the circles he moves in and gain access to his/her information!

Things to consider before selecting kNN:

KNN is computationally expensive Variables should be normalized else higher range variables can bias it Works on pre-processing stage more before going for kNN like an outlier, noise removal

It is a type of unsupervised algorithm which solves the clustering problem. Its procedure follows a simple and easy way to classify a given data set through a certain number of clusters (assume k clusters). Data points inside a cluster are homogeneous and heterogeneous to peer groups.

Remember figuring out shapes from ink blots? k means is somewhat similar this activity. You look at the shape and spread to decipher how many different clusters / population are present!

splatter_ink_blot_texture_by_maki_tak-d5p6zph

How K-means forms cluster:

K-means picks k number of points for each cluster known as centroids. Each data point forms a cluster with the closest centroids i.e. k clusters. Finds the centroid of each cluster based on existing cluster members. Here we have new centroids. As we have new centroids, repeat step 2 and 3. Find the closest distance for each data point from new centroids and get associated with new k-clusters. Repeat this process until convergence occurs i.e. centroids does not change.

How to determine value of K: In K-means, we have clusters and each cluster has its own centroid. Sum of square of difference between centroid and the data points within a cluster constitutes within sum of square value for that cluster. Also, when the sum of square values for all the clusters are added, it becomes total within sum of square value for the cluster solution.

We know that as the number of cluster increases, this value keeps on decreasing but if you plot the result you may see that the sum of squared distance decreases sharply up to some value of k, and then much more slowly after that. Here, we can find the optimum number of cluster.

Random Forest is a trademark term for an ensemble of decision trees. In Random Forest, we’ve collection of decision trees (so known as “Forest”). To classify a new object based on attributes, each tree gives a classification and we say the tree “votes” for that class. The forest chooses the classification having the most votes (over all the trees in the forest).

Each tree is planted & grown as follows:

If the number of cases in the training set is N, then sample of N cases is taken at random but with replacement. This sample will be the training set for growing the tree. If there are M input variables, a number m<<M is specified such that at each node, m variables are selected at random out of the M and the best split on these m is used to split the node. The value of m is held constant during the forest growing. Each tree is grown to the largest extent possible. There is no pruning. For more details on this algorithm, comparing with decision tree and tuning model parameters, I would suggest you to read these articles:

In the last 4-5 years, there has been an exponential increase in data capturing at every possible stages. Corporates/ Government Agencies/ Research organisations are not only coming with new sources but also they are capturing data in great detail.

For example: E-commerce companies are capturing more details about customer like their demographics, web crawling history, what they like or dislike, purchase history, feedback and many others to give them personalized attention more than your nearest grocery shopkeeper.

As a data scientist, the data we are offered also consist of many features, this sounds good for building good robust model but there is a challenge. How’d you identify highly significant variable(s) out 1000 or 2000? In such cases, dimensionality reduction algorithm helps us along with various other algorithms like Decision Tree, Random Forest, PCA, Factor Analysis, Identify based on correlation matrix, missing value ratio and others.

10.1. GBM: GBM is a boosting algorithm used when we deal with plenty of data to make a prediction with high prediction power. Boosting is actually an ensemble of learning algorithms which combines the prediction of several base estimators in order to improve robustness over a single estimator. It combines multiple weak or average predictors to a build strong predictor. These boosting algorithms always work well in data science competitions like Kaggle, AV Hackathon, CrowdAnalytix.

10.2. XGBoost: Another classic gradient boosting algorithm that’s known to be the decisive choice between winning and losing in some Kaggle competitions. The XGBoost has an immensely high predictive power which makes it the best choice for accuracy in events as it possesses both linear model and the tree learning algorithm, making the algorithm almost 10x faster than existing gradient booster techniques. The support includes various objective functions, including regression, classification and ranking. One of the most interesting things about the XGBoost is that it is also called a regularized boosting technique. This helps to reduce overfit modelling and has a massive support for a range of languages such as Scala, Java, R, Python, Julia and C++. Supports distributed and widespread training on many machines that encompass GCE, AWS, Azure and Yarn clusters. XGBoost can also be integrated with Spark, Flink and other cloud dataflow systems with a built in cross validation at each iteration of the boosting process.

10.3. LightGBM: LightGBM is a gradient boosting framework that uses tree based learning algorithms. It is designed to be distributed and efficient with the following advantages: Faster training speed and higher efficiency Lower memory usage Better accuracy Parallel and GPU learning supported Capable of handling large-scale data The framework is a fast and high-performance gradient boosting one based on decision tree algorithms, used for ranking, classification and many other machine learning tasks. It was developed under the Distributed Machine Learning Toolkit Project of Microsoft.

Since the LightGBM is based on decision tree algorithms, it splits the tree leaf wise with the best fit whereas other boosting algorithms split the tree depth wise or level wise rather than leaf-wise. So when growing on the same leaf in Light GBM, the leaf-wise algorithm can reduce more loss than the level-wise algorithm and hence results in much better accuracy which can rarely be achieved by any of the existing boosting algorithms.

Also, it is surprisingly very fast, hence the word ‘Light’.

10.4. Catboost: CatBoost is a recently open-sourced machine learning algorithm from Yandex. It can easily integrate with deep learning frameworks like Google’s TensorFlow and Apple’s Core ML. The best part about CatBoost is that it does not require extensive data training like other ML models, and can work on a variety of data formats; not undermining how robust it can be. Make sure you handle missing data well before you proceed with the implementation. Catboost can automatically deal with categorical variables without showing the type conversion error, which helps you to focus on tuning your model better rather than sorting out trivial errors.