In this blog we’re going to talk about Autonomous vehicles, which are considered to be the future of the Motor Industry. I’ll try to explain how this piece of technology was made possible thanks to the advancements in the field of High Performance Computing (HPC).

What is an Autonomous Vehicle ?

• A self-driving car, also known as an autonomous vehicle (AV), is a vehicle that is capable of sensing its environment and moving safely with little or no human input.
• Self-driving cars combine a variety of sensors to perceive their surroundings, such as radar, lidar, sonar, GPS, odometry and inertial measurement units. Advanced control systems interpret sensory information to identify appropriate navigation paths, as well as obstacles and relevant signage.
• One of the major modern-day application of autonomous vehicles is Long Distance Trucking.

Most of the driver-less cars in the market are simple applications of a type of Machine Learning algorithm which is called as Reinforcement Learning.

Autonomous Vehicles : An Example of Reinforcement Learning

• Reinforcement Learning is about taking suitable actions to maximize reward in a particular situation. It helps in finding the best possible behavior or path to take in a specific situation.
• In reinforcement learning, there is no answer and the reinforcement agent decides what to do in order to perform the given task.
• In the absence of training data set, it is bound to learn from its experience.

Reinforcement Learning v/s Supervised Learning

Reinforcement learning differs from the supervised learning in a way that in supervised learning the training data has the answer key with it, so the model is trained with the correct answer itself whereas in reinforcement learning, there is no answer and the reinforcement agent decides what to do in order to perform the given task.

Any self driving car to gather data from the real-time surroundings is required to be equipped with numerous sensors. All this data has to be processed, analyzed and managed.

Data Generated by Autonomous vehicles is HUGE !!

• Driver-less vehicles will generate and use a variety of data to analyze various things like condition of the road, vehicle condition, passenger comfort and safety and also the geographical position of the car.
• Driver-less cars are equipped with a large number of sensors, embedded cameras, in-car computers, high precision GPS and satellite receivers, short-range wireless network and 4G/5G interfaces to connect to the Internet.
• Real-time information such as the local environment, traffic situation, emergency alerts and weather conditions are constantly fed to large data-bases that exchange data with management systems and supervising systems of the vehicle.
• The transmitted information will be used by predictive driving functions to avoid road hazards and increase passenger safety.

According to an estimate, a Google car produces around 1GB of sensor data per second !!. That would add up to 2 Petabytes of data per year (2 million GB) per car, assuming that the average American drives 600 hours a year — (Apr 6, 2017)

To manage all this data and ensure the safety of the passengers, we need High Performance Computing.

Whenever these kinds of vehicles are tested for failure, it is done under lab conditions with people trying to simulate Real-life conditions. Their too the data generated by them is Humongous to process and analyze but imagine the amount of data generated when these vehicles hit the Real Roads !!

A big challenge to Autonomous Vehicles : Amount of Data and its Processing

• When these types of vehicles will go mainstream (well that process has already started), the amount of data generated will grow exponentially.
• According to Intel, driver-less vehicles will send more than four terabytes of data (approximately 1000 DVDs) in about an hour and a half of driving to the cloud.
• To implement the predictive decision system based on AI to process and evaluate such enormous amount of data we’ll require the next generation High Performance Computing (HPC) and Big Data capabilities.

In the future, cars will create significantly more data than today the entire Internet community.

Conclusion :

Autonomous Vehicles are still a work in progress. While some of these vehicles have started hitting the consumer markets, it is important to understand that the amount of data being generated by these vehicles is enormous and is expected to keep increasing at a phenomenal rate. Only the next generation of High Performance Computers (HPC) and Distributed Processing Systems can help us advance into the world of Self-driving cars.

Connect with me

LinkedIn Instagram Twitter Facebook

In this blog I am going to provide a brief overview about Hadoop, which is the most popular frameworks for distributed processing of large data sets.

We will try to understand the architecture of Hadoop, its modules and will also look at the advantages of using Hadoop. In the end, we’ll take a look at its history to highlight some key people involved in the actual development of this great framework.

What is Hadoop ?

Hadoop is an open-source, Java-based framework from Apache which is used for storing, processing and analyzing data which are very huge in volume. Hadoop is used for batch/ offline processing.

It is a collection of software utilities which uses a network of many computers to solve problems involving large amounts of data and computation.

Architecture :

The architecture of Hadoop involves a package of the file system and operating system abstractions which is called the Hadoop Common package, containing a MapReduce engine (processing part) and the Hadoop Distributed File System (storage part).

The Hadoop Common package contains the JAR (Java ARchive) files and scripts needed to start Hadoop.

Every Hadoop cluster consists of a single master and multiple worker nodes. The Master node has a Job Tracker, Task Tracker, Name Node and Data Node while the Slave (worker node) can act as both a DataNode and TaskTracker. Also it is possible to have data-only and compute only worker nodes.

The standard startup and shutdown scripts require that Secure Shell (SSH) to be set up between nodes in the cluster. Apart from this Hadoop requires a JRE (Java Runtime Environment) of 1.6 or greater.

Modules of Hadoop :

The Hadoop framework is composed of the following modules :

1. Hadoop Distributed File System (HDFS) : It includes the files that will be broken into blocks and will be stored in nodes over a distributed architecture. Using a distributed file system provides very high aggregate bandwidth across clusters
2. Hadoop Yarn (Yet another Resource Negotiator) : Used for job scheduling and managing the computing resources in clusters.
3. Hadoop MapReduce : It is an algorithm which distributes the task into small pieces and assigns those pieces to many computers joined over the network, and assembles all the events to form the last event dataset.
4. Hadoop Common : Includes Java Libraries that are used to start Hadoop and utilities which are needed by other Hadoop modules.

Advantages of Hadoop :

1. Hadoop is fast, as it allows the users to quickly write and test distributed systems. It stores and processes huge  amounts of any kind of data, quickly.
2. Hadoop is efficient, and it automatically distributes the data and work across the machines. Through this it utilizes the underlying parallelism of the CPU cores.
3. Hadoop is Scalable, servers can be added or removed from the cluster dynamically. You can easily grow your system to handle more data by simple adding nodes.
4. Hadoop is cost-effective in comparison to other traditional databases when it comes to storing and performing computations on data.
5. Hadoop is an open-source framework and is compatible on all the platforms since it is based on Java.
6. Hadoop library has been designed to detect and handle failures at the application layer. It does not rely on the hardware to provide fault-tolerance and high availability (FTHA).

History

Conclusion

So that’s it folks you’ve learned the basics of Hadoop framework. We saw the architecture of Hadoop and looked at the modules which make it the best framework for distributed processing. Also, we looked at some of the advantages of using Hadoop and tried to understand the timeline involved in its development process.

Support Vector Machines (SVMs) are a set of supervised learning methods which learn from the dataset and can be used for both regression and classification. An SVM is a kind of large-margin classifier : it is a vector space based machine learning method where the goal is to find a decision boundary between two classes that is maximally far from any point in the training data.

Support Vectors :

The term Support Vectors refers to the co-ordinates of individual observation. Support Vector Machine is a frontier which best segregates the two classes using a hyperplane/ line.

Working of SVM :

An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall.

In addition to performing linear classification, SVMs can efficiently perform a non-linear classification using what is called the kernel trick, implicitly mapping their inputs into high-dimensional feature spaces.

Kernel Method :

Kernel method is used by SVM to perform a non-linear classification. They take low dimensional input space and convert them into high dimensional input space. It converts non-separable classes into the separable one, it finds out a way to separate the data on the basis of the data labels defined by us.

Features and Advantages of SVMs :

1. They maximize the margin of the decision boundary using quadratic optimization techniques which find the optimal hyperplane.
2. It has the ability to handle large feature spaces.
3. SVM’s are very good when we have no idea about our data.
4. Works well with even unstructured and semi-structured data like text, Images and trees.
5. The kernel trick is real strength of SVM. With an appropriate kernel function, we can solve any complex problem.
6. It scales relatively well to high dimensional data.
7. SVM models have generalization in practice, the risk of over-fitting is less in SVM.

Limitations of SVM :

1. It is sensitive to noise.
2. The extension of classification to more than two classes is problematic.
3. Choosing a “good” kernel function is not easy.
4. Long training time for large datasets.
5. Difficult to understand and interpret the final model, variable weights and individual impact.
6. Since the final model is not so easy to see, we can not do small calibrations to the model hence its tough to incorporate our business logic.
7. The SVM hyper parameters are Cost -C and gamma. It is not that easy to fine-tune these hyper-parameters. It is hard to visualize their impact

Some applications of SVM :

1. Text (and hypertext) categorization.
2. Image Classification.
3. Bioinformatics (Protein classification, Cancer classification).
4. Hand-written character recognition.
5. Determination of SPAM email.
6. Time Series Analysis.
7. Anomaly Detection.

You can find the code over here.

Random Forest Classifier :

It is an ensemble tree-based learning algorithm. The Random Forest Classifier is a set of decision trees from randomly selected subset of training set. It aggregates the votes from different decision trees to decide the final class of the test object.

Ensemble Algorithm :

Ensemble algorithms are those which combines more than one algorithms of same or different kind for classifying objects. For example, running prediction over Naive Bayes, SVM and Decision Tree and then taking vote for final consideration of class for test object.

Types of Random Forest models:

1. Random Forest Prediction for a classification problem:

f(x) = majority vote of all predicted classes over B trees

2. Random Forest Prediction for a regression problem:

f(x) = sum of all sub-tree predictions divided over B trees

An Example of Random Forest Classification :

The 9 decision tree classifiers shown above can be aggregated into a random forest ensemble which combines their input (on the right). The horizontal and vertical axes of the above decision tree outputs can be thought of as features x1 and x2. At certain values of each feature, the decision tree outputs a classification of “blue”, “green”, “red”, etc.

These above results are aggregated, through model votes or averaging, into a single
ensemble model that ends up outperforming any individual decision tree’s output.

Feature and Advantages of Random Forest :

1. It is one of the most accurate learning algorithms available. For many data sets, it produces a highly accurate classifier.
2. It runs efficiently on large databases.
3. It can handle thousands of input variables without variable deletion.
4. It gives estimates of what variables that are important in the classification.
5. It generates an internal unbiased estimate of the generalization error as the forest building progresses.
6. It has an effective method for estimating missing data and maintains accuracy when a large proportion of the data are missing.

Disadvantages of Random Forest :

1. Random forests have been observed to overfit for some datasets with noisy classification/regression tasks.
2. For data including categorical variables with different number of levels, random forests are biased in favor of those attributes with more levels. Therefore, the variable importance scores from random forest are not reliable for this type of data.

Implementation of Random Forest Classification on real life dataset:

1. Importing Python Libraries and Loading our Dataset into a Data Frame

2. Splitting our dataset into training set and test set

3. Creating a Random Forest Regression model and fitting it to the training data

4. Predicting the test set results and making the Confusion matrix

Conclusion :

In this blog we have learned about the Random forest classifier and its implementation. We looked at the ensembled learning algorithm in action and tried to understand what makes Random Forest different form other machine learning algorithms.

Logistic regression uses an equation as the representation, very much like linear regression. It is not much different from linear regression, except that a Sigmoid function is being fit in the equation of linear regression.

Simple Linear and Multiple Linear Regression Equation:

y = b0 + b1x1 + b2x2 + ... + e

Sigmoid function :

p = 1 / (1 + e ^ -(y))

Logistic Regression Equation :

p = 1 / (1 + e ^ -(b0 + b1x1 + b2x2 +... + e))

where,

p is the probability of outcome

y is the predicted output

b0 is the bias or intercept term

Each column in your input data has an associated b coefficient (a constant real value) that must be learned from your training data.

Difference between Linear Regression and Logistic Regression :

• In Linear Regression the target is an continuous (real value) variable while in Logistic Regression the target is a discrete (binary or ordinal) variable.
• The Predicted values in case of Linear Regression are the mean of the target variable at the given values of the input variables. While the Predicted values in Logistic regression are the probability of a particular level(s) of the target variable at the given values of the input variables.

Types of Logistic Regression:

1. Binary Logistic Regression: The target variable has only two 2 possible outcomes. For example, classifying e-mails as Spam or not Spam.

2. Multinomial Logistic Regression: The target variable has three or more categories without ordering. For example,  predicting which food is preferred more (Veg, Non-Veg, Vegan)

3. Ordinal Logistic Regression: The target variable has three or more categories with ordering. For example, rating for a movie rating from 1 to 5.

Decision Boundary

To predict which class a data belongs, a threshold can be set. Based upon this threshold, the obtained estimated probability is classified into classes. This threshold is called the Decision Boundary.

Say, if predicted_value ≥ 0.5, then classify email as spam else as not spam.

Decision boundary can be linear or non-linear. Polynomial order can be increased to get complex decision boundary.

Advantages of Linear Regression :

1. It makes no assumptions about distributions of classes in feature space.
2. Easily extended to multiple classes (multinomial regression)
3. Natural probabilistic view of class predictions.
4. Quick to train and very fast at classifying unknown records.
5. Good accuracy for many simple data sets.
6. Resistant to overfitting.

Disadvantages of Logistic Regression :

1. It can not handle continuous variables.
2. If independent variables are not correlated with the target variable, then Logistic regression does not work.
3. Requires large sample size for stable results.

Logistic Regression Assumptions :

1. Binary logistic regression requires the dependent variable to be binary.
2. Dependent variables are not measured on a ratio scale.
3. Only the meaningful variables should be included.
4. The independent variables should be independent of each other. That is, the model should have little or no multi-collinearity.
5. Logistic regression requires quite large sample sizes.

A Decision Tree is a simple representation for classifying examples. It is a Supervised Machine Learning where the data is continuously split according to a certain parameter.

Decision Tree consists of :

1. Nodes : Test for the value of a certain attribute.
2. Edges/ Branch : Correspond to the outcome of a test and connect to the next node or leaf.
3. Leaf nodes : Terminal nodes that predict the outcome (represent class labels or class distribution).

To understand the concept of Decision Tree consider the above example. Let’s say you want to predict whether a person is fit or unfit, given their information like age, eating habits, physical activity, etc. The decision nodes are the questions like ‘What’s the age?’, ‘Does he exercise?’, ‘Does he eat a lot of pizzas’? And the leaves represent outcomes like either ‘fit’, or ‘unfit’.

There are two main types of Decision Trees:

1. Classification Trees.
2. Regression Trees.

1. Classification trees (Yes/No types) :

What we’ve seen above is an example of classification tree, where the outcome was a variable like ‘fit’ or ‘unfit’. Here the decision variable is Categorical/ discrete.

Such a tree is built through a process known as binary recursive partitioning. This is an iterative process of splitting the data into partitions, and then splitting it up further on each of the branches.

2. Regression trees (Continuous data types) :

Decision trees where the target variable can take continuous values (typically real numbers) are called regression trees. (e.g. the price of a house, or a patient’s length of stay in a hospital)

Creation of Decision Tree :

In this method a set of training examples is broken down into smaller and smaller subsets while at the same time an associated decision tree get incrementally developed. At the end of the learning process, a decision tree covering the training set is returned.

The key idea is to use a decision tree to partition the data space into cluster (or dense) regions and empty (or sparse) regions.

In Decision Tree Classification a new example is classified by submitting it to a series of tests that determine the class label of the example. These tests are organized in a hierarchical structure called a decision tree. Decision Trees follow Divide-and-Conquer Algorithm.

Divide and Conquer

Decision trees are built using a heuristic called recursive partitioning. This approach is also commonly known as divide and conquer because it splits the data into subsets, which are then split repeatedly into even smaller subsets, and so on and so forth until the process stops when the algorithm determines the data within the subsets are sufficiently homogenous, or another stopping criterion has been met.

Basic Divide-and-Conquer Algorithm :

1. Select a test for root node. Create branch for each possible outcome of the test.
2. Split instances into subsets. One for each branch extending from the node.
3. Repeat recursively for each branch, using only instances that reach the branch.
4. Stop recursion for a branch if all its instances have the same class.

Decision Tree Classifier

• Using the decision algorithm, we start at the tree root and split the data on the feature that results in the largest information gain (IG) (reduction in uncertainty towards the final decision).
• In an iterative process, we can then repeat this splitting procedure at each child node until the leaves are pure. This means that the samples at each leaf node all belong to the same class.
• In practice, we may set a limit on the depth of the tree to prevent overfitting. We compromise on purity here somewhat as the final leaves may still have some impurity.

Advantages of Classification with Decision Trees:

1. Inexpensive to construct.
2. Extremely fast at classifying unknown records.
3. Easy to interpret for small-sized trees
4. Accuracy comparable to other classification techniques for many simple data sets.
5. Excludes unimportant features.

Disadvantages of Classification with Decision Trees:

1. Easy to overfit.
2. Decision Boundary restricted to being parallel to attribute axes.
3. Decision tree models are often biased toward splits on features having a large number of levels.
4. Small changes in the training data can result in large changes to decision logic.
5. Large trees can be difficult to interpret and the decisions they make may seem counter intuitive.

Applications of Decision trees in real life :

1. Biomedical Engineering (decision trees for identifying features to be used in implantable devices).
2. Financial analysis (Customer Satisfaction with a product or service).
3. Astronomy (classify galaxies).
4. System Control.
5. Manufacturing and Production (Quality control, Semiconductor manufacturing, etc).
6. Medicines (diagnosis, cardiology, psychiatry).
7. Physics (Particle detection).

In this blog we’ll talk about one of the most widely used machine learning algorithm for classification, which is the K-Nearest Neighbors (K-NN) algorithm. K-Nearest Neighbor (K-NN) is a simple, easy to understand, versatile and one of the topmost machine learning algorithms that finds its applications in a variety of fields.

What is K-NN ?

K-NN is a non-parametric and lazy learning algorithm. Non-parametric means there is no assumption for underlying data distribution i.e. the model structure determined from the dataset.

It is called Lazy algorithm because it does not need any training data points for model generation. All training data is used in the testing phase which makes training faster and testing phase slower and costlier.

K-Nearest Neighbor (K-NN) is a simple algorithm that stores all the available cases and classifies the new data or case based on a similarity measure.

K-NN classification

In K-NN classification, the output is a class membership. An object is classified by a plurality vote of its neighbors, with the object being assigned to the class most common among its k nearest neighbors (k is a positive integer, typically small). If k = 1, then the object is simply assigned to the class of that single nearest neighbor.

To determine which of the K instances in the training dataset are most similar to a new input, a distance measure is used. For real-valued input variables, the most popular distance measure is the Euclidean distance.

The Euclidean distance

• The Euclidean distance is the most common distance metric used in low dimensional data sets. It is also known as the L² norm. The Euclidean distance is the usual manner in which distance is measured in real world.

where p and q are n-dimensional vectors and denoted by p = (p₁, p₂,…, p_n) and q = (q₁, q₂,…, q_n) represent the n attribute values of two records.

• While Euclidean distance is useful in low dimensions, it doesn’t work well in high dimensions and for categorical variables. The drawback of Euclidean distance is that it ignores the similarity between attributes. Each attribute is treated as totally different from all of the attributes.

Other popular distance measures :

• Hamming Distance: Calculate the distance between binary vectors.
• Manhattan Distance:  Calculate the distance between real vectors using the sum of their absolute difference. Also called City Block Distance.
• Minkowski Distance:  Generalization of Euclidean and Manhattan distance.

Steps to be carries out during the K-NN algorithm are as follows :

1. Divide the data into training and test data.
2. Select a value K.
3. Determine which distance function is to be used.
4. Choose a sample from the test data that needs to be classified and compute the distance to its n training samples.
5. Sort the distances obtained and take the k-nearest data samples.
6. Assign the test class to the class based on the majority vote of its k neighbors.

Performance of the K-NN algorithm is influenced by three main factors :

1. The distance function or distance metric used to determine the nearest neighbors.
2. The decision rule used to derive a classification from the K-nearest neighbors.
3. The number of neighbors used to classify the new example.

Advantages of K-NN :

1. The K-NN algorithm is very easy to implement.
2. Nearly optimal in the large sample limit.
3. Uses local information, which can yield highly adaptive behavior.
4. Lends itself very easily to parallel implementation.

Disadvantages of K-NN :

1. Large storage requirements.
2. Computationally intensive recall.
3. Highly susceptible to the curse of dimensionality.

K-NN Algorithm finds its applications in :

1. Finance – financial institutes will predict the credit rating of customers.
2. Healthcare – gene expression.
3. Political Science – classifying potential voters in two classes will vote or won’t vote.
4. Handwriting detection.
5. Image Recognition.
6. Video Recognition.
7. Pattern Recognition.

In this blog we’ll try to understand one of the most important algorithms in machine learning i.e. Random Forest Algorithm. We will try to look at the things that make Random Forest so special and will try to implement it on a real life dataset.

The Code along with the dataset can be found here.

Ensemble Learning

An Ensemble method is a technique that combines the predictions from multiple machine learning algorithms together to make more accurate predictions than any individual model. A model comprised of many models is called an Ensemble model.

Types of Ensemble Learning:

1. Boosting.
2. Bootstrap Aggregation (Bagging).

1. Boosting

Boosting refers to a group of algorithms that utilize weighted averages to make weak learners into stronger learners. Boosting is all about “teamwork”. Each model that runs, dictates what features the next model will focus on.

In boosting as the name suggests, one is learning from other which in turn boosts the learning.

2. Bootstrap Aggregation (Bagging)

Bootstrap refers to random sampling with replacement. Bootstrap allows us to better understand the bias and the variance with the dataset. Bootstrap involves random sampling of small subset of data from the dataset.

It is a general procedure that can be used to reduce the variance for those algorithm that have high variance, typically decision trees. Bagging makes each model run independently and then aggregates the outputs at the end without preference to any model.

Problems with Decision Trees

Decision trees are sensitive to the specific data on which they are trained. If the training data is changed the resulting decision tree can be quite different and in turn the predictions can be quite different.

Also Decision trees are computationally expensive to train, carry a big risk of overfitting, and tend to find local optima because they can’t go back after they have made a split.

To address these weaknesses, we turn to Random Forest which illustrates the power of combining many decision trees into one model.

Random Forest

Random forest is a Supervised Learning algorithm which uses ensemble learning method for classification and regression.

Random forest is a bagging technique and not a boosting technique. The trees in random forests are run in parallel. There is no interaction between these trees while building the trees.

It operates by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees.

A random forest is a meta-estimator (i.e. it combines the result of multiple predictions) which aggregates many decision trees, with some helpful modifications:

1. The number of features that can be split on at each node is limited to some percentage of the total (which is known as the hyperparameter). This ensures that the ensemble model does not rely too heavily on any individual feature, and makes fair use of all potentially predictive features.
2. Each tree draws a random sample from the original data set when generating its splits, adding a further element of randomness that prevents overfitting.

The above modifications help prevent the trees from being too highly correlated.

For Example, See these nine decision tree classifiers below :

These decision tree classifiers can be aggregated into a random forest ensemble which combines their input. Think of the horizontal and vertical axes of the above decision tree outputs as features x1 and x2. At certain values of each feature, the decision tree outputs a classification of “blue”, “green”, “red”, etc.

These above results are aggregated, through model votes or averaging, into a single
ensemble model that ends up outperforming any individual decision tree’s output.

The aggregated result for the nine decision tree classifiers is shown below :

Feature and Advantages of Random Forest :

1. It is one of the most accurate learning algorithms available. For many data sets, it produces a highly accurate classifier.
2. It runs efficiently on large databases.
3. It can handle thousands of input variables without variable deletion.
4. It gives estimates of what variables that are important in the classification.
5. It generates an internal unbiased estimate of the generalization error as the forest building progresses.
6. It has an effective method for estimating missing data and maintains accuracy when a large proportion of the data are missing.

Disadvantages of Random Forest :

1. Random forests have been observed to overfit for some datasets with noisy classification/regression tasks.
2. For data including categorical variables with different number of levels, random forests are biased in favor of those attributes with more levels. Therefore, the variable importance scores from random forest are not reliable for this type of data.

Implementation of Random Forest Regression on real life dataset:

1. Importing Python Libraries and Loading our Dataset into a Data Frame

2. Splitting our dataset into training set and test set

3. Creating a Random Forest Regression model and fitting it to the training data

For this model I’ve chosen 10 trees (n_estimator=10)

4. Visualizing the Random Forest Regression results

Conclusion :

So here you go, we have learned about the various kinds of ensemble learning algorithms and how these algorithms help in making Random Forest work. We have got an intuitive understanding of the features and advantages of using Random Forest over other Machine Learning algorithms. Apart from it we also saw some of its disadvantages along with the implementation Random Forest Regression on an actual dataset.