This article discusses the basics of Softmax Regression and its implementation in Python using TensorFlow library.

What is Softmax Regression?

Softmax regression (or multinomial logistic regression) is a generalization of logistic regression to the case where we want to handle multiple classes.

A gentle introduction to linear regression can be found here:
Understanding Logistic Regression

In binary logistic regression we assumed that the labels were binary, i.e. for observation,

But consider a scenario where we need to classify an observation out of two or more class labels. For example, digit classification. Here, the possible labels are:

In such cases, we can use Softmax Regression.

Let us first define our model:

• Let the dataset have ‘m’ features and ‘n’ observations. Also, there are ‘k’ class labels, i.e every observation can be classified as one of the ‘k’ possible target values. For example, if we have a dataset of 100 handwritten digit images of vector size 28×28 for digit classification, we have, n = 100, m = 28×28 = 784 and k = 10.

• Feature matrix
  The feature matrix, , is represented as: Here, denotes the values of feature for observation. The matrix has dimensions:

• Weight matrix
  We define a weight matrix, as: Here, represents the weight assigned to feature for class label. The matrix has dimensions:. Initially, weight matrix is filled using some normal distribution.

• Logit score matrix
  Then, we define our net input matrix(also called logit score matrix), , as: The matrix has dimensions: .

  Currently, we are taking an extra column in feature matrix, and an extra row in weight matrix, . These extra columns and rows correspond to the bias terms associated with each prediction. This could be simplified by defining an extra matrix for bias, of size where . (In practice, all we need is a vector of size and some broadcasting techniques for the bias terms!)

  So, the final score matrix, is:

  

  where matrix has dimensions while has dimensions . But matrix still has same value and dimensions!

  But what does matrix signify? Actually, is the likelihood of label j for observation. It is not a proper probability value but can be considered as a score given to each class label for each observation!

  Let us define as the logit score vector for observation.

  For example, let the vector represents the score for each of the class labels in handwritten digit classification problem for observation. Here, the max score is 5.2 which corresponds to class label ‘3’. Hence, our model currently predicts the observation/image as ‘3’.

• Softmax layer
  It is harder to train the model using score values since it is hard to differentiate them while implementing Gradient Descent algorithm for minimizing the cost function. So, we need some function which normalizes the logit scores as well as makes them easily differentiable!In order to convert the score matrix to probabilities, we use Softmax function.For a vector , softmax function is defined as: So, softmax function will do 2 things:

  1. convert all scores to probabilities.
  2. sum of all probabilities is 1.
  

  Diagram given below explains how the softmax function works:

  

  Recall that in Binary Logistic classifier, we used sigmoid function for the same task. Softmax function is nothing but a generalization of sigmoid function! Now, this softmax function computes the probability that the training sample belongs to class given the logits vector as:

  In vector form, we can simply write:

  For simplicity, let denote the softmax probability vector for observation.

• One-hot encoded target vector
  Since softmax function provides us with a vector of probabilities of each class label for a given observation, we need to convert corresponding target label in the same format to calculate the cost function!Corresponding to each observation, we create a target vector composed of only zeros and ones where only correct class label is set as 1. This technique is called one-hot encoding.

  See the diagram given below for a better understanding:

  Now, we denote one-hot encoding vector for observation as

• Cost function
  Now, we need to define a cost function for which, we have to compare the softmax probabilities and one-hot encoded target vector for similarity. We use the concept of Cross-Entropy for the same. The Cross-entropy is a distance calculation function which takes the calculated probabilities from softmax function and the created one-hot-encoding matrix to calculate the distance. For the right target classes, the distance values will be lesser, and the distance values will be larger for the wrong target classes.We define cross entropy, for observation with softmax probability vector, and one-hot target vector, as: And now, cost function, can be defined as the average cross entropy, i.e: and the task is to minimize this cost function!

• Gradient Descent algorithm
  In order to learn our softmax model via gradient descent, we need to compute the derivative: and which we then use to update the weights and biases in opposite direction of the gradient: and for each class where and is learning rate.Using this cost gradient, we iteratively update the weight matrix until we reach a specified number of epochs (passes over the training set) or reach the desired cost threshold.

Implementation

Let us now implement Softmax Regression on the MNIST handwritten digit dataset using TensorFlow library.

For a gentle introduction to TensorFlow, follow this tutorial:
Introduction to TensorFlow

Step 1: Import the dependencies

First of all, we import the dependencies.

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

Step 2: Download the data

TensorFlow allows you to download and read in the MNIST data automatically. Consider the code given below. It will download and save data to the folder, MNIST_data, in your current project directory and load it in current program.

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

Extracting MNIST_data/train-images-idx3-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/t10k-images-idx3-ubyte.gz
Extracting MNIST_data/t10k-labels-idx1-ubyte.gz

Step 3: Understanding data

Now, we try to understand the structure of the dataset.

The MNIST data is split into three parts: 55,000 data points of training data (mnist.train), 10,000 points of test data (mnist.test), and 5,000 points of validation data (mnist.validation).

Each image is 28 pixels by 28 pixels which has been flattened into 1-D numpy array of size 784. Number of class labels is 10. Each target label is already provided in one-hot encoded form.

print("Shape of feature matrix:", mnist.train.images.shape)
print("Shape of target matrix:", mnist.train.labels.shape)
print("One-hot encoding for 1st observation:\n", mnist.train.labels[0])

# visualize data by plotting images
fig,ax = plt.subplots(10,10)
k = 0
for i in range(10):
    for j in range(10):
        ax[i][j].imshow(mnist.train.images[k].reshape(28,28), aspect='auto')
        k += 1
plt.show()

Output:

Shape of feature matrix: (55000, 784)
Shape of target matrix: (55000, 10)
One-hot encoding for 1st observation:
 [ 0.  0.  0.  0.  0.  0.  0.  1.  0.  0.]

Step 4: Defining computation graph

Now, we create a computation graph.

# number of features
num_features = 784
# number of target labels
num_labels = 10
# learning rate (alpha)
learning_rate = 0.05
# batch size
batch_size = 128
# number of epochs
num_steps = 5001

# input data
train_dataset = mnist.train.images
train_labels = mnist.train.labels
test_dataset = mnist.test.images
test_labels = mnist.test.labels
valid_dataset = mnist.validation.images
valid_labels = mnist.validation.labels

# initialize a tensorflow graph
graph = tf.Graph()

with graph.as_default():
    """
    defining all the nodes
    """

    # Inputs
    tf_train_dataset = tf.placeholder(tf.float32, shape=(batch_size, num_features))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)

    # Variables.
    weights = tf.Variable(tf.truncated_normal([num_features, num_labels]))
    biases = tf.Variable(tf.zeros([num_labels]))

    # Training computation.
    logits = tf.matmul(tf_train_dataset, weights) + biases
    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
                        labels=tf_train_labels, logits=logits))

    # Optimizer.
    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

    # Predictions for the training, validation, and test data.
    train_prediction = tf.nn.softmax(logits)
    valid_prediction = tf.nn.softmax(tf.matmul(tf_valid_dataset, weights) + biases)
    test_prediction = tf.nn.softmax(tf.matmul(tf_test_dataset, weights) + biases)

Some important points to note:

• For the training data, we use a placeholder that will be fed at run time with a training minibatch. The technique of using minibatches for training model using gradient descent is termed as Stochastic Gradient Descent.
  

  In both gradient descent (GD) and stochastic gradient descent (SGD), you update a set of parameters in an iterative manner to minimize an error function. While in GD, you have to run through ALL the samples in your training set to do a single update for a parameter in a particular iteration, in SGD, on the other hand, you use ONLY ONE or SUBSET of training sample from your training set to do the update for a parameter in a particular iteration. If you use SUBSET, it is called Minibatch Stochastic gradient Descent. Thus, if the number of training samples are large, in fact very large, then using gradient descent may take too long because in every iteration when you are updating the values of the parameters, you are running through the complete training set. On the other hand, using SGD will be faster because you use only one training sample and it starts improving itself right away from the first sample. SGD often converges much faster compared to GD but the error function is not as well minimized as in the case of GD. Often in most cases, the close approximation that you get in SGD for the parameter values are enough because they reach the optimal values and keep oscillating there.

• The weight matrix is initialized using random values following a (truncated) normal distribution. This is achieved using tf.truncated_normal method. The biases get initialized to zero using tf.zeros method.
• Now, we multiply the inputs with the weight matrix, and add biases. We compute the softmax and cross-entropy using tf.nn.softmax_cross_entropy_with_logits (it’s one operation in TensorFlow, because it’s very common, and it can be optimized). We take the average of this cross-entropy across all training examples using tf.reduce_mean method.
• We are going to minimize the loss using gradient descent. For this, we use tf.train.GradientDescentOptimizer.
• train_prediction, valid_prediction and test_prediction are not part of training, but merely here so that we can report accuracy figures as we train.

Step 5: Running the computation graph

Since we have already built the computation graph, now it’s time to run it through a session.

# utility function to calculate accuracy
def accuracy(predictions, labels):
    correctly_predicted = np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))
    accu = (100.0 * correctly_predicted) / predictions.shape[0]
    return accu

with tf.Session(graph=graph) as session:
    # initialize weights and biases
    tf.global_variables_initializer().run()
    print("Initialized")

    for step in range(num_steps):
        # pick a randomized offset
        offset = np.random.randint(0, train_labels.shape[0] - batch_size - 1)

        # Generate a minibatch.
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]

        # Prepare the feed dict
        feed_dict = {tf_train_dataset : batch_data,
                     tf_train_labels : batch_labels}

        # run one step of computation
        _, l, predictions = session.run([optimizer, loss, train_prediction],
                                        feed_dict=feed_dict)

        if (step % 500 == 0):
            print("Minibatch loss at step {0}: {1}".format(step, l))
            print("Minibatch accuracy: {:.1f}%".format(
                accuracy(predictions, batch_labels)))
            print("Validation accuracy: {:.1f}%".format(
                accuracy(valid_prediction.eval(), valid_labels)))

    print("\nTest accuracy: {:.1f}%".format(
        accuracy(test_prediction.eval(), test_labels)))

Output:

Initialized
Minibatch loss at step 0: 11.68728256225586
Minibatch accuracy: 10.2%
Validation accuracy: 14.3%
Minibatch loss at step 500: 2.239773750305176
Minibatch accuracy: 46.9%
Validation accuracy: 67.6%
Minibatch loss at step 1000: 1.0917563438415527
Minibatch accuracy: 78.1%
Validation accuracy: 75.0%
Minibatch loss at step 1500: 0.6598564386367798
Minibatch accuracy: 78.9%
Validation accuracy: 78.6%
Minibatch loss at step 2000: 0.24766433238983154
Minibatch accuracy: 91.4%
Validation accuracy: 81.0%
Minibatch loss at step 2500: 0.6181786060333252
Minibatch accuracy: 84.4%
Validation accuracy: 82.5%
Minibatch loss at step 3000: 0.9605385065078735
Minibatch accuracy: 85.2%
Validation accuracy: 83.9%
Minibatch loss at step 3500: 0.6315320730209351
Minibatch accuracy: 85.2%
Validation accuracy: 84.4%
Minibatch loss at step 4000: 0.812285840511322
Minibatch accuracy: 82.8%
Validation accuracy: 85.0%
Minibatch loss at step 4500: 0.5949224233627319
Minibatch accuracy: 80.5%
Validation accuracy: 85.6%
Minibatch loss at step 5000: 0.47554320096969604
Minibatch accuracy: 89.1%
Validation accuracy: 86.2%

Test accuracy: 86.5%

Some important points to note:

• In every iteration, a minibatch is selected by choosing a random offset value using np.random.randint method.
• To feed the placeholders tf_train_dataset and tf_train_label, we create a feed_dict like this:

  feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
  

• A shortcut way of performing one step of computation is:

  _, l, predictions = session.run([optimizer, loss, train_prediction], feed_dict=feed_dict)
  

  This node returns the new values of loss and predictions after performing optimization step.

This brings us to the end of the implementation. Complete code can be found here.

Finally, here are some points to ponder upon:

• You can try to tweak with the parameters like learning rate, batch size, number of epochs, etc and achieve better results. You can also try a different optimizer like tf.train.AdamOptimizer.
• Accuracy of above model can be improved by using a neural network with one or more hidden layers. We will discuss its implementation using TensorFlow in some upcoming articles.
• Softmax Regression vs. k Binary Classifiers
  One should be aware of the scenarios where softmax regression works and where it doesn’t. In many cases, you may need to use k different binary logistic classifiers for each of the k possible values of the class label.Suppose you are working on a computer vision problem where you’re trying to classify images into three different classes:
  Case 1: Suppose that your classes are indoor_scene, outdoor_urban_scene, and outdoor_wilderness_scene.

  Case 2: Suppose your classes are indoor_scene, black_and_white_image, and image_has_people.

  In which case you would use Softmax Regression classifier and in which case you would use 3 Binary Logistic Regression classifiers?
  This will depend on whether the 3 classes are mutually exclusive.
  In case 1, a scene can be either indoor_scene, outdoor_urban_scene or outdoor_wilderness_scene. So, assuming that each training example is labeled with exactly one of the 3 classes, we should build a softmax classifier with k = 3.

  However, in case 2, the classes are not mutually exclusive since a scene can be both indoor and have people in it. So, in this case, it would be more appropriate to build 3 binary logistic regression classifiers. This way, for each new scene, your algorithm can separately decide whether it falls into each of the 3 categories.

References:

• http://www.kdnuggets.com/2016/07/softmax-regression-related-logistic-regression.html
• https://classroom.udacity.com/courses/ud730
• http://ufldl.stanford.edu/wiki/index.php/Softmax_Regression

This article is a brief introduction to TensorFlow library using Python programming language.

Introduction

TensorFlow is an open-source software library. TensorFlow was originally developed by researchers and engineers working on the Google Brain Team within Google’s Machine Intelligence research organization for the purposes of conducting machine learning and deep neural networks research, but the system is general enough to be applicable in a wide variety of other domains as well!

Let us first try to understand what the word TensorFlow actually mean!

TensorFlow is basically a software library for numerical computation using data flow graphs where:

• nodes in the graph represent mathematical operations.
• edges in the graph represent the multidimensional data arrays (called tensors) communicated between them. (Please note that tensor is the central unit of data in TensorFlow).

Consider the diagram given below:

Here, add is a node which represents addition operation. a and b are input tensors and c is the resultant tensor.

This flexible architecture allows you to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API!

TensorFlow APIs

TensorFlow provides multiple APIs (Application Programming Interfaces). These can be classified into 2 major categories:

1. Low level API:
   • complete programming control
   • recommended for machine learning researchers
   • provides fine levels of control over the models
   • TensorFlow Core is the low level API of TensorFlow.
2. High level API:
   • built on top of TensorFlow Core
   • easier to learn and use than TensorFlow Core
   • make repetitive tasks easier and more consistent between different users
   • tf.contrib.learn is an example of a high level API.

In this article, we first discuss the basics of TensorFlow Core and then explore the higher level API, tf.contrib.learn.

TensorFlow Core

1. Installing TensorFlow

An easy to follow guide for TensorFlow installation is available here:
Installing TensorFlow.

Once installed, you can ensure a successful installation by running this command in python interpreter:

import tensorflow as tf

2. The Computational Graph

Any TensorFlow Core program can be divided into two discrete sections:

• Building the computational graph.A computational graph is nothing but a series of TensorFlow operations arranged into a graph of nodes.

• Running the computational graph.To actually evaluate the nodes, we must run the computational graph within a session. A session encapsulates the control and state of the TensorFlow runtime.

Now, let us write our very first TensorFlow program to understand above concept:

# importing tensorflow
import tensorflow as tf

# creating nodes in computation graph
node1 = tf.constant(3, dtype=tf.int32)
node2 = tf.constant(5, dtype=tf.int32)
node3 = tf.add(node1, node2)

# create tensorflow session object
sess = tf.Session()

# evaluating node3 and printing the result
print("Sum of node1 and node2 is:",sess.run(node3))

# closing the session
sess.close()

Output:

Sum of node1 and node2 is: 8

Let us try to understand above code:

• Step 1 : Create a computational graph
  By creating computational graph, we mean defining the nodes. Tensorflow provides different types of nodes for a variety of tasks. Each node takes zero or more tensors as inputs and produces a tensor as an output.

• In above program, the nodes node1 and node2 are of tf.constant type. A constant node takes no inputs, and it outputs a value it stores internally. Note that we can also specify the data type of output tensor using dtype argument.

  node1 = tf.constant(3, dtype=tf.int32)
  node2 = tf.constant(5, dtype=tf.int32)
  

• node3 is of tf.add type. It takes two tensors as input and returns their sum as output tensor.

  node3 = tf.add(node1, node2)
  

• Step 2 : Run the computational graph
  In order to run the computational graph, we need to create a session. To create a session, we simply do:

  sess = tf.Session()
  

  Now, we can invoke the run method of session object to perform computations on any node:

  print("Sum of node1 and node2 is:",sess.run(node3))
  

  Here, node3 gets evaluated which further invokes node1 and node2. Finally, we close the session using:

  sess.close()
  

Note: Another(and better) method of working with sessions is to use with block like this:

with tf.Session() as sess:
    print("Sum of node1 and node2 is:",sess.run(node3))

The benefit of this approach is that you do not need to close the session explicitly as it gets automatically closed once control goes out of the scope of with block.

3. Variables

TensorFlow has Variable nodes too which can hold variable data. They are mainly used to hold and update parameters of a training model.

Variables are in-memory buffers containing tensors. They must be explicitly initialized and can be saved to disk during and after training. You can later restore saved values to exercise or analyze the model.

An important difference to note between a constant and Variable is:

A constant’s value is stored in the graph and its value is replicated wherever the graph is loaded. A variable is stored separately, and may live on a parameter server.

Given below is an example using Variable:

# importing tensorflow
import tensorflow as tf

# creating nodes in computation graph
node = tf.Variable(tf.zeros([2,2]))

# running computation graph
with tf.Session() as sess:
    # initialize all global variables
    sess.run(tf.global_variables_initializer())
    # evaluating node
    print("Tensor value before addition:\n",sess.run(node))
    # elementwise addition to tensor
    node = node.assign(node + tf.ones([2,2]))
    # evaluate node again
    print("Tensor value after addition:\n", sess.run(node))

Output:

Tensor value before addition:
 [[ 0.  0.]
 [ 0.  0.]]
Tensor value after addition:
 [[ 1.  1.]
 [ 1.  1.]]

In above program:

• We define a node of type Variable and assign it some initial value.

  node = tf.Variable(tf.zeros([2,2]))
  

• To initialize the variable node in current session’s scope, we do:

  sess.run(tf.global_variables_initializer())
  

• To assign a new value to a variable node, we can use assign method like this:

  node = node.assign(node + tf.ones([2,2]))
  

4. Placeholders

A graph can be parameterized to accept external inputs, known as placeholders. A placeholder is a promise to provide a value later.

While evaluating the graph involving placeholder nodes, a feed_dict parameter is passed to the session’s run method to specify Tensors that provide concrete values to these placeholders.

Consider the example given below:

# importing tensorflow
import tensorflow as tf

# creating nodes in computation graph
a = tf.placeholder(tf.int32, shape=(3,1))
b = tf.placeholder(tf.int32, shape=(1,3))
c = tf.matmul(a,b)

# running computation graph
with tf.Session() as sess:
    print(sess.run(c, feed_dict={a:[[3],[2],[1]], b:[[1,2,3]]}))

Output:

[[3 6 9]
 [2 4 6]
 [1 2 3]]

Let us try to understand above program:

• We define placeholder nodes a and b like this:

  a = tf.placeholder(tf.int32, shape=(3,1))
  b = tf.placeholder(tf.int32, shape=(1,3))
  

  The first argument is the data type of the tensor and one of the optional argument is shape of the tensor.

• We define another node c which does the operation of matrix multiplication (matmul). We pass the two placeholder nodes as argument.

  c = tf.matmul(a,b)
  

• Finally, when we run the session, we pass the value of placeholder nodes in feed_dict argument of sess.run:

  print(sess.run(c, feed_dict={a:[[3],[2],[1]], b:[[1,2,3]]}))
  

  Consider the diagrams shown below to clear the concept:

• Initially:
  
• After sess.run:
  

5. An example : Linear Regression model

Given below is an implementation of a Linear Regression model using TensorFlow Core API.

# importing the dependencies
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

# Model Parameters
learning_rate = 0.01
training_epochs = 2000
display_step = 200

# Training Data
train_X = np.asarray([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
                         7.042,10.791,5.313,7.997,5.654,9.27,3.1])
train_y = np.asarray([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
                         2.827,3.465,1.65,2.904,2.42,2.94,1.3])
n_samples = train_X.shape[0]

# Test Data
test_X = np.asarray([6.83, 4.668, 8.9, 7.91, 5.7, 8.7, 3.1, 2.1])
test_y = np.asarray([1.84, 2.273, 3.2, 2.831, 2.92, 3.24, 1.35, 1.03])

# Set placeholders for feature and target vectors
X = tf.placeholder(tf.float32)
y = tf.placeholder(tf.float32)

# Set model weights and bias
W = tf.Variable(np.random.randn(), name="weight")
b = tf.Variable(np.random.randn(), name="bias")

# Construct a linear model
linear_model = W*X + b

# Mean squared error
cost = tf.reduce_sum(tf.square(linear_model - y)) / (2*n_samples)

# Gradient descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

# Initializing the variables
init = tf.global_variables_initializer()

# Launch the graph
with tf.Session() as sess:
    # Load initialized variables in current session
    sess.run(init)

    # Fit all training data
    for epoch in range(training_epochs):
        # perform gradient descent step
        sess.run(optimizer, feed_dict={X: train_X, y: train_y})

        # Display logs per epoch step
        if (epoch+1) % display_step == 0:
            c = sess.run(cost, feed_dict={X: train_X, y: train_y})
            print("Epoch:{0:6} \t Cost:{1:10.4} \t W:{2:6.4} \t b:{3:6.4}".
                  format(epoch+1, c, sess.run(W), sess.run(b)))

    # Print final parameter values
    print("Optimization Finished!")
    training_cost = sess.run(cost, feed_dict={X: train_X, y: train_y})
    print("Final training cost:", training_cost, "W:", sess.run(W), "b:",
          sess.run(b), '\n')

    # Graphic display
    plt.plot(train_X, train_y, 'ro', label='Original data')
    plt.plot(train_X, sess.run(W) * train_X + sess.run(b), label='Fitted line')
    plt.legend()
    plt.show()

    # Testing the model
    testing_cost = sess.run(tf.reduce_sum(tf.square(linear_model - y)) / (2 * test_X.shape[0]),
                            feed_dict={X: test_X, y: test_y})

    print("Final testing cost:", testing_cost)
    print("Absolute mean square loss difference:", abs(training_cost - testing_cost))

    # Display fitted line on test data
    plt.plot(test_X, test_y, 'bo', label='Testing data')
    plt.plot(train_X, sess.run(W) * train_X + sess.run(b), label='Fitted line')
    plt.legend()
    plt.show()

Epoch:   200 	 Cost:    0.1715 	 W: 0.426 	 b:-0.4371
Epoch:   400 	 Cost:    0.1351 	 W:0.3884 	 b:-0.1706
Epoch:   600 	 Cost:    0.1127 	 W:0.3589 	 b:0.03849
Epoch:   800 	 Cost:   0.09894 	 W:0.3358 	 b:0.2025
Epoch:  1000 	 Cost:   0.09047 	 W:0.3176 	 b:0.3311
Epoch:  1200 	 Cost:   0.08526 	 W:0.3034 	 b:0.4319
Epoch:  1400 	 Cost:   0.08205 	 W:0.2922 	 b:0.5111
Epoch:  1600 	 Cost:   0.08008 	 W:0.2835 	 b:0.5731
Epoch:  1800 	 Cost:   0.07887 	 W:0.2766 	 b:0.6218
Epoch:  2000 	 Cost:   0.07812 	 W:0.2712 	 b:  0.66
Optimization Finished!
Final training cost: 0.0781221 W: 0.271219 b: 0.65996 



Final testing cost: 0.0756337
Absolute mean square loss difference: 0.00248838

Let us try to understand the above code.

• First of all, we define some parameters for training our model, like:

  learning_rate = 0.01
  training_epochs = 2000
  display_step = 200
  

• Then we define placeholder nodes for feature and target vector.

  X = tf.placeholder(tf.float32)
  y = tf.placeholder(tf.float32)
  

• Then, we define variable nodes for weight and bias.

  W = tf.Variable(np.random.randn(), name="weight")
  b = tf.Variable(np.random.randn(), name="bias")
  

• linear_model is an operational node which calculates the hypothesis for the linear regression model.

  linear_model = W*X + b
  

• Loss (or cost) per gradient descent is calculated as the mean squared error and its node is defined as:

  cost = tf.reduce_sum(tf.square(linear_model - y)) / (2*n_samples)
  

• Finally, we have the optimizer node which implements the Gradient Descent Algorithm.

  optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
  

• Now, the training data is fit into the linear model by applying the Gradient Descent Algorithm. The task is repeated training_epochs number of times. In each epoch, we perform the gradient descent step like this:

  sess.run(optimizer, feed_dict={X: train_X, y: train_y})
  

• After every display_step number of epochs, we print the value of current loss which is found using:

  c = sess.run(cost, feed_dict={X: train_X, y: train_y})
  

• The model is evaluated on test data and testing_cost is calculated using:

  testing_cost = sess.run(tf.reduce_sum(tf.square(linear_model - y)) / (2 * test_X.shape[0]),
                            feed_dict={X: test_X, y: test_y})
  

tf.contrib.learn

tf.contrib.learn is a high-level TensorFlow library that simplifies the mechanics of machine learning, including the following:

• running training loops
• running evaluation loops
• managing data sets
• managing feeding

Let us try to see the implementation of Linear regression on same data we used above using tf.contrib.learn.

# importing the dependencies
import tensorflow as tf
import numpy as np

# declaring list of features
features = [tf.contrib.layers.real_valued_column("X")]

# creating a linear regression estimator
estimator = tf.contrib.learn.LinearRegressor(feature_columns=features)

# training and test data
train_X = np.asarray([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
                         7.042,10.791,5.313,7.997,5.654,9.27,3.1])
train_y = np.asarray([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
                         2.827,3.465,1.65,2.904,2.42,2.94,1.3])
test_X = np.asarray([6.83, 4.668, 8.9, 7.91, 5.7, 8.7, 3.1, 2.1])
test_y = np.asarray([1.84, 2.273, 3.2, 2.831, 2.92, 3.24, 1.35, 1.03])

# function to feed dict of numpy arrays into the model for training
input_fn = tf.contrib.learn.io.numpy_input_fn({"X":train_X}, train_y,
                                              batch_size=4, num_epochs=2000)

# function to feed dict of numpy arrays into the model for testing
test_input_fn = tf.contrib.learn.io.numpy_input_fn({"X":test_X}, test_y)

# fit training data into estimator
estimator.fit(input_fn=input_fn)

# print value of weight and bias
W = estimator.get_variable_value('linear/X/weight')[0][0]
b = estimator.get_variable_value('linear/bias_weight')[0]
print("W:", W, "\tb:", b)

# evaluating the final loss
train_loss = estimator.evaluate(input_fn=input_fn)['loss']
test_loss = estimator.evaluate(input_fn=test_input_fn)['loss']
print("Final training loss:", train_loss)
print("Final testing loss:", test_loss)

W: 0.252928 	b: 0.802972
Final training loss: 0.153998
Final testing loss: 0.0777036

Let us try to understand the above code.

• The shape and type of feature matrix is declared using a list. Each element of the list defines the structure of a column. In above example, we have only 1 feature which stores real values and has been given a name X.

  features = [tf.contrib.layers.real_valued_column("X")]
  

• Then, we need an estimator. An estimator is nothing but a pre-defined model with many useful methods and parameters. In above example, we use a Linear Regression model estimator.

  estimator = tf.contrib.learn.LinearRegressor(feature_columns=features)
  

• For training purpose, we need to use an input function which is responsible for feeding data to estimator while training. It takes the feature column values as dictionary. Many other parameters like batch size, number of epochs, etc can be specified.

  input_fn = tf.contrib.learn.io.numpy_input_fn({"X":train_X}, 
                train_y, batch_size=4, num_epochs=2000)
  

• To fit training data to estimator, we simply use fit method of estimator in which input function is passed as an argument.

  estimator.fit(input_fn=input_fn)
  

• Once training is complete, we can get the value of different variables using get_variable_value method of estimator. You can get a list of all variables using get_variable_names method.

  W = estimator.get_variable_value('linear/X/weight')[0][0]
  b = estimator.get_variable_value('linear/bias_weight')[0]
  

• The mean squared error/loss can be computed as:

  train_loss = estimator.evaluate(input_fn=input_fn)['loss']
  test_loss = estimator.evaluate(input_fn=test_input_fn)['loss']
  

This brings us to the end of this Introduction to TensorFlow article!

From here, you can try to explore this tutorial: MNIST For ML Beginners.

I will upload a few more tutorials on TensorFlow soon!

This article discusses the basics of Logistic Regression and its implementation in Python. Logistic regression is basically a supervised classification algorithm.

In a classification problem, the target variable(or output), y, can take only discrete values for given set of features(or inputs), X.

We can also say that the target variable is categorical. Based on the number of categories, Logistic regression can be classified as:

• binomial: target variable can have only 2 possible types: “0” or “1” which may represent “win” vs “loss”, “pass” vs “fail”, “dead” vs “alive”, etc.
• multinomial: target variable can have 3 or more possible types which are not ordered(i.e. types have no quantitative significance) like “disease A” vs “disease B” vs “disease C”.
• ordinal: it deals with target variables with ordered categories. For example, a test score can be categorized as:“very poor”, “poor”, “good”, “very good”. Here, each category can be given a score like 0, 1, 2, 3.

First of all, we explore the simplest form of Logistic Regression, i.e Binomial Logistic Regression.

Binomial Logistic Regression

Consider an example dataset which maps the number of hours of study with the result of an exam. The result can take only two values, namely passed(1) or failed(0):

Hours(x)	0.50	0.75	1.00	1.25	1.50	1.75	2.00	2.25	2.50	2.75	3.00	3.25	3.50	3.75	4.00	4.25	4.50	4.75	5.00	5.50
Pass(y)	0	0	0	0	0	0	1	0	1	0	1	0	1	0	1	1	1	1	1	1

So, we have

i.e. y is a categorical target variable which can take only two possible type:“0” or “1”.
In order to generalize our model, we assume that:

• The dataset has ‘p’ feature variables and ‘n’ observations.
• The feature matrix is represented as:

Here, denotes the values of feature for observation.
Here, we are keeping the convention of letting = 1. (Keep reading, you will understand the logic in a few moments).

The observation, , can be represented as:

represents the predicted response for observation, i.e. . The formula we use for calculating is called hypothesis.

If you have gone though Linear Regression, you should recall that in Linear Regression, the hypothesis we used for prediction was:

where, are the regression coefficients.
Let regression coefficient matrix/vector, be:

Then, in a more compact form,

The reason for taking = 1 is pretty clear now.

We needed to do a matrix product, but there was no actual multiplied to in original hypothesis formula. So, we defined = 1.

Now, if we try to apply Linear Regression on above problem, we are likely to get continuous values using the hypothesis we discussed above. Also, it does not make sense for to take values larger that 1 or smaller than 0.
So, some modifications are made to the hypothesis for classification:

where,

is called logistic function or the sigmoid function.
Here is a plot showing g(z):

We can infer from above graph that:

• g(z) tends towards 1 as
• g(z) tends towards 0 as
• g(z) is always bounded between 0 and 1

So, now, we can define conditional probabilities for 2 labels(0 and 1) for observation as:

We can write it more compactly as:

Now, we define another term, likelihood of parameters as:

Likelihood is nothing but the probability of data(training examples), given a model and specific parameter values(here, ). It measures the support provided by the data for each possible value of the . We obtain it by multiplying all for given .

And for easier calculations, we take log likelihood:

The cost function for logistic regression is proportional to inverse of likelihood of parameters. Hence, we can obtain an expression for cost function, J using log likelihood equation as:

and our aim is to estimate so that cost function is minimized !!

Using Gradient descent algorithm

Firstly, we take partial derivatives of w.r.t each to derive the stochastic gradient descent rule(we present only the final derived value here):

Here, y and h(x) represent the response vector and predicted response vector(respectively). Also, is the vector representing the observation values for feature.
Now, in order to get min ,

where is called learning rate and needs to be set explicitly.
Let us see the python implementation of above technique on a sample dataset (download it from here):

import csv
import numpy as np
import matplotlib.pyplot as plt
 
 
def loadCSV(filename):
    '''
    function to load dataset
    '''
    with open(filename,"r") as csvfile:
        lines = csv.reader(csvfile)
        dataset = list(lines)
        for i in range(len(dataset)):
            dataset[i] = [float(x) for x in dataset[i]]     
    return np.array(dataset)
 
 
def normalize(X):
    '''
    function to normalize feature matrix, X
    '''
    mins = np.min(X, axis = 0)
    maxs = np.max(X, axis = 0)
    rng = maxs - mins
    norm_X = 1 - ((maxs - X)/rng)
    return norm_X
 
 
def logistic_func(beta, X):
    '''
    logistic(sigmoid) function
    '''
    return 1.0/(1 + np.exp(-np.dot(X, beta.T)))
 
 
def log_gradient(beta, X, y):
    '''
    logistic gradient function
    '''
    first_calc = logistic_func(beta, X) - y.reshape(X.shape[0], -1)
    final_calc = np.dot(first_calc.T, X)
    return final_calc
 
 
def cost_func(beta, X, y):
    '''
    cost function, J
    '''
    log_func_v = logistic_func(beta, X)
    y = np.squeeze(y)
    step1 = y * np.log(log_func_v)
    step2 = (1 - y) * np.log(1 - log_func_v)
    final = -step1 - step2
    return np.mean(final)
 
 
def grad_desc(X, y, beta, lr=.01, converge_change=.001):
    '''
    gradient descent function
    '''
    cost = cost_func(beta, X, y)
    change_cost = 1
    num_iter = 1
     
    while(change_cost > converge_change):
        old_cost = cost
        beta = beta - (lr * log_gradient(beta, X, y))
        cost = cost_func(beta, X, y)
        change_cost = old_cost - cost
        num_iter += 1
     
    return beta, num_iter 
 
 
def pred_values(beta, X):
    '''
    function to predict labels
    '''
    pred_prob = logistic_func(beta, X)
    pred_value = np.where(pred_prob >= .5, 1, 0)
    return np.squeeze(pred_value)
 
 
def plot_reg(X, y, beta):
    '''
    function to plot decision boundary
    '''
    # labelled observations
    x_0 = X[np.where(y == 0.0)]
    x_1 = X[np.where(y == 1.0)]
     
    # plotting points with diff color for diff label
    plt.scatter(x_0[:, 1], x_0[:, 2], c='b', label='y = 0')
    plt.scatter(x_1[:, 1], x_1[:, 2], c='r', label='y = 1')
     
    # plotting decision boundary
    x1 = np.arange(0, 1, 0.1)
    x2 = -(beta[0,0] + beta[0,1]*x1)/beta[0,2]
    plt.plot(x1, x2, c='k', label='reg line')
 
    plt.xlabel('x1')
    plt.ylabel('x2')
    plt.legend()
    plt.show()
     
 
     
if __name__ == "__main__":
    # load the dataset
    dataset = loadCSV('dataset1.csv')
     
    # normalizing feature matrix
    X = normalize(dataset[:, :-1])
     
    # stacking columns wth all ones in feature matrix
    X = np.hstack((np.matrix(np.ones(X.shape[0])).T, X))
 
    # response vector
    y = dataset[:, -1]
 
    # initial beta values
    beta = np.matrix(np.zeros(X.shape[1]))
 
    # beta values after running gradient descent
    beta, num_iter = grad_desc(X, y, beta)
 
    # estimated beta values and number of iterations
    print("Estimated regression coefficients:", beta)
    print("No. of iterations:", num_iter)
 
    # predicted labels
    y_pred = pred_values(beta, X)
     
    # number of correctly predicted labels
    print("Correctly predicted labels:", np.sum(y == y_pred))
     
    # plotting regression line
    plot_reg(X, y, beta)

Estimated regression coefficients: [[  1.70474504  15.04062212 -20.47216021]]
No. of iterations: 2612
Correctly predicted labels: 100

Note: Gradient descent is one of the many way to estimate .
Basically, these are more advanced algorithms which can be easily run in Python once you have defined your cost function and your gradients. These algorithms are:

• BFGS(Broyden–Fletcher–Goldfarb–Shanno algorithm)
• L-BFGS(Like BFGS but uses limited memory)
• Conjugate Gradient

Advantages/disadvantages of using any one of these algorithms over Gradient descent:

• Advantages
  • Don’t need to pick learning rate
  • Often run faster (not always the case)
  • Can numerically approximate gradient for you (doesn’t always work out well)
• Disadvantages
  • More complex
  • More of a black box unless you learn the specifics

Multinomial Logistic Regression

In Multinomial Logistic Regression, the output variable can have more than two possible discrete outputs.

Consider the Digit Dataset. Here, the output variable is the digit value which can take values out of (0, 12, 3, 4, 5, 6, 7, 8, 9).

Given below is the implementation of Multinomial Logisitc Regression using scikit-learn to make predictions on digit dataset.

from sklearn import datasets, linear_model, metrics
  
# load the digit dataset
digits = datasets.load_digits()
  
# defining feature matrix(X) and response vector(y)
X = digits.data
y = digits.target
 
# splitting X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4,
                                                    random_state=1)
  
# create logistic regression object
reg = linear_model.LogisticRegression()
  
# train the model using the training sets
reg.fit(X_train, y_train)
 
# making predictions on the testing set
y_pred = reg.predict(X_test)
  
# comparing actual response values (y_test) with predicted response values (y_pred)
print("Logistic Regression model accuracy(in %):", 
metrics.accuracy_score(y_test, y_pred)*100)

Logistic Regression model accuracy(in %): 95.6884561892

At last, here are some points about Logistic regression to ponder upon:

• Does NOT assume a linear relationship between the dependent variable and the independent variables, but it does assume linear relationship between the logit of the explanatory variables and the response.
• Independent variables can be even the power terms or some other nonlinear transformations of the original independent variables.
• The dependent variable does NOT need to be normally distributed, but it typically assumes a distribution from an exponential family (e.g. binomial, Poisson, multinomial, normal,…); binary logistic regression assume binomial distribution of the response.
• The homogeneity of variance does NOT need to be satisfied.
• Errors need to be independent but NOT normally distributed.
• It uses maximum likelihood estimation (MLE) rather than ordinary least squares (OLS) to estimate the parameters, and thus relies on large-sample approximations.

References:

• http://cs229.stanford.edu/notes/cs229-notes1.pdf
• http://machinelearningmastery.com/logistic-regression-for-machine-learning/
• https://onlinecourses.science.psu.edu/stat504/node/164

This article discusses the basics of linear regression and its implementation in Python programming language.

Linear regression is a statistical approach for modelling relationship between a dependent variable with a given set of independent variables.

Note: In this article, we refer dependent variables as response and independent variables as features for simplicity.

In order to provide a basic understanding of linear regression, we start with the most basic version of linear regression, i.e. Simple linear regression.

Simple Linear Regression

Simple linear regression is an approach for predicting a response using a single feature.

It is assumed that the two variables are linearly related. Hence, we try to find a linear function that predicts the response value(y) as accurately as possible as a function of the feature or independent variable(x).

Let us consider a dataset where we have a value of response y for every feature x:

For generality, we define:

x as feature vector, i.e ,

y as response vector, i.e

for n observations (in above example, n=10).

A scatter plot of above dataset looks like:-

Now, the task is to find a line which fits best in above scatter plot so that we can predict the response for any new feature values. (i.e a value of x not present in dataset)

This line is called regression line.

The equation of regression line is represented as:

Here,

• represents the predicted response value for observation.
• and are regression coefficients and represent y-intercept and slope of regression line respectively.

To create our model, we must “learn” or estimate the values of regression coefficients and . And once we’ve estimated these coefficients, we can use the model to predict responses!

In this article, we are going to use the Least Squares technique.

Now consider:

Here, is residual error in observation.
So, our aim is to minimize .

We define the squared error or cost function, J as:

and our task is to find the value of and for which is minimum!

Without going into the mathematical details, we present the result here:

where is the sum of cross-deviations of y and x:

and is the sum of squared deviations of x:

Note: The complete derivation for finding least squares estimates in simple linear regression can be found here.

Given below is the python implementation of above technique on our small dataset:

import numpy as np
import matplotlib.pyplot as plt

def estimate_coef(x, y):
    # number of observations/points
    n = np.size(x)

    # mean of x and y vector
    m_x, m_y = np.mean(x), np.mean(y)

    # calculating cross-deviation and deviation about x
    SS_xy = np.sum(y*x - n*m_y*m_x)
    SS_xx = np.sum(x*x - n*m_x*m_x)

    # calculating regression coefficients
    b_1 = SS_xy / SS_xx
    b_0 = m_y - b_1*m_x

    return(b_0, b_1)

def plot_regression_line(x, y, b):
    # plotting the actual points as scatter plot
    plt.scatter(x, y, color = "m",
               marker = "o", s = 30)

    # predicted response vector
    y_pred = b[0] + b[1]*x

    # plotting the regression line
    plt.plot(x, y_pred, color = "g")

    # putting labels
    plt.xlabel('x')
    plt.ylabel('y')

    # function to show plot
    plt.show()

def main():
    # observations
    x = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
    y = np.array([1, 3, 2, 5, 7, 8, 8, 9, 10, 12])

    # estimating coefficients
    b = estimate_coef(x, y)
    print("Estimated coefficients:\nb_0 = {}  \
          \nb_1 = {}".format(b[0], b[1]))

    # plotting regression line
    plot_regression_line(x, y, b)

if __name__ == "__main__":
    main()

Click here for NumPy tutorial

Output of above piece of code is:

Estimated coefficients:
b_0 = -0.0586206896552
b_1 = 1.45747126437

And graph obtained looks like this:

Multiple linear regression

Multiple linear regression attempts to model the relationship between two or more features and a response by fitting a linear equation to observed data.

Clearly, it is nothing but an extension of Simple linear regression.

Consider a dataset with p features(or independent variables) and one response(or dependent variable).
Also, the dataset contains n rows/observations.

We define:

X (feature matrix) = a matrix of size n X p where denotes the values of feature for observation.

So,

and

y (response vector) = a vector of size n where denotes the value of response for observation.

The regression line for p features is represented as:

where is predicted response value for observation and are the regression coefficients.

Also, we can write:

where represents residual error in observation.

We can generalize our linear model a little bit more by representing feature matrix X as:

So now, the linear model can be expressed in terms of matrices as:

where,

and

Now, we determine estimate of , i.e. using Least Squares method.

As already explained, Least Squares method tends to determine for which is minimized.

We present the result directly here:

where represents the transpose of the matrix while represents the matrix inverse.

Knowing the least square estimates, , the multiple linear regression model can now be estimated as:

where is estimated response vector.

Note: The complete derivation for obtaining least square estimates in multiple linear regression can be found here.

Given below is the implementation of multiple linear regression technique on the Boston house pricing dataset dataset using Scikit-learn.

import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model, metrics

# load the boston dataset
boston = datasets.load_boston(return_X_y=False)

# defining feature matrix(X) and response vector(y)
X = boston.data
y = boston.target

# splitting X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4,
                                                    random_state=1)

# create linear regression object
reg = linear_model.LinearRegression()

# train the model using the training sets
reg.fit(X_train, y_train)

# regression coefficients
print('Coefficients: \n', reg.coef_)

# variance score: 1 means perfect prediction
print('Variance score: {}'.format(reg.score(X_test, y_test)))

# plot for residual error

## setting plot style
plt.style.use('fivethirtyeight')

## plotting residual errors in training data
plt.scatter(reg.predict(X_train), reg.predict(X_train) - y_train,
            color = "green", s = 10, label = 'Train data')

## plotting residual errors in test data
plt.scatter(reg.predict(X_test), reg.predict(X_test) - y_test,
            color = "blue", s = 10, label = 'Test data')

## plotting line for zero residual error
plt.hlines(y = 0, xmin = 0, xmax = 50, linewidth = 2)

## plotting legend
plt.legend(loc = 'upper right')

## plot title
plt.title("Residual errors")

## function to show plot
plt.show()

Click here for Scikit-learn tutorial

Output of above program looks like this:

Coefficients:
[ -8.80740828e-02   6.72507352e-02   5.10280463e-02   2.18879172e+00
-1.72283734e+01   3.62985243e+00   2.13933641e-03  -1.36531300e+00
2.88788067e-01  -1.22618657e-02  -8.36014969e-01   9.53058061e-03
-5.05036163e-01]
Variance score: 0.720898784611

and Residual Error plot looks like this:

In above example, we determine accuracy score using Explained Variance Score.
We define:

where is the estimated target output, the corresponding (correct) target output, and is Variance, the square of the standard deviation.
The best possible score is 1.0, lower values are worse.

Assumptions

Given below are the basic assumptions that a linear regression model makes regarding a dataset on which it is applied:

• Linear relationship: Relationship between response and feature variables should be linear. The linearity assumption can be tested using scatter plots. As shown below, 1st figure represents linearly related variables where as variables in 2nd and 3rd figure are most likely non-linear. So, 1st figure will give better predictions using linear regression.
  
• Little or no multi-collinearity: It is assumed that there is little or no multicollinearity in the data.  Multicollinearity occurs when the features (or independent variables) are not independent from each other.
• Little or no auto-correlation:  Another assumption is that there is little or no autocorrelation in the data.  Autocorrelation occurs when the residual errors are not independent from each other. You can refer here for more insight into this topic.
• Homoscedasticity: Homoscedasticity describes a situation in which the error term (that is, the “noise” or random disturbance in the relationship between the independent variables and the dependent variable) is the same across all values of the independent variables. As shown below, figure 1 has homoscedasticity while figure 2 has heteroscedasticity.
  

As we reach to the end of this article, we discuss some applications of linear regression below.

Applications:

1. Trend lines: A trend line represents the variation in some quantitative data with passage of time (like GDP, oil  prices, etc.). These trends usually follow a linear relationship. Hence, linear regression can be applied to predict future values. However, this method suffers from a lack of scientific validity in cases where other potential changes can affect the data.

2. Economics: Linear regression is the predominant empirical tool in economics. For example, it is used to predict consumption spending, fixed investment spending, inventory investment, purchases of a country’s exports, spending on imports, the demand to hold liquid assets, labor demand, and labor supply.

3. Finance: Capital price asset model uses linear regression to analyze and quantify the systematic risks of an investment.

4. Biology: Linear regression is used to model causal relationships between parameters in biological systems.

References:

• https://en.wikipedia.org/wiki/Linear_regression
• https://en.wikipedia.org/wiki/Simple_linear_regression
• http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html
• http://www.statisticssolutions.com/assumptions-of-linear-regression/

This article discusses the theory behind the Naive Bayes classifiers and their implementation.

Assumption:

X = (Rainy, Hot, High, False)
y = No

today = (Sunny, Hot, Normal, False)

# load the iris dataset
from sklearn.datasets import load_iris
iris = load_iris()

# store the feature matrix (X) and response vector (y)
X = iris.data
y = iris.target

# splitting X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=1)

# training the model on training set
from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB()
gnb.fit(X_train, y_train)

# making predictions on the testing set
y_pred = gnb.predict(X_test)

# comparing actual response values (y_test) with predicted response values (y_pred)
from sklearn import metrics
print("Gaussian Naive Bayes model accuracy(in %):", metrics.accuracy_score(y_test, y_pred)*100)

Gaussian Naive Bayes model accuracy(in %): 95.0

scikit-learn is an open source Python library that implements a range of machine learning, pre-processing, cross-validation and visualization algorithms using a unified interface.

Important features of scikit-learn:

• Simple and efficient tools for data mining and data analysis. It features various classification, regression and clustering algorithms including support vector machines, random forests, gradient boosting, k-means, etc.
• Accessible to everybody and reusable in various contexts.
• Built on the top of NumPy, SciPy, and matplotlib.
• Open source, commercially usable – BSD license.

In this article, we are going to see how we can easily build a machine learning model using scikit-learn.

Installation:

Scikit-learn requires:

• NumPy
• SciPy as its dependencies.

Before installing scikit-learn, ensure that you have NumPy and SciPy installed. Once you have a working installation of NumPy and SciPy, the easiest way to install scikit-learn is using pip:

pip install -U scikit-learn

Let us get started with the modelling process now.

Step 1: Load a dataset

A dataset is nothing but a collection of data. A dataset generally has two main components:

• Features: (also known as predictors, inputs, or attributes) they are simply the variables of our data. They can be more than one and hence represented by a feature matrix (‘X’ is a common notation to represent feature matrix). A list of all the feature names is termed as feature names.
• Response: (also known as the target, label, or output) This is the output variable depending on the feature variables. We generally have a single response column and it is represented by a response vector (‘y’ is a common notation to represent response vector). All the possible values taken by a response vector is termed as target names.

Loading exemplar dataset: scikit-learn comes loaded with a few example datasets like the iris and digits datasets for classification and the boston house prices dataset for regression.
Given below is an example of how one can load an exemplar dataset:

# load the iris dataset as an example
from sklearn.datasets import load_iris
iris = load_iris()

# store the feature matrix (X) and response vector (y)
X = iris.data
y = iris.target

# store the feature and target names
feature_names = iris.feature_names
target_names = iris.target_names

# printing features and target names of our dataset
print("Feature names:", feature_names)
print("Target names:", target_names)

# X and y are numpy arrays
print("\nType of X is:", type(X))

# printing first 5 input rows
print("\nFirst 5 rows of X:\n", X[:5])

Output:

Feature names: ['sepal length (cm)','sepal width (cm)','petal length (cm)','petal width (cm)']
Target names: ['setosa' 'versicolor' 'virginica']

Type of X is: 

First 5 rows of X:
 [[ 5.1  3.5  1.4  0.2]
 [ 4.9  3.   1.4  0.2]
 [ 4.7  3.2  1.3  0.2]
 [ 4.6  3.1  1.5  0.2]
 [ 5.   3.6  1.4  0.2]]

Loading external dataset: Now, consider the case when we want to load an external dataset. For this purpose, we can use pandas library for easily loading and manipulating dataset.

To install pandas, use the following pip command:

pip install pandas

In pandas, important data types are:

Series: Series is a one-dimensional labeled array capable of holding any data type.

DataFrame: It is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table, or a dict of Series objects. It is generally the most commonly used pandas object.

Note: The CSV file used in example below can be downloaded from here: weather.csv

import pandas as pd

# reading csv file
data = pd.read_csv('weather.csv')

# shape of dataset
print("Shape:", data.shape)

# column names
print("\nFeatures:", data.columns)

# storing the feature matrix (X) and response vector (y)
X = data[data.columns[:-1]]
y = data[data.columns[-1]]

# printing first 5 rows of feature matrix
print("\nFeature matrix:\n", X.head())

# printing first 5 values of response vector
print("\nResponse vector:\n", y.head())

Output:

Shape: (14, 5)

Features: Index([u'Outlook', u'Temperature', u'Humidity', u'Windy', u'Play'], dtype='object')

Feature matrix:
     Outlook Temperature Humidity  Windy
0  overcast         hot     high  False
1  overcast        cool   normal   True
2  overcast        mild     high   True
3  overcast         hot   normal  False
4     rainy        mild     high  False

Response vector:
0    yes
1    yes
2    yes
3    yes
4    yes
Name: Play, dtype: object

Step 2: Splitting the dataset

One important aspect of all machine learning models is to determine their accuracy. Now, in order to determine their accuracy, one can train the model using the given dataset and then predict the response values for same dataset using that model and hence, find the accuracy of model.
But this method has several flaws in it, like:

• Goal is to estimate likely performance of a model on an out-of-sample data.
• Maximizing training accuracy rewards overly complex models that won’t necessarily generalize our model.
• Unnecessarily complex models may over-fit the training data.

A better option is to split our data into two parts: first one for training our machine learning model, and second one for testing our model.
To summarize:

• Split the dataset into two pieces: a training set and a testing set.
• Train the model on the training set.
• Test the model on the testing set, and evaluate how well our model did.

Advantages of train/test split:

• Model can be trained and tested on different data than the one used for training.
• Response values are known for the test dataset, hence predictions can be evaluated
• Testing accuracy is a better estimate than training accuracy of out-of-sample performance.

Consider the example below:

# load the iris dataset as an example
from sklearn.datasets import load_iris
iris = load_iris()

# store the feature matrix (X) and response vector (y)
X = iris.data
y = iris.target

# splitting X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=1)

# printing the shapes of the new X objects
print(X_train.shape)
print(X_test.shape)

# printing the shapes of the new y objects
print(y_train.shape)
print(y_test.shape)

Output:

(90L, 4L)
(60L, 4L)
(90L,)
(60L,)

The train_test_split function takes several arguments which are explained below:

• X, y: These are the feature matrix and response vector which need to be splitted.
• test_size: It is the ratio of test data to the given data. For example, setting test_size = 0.4 for 150 rows of X produces test data of 150 x 0.4 = 60 rows.
• random_state: If you use random_state = some_number, then you can guarantee that your split will be always the same. This is useful if you want reproducible results, for example in testing for consistency in the documentation (so that everybody can see the same numbers).

Step 3: Training the model

Now, its time to train some prediction-model using our dataset. Scikit-learn provides a wide range of machine learning algorithms which have a unified/consistent interface for fitting, predicting accuracy, etc.

The example given below uses KNN (K nearest neighbors) classifier.

Note: We will not go into the details of how the algorithm works as we are interested in understanding its implementation only.

Now, consider the example below:

# load the iris dataset as an example
from sklearn.datasets import load_iris
iris = load_iris()

# store the feature matrix (X) and response vector (y)
X = iris.data
y = iris.target

# splitting X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=1)

# training the model on training set
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_train, y_train)

# making predictions on the testing set
y_pred = knn.predict(X_test)

# comparing actual response values (y_test) with predicted response values (y_pred)
from sklearn import metrics
print("kNN model accuracy:", metrics.accuracy_score(y_test, y_pred))

# making prediction for out of sample data
sample = [[3, 5, 4, 2], [2, 3, 5, 4]]
preds = knn.predict(sample)
pred_species = [iris.target_names[p] for p in preds]
print("Predictions:", pred_species)

# saving the model
from sklearn.externals import joblib
joblib.dump(knn, 'iris_knn.pkl')

Output:

kNN model accuracy: 0.983333333333
Predictions: ['versicolor', 'virginica']

Important points to note from above code:

• We create a knn classifier object using:

  knn = KNeighborsClassifier(n_neighbors=3)
  

• The classifier is trained using X_train data. The process is termed as fitting. We pass the feature matrix and the corresponding response vector.

  knn.fit(X_train, y_train)
  

• Now, we need to test our classifier on the X_test data. knn.predict method is used for this purpose. It returns the predicted response vector, y_pred.

  y_pred = knn.predict(X_test)
  

• Now, we are interested in finding the accuracy of our model by comparing y_test and y_pred. This is done using metrics module’s method accuracy_score:

  print(metrics.accuracy_score(y_test, y_pred))
  

• Consider the case when you want your model to make prediction on out of sample data. Then, the sample input can simply pe passed in the same way as we pass any feature matrix.

  sample = [[3, 5, 4, 2], [2, 3, 5, 4]]
  preds = knn.predict(sample)
  

• If you are not interested in training your classifier again and again and use the pre-trained classifier, one can save their classifier using joblib. All you need to do is:

  joblib.dump(knn, 'iris_knn.pkl')
  

• In case you want to load an already saved classifier, use the following method:

  knn = joblib.load('iris_knn.pkl')

As we approach to end of this article, here are some benefits of using scikit-learn over some other machine learning libraries(like R):

• Consistent interface to machine learning models
• Provides many tuning parameters but with sensible defaults
• Exceptional documentation
• Rich set of functionality for companion tasks.
• Active community for development and support.

References:

• http://scikit-learn.org/stable/documentation.html
• https://github.com/justmarkham/scikit-learn-videos

This article covers the sentiment analysis of any topic by parsing the tweets fetched from Twitter using Python.

What is sentiment analysis?
Sentiment Analysis is the process of ‘computationally’ determining whether a piece of writing is positive, negative or neutral. It’s also known as opinion mining, deriving the opinion or attitude of a speaker.

Why sentiment analysis?

• Business: In marketing field companies use it to develop their strategies, to understand customers’ feelings towards products or brand, how people respond to their campaigns or product launches and why consumers don’t buy some
  products.
• Politics: In political field, it is used to keep track of political view, to detect consistency and inconsistency between statements and actions at the government level. It can be used to predict election results as well!
• Public Actions: Sentiment analysis also is used to monitor and analyse social phenomena, for the spotting of potentially dangerous situations and determining the general mood of the blogosphere.

Installation:

• Tweepy: tweepy is the python client for the official Twitter API.
  Install it using following pip command:

  pip install tweepy

• TextBlob: textblob is the python library for processing textual data.
  Install it using following pip command:

  pip install textblob

  Also, we need to install some NLTK corpora using following command:

  python -m textblob.download_corpora

  (Corpora is nothing but a large and structured set of texts.)

Authentication:
In order to fetch tweets through Twitter API, one needs to register an App through their twitter account. Follow these steps for the same:

• Open this link and click the button: ‘Create New App’
• Fill the application details. You can leave the callback url field empty.
• Once the app is created, you will be redirected to the app page.
• Open the ‘Keys and Access Tokens’ tab.
• Copy ‘Consumer Key’, ‘Consumer Secret’, ‘Access token’ and ‘Access Token Secret’.

Implementation:

import re
import tweepy
from tweepy import OAuthHandler
from textblob import TextBlob

class TwitterClient(object):
	'''
	Generic Twitter Class for sentiment analysis.
	'''
	def __init__(self):
		'''
		Class constructor or initialization method.
		'''
		# keys and tokens from the Twitter Dev Console
		consumer_key = 'XXXXXXXXXXXXXXXXXXXXXXXX'
		consumer_secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXX'
		access_token = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXX'
		access_token_secret = 'XXXXXXXXXXXXXXXXXXXXXXXXX'

		# attempt authentication
		try:
			# create OAuthHandler object
			self.auth = OAuthHandler(consumer_key, consumer_secret)
			# set access token and secret
			self.auth.set_access_token(access_token, access_token_secret)
			# create tweepy API object to fetch tweets
			self.api = tweepy.API(self.auth)
		except:
			print("Error: Authentication Failed")

	def clean_tweet(self, tweet):
		'''
		Utility function to clean tweet text by removing links, special characters
		using simple regex statements.
		'''
		return ' '.join(re.sub("(@[A-Za-z0-9]+)|([^0-9A-Za-z \t])|(\w+:\/\/\S+)", " ", tweet).split())

	def get_tweet_sentiment(self, tweet):
		'''
		Utility function to classify sentiment of passed tweet
		using textblob's sentiment method
		'''
		# create TextBlob object of passed tweet text
		analysis = TextBlob(self.clean_tweet(tweet))
		# set sentiment
		if analysis.sentiment.polarity > 0:
			return 'positive'
		elif analysis.sentiment.polarity == 0:
			return 'neutral'
		else:
			return 'negative'

	def get_tweets(self, query, count = 10):
		'''
		Main function to fetch tweets and parse them.
		'''
		# empty list to store parsed tweets
		tweets = []

		try:
			# call twitter api to fetch tweets
			fetched_tweets = self.api.search(q = query, count = count)

			# parsing tweets one by one
			for tweet in fetched_tweets:
				# empty dictionary to store required params of a tweet
				parsed_tweet = {}

				# saving text of tweet
				parsed_tweet['text'] = tweet.text
				# saving sentiment of tweet
				parsed_tweet['sentiment'] = self.get_tweet_sentiment(tweet.text)

				# appending parsed tweet to tweets list
				if tweet.retweet_count > 0:
					# if tweet has retweets, ensure that it is appended only once
					if parsed_tweet not in tweets:
						tweets.append(parsed_tweet)
				else:
					tweets.append(parsed_tweet)

			# return parsed tweets
			return tweets

		except tweepy.TweepError as e:
			# print error (if any)
			print("Error : " + str(e))

def main():
	# creating object of TwitterClient Class
	api = TwitterClient()
	# calling function to get tweets
	tweets = api.get_tweets(query = 'Donald Trump', count = 200)

	# picking positive tweets from tweets
	ptweets = [t for  in tweets if t['sentiment'] == 'positive']
	# percentage of positive tweets
	print("Positive tweets percentage: {} %".format(100*len(ptweets)/len(tweets)))
	# picking negative tweets from tweets
	ntweets = [t for t in tweets if t['sentiment'] == 'negative']
	# percentage of negative tweets
	print("Negative tweets percentage: {} %".format(100*len(ntweets)/len(tweets)))
	# percentage of neutral tweets
	print("Neutral tweets percentage: {} %".format(100*(len(tweets) - len(ntweets) - len(ptweets))/len(tweets)))

	# printing first 5 positive tweets
	print("\n\nPositive tweets:")
	for tweet in ptweets[:10]:
		print(tweet['text'])

	# printing first 5 negative tweets
	print("\n\nNegative tweets:")
	for tweet in ntweets[:10]:
		print(tweet['text'])

if __name__ == "__main__":
	# calling main function
	main()

Here is how a sample output looks like when above program is run:

Positive tweets percentage: 22 %
Negative tweets percentage: 15 %


Positive tweets:
RT @JohnGGalt: Amazing—after years of attacking Donald Trump the media managed
to turn #InaugurationDay into all about themselves.
#MakeAme…
RT @vooda1: CNN Declines to Air White House Press Conference Live YES! 
THANK YOU @CNN FOR NOT LEGITIMI…
RT @Muheeb_Shawwa: Donald J. Trump's speech sounded eerily familiar...
POTUS plans new deal for UK as Theresa May to be first foreign leader to meet new 
president since inauguration 
.@realdonaldtrump #Syria #Mexico #Russia & now #Afghanistan. 
Another #DearDonaldTrump Letter worth a read @AJEnglish 


Negative tweets:
RT @Slate: Donald Trump’s administration: “Government by the worst men.” 
RT @RVAwonk: Trump, Sean Spicer, et al. lie for a reason. 
Their lies are not just lies. Their lies are authoritarian propaganda.  
RT @KomptonMusic: Me: I hate corn 
Donald Trump: I hate corn too
Me: https://t.co/GPgy8R8HB5
It's ridiculous that people are more annoyed at this than Donald Trump's sexism.
RT @tony_broach: Chris Wallace on Fox news right now talking crap 
about Donald Trump news conference it seems he can't face the truth eithe…
RT @fravel: With False Claims, Donald Trump Attacks Media on Crowd Turnout 
Aziz Ansari Just Hit Donald Trump Hard In An Epic Saturday NIght Live Monologue

We follow these 3 major steps in our program:

• Authorize twitter API client.
• Make a GET request to Twitter API to fetch tweets for a particular query.
• Parse the tweets. Classify each tweet as positive, negative or neutral.

Now, let us try to understand the above piece of code:

• First of all, we create a TwitterClient class. This class contains all the methods to interact with Twitter API and parsing tweets. We use __init__ function to handle the authentication of API client.
• In get_tweets function, we use:

  fetched_tweets = self.api.search(q = query, count = count)

  to call the Twitter API to fetch tweets.

• In get_tweet_sentiment we use textblob module.

  analysis = TextBlob(self.clean_tweet(tweet))

  TextBlob is actually a high level library built over top of NLTK library. First we call clean_tweet method to remove links, special characters, etc. from the tweet using some simple regex.
  Then, as we pass tweet to create a TextBlob object, following processing is done over text by textblob library:

  • Tokenize the tweet ,i.e split words from body of text.
  • Remove stopwords from the tokens.(stopwords are the commonly used words which are irrelevant in text analysis like I, am, you, are, etc.)
  • Do POS( part of speech) tagging of the tokens and select only significant features/tokens like adjectives, adverbs, etc.
  • Pass the tokens to a sentiment classifier which classifies the tweet sentiment as positive, negative or neutral by assigning it a polarity between -1.0 to 1.0 .

  Here is how sentiment classifier is created:

  • TextBlob uses a Movies Reviews dataset in which reviews have already been labelled as positive or negative.
  • Positive and negative features are extracted from each positive and negative review respectively.
  • Training data now consists of labelled positive and negative features. This data is trained on a Naive Bayes Classifier.

  Then, we use sentiment.polarity method of TextBlob class to get the polarity of tweet between -1 to 1.
  Then, we classify polarity as:

  if analysis.sentiment.polarity > 0:
         return 'positive'
  elif analysis.sentiment.polarity == 0:
         return 'neutral'
  else:
         return 'negative'

• Finally, parsed tweets are returned. Then, we can do various type of statistical analysis on the tweets. For example, in above program, we tried to find the percentage of positive, negative and neutral tweets about a query.

References:

• http://www.ijcaonline.org/research/volume125/number3/dandrea-2015-ijca-905866.pdf
• https://textblob.readthedocs.io/en/dev/quickstart.html#sentiment-analysis
• textblob.readthedocs.io/en/dev/_modules/textblob/en/sentiments.html