In this article let’s try to demystify these *.pyc files!

But before we start, I recommend you to read this article for a better understanding of how Python runs:  How Python runs?

After reading the article given below, you will be able to understand:

• What are pyc files?
• When are pyc files created?
• Can I run pyc files?
• How to create pyc files without any import?
• Can I decompile pyc files?
• Should I ignore pyc files while adding code to git?

So, let’s get started!

What are pyc files?

pyc files are simply the compiled python files which contain the bytecode representation of your source code.

The only difference between python source code and bytecode is that former one is uncompiled while latter one is compiled.

This byte code translation is performed to speed up the execution—byte code can be run much quicker than the original source code statements. 

When are pyc files created?

Whenever a Python script is executed, the byte code is generated in memory and simply discarded when program exits.

But, if a Python module is imported, a .pyc file for the module is generated (by default) which contains its bytecode.
Thus, when the module is imported next time, the byte code from .pyc file is used. This makes loading of Python modules much faster because the compilation phase can be bypassed!

There’s no harm in deleting them (.pyc), but they will save compilation time if you’re doing lots of processing.

In 3.2 and later, Python saves .pyc compiled byte code files in a sub-directory named __pycache__ located in the directory where your source files reside with filenames that identify the Python version that created them (e.g. script.cpython-36.pyc)

Here is a flowchart which clears the concept:

Let’s try to understand this by an example.

Consider two python scripts, namely, myadd.py and test.py.
The add function of myadd.py has been imported in test.py.

Now, if we try to execute test.py, a folder named __pycache__ gets created in the current directory. It contains the following file:

Here, myadd.cpython-36.pyc is the name of generated pyc file. cpython-36 is the specification of the interpreter which created this pyc file. First few bytes of a pyc file specify the interpreter version (also called magic number).

Can I run pyc files?

Of course, you can run pyc files!

In a CPython interpreter, bytecode is fed to PVM (Python Virtual Machine) which is responsible for running your code.

Since, pyc files contain nothing but bytecode representation of your source code, we can execute them directly (just like the normal py files):

$ python myadd.cpython-36.pyc

Note: A pyc file generated by a Python3 compiler can’t be executed using a Python2 compiler and will throw the error: RuntimeError: Bad magic number in .pyc file

How to create pyc files without any import?

If you need to create a pyc file for a module that is not imported, you can use the py_compile module.

The py_compile module can manually compile any module. One way is to use the py_compile.compile function in that module interactively:

import py_compile
py_compile.compile("myadd.py")

This will write the .pyc in the __pycache__ folder in the same location as myadd.py.

You can also automatically compile all files in a directory or directories using the compileall module.

$ python -m compileall .

Can I decompile pyc files?

Yes, pyc files can be decompiled but the generated source code may or may not be totally identical to your original source code.

Also, there is no built-in module for decompilation. For this purpose, you can use a 3rd party python package uncompyle6 .

Simply install it using:

$ pip install uncompyle6

Now, using the terminal, you can decompile any pyc file as:

$ uncompyle6 myadd.cpython-36.pyc

The obtained output looks something like this:

# uncompyle6 version 2.14.1
# Python bytecode 3.6 (3379)
# Decompiled from: Python 3.6.3 (default, Oct 3 2017, 21:45:48) 
# [GCC 7.2.0]
# Embedded file name: /home/nikhil/Desktop/myadd.py
# Compiled at: 2018-01-05 09:53:32
# Size of source mod 2**32: 72 bytes

def add(a, b):
 return a + b

if __name__ == '__main__':
 print(add(2, 3))
# okay decompiling myadd.cpython-36.pyc

Should I ignore pyc files while adding code to git?

Since pyc files can be generated automatically when you import python modules, it is useless to add them to git repositories.

In order to not share your pyc files with others, you would add the entries:

*.pyc
__pycache__

in .gitignore file and git will start ignoring any new pyc files and __pycache__  folders in the repo.

But what about those files that are already being tracked by git? To fix this,
we need to ask git to remove these paths from it’s index by running the git rm command with the --cached option.

$ git rm --cached *.pyc

So, this was all about pyc files! If you have any doubts or find anything incorrect, please share in the comments section below. Thanks for reading!  

Have you ever thought how the Python code is actually executed by the Python interpreter? What steps are carried out to generate the final output of your Python script? This article answers all these questions in a simplistic manner!

When the Python software is installed on your machine, minimally, it has:

• an interpreter
• a support library.

The interpreter

Interpreter is nothing but a software which can run your Python scripts.

Interestingly, it can be implemented in any programming language!

• CPython is the default interpreter for Python which is written in C programming language.
• Jython is another popular implementation of python interpreter written using Java programming language.

Programmer’s view of interpreter

If you have been coding in Python for sometime, you must have heard about the interpreter  at least a few times. From a programmer’s perspective, an interpreter is simply a software which executes the source code line by line.

For most of the Python programmers, an interpreter is like a black box.

Python’s view of interpreter

Now, let us scan through the python interpreter and try to understand how it works.

Have a look at the diagram shown below:

I hope you didn’t get amazed to see a compiler inside an interpreter!

From the figure above, it can be inferred that interpreter is made up of two parts:

• compiler
• virtual machine 

What does compiler do?

Compiler compiles your source code (the statements in your file) into a format known as byte code. Compilation is simply a translation step!

Byte code is a:

• lower level,
• platform independent,
• efficient and
• intermediate

representation of your source code!

Roughly, each of your source statements is translated into a group of byte code instructions.

The process of compilation in CPython interpreter’s compiler can be divided into 4 main parts:

• Parse source code into a parse tree
  Based on grammar rules of Python programming language, the source code is converted to a parse tree. Every node of the parse tree contains a part of your code.
  Consider a simple arithmetic expression:

  14 + 2 * 3 - 6 / 2

  The parse tree for above expression looks like this:
  

• Transform parse tree into an Abstract Syntax Tree
  The abstract syntax tree (AST) is a high-level representation of the program structure.
  Each node of the tree denotes a construct occurring in the source code. The syntax is “abstract” in not representing every detail appearing in the real syntax.Consider the AST shown below for the parse tree example discussed above:
  
• Transform AST into a Control Flow Graph
  A control flow graph is a directed graph that models the flow of a program using basic blocks. Each block contains the bytecode representation of program code inside it.
  
  
• Byte Code generation from CFG
  CFGs are usually one step away from final code output. Code is directly generated from the basic blocks by doing a post-order depth-first search on the CFG following the edges.

This byte code translation is performed to speed up the execution—byte code can be run much quicker than the original source code statements.

What does Virtual Machine do?

As soon as source code gets converted to byte code, it is fed into PVM (Python Virtual Machine).

The PVM sounds more impressive than it is!

It’s just a big loop that iterates through your byte code instructions, one by one, to carry out their operations. The PVM is the runtime engine of Python; it’s always present as part of the Python system, and is the component that truly runs your scripts. Technically, it’s just the last step of what is called the Python interpreter.

So, this is how a python interpreter runs your python code!

Lastly, here are a few points to ponder upon:

• PyPy is an implementation of Python which does not use an interpreter! It is implemented using something called just-in-time compiler!
  Interestingly, it often runs faster than the standard implementation of Python, CPython.
• Whenever a Python script is executed, the byte code is generated in memory and simply discarded when program exits.
• But, if a Python module is imported, a .pyc file for the module is generated which contains its Byte code.
  Thus, when the module is imported next time, the byte code from .pyc file is used, hence skipping the compilation step!

If you have any doubts or find anything incorrect, please share in the comments section below. Thanks for reading! 

This article discusses the basics of Softmax Regression and its implementation in Python using TensorFlow library.

What is Softmax Regression?

Softmax regression (or multinomial logistic regression) is a generalization of logistic regression to the case where we want to handle multiple classes.

A gentle introduction to linear regression can be found here:
Understanding Logistic Regression

In binary logistic regression we assumed that the labels were binary, i.e. for observation,

But consider a scenario where we need to classify an observation out of two or more class labels. For example, digit classification. Here, the possible labels are:

In such cases, we can use Softmax Regression.

Let us first define our model:

• Let the dataset have ‘m’ features and ‘n’ observations. Also, there are ‘k’ class labels, i.e every observation can be classified as one of the ‘k’ possible target values. For example, if we have a dataset of 100 handwritten digit images of vector size 28×28 for digit classification, we have, n = 100, m = 28×28 = 784 and k = 10.

• Feature matrix
  The feature matrix, , is represented as: Here, denotes the values of feature for observation. The matrix has dimensions:

• Weight matrix
  We define a weight matrix, as: Here, represents the weight assigned to feature for class label. The matrix has dimensions:. Initially, weight matrix is filled using some normal distribution.

• Logit score matrix
  Then, we define our net input matrix(also called logit score matrix), , as: The matrix has dimensions: .

  Currently, we are taking an extra column in feature matrix, and an extra row in weight matrix, . These extra columns and rows correspond to the bias terms associated with each prediction. This could be simplified by defining an extra matrix for bias, of size where . (In practice, all we need is a vector of size and some broadcasting techniques for the bias terms!)

  So, the final score matrix, is:

  

  where matrix has dimensions while has dimensions . But matrix still has same value and dimensions!

  But what does matrix signify? Actually, is the likelihood of label j for observation. It is not a proper probability value but can be considered as a score given to each class label for each observation!

  Let us define as the logit score vector for observation.

  For example, let the vector represents the score for each of the class labels in handwritten digit classification problem for observation. Here, the max score is 5.2 which corresponds to class label ‘3’. Hence, our model currently predicts the observation/image as ‘3’.

• Softmax layer
  It is harder to train the model using score values since it is hard to differentiate them while implementing Gradient Descent algorithm for minimizing the cost function. So, we need some function which normalizes the logit scores as well as makes them easily differentiable!In order to convert the score matrix to probabilities, we use Softmax function.For a vector , softmax function is defined as: So, softmax function will do 2 things:

  1. convert all scores to probabilities.
  2. sum of all probabilities is 1.
  

  Diagram given below explains how the softmax function works:

  

  Recall that in Binary Logistic classifier, we used sigmoid function for the same task. Softmax function is nothing but a generalization of sigmoid function! Now, this softmax function computes the probability that the training sample belongs to class given the logits vector as:

  In vector form, we can simply write:

  For simplicity, let denote the softmax probability vector for observation.

• One-hot encoded target vector
  Since softmax function provides us with a vector of probabilities of each class label for a given observation, we need to convert corresponding target label in the same format to calculate the cost function!Corresponding to each observation, we create a target vector composed of only zeros and ones where only correct class label is set as 1. This technique is called one-hot encoding.

  See the diagram given below for a better understanding:

  Now, we denote one-hot encoding vector for observation as

• Cost function
  Now, we need to define a cost function for which, we have to compare the softmax probabilities and one-hot encoded target vector for similarity. We use the concept of Cross-Entropy for the same. The Cross-entropy is a distance calculation function which takes the calculated probabilities from softmax function and the created one-hot-encoding matrix to calculate the distance. For the right target classes, the distance values will be lesser, and the distance values will be larger for the wrong target classes.We define cross entropy, for observation with softmax probability vector, and one-hot target vector, as: And now, cost function, can be defined as the average cross entropy, i.e: and the task is to minimize this cost function!

• Gradient Descent algorithm
  In order to learn our softmax model via gradient descent, we need to compute the derivative: and which we then use to update the weights and biases in opposite direction of the gradient: and for each class where and is learning rate.Using this cost gradient, we iteratively update the weight matrix until we reach a specified number of epochs (passes over the training set) or reach the desired cost threshold.

Implementation

Let us now implement Softmax Regression on the MNIST handwritten digit dataset using TensorFlow library.

For a gentle introduction to TensorFlow, follow this tutorial:
Introduction to TensorFlow

Step 1: Import the dependencies

First of all, we import the dependencies.

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

Step 2: Download the data

TensorFlow allows you to download and read in the MNIST data automatically. Consider the code given below. It will download and save data to the folder, MNIST_data, in your current project directory and load it in current program.

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

Extracting MNIST_data/train-images-idx3-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/t10k-images-idx3-ubyte.gz
Extracting MNIST_data/t10k-labels-idx1-ubyte.gz

Step 3: Understanding data

Now, we try to understand the structure of the dataset.

The MNIST data is split into three parts: 55,000 data points of training data (mnist.train), 10,000 points of test data (mnist.test), and 5,000 points of validation data (mnist.validation).

Each image is 28 pixels by 28 pixels which has been flattened into 1-D numpy array of size 784. Number of class labels is 10. Each target label is already provided in one-hot encoded form.

print("Shape of feature matrix:", mnist.train.images.shape)
print("Shape of target matrix:", mnist.train.labels.shape)
print("One-hot encoding for 1st observation:\n", mnist.train.labels[0])

# visualize data by plotting images
fig,ax = plt.subplots(10,10)
k = 0
for i in range(10):
    for j in range(10):
        ax[i][j].imshow(mnist.train.images[k].reshape(28,28), aspect='auto')
        k += 1
plt.show()

Output:

Shape of feature matrix: (55000, 784)
Shape of target matrix: (55000, 10)
One-hot encoding for 1st observation:
 [ 0.  0.  0.  0.  0.  0.  0.  1.  0.  0.]

Step 4: Defining computation graph

Now, we create a computation graph.

# number of features
num_features = 784
# number of target labels
num_labels = 10
# learning rate (alpha)
learning_rate = 0.05
# batch size
batch_size = 128
# number of epochs
num_steps = 5001

# input data
train_dataset = mnist.train.images
train_labels = mnist.train.labels
test_dataset = mnist.test.images
test_labels = mnist.test.labels
valid_dataset = mnist.validation.images
valid_labels = mnist.validation.labels

# initialize a tensorflow graph
graph = tf.Graph()

with graph.as_default():
    """
    defining all the nodes
    """

    # Inputs
    tf_train_dataset = tf.placeholder(tf.float32, shape=(batch_size, num_features))
    tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
    tf_valid_dataset = tf.constant(valid_dataset)
    tf_test_dataset = tf.constant(test_dataset)

    # Variables.
    weights = tf.Variable(tf.truncated_normal([num_features, num_labels]))
    biases = tf.Variable(tf.zeros([num_labels]))

    # Training computation.
    logits = tf.matmul(tf_train_dataset, weights) + biases
    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
                        labels=tf_train_labels, logits=logits))

    # Optimizer.
    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

    # Predictions for the training, validation, and test data.
    train_prediction = tf.nn.softmax(logits)
    valid_prediction = tf.nn.softmax(tf.matmul(tf_valid_dataset, weights) + biases)
    test_prediction = tf.nn.softmax(tf.matmul(tf_test_dataset, weights) + biases)

Some important points to note:

• For the training data, we use a placeholder that will be fed at run time with a training minibatch. The technique of using minibatches for training model using gradient descent is termed as Stochastic Gradient Descent.
  

  In both gradient descent (GD) and stochastic gradient descent (SGD), you update a set of parameters in an iterative manner to minimize an error function. While in GD, you have to run through ALL the samples in your training set to do a single update for a parameter in a particular iteration, in SGD, on the other hand, you use ONLY ONE or SUBSET of training sample from your training set to do the update for a parameter in a particular iteration. If you use SUBSET, it is called Minibatch Stochastic gradient Descent. Thus, if the number of training samples are large, in fact very large, then using gradient descent may take too long because in every iteration when you are updating the values of the parameters, you are running through the complete training set. On the other hand, using SGD will be faster because you use only one training sample and it starts improving itself right away from the first sample. SGD often converges much faster compared to GD but the error function is not as well minimized as in the case of GD. Often in most cases, the close approximation that you get in SGD for the parameter values are enough because they reach the optimal values and keep oscillating there.

• The weight matrix is initialized using random values following a (truncated) normal distribution. This is achieved using tf.truncated_normal method. The biases get initialized to zero using tf.zeros method.
• Now, we multiply the inputs with the weight matrix, and add biases. We compute the softmax and cross-entropy using tf.nn.softmax_cross_entropy_with_logits (it’s one operation in TensorFlow, because it’s very common, and it can be optimized). We take the average of this cross-entropy across all training examples using tf.reduce_mean method.
• We are going to minimize the loss using gradient descent. For this, we use tf.train.GradientDescentOptimizer.
• train_prediction, valid_prediction and test_prediction are not part of training, but merely here so that we can report accuracy figures as we train.

Step 5: Running the computation graph

Since we have already built the computation graph, now it’s time to run it through a session.

# utility function to calculate accuracy
def accuracy(predictions, labels):
    correctly_predicted = np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))
    accu = (100.0 * correctly_predicted) / predictions.shape[0]
    return accu

with tf.Session(graph=graph) as session:
    # initialize weights and biases
    tf.global_variables_initializer().run()
    print("Initialized")

    for step in range(num_steps):
        # pick a randomized offset
        offset = np.random.randint(0, train_labels.shape[0] - batch_size - 1)

        # Generate a minibatch.
        batch_data = train_dataset[offset:(offset + batch_size), :]
        batch_labels = train_labels[offset:(offset + batch_size), :]

        # Prepare the feed dict
        feed_dict = {tf_train_dataset : batch_data,
                     tf_train_labels : batch_labels}

        # run one step of computation
        _, l, predictions = session.run([optimizer, loss, train_prediction],
                                        feed_dict=feed_dict)

        if (step % 500 == 0):
            print("Minibatch loss at step {0}: {1}".format(step, l))
            print("Minibatch accuracy: {:.1f}%".format(
                accuracy(predictions, batch_labels)))
            print("Validation accuracy: {:.1f}%".format(
                accuracy(valid_prediction.eval(), valid_labels)))

    print("\nTest accuracy: {:.1f}%".format(
        accuracy(test_prediction.eval(), test_labels)))

Output:

Initialized
Minibatch loss at step 0: 11.68728256225586
Minibatch accuracy: 10.2%
Validation accuracy: 14.3%
Minibatch loss at step 500: 2.239773750305176
Minibatch accuracy: 46.9%
Validation accuracy: 67.6%
Minibatch loss at step 1000: 1.0917563438415527
Minibatch accuracy: 78.1%
Validation accuracy: 75.0%
Minibatch loss at step 1500: 0.6598564386367798
Minibatch accuracy: 78.9%
Validation accuracy: 78.6%
Minibatch loss at step 2000: 0.24766433238983154
Minibatch accuracy: 91.4%
Validation accuracy: 81.0%
Minibatch loss at step 2500: 0.6181786060333252
Minibatch accuracy: 84.4%
Validation accuracy: 82.5%
Minibatch loss at step 3000: 0.9605385065078735
Minibatch accuracy: 85.2%
Validation accuracy: 83.9%
Minibatch loss at step 3500: 0.6315320730209351
Minibatch accuracy: 85.2%
Validation accuracy: 84.4%
Minibatch loss at step 4000: 0.812285840511322
Minibatch accuracy: 82.8%
Validation accuracy: 85.0%
Minibatch loss at step 4500: 0.5949224233627319
Minibatch accuracy: 80.5%
Validation accuracy: 85.6%
Minibatch loss at step 5000: 0.47554320096969604
Minibatch accuracy: 89.1%
Validation accuracy: 86.2%

Test accuracy: 86.5%

Some important points to note:

• In every iteration, a minibatch is selected by choosing a random offset value using np.random.randint method.
• To feed the placeholders tf_train_dataset and tf_train_label, we create a feed_dict like this:

  feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}
  

• A shortcut way of performing one step of computation is:

  _, l, predictions = session.run([optimizer, loss, train_prediction], feed_dict=feed_dict)
  

  This node returns the new values of loss and predictions after performing optimization step.

This brings us to the end of the implementation. Complete code can be found here.

Finally, here are some points to ponder upon:

• You can try to tweak with the parameters like learning rate, batch size, number of epochs, etc and achieve better results. You can also try a different optimizer like tf.train.AdamOptimizer.
• Accuracy of above model can be improved by using a neural network with one or more hidden layers. We will discuss its implementation using TensorFlow in some upcoming articles.
• Softmax Regression vs. k Binary Classifiers
  One should be aware of the scenarios where softmax regression works and where it doesn’t. In many cases, you may need to use k different binary logistic classifiers for each of the k possible values of the class label.Suppose you are working on a computer vision problem where you’re trying to classify images into three different classes:
  Case 1: Suppose that your classes are indoor_scene, outdoor_urban_scene, and outdoor_wilderness_scene.

  Case 2: Suppose your classes are indoor_scene, black_and_white_image, and image_has_people.

  In which case you would use Softmax Regression classifier and in which case you would use 3 Binary Logistic Regression classifiers?
  This will depend on whether the 3 classes are mutually exclusive.
  In case 1, a scene can be either indoor_scene, outdoor_urban_scene or outdoor_wilderness_scene. So, assuming that each training example is labeled with exactly one of the 3 classes, we should build a softmax classifier with k = 3.

  However, in case 2, the classes are not mutually exclusive since a scene can be both indoor and have people in it. So, in this case, it would be more appropriate to build 3 binary logistic regression classifiers. This way, for each new scene, your algorithm can separately decide whether it falls into each of the 3 categories.

References:

• http://www.kdnuggets.com/2016/07/softmax-regression-related-logistic-regression.html
• https://classroom.udacity.com/courses/ud730
• http://ufldl.stanford.edu/wiki/index.php/Softmax_Regression

This article is a brief introduction to TensorFlow library using Python programming language.

Introduction

TensorFlow is an open-source software library. TensorFlow was originally developed by researchers and engineers working on the Google Brain Team within Google’s Machine Intelligence research organization for the purposes of conducting machine learning and deep neural networks research, but the system is general enough to be applicable in a wide variety of other domains as well!

Let us first try to understand what the word TensorFlow actually mean!

TensorFlow is basically a software library for numerical computation using data flow graphs where:

• nodes in the graph represent mathematical operations.
• edges in the graph represent the multidimensional data arrays (called tensors) communicated between them. (Please note that tensor is the central unit of data in TensorFlow).

Consider the diagram given below:

Here, add is a node which represents addition operation. a and b are input tensors and c is the resultant tensor.

This flexible architecture allows you to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API!

TensorFlow APIs

TensorFlow provides multiple APIs (Application Programming Interfaces). These can be classified into 2 major categories:

1. Low level API:
   • complete programming control
   • recommended for machine learning researchers
   • provides fine levels of control over the models
   • TensorFlow Core is the low level API of TensorFlow.
2. High level API:
   • built on top of TensorFlow Core
   • easier to learn and use than TensorFlow Core
   • make repetitive tasks easier and more consistent between different users
   • tf.contrib.learn is an example of a high level API.

In this article, we first discuss the basics of TensorFlow Core and then explore the higher level API, tf.contrib.learn.

TensorFlow Core

1. Installing TensorFlow

An easy to follow guide for TensorFlow installation is available here:
Installing TensorFlow.

Once installed, you can ensure a successful installation by running this command in python interpreter:

import tensorflow as tf

2. The Computational Graph

Any TensorFlow Core program can be divided into two discrete sections:

• Building the computational graph.A computational graph is nothing but a series of TensorFlow operations arranged into a graph of nodes.

• Running the computational graph.To actually evaluate the nodes, we must run the computational graph within a session. A session encapsulates the control and state of the TensorFlow runtime.

Now, let us write our very first TensorFlow program to understand above concept:

# importing tensorflow
import tensorflow as tf

# creating nodes in computation graph
node1 = tf.constant(3, dtype=tf.int32)
node2 = tf.constant(5, dtype=tf.int32)
node3 = tf.add(node1, node2)

# create tensorflow session object
sess = tf.Session()

# evaluating node3 and printing the result
print("Sum of node1 and node2 is:",sess.run(node3))

# closing the session
sess.close()

Output:

Sum of node1 and node2 is: 8

Let us try to understand above code:

• Step 1 : Create a computational graph
  By creating computational graph, we mean defining the nodes. Tensorflow provides different types of nodes for a variety of tasks. Each node takes zero or more tensors as inputs and produces a tensor as an output.

• In above program, the nodes node1 and node2 are of tf.constant type. A constant node takes no inputs, and it outputs a value it stores internally. Note that we can also specify the data type of output tensor using dtype argument.

  node1 = tf.constant(3, dtype=tf.int32)
  node2 = tf.constant(5, dtype=tf.int32)
  

• node3 is of tf.add type. It takes two tensors as input and returns their sum as output tensor.

  node3 = tf.add(node1, node2)
  

• Step 2 : Run the computational graph
  In order to run the computational graph, we need to create a session. To create a session, we simply do:

  sess = tf.Session()
  

  Now, we can invoke the run method of session object to perform computations on any node:

  print("Sum of node1 and node2 is:",sess.run(node3))
  

  Here, node3 gets evaluated which further invokes node1 and node2. Finally, we close the session using:

  sess.close()
  

Note: Another(and better) method of working with sessions is to use with block like this:

with tf.Session() as sess:
    print("Sum of node1 and node2 is:",sess.run(node3))

The benefit of this approach is that you do not need to close the session explicitly as it gets automatically closed once control goes out of the scope of with block.

3. Variables

TensorFlow has Variable nodes too which can hold variable data. They are mainly used to hold and update parameters of a training model.

Variables are in-memory buffers containing tensors. They must be explicitly initialized and can be saved to disk during and after training. You can later restore saved values to exercise or analyze the model.

An important difference to note between a constant and Variable is:

A constant’s value is stored in the graph and its value is replicated wherever the graph is loaded. A variable is stored separately, and may live on a parameter server.

Given below is an example using Variable:

# importing tensorflow
import tensorflow as tf

# creating nodes in computation graph
node = tf.Variable(tf.zeros([2,2]))

# running computation graph
with tf.Session() as sess:
    # initialize all global variables
    sess.run(tf.global_variables_initializer())
    # evaluating node
    print("Tensor value before addition:\n",sess.run(node))
    # elementwise addition to tensor
    node = node.assign(node + tf.ones([2,2]))
    # evaluate node again
    print("Tensor value after addition:\n", sess.run(node))

Output:

Tensor value before addition:
 [[ 0.  0.]
 [ 0.  0.]]
Tensor value after addition:
 [[ 1.  1.]
 [ 1.  1.]]

In above program:

• We define a node of type Variable and assign it some initial value.

  node = tf.Variable(tf.zeros([2,2]))
  

• To initialize the variable node in current session’s scope, we do:

  sess.run(tf.global_variables_initializer())
  

• To assign a new value to a variable node, we can use assign method like this:

  node = node.assign(node + tf.ones([2,2]))
  

4. Placeholders

A graph can be parameterized to accept external inputs, known as placeholders. A placeholder is a promise to provide a value later.

While evaluating the graph involving placeholder nodes, a feed_dict parameter is passed to the session’s run method to specify Tensors that provide concrete values to these placeholders.

Consider the example given below:

# importing tensorflow
import tensorflow as tf

# creating nodes in computation graph
a = tf.placeholder(tf.int32, shape=(3,1))
b = tf.placeholder(tf.int32, shape=(1,3))
c = tf.matmul(a,b)

# running computation graph
with tf.Session() as sess:
    print(sess.run(c, feed_dict={a:[[3],[2],[1]], b:[[1,2,3]]}))

Output:

[[3 6 9]
 [2 4 6]
 [1 2 3]]

Let us try to understand above program:

• We define placeholder nodes a and b like this:

  a = tf.placeholder(tf.int32, shape=(3,1))
  b = tf.placeholder(tf.int32, shape=(1,3))
  

  The first argument is the data type of the tensor and one of the optional argument is shape of the tensor.

• We define another node c which does the operation of matrix multiplication (matmul). We pass the two placeholder nodes as argument.

  c = tf.matmul(a,b)
  

• Finally, when we run the session, we pass the value of placeholder nodes in feed_dict argument of sess.run:

  print(sess.run(c, feed_dict={a:[[3],[2],[1]], b:[[1,2,3]]}))
  

  Consider the diagrams shown below to clear the concept:

• Initially:
  
• After sess.run:
  

5. An example : Linear Regression model

Given below is an implementation of a Linear Regression model using TensorFlow Core API.

# importing the dependencies
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt

# Model Parameters
learning_rate = 0.01
training_epochs = 2000
display_step = 200

# Training Data
train_X = np.asarray([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
                         7.042,10.791,5.313,7.997,5.654,9.27,3.1])
train_y = np.asarray([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
                         2.827,3.465,1.65,2.904,2.42,2.94,1.3])
n_samples = train_X.shape[0]

# Test Data
test_X = np.asarray([6.83, 4.668, 8.9, 7.91, 5.7, 8.7, 3.1, 2.1])
test_y = np.asarray([1.84, 2.273, 3.2, 2.831, 2.92, 3.24, 1.35, 1.03])

# Set placeholders for feature and target vectors
X = tf.placeholder(tf.float32)
y = tf.placeholder(tf.float32)

# Set model weights and bias
W = tf.Variable(np.random.randn(), name="weight")
b = tf.Variable(np.random.randn(), name="bias")

# Construct a linear model
linear_model = W*X + b

# Mean squared error
cost = tf.reduce_sum(tf.square(linear_model - y)) / (2*n_samples)

# Gradient descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

# Initializing the variables
init = tf.global_variables_initializer()

# Launch the graph
with tf.Session() as sess:
    # Load initialized variables in current session
    sess.run(init)

    # Fit all training data
    for epoch in range(training_epochs):
        # perform gradient descent step
        sess.run(optimizer, feed_dict={X: train_X, y: train_y})

        # Display logs per epoch step
        if (epoch+1) % display_step == 0:
            c = sess.run(cost, feed_dict={X: train_X, y: train_y})
            print("Epoch:{0:6} \t Cost:{1:10.4} \t W:{2:6.4} \t b:{3:6.4}".
                  format(epoch+1, c, sess.run(W), sess.run(b)))

    # Print final parameter values
    print("Optimization Finished!")
    training_cost = sess.run(cost, feed_dict={X: train_X, y: train_y})
    print("Final training cost:", training_cost, "W:", sess.run(W), "b:",
          sess.run(b), '\n')

    # Graphic display
    plt.plot(train_X, train_y, 'ro', label='Original data')
    plt.plot(train_X, sess.run(W) * train_X + sess.run(b), label='Fitted line')
    plt.legend()
    plt.show()

    # Testing the model
    testing_cost = sess.run(tf.reduce_sum(tf.square(linear_model - y)) / (2 * test_X.shape[0]),
                            feed_dict={X: test_X, y: test_y})

    print("Final testing cost:", testing_cost)
    print("Absolute mean square loss difference:", abs(training_cost - testing_cost))

    # Display fitted line on test data
    plt.plot(test_X, test_y, 'bo', label='Testing data')
    plt.plot(train_X, sess.run(W) * train_X + sess.run(b), label='Fitted line')
    plt.legend()
    plt.show()

Epoch:   200 	 Cost:    0.1715 	 W: 0.426 	 b:-0.4371
Epoch:   400 	 Cost:    0.1351 	 W:0.3884 	 b:-0.1706
Epoch:   600 	 Cost:    0.1127 	 W:0.3589 	 b:0.03849
Epoch:   800 	 Cost:   0.09894 	 W:0.3358 	 b:0.2025
Epoch:  1000 	 Cost:   0.09047 	 W:0.3176 	 b:0.3311
Epoch:  1200 	 Cost:   0.08526 	 W:0.3034 	 b:0.4319
Epoch:  1400 	 Cost:   0.08205 	 W:0.2922 	 b:0.5111
Epoch:  1600 	 Cost:   0.08008 	 W:0.2835 	 b:0.5731
Epoch:  1800 	 Cost:   0.07887 	 W:0.2766 	 b:0.6218
Epoch:  2000 	 Cost:   0.07812 	 W:0.2712 	 b:  0.66
Optimization Finished!
Final training cost: 0.0781221 W: 0.271219 b: 0.65996 



Final testing cost: 0.0756337
Absolute mean square loss difference: 0.00248838

Let us try to understand the above code.

• First of all, we define some parameters for training our model, like:

  learning_rate = 0.01
  training_epochs = 2000
  display_step = 200
  

• Then we define placeholder nodes for feature and target vector.

  X = tf.placeholder(tf.float32)
  y = tf.placeholder(tf.float32)
  

• Then, we define variable nodes for weight and bias.

  W = tf.Variable(np.random.randn(), name="weight")
  b = tf.Variable(np.random.randn(), name="bias")
  

• linear_model is an operational node which calculates the hypothesis for the linear regression model.

  linear_model = W*X + b
  

• Loss (or cost) per gradient descent is calculated as the mean squared error and its node is defined as:

  cost = tf.reduce_sum(tf.square(linear_model - y)) / (2*n_samples)
  

• Finally, we have the optimizer node which implements the Gradient Descent Algorithm.

  optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
  

• Now, the training data is fit into the linear model by applying the Gradient Descent Algorithm. The task is repeated training_epochs number of times. In each epoch, we perform the gradient descent step like this:

  sess.run(optimizer, feed_dict={X: train_X, y: train_y})
  

• After every display_step number of epochs, we print the value of current loss which is found using:

  c = sess.run(cost, feed_dict={X: train_X, y: train_y})
  

• The model is evaluated on test data and testing_cost is calculated using:

  testing_cost = sess.run(tf.reduce_sum(tf.square(linear_model - y)) / (2 * test_X.shape[0]),
                            feed_dict={X: test_X, y: test_y})
  

tf.contrib.learn

tf.contrib.learn is a high-level TensorFlow library that simplifies the mechanics of machine learning, including the following:

• running training loops
• running evaluation loops
• managing data sets
• managing feeding

Let us try to see the implementation of Linear regression on same data we used above using tf.contrib.learn.

# importing the dependencies
import tensorflow as tf
import numpy as np

# declaring list of features
features = [tf.contrib.layers.real_valued_column("X")]

# creating a linear regression estimator
estimator = tf.contrib.learn.LinearRegressor(feature_columns=features)

# training and test data
train_X = np.asarray([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
                         7.042,10.791,5.313,7.997,5.654,9.27,3.1])
train_y = np.asarray([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
                         2.827,3.465,1.65,2.904,2.42,2.94,1.3])
test_X = np.asarray([6.83, 4.668, 8.9, 7.91, 5.7, 8.7, 3.1, 2.1])
test_y = np.asarray([1.84, 2.273, 3.2, 2.831, 2.92, 3.24, 1.35, 1.03])

# function to feed dict of numpy arrays into the model for training
input_fn = tf.contrib.learn.io.numpy_input_fn({"X":train_X}, train_y,
                                              batch_size=4, num_epochs=2000)

# function to feed dict of numpy arrays into the model for testing
test_input_fn = tf.contrib.learn.io.numpy_input_fn({"X":test_X}, test_y)

# fit training data into estimator
estimator.fit(input_fn=input_fn)

# print value of weight and bias
W = estimator.get_variable_value('linear/X/weight')[0][0]
b = estimator.get_variable_value('linear/bias_weight')[0]
print("W:", W, "\tb:", b)

# evaluating the final loss
train_loss = estimator.evaluate(input_fn=input_fn)['loss']
test_loss = estimator.evaluate(input_fn=test_input_fn)['loss']
print("Final training loss:", train_loss)
print("Final testing loss:", test_loss)

W: 0.252928 	b: 0.802972
Final training loss: 0.153998
Final testing loss: 0.0777036

Let us try to understand the above code.

• The shape and type of feature matrix is declared using a list. Each element of the list defines the structure of a column. In above example, we have only 1 feature which stores real values and has been given a name X.

  features = [tf.contrib.layers.real_valued_column("X")]
  

• Then, we need an estimator. An estimator is nothing but a pre-defined model with many useful methods and parameters. In above example, we use a Linear Regression model estimator.

  estimator = tf.contrib.learn.LinearRegressor(feature_columns=features)
  

• For training purpose, we need to use an input function which is responsible for feeding data to estimator while training. It takes the feature column values as dictionary. Many other parameters like batch size, number of epochs, etc can be specified.

  input_fn = tf.contrib.learn.io.numpy_input_fn({"X":train_X}, 
                train_y, batch_size=4, num_epochs=2000)
  

• To fit training data to estimator, we simply use fit method of estimator in which input function is passed as an argument.

  estimator.fit(input_fn=input_fn)
  

• Once training is complete, we can get the value of different variables using get_variable_value method of estimator. You can get a list of all variables using get_variable_names method.

  W = estimator.get_variable_value('linear/X/weight')[0][0]
  b = estimator.get_variable_value('linear/bias_weight')[0]
  

• The mean squared error/loss can be computed as:

  train_loss = estimator.evaluate(input_fn=input_fn)['loss']
  test_loss = estimator.evaluate(input_fn=test_input_fn)['loss']
  

This brings us to the end of this Introduction to TensorFlow article!

From here, you can try to explore this tutorial: MNIST For ML Beginners.

I will upload a few more tutorials on TensorFlow soon!

Multithreading in Python | Part-1

This article discusses the concept of thread synchronization in case of multithreading in Python programming language.

Synchronization between threads

Thread synchronization is defined as a mechanism which ensures that two or more concurrent threads do not simultaneously execute some particular program segment known as critical section.

Critical section refers to the parts of the program where the shared resource is accessed.

For example, in the diagram below, 3 threads try to access shared resource or critical section at the same time.

Concurrent accesses to shared resource can lead to race condition.

A race condition occurs when two or more threads can access shared data and they try to change it at the same time. As a result, the values of variables may be unpredictable and vary depending on the timings of context switches of the processes.

Consider the program below to understand the concept of race condition:

import threading

# global variable x
x = 0

def increment():
"""
function to increment global variable x
"""
global x
x += 1

def thread_task():
"""
task for thread
calls increment function 100000 times.
"""
for _ in range(100000):
increment()

def main_task():
global x
# setting global variable x as 0
x = 0

# creating threads
t1 = threading.Thread(target=thread_task)
t2 = threading.Thread(target=thread_task)

# start threads
t1.start()
t2.start()

# wait until threads finish their job
t1.join()
t2.join()

if __name__ == "__main__":
for i in range(10):
main_task()
print("Iteration {0}: x = {1}".format(i,x))

Output:

Iteration 0: x = 175005
Iteration 1: x = 200000
Iteration 2: x = 200000
Iteration 3: x = 169432
Iteration 4: x = 153316
Iteration 5: x = 200000
Iteration 6: x = 167322
Iteration 7: x = 200000
Iteration 8: x = 169917
Iteration 9: x = 153589

In above program:

• Two threads t1 and t2 are created in main_task function and global variable x is set to 0.
• Each thread has a target function thread_task in which increment function is called 100000 times.
• increment function will increment the global variable x by 1 in each call.

The expected final value of x is 200000 but what we get in 10 iterations of main_task function is some different values.

This happens due to concurrent access of threads to the shared variable x. This unpredictability in value of x is nothing but race condition.

Given below is a diagram which shows how can race condition occur in above program:

Notice that expected value of x in above diagram is 12 but due to race condition, it turns out to be 11!

Hence, we need a tool for proper synchronization between multiple threads.

Using Locks

threading module provides a Lock class to deal with the race conditions. Lock is implemented using a Semaphore object provided by the Operating System.

A semaphore is a synchronization object that controls access by multiple processes/threads to a common resource in a parallel programming environment. It is simply a value in a designated place in operating system (or kernel) storage that each process/thread can check and then change. Depending on the value that is found, the process/thread can use the resource or will find that it is already in use and must wait for some period before trying again. Semaphores can be binary (0 or 1) or can have additional values. Typically, a process/thread using semaphores checks the value and then, if it using the resource, changes the value to reflect this so that subsequent semaphore users will know to wait.

Lock class provides following methods:

• acquire([blocking]) : To acquire a lock. A lock can be blocking or non-blocking.
  • When invoked with the blocking argument set to True (the default), thread execution is blocked until the lock is unlocked, then lock is set to locked and return True.
  • When invoked with the blocking argument set to False, thread execution is not blocked. If lock is unlocked, then set it to locked and return True else return False immediately.
• release() : To release a lock.
  • When the lock is locked, reset it to unlocked, and return. If any other threads are blocked waiting for the lock to become unlocked, allow exactly one of them to proceed.
  • If lock is already unlocked, a ThreadError is raised.

Consider the example given below:

import threading

# global variable x
x = 0

def increment():
"""
function to increment global variable x
"""
global x
x += 1

def thread_task(lock):
"""
task for thread
calls increment function 100000 times.
"""
for _ in range(100000):
lock.acquire()
increment()
lock.release()

def main_task():
global x
# setting global variable x as 0
x = 0

# creating a lock
lock = threading.Lock()

# creating threads
t1 = threading.Thread(target=thread_task, args=(lock,))
t2 = threading.Thread(target=thread_task, args=(lock,))

# start threads
t1.start()
t2.start()

# wait until threads finish their job
t1.join()
t2.join()

if __name__ == "__main__":
for i in range(10):
main_task()
print("Iteration {0}: x = {1}".format(i,x))

Output:

Iteration 0: x = 200000
Iteration 1: x = 200000
Iteration 2: x = 200000
Iteration 3: x = 200000
Iteration 4: x = 200000
Iteration 5: x = 200000
Iteration 6: x = 200000
Iteration 7: x = 200000
Iteration 8: x = 200000
Iteration 9: x = 200000

Let us try to understand the above code step by step:

• Firstly, a Lock object is created using:

    lock = threading.Lock()
  

• Then, lock is passed as target function argument:

    t1 = threading.Thread(target=thread_task, args=(lock,))
    t2 = threading.Thread(target=thread_task, args=(lock,))
  

• In the critical section of target function, we apply lock using lock.acquire() method. As soon as a lock is acquired, no other thread can access the critical section (here, increment function) until the lock is released using lock.release() method.

    lock.acquire()
    increment()
    lock.release()
  

  As you can see in the results, the final value of x comes out to be 200000 every time (which is the expected final result).

Here is a diagram given below which depicts the implementation of locks in above program:

This brings us to the end of this tutorial series on Multithreading in Python.
Finally, here are are a few advantages and disadvantages of multithreading:

Advantages:

• It doesn’t block the user. This is because threads are independent of each other.
• Better use of system resources is possible since threads execute tasks parallely.
• Enhanced performance on multi-processor machines.
• Multi-threaded servers and interactive GUIs use multithreading exclusively.

Disadvantages:

• As number of threads increase, complexity increases.
• Synchronization of shared resources (objects, data) is necessary.
• It is difficult to debug, result is sometimes unpredictable.
• Potential deadlocks which leads to starvation, i.e. some threads may not be served with a bad design
• Constructing and synchronizing threads is CPU/memory intensive.

Upcoming articles are gonna cover asyncio!

This article covers the basics of multithreading in Python programming language.

Just like multiprocessing, multithreading is a way of achieving multitasking. In multithreading, the concept of threads is used.

Let us first understand the concept of thread in computer architecture.

Thread

In computing, a process is an instance of a computer program that is being executed. Any process has 3 basic components:

• An executable program.
• The associated data needed by the program (variables, work space, buffers, etc.)
• The execution context of the program (State of process)

A thread is an entity within a process that can be scheduled for execution. Also, it is the smallest unit of processing that can be performed in an OS (Operating System).

In simple words, a thread is a sequence of such instructions within a program that can be executed independently of other code. For simplicity, you can assume that a thread is simply a subset of a process!

A thread contains all this information in a Thread Control Block (TCB):

• Thread Identifier: Unique id (TID) is assigned to every new thread
• Stack pointer: Points to thread’s stack in the process. Stack contains the local variables under thread’s scope.
• Program counter: a register which stores the address of the instruction currently being executed by thread.
• Thread state: can be running, ready, waiting, start or done.
• Thread’s register set: registers assigned to thread for computations.
• Parent process Pointer: A pointer to the Process control block (PCB) of the process that the thread lives on.

Consider the diagram below to understand the relation between process and its thread:

Multithreading

Multiple threads can exist within one process where:

• Each thread contains its own register set and local variables (stored in stack).
• All thread of a process share global variables (stored in heap) and the program code.

Consider the diagram below to understand how multiple threads exist in memory:

Multithreading is defined as the ability of a processor to execute multiple threads concurrently.

In a simple, single-core CPU, it is achieved using frequent switching between threads. This is termed as context switching. In context switching, the state of a thread is saved and state of another thread is loaded whenever any interrupt (due to I/O or manually set) takes place. Context switching takes place so frequently that all the threads appear to be running parallely (this is termed as multitasking).

Consider the diagram below in which a process contains two active threads:

Multithreading in Python

In Python, the threading module provides a very simple and intuitive API for spawning multiple threads in a program.

Let us consider a simple example using threading module:

# importing the threading module
import threading

def print_cube(num):
    """
    function to print cube of given num
    """
    print("Cube: {}".format(num * num * num))

def print_square(num):
    """
    function to print square of given num
    """
    print("Square: {}".format(num * num))

if __name__ == "__main__":
    # creating thread
    t1 = threading.Thread(target=print_square, args=(10,))
    t2 = threading.Thread(target=print_cube, args=(10,))

    # starting thread 1
    t1.start()
    # starting thread 2
    t2.start()

    # wait until thread 1 is completely executed
    t1.join()
    # wait until thread 2 is completely executed
    t2.join()

    # both threads completely executed
    print("Done!")

Square: 100
Cube: 1000
Done!

Let us try to understand the above code:

• To import the threading module, we do:

  import threading
  

• To create a new thread, we create an object of Thread class. It takes following arguments:
  • target: the function to be executed by thread
  • args: the arguments to be passed to the target function

  In above example, we created 2 threads with different target functions:

  t1 = threading.Thread(target=print_square, args=(10,))
  t2 = threading.Thread(target=print_cube, args=(10,))
  

• To start a thread, we use start method of Thread class.

  t1.start()
  t2.start()
  

• Once the threads start, the current program (you can think of it like a main thread) also keeps on executing. In order to stop execution of current program until a thread is complete, we use join method.

  t1.join()
  t2.join()
  

  As a result, the current program will first wait for the completion of t1 and then t2. Once, they are finished, the remaining statements of current program are executed.

Consider the diagram below for a better understanding of how above program works:

Now, consider the python program given below in which we print thread name and corresponding process for each task:

import threading
import os

def task1():
    print("Task 1 assigned to thread: {}".format(threading.current_thread().name))
    print("ID of process running task 1: {}".format(os.getpid()))

def task2():
    print("Task 2 assigned to thread: {}".format(threading.current_thread().name))
    print("ID of process running task 2: {}".format(os.getpid()))

if __name__ == "__main__":

    # print ID of current process
    print("ID of process running main program: {}".format(os.getpid()))

    # print name of main thread
    print("Main thread name: {}".format(threading.main_thread().name))

    # creating threads
    t1 = threading.Thread(target=task1, name='t1')
    t2 = threading.Thread(target=task2, name='t2')  

    # starting threads
    t1.start()
    t2.start()

    # wait until all threads finish
    t1.join()
    t2.join()

ID of process running main program: 11758
Main thread name: MainThread
Task 1 assigned to thread: t1
ID of process running task 1: 11758
Task 2 assigned to thread: t2
ID of process running task 2: 11758

Let us try to understand the above code:

• We use os.getpid() function to get ID of current process.

  print("ID of process running main program: {}".format(os.getpid()))
  

  As it is clear from the output, the process ID remains same for all threads.

• We use threading.main_thread() function to get the main thread object. In normal conditions, the main thread is the thread from which the Python interpreter was started. name attribute of thread object is used to get the name of thread.

  print("Main thread name: {}".format(threading.main_thread().name))
  

• We use the threading.current_thread() function to get the current thread object.

  print("Task 1 assigned to thread: {}".format(threading.current_thread().name))
  

The diagram given below clears the above concept:

So, this was a brief introduction to multithreading in Python. The next article in this series covers synchronization between multiple threads.

This article discusses two important concepts related to multiprocessing in Python:

• Synchronization between processes
• Pooling of processes

Synchronization between processes

Process synchronization is defined as a mechanism which ensures that two or more concurrent processes do not simultaneously execute some particular program segment known as critical section.

Critical section refers to the parts of the program where the shared resource is accessed.

For example, in the diagram below, 3 processes try to access shared resource or critical section at the same time.

Concurrent accesses to shared resource can lead to race condition.

A race condition occurs when two or more processes can access shared data and they try to change it at the same time. As a result, the values of variables may be unpredictable and vary depending on the timings of context switches of the processes.

Consider the program below to understand the concept of race condition:

import multiprocessing

def withdraw(balance):
    # function to withdraw from account
    for _ in range(10000):
        balance.value = balance.value - 1

def deposit(balance):
    # function to deposit to account
    for _ in range(10000):
        balance.value = balance.value + 1

def perform_transactions():
    # initial balance (in shared memory)
    balance = multiprocessing.Value('i', 100)

    # creating new processes
    p1 = multiprocessing.Process(target=withdraw, args=(balance,))
    p2 = multiprocessing.Process(target=deposit, args=(balance,))

    # starting processes
    p1.start()
    p2.start()

    # wait until processes are finished
    p1.join()
    p2.join()

    # print final balance
    print("Final balance = {}".format(balance.value))

if __name__ == "__main__":
    for _ in range(10):
        # perform same transaction process 10 times
        perform_transactions()

If you run above program, you will get some unexpected values like this:

Final balance = 1311
Final balance = 199
Final balance = 558
Final balance = -2265
Final balance = 1371
Final balance = 1158
Final balance = -577
Final balance = -1300
Final balance = -341
Final balance = 157

In above program, 10000 withdraw and 10000 deposit transactions are carried out with initial balance as 100. The expected final balance is 100 but what we get in 10 iterations of perform_transactions function is some different values.

This happens due to concurrent access of processes to the shared data balance. This unpredictability in balance value is nothing but race condition.

Let us try to understand it better using the sequence diagrams given below. These are the different sequences which can be produced in above example for a single withdraw and deposit action.

• This is a possible sequence which gives wrong answer as both processes read the same value and write it back accordingly.
  
• These are 2 possible sequences which are desired in above scenario.
  

Using Locks

multiprocessing module provides a Lock class to deal with the race conditions. Lock is implemented using a Semaphore object provided by the Operating System.

A semaphore is a synchronization object that controls access by multiple processes to a common resource in a parallel programming environment. It is simply a value in a designated place in operating system (or kernel) storage that each process can check and then change. Depending on the value that is found, the process can use the resource or will find that it is already in use and must wait for some period before trying again. Semaphores can be binary (0 or 1) or can have additional values. Typically, a process using semaphores checks the value and then, if it using the resource, changes the value to reflect this so that subsequent semaphore users will know to wait.

Consider the example given below:

import multiprocessing

def withdraw(balance, lock):
    # function to withdraw from account
    for _ in range(10000):
        lock.acquire()
        balance.value = balance.value - 1
        lock.release()

def deposit(balance, lock):
    # function to deposit to account
    for _ in range(10000):
        lock.acquire()
        balance.value = balance.value + 1
        lock.release()

def perform_transactions():
    # intitial balance (in shared memory)
    balance = multiprocessing.Value('i', 100)

    # creating a lock object
    lock = multiprocessing.Lock()

    # creating new processes
    p1 = multiprocessing.Process(target=withdraw, args=(balance,lock))
    p2 = multiprocessing.Process(target=deposit, args=(balance,lock))

    # starting processes
    p1.start()
    p2.start()

    # wait until processes are finished
    p1.join()
    p2.join()

    # print final balance
    print("Final balance = {}".format(balance.value))

if __name__ == "__main__":
    for _ in range(10):
        # perform same transaction process 10 times
        perform_transactions()

Output:

Final balance = 100
Final balance = 100
Final balance = 100
Final balance = 100
Final balance = 100
Final balance = 100
Final balance = 100
Final balance = 100
Final balance = 100
Final balance = 100

Let us try to understand the above code step by step:

• Firstly, a Lock object is created using:

  lock = multiprocessing.Lock()
  

• Then, lock is passed as target function argument:

   p1 = multiprocessing.Process(target=withdraw, args=(balance,lock))
   p2 = multiprocessing.Process(target=deposit, args=(balance,lock))
  

• In the critical section of target function, we apply lock using lock.acquire() method. As soon as a lock is acquired, no other process can access its critical section until the lock is released using lock.release() method.

  lock.acquire()
  balance.value = balance.value - 1
  lock.release()
  

  As you can see in the results, the final balance comes out to be 100 every time (which is the expected final result).

Pooling between processes

Let us consider a simple program to find squares of numbers in a given list.

def square(n):
    return (n*n)

if __name__ == "__main__":
    # input list
    mylist = [1,2,3,4,5]
    # empty list to store result
    result = []

    for num in mylist:
        result.append(square(num))

    print(result)

Output:

[1, 4, 9, 16, 25]

It is a simple program to calculate squares of elements of a given list. In a multi-core/multi-processor system, consider the diagram below to understand how above program will work:

Only one of the cores is used for program execution and it’s quite possible that other cores remain idle.

In order to utilize all the cores, multiprocessing module provides a Pool class. The Pool class represents a pool of worker processes. It has methods which allows tasks to be offloaded to the worker processes in a few different ways. Consider the diagram below:

Here, the task is offloaded/distributed among the cores/processes automatically by Pool object. User doesn’t need to worry about creating processes explicitly.

Consider the program given below:

import multiprocessing
import os

def square(n):
    print("Worker process id for {0}: {1}".format(n, os.getpid()))
    return (n*n)

if __name__ == "__main__":
    # input list
    mylist = [1,2,3,4,5]

    # creating a pool object
    p = multiprocessing.Pool()

    # map list to target function
    result = p.map(square, mylist)

    print(result)

Output:

Worker process id for 2: 4152
Worker process id for 1: 4151
Worker process id for 4: 4151
Worker process id for 3: 4153
Worker process id for 5: 4152
[1, 4, 9, 16, 25]

Let us try to understand above code step by step:

• We create a Pool object using:

    p = multiprocessing.Pool()
  

  There are a few arguments for gaining more control over offloading of task. These are:

  • processes: specify the number of worker processes.
  • maxtasksperchild: specify the maximum number of task to be assigned per child.

  All the processes in a pool can be made to perform some initialization using these arguments:

  • initializer: specify an initialization function for worker processes.
  • initargs: arguments to be passed to initializer.
• Now, in order to perform some task, we have to map it to some function. In the example above, we map mylist to square function. As a result, the contents of mylist and definition of square will be distributed among the cores.

    result = p.map(square, mylist)
  

• Once all the worker processes finish their task, a list is returned with the final result.

This brings us to the end of this tutorial series on multiprocessing in Python!
Next few articles will cover multithreading and asyncio libraries in Python.

• run independently
• have their own memory space.

Consider the program below to understand this concept:

import multiprocessing

# empty list with global scope
result = []

def square_list(mylist):
    """
    function to square a given list
    """
    global result
    # append squares of mylist to global list result
    for num in mylist:
        result.append(num * num)
    # print global list result
    print("Result(in process p1): {}".format(result))

if __name__ == "__main__":
    # input list
    mylist = [1,2,3,4]

    # creating new process
    p1 = multiprocessing.Process(target=square_list, args=(mylist,))
    # starting process
    p1.start()
    # wait until process is finished
    p1.join()

    # print global result list
    print("Result(in main program): {}".format(result))

Result(in process p1): [1, 4, 9, 16]
Result(in main program): []

In above example, we try to print contents of global list result at two places:

• In square_list function. Since, this function is called by process p1, result list is changed in memory space of process p1 only.
• After the completion of process p1 in main program. Since main program is run by a different process, its memory space still contains the empty result list.

Diagram shown below clears this concept:

Sharing data between processes

1. Shared memory : multiprocessing module provides Array and Value objects to share data between processes.
   • Array: a ctypes array allocated from shared memory.
   • Value: a ctypes object allocated from shared memory.

   Given below is a simple example showing use of Array and Value for sharing data between processes.

   import multiprocessing
   
   def square_list(mylist, result, square_sum):
       """
       function to square a given list
       """
       # append squares of mylist to result array
       for idx, num in enumerate(mylist):
           result[idx] = num * num
   
       # square_sum value
       square_sum.value = sum(result)
   
       # print result Array
       print("Result(in process p1): {}".format(result[:]))
   
       # print square_sum Value
       print("Sum of squares(in process p1): {}".format(square_sum.value))
   
   if __name__ == "__main__":
       # input list
       mylist = [1,2,3,4]
   
       # creating Array of int data type with space for 4 integers
       result = multiprocessing.Array('i', 4)
   
       # creating Value of int data type
       square_sum = multiprocessing.Value('i')
   
       # creating new process
       p1 = multiprocessing.Process(target=square_list, args=(mylist, result, square_sum))
   
       # starting process
       p1.start()
   
       # wait until process is finished
       p1.join()
   
       # print result array
       print("Result(in main program): {}".format(result[:]))
   
       # print square_sum Value
       print("Sum of squares(in main program): {}".format(square_sum.value))
   

   Result(in process p1): [1, 4, 9, 16]
   Sum of squares(in process p1): 30
   Result(in main program): [1, 4, 9, 16]
   Sum of squares(in main program): 30
   

   Let us try to understand the above code line by line:

   • First of all, we create an Array result like this:

       result = multiprocessing.Array('i', 4)
     

     • First argument is the data type. ‘i’ stands for integer whereas ‘d’ stands for float data type.
     • Second argument is the size of array. Here, we create an array of 4 elements.

     Similarly, we create a Value square_sum like this:

       square_sum = multiprocessing.Value('i')
     

     Here, we only need to specify data type. The value can be given an initial value(say 10) like this:

       square_sum = multiprocessing.Value('i', 10)
     

   • Secondly, we pass result and square_sum as arguments while creating Process object.

       p1 = multiprocessing.Process(target=square_list, args=(mylist, result, square_sum))
     

   • result array elements are given a value by specifying index of array element.

       for idx, num in enumerate(mylist):
           result[idx] = num * num
     

     square_sum is given a value by using its value attribute:

       square_sum.value = sum(result)
     

   • In order to print result array elements, we use result[:] to print complete array.

       print("Result(in process p1): {}".format(result[:]))
     

     Value of square_sum is simply printed as:

       print("Sum of squares(in process p1): {}".format(square_sum.value))
     

   Here is a diagram depicting how processes share Array and Value object:
   

2. Server process : Whenever a python program starts, a server process is also started. From there on, whenever a new process is needed, the parent process connects to the server and requests it to fork a new process.
   A server process can hold Python objects and allows other processes to manipulate them using proxies.
   multiprocessing module provides a Manager class which controls a server process. Hence, managers provide a way to create data which can be shared between different processes.

Server process managers are more flexible than using shared memory objects because they can be made to support arbitrary object types like lists, dictionaries, Queue, Value, Array, etc. Also, a single manager can be shared by processes on different computers over a network. They are, however, slower than using shared memory.

Consider the example given below:

import multiprocessing

def print_records(records):
    """
    function to print record(tuples) in records(list)
    """
    for record in records:
        print("Name: {0}\nScore: {1}\n".format(record[0], record[1]))

def insert_record(record, records):
    """
    function to add a new record to records(list)
    """
    records.append(record)
    print("New record added!\n")

if __name__ == '__main__':
    with multiprocessing.Manager() as manager:
        # creating a list in server process memory
        records = manager.list([('Sam', 10), ('Adam', 9), ('Kevin',9)])
        # new record to be inserted in records
        new_record = ('Jeff', 8)

        # creating new processes
        p1 = multiprocessing.Process(target=insert_record, args=(new_record, records))
        p2 = multiprocessing.Process(target=print_records, args=(records,))

        # running process p1 to insert new record
        p1.start()
        p1.join()

        # running process p2 to print records
        p2.start()
        p2.join()

New record added!

Name: Sam
Score: 10

Name: Adam
Score: 9

Name: Kevin
Score: 9

Name: Jeff
Score: 8

Let us try to understand above piece of code:

• First of all, we create a manager object using:

    with multiprocessing.Manager() as manager:
  

  All the lines under with statement block are under the scope of manager object.

• Then, we create a list records in server process memory using:

    records = manager.list([('Sam', 10), ('Adam', 9), ('Kevin',9)])
  

  Similarly, you can create a dictionary as manager.dict method.

• Finally, we create to processes p1 (to insert a new record in records list) and p2 (to print records) and run them while passing records as one of the arguments.

The concept of server process is depicted in the diagram shown below:

Communication between processes

Effective use of multiple processes usually requires some communication between them, so that work can be divided and results can be aggregated.
multiprocessing supports two types of communication channel between processes:

• Queue
• Pipe

1. Queue : A simple way to communicate between process with multiprocessing is to use a Queue to pass messages back and forth. Any Python object can pass through a Queue.
   Note: The multiprocessing.Queue class is a near clone of queue.Queue.
   Consider the example program given below:

   import multiprocessing
   
   def square_list(mylist, q):
       """
       function to square a given list
       """
       # append squares of mylist to queue
       for num in mylist:
           q.put(num * num)
   
   def print_queue(q):
       """
       function to print queue elements
       """
       print("Queue elements:")
       while not q.empty():
           print(q.get())
       print("Queue is now empty!")
   
   if __name__ == "__main__":
       # input list
       mylist = [1,2,3,4]
   
       # creating multiprocessing Queue
       q = multiprocessing.Queue()
   
       # creating new processes
       p1 = multiprocessing.Process(target=square_list, args=(mylist, q))
       p2 = multiprocessing.Process(target=print_queue, args=(q,))
   
       # running process p1 to square list
       p1.start()
       p1.join()
   
       # running process p2 to get queue elements
       p2.start()
       p2.join()
   

   Queue elements:
   1
   4
   9
   16
   Queue is now empty!
   

   Let us try to understand the above code step by step:

   • Firstly, we create a multiprocessing Queue using:

       q = multiprocessing.Queue()
     

   • Then we pass empty queue q to square_list function through process p1. Elements are inserted to queue using put method.

       q.put(num * num)
     

   • In order to print queue elements, we use get method until queue is not empty.

       while not q.empty():
           print(q.get())
     

   Given below is a simple diagram depicting the operations on queue:
   

2. Pipes : A pipe can have only two endpoints. Hence, it is preferred over queue when only two-way communication is required.multiprocessing module provides Pipe() function which returns a pair of connection objects connected by a pipe. The two connection objects returned by Pipe() represent the two ends of the pipe. Each connection object has send() and recv() methods (among others).
   Consider the program given below:

   import multiprocessing
   
   def sender(conn, msgs):
       """
       function to send messages to other end of pipe
       """
       for msg in msgs:
           conn.send(msg)
           print("Sent the message: {}".format(msg))
       conn.close()
   
   def receiver(conn):
       """
       function to print the messages received from other
       end of pipe
       """
       while 1:
           msg = conn.recv()
           if msg == "END":
               break
           print("Received the message: {}".format(msg))
   
   if __name__ == "__main__":
       # messages to be sent
       msgs = ["hello", "hey", "hru?", "END"]
   
       # creating a pipe
       parent_conn, child_conn = multiprocessing.Pipe()
   
       # creating new processes
       p1 = multiprocessing.Process(target=sender, args=(parent_conn,msgs))
       p2 = multiprocessing.Process(target=receiver, args=(child_conn,))
   
       # running processes
       p1.start()
       p2.start()
   
       # wait until processes finish
       p1.join()
       p2.join()
   

   Sent the message: hello
   Sent the message: hey
   Sent the message: hru?
   Received the message: hello
   Sent the message: END
   Received the message: hey
   Received the message: hru?
   

   Let us try to understand above code:

   • A pipe was created simply using:

       parent_conn, child_conn = multiprocessing.Pipe()
     

     The function returned two connection objects for the two ends of the pipe.

   • Message is sent from one end of pipe to another using send method.

       conn.send(msg)
     

   • To receive any messages at one end of a pipe, we use recv method.

       msg = conn.recv()
     

   • In above program, we send a list of messages from one end to another. At the other end, we read messages until we receive “END” message.

   Consider the diagram given below which shows the relation b/w pipe and processes:
   

Note: Data in a pipe may become corrupted if two processes (or threads) try to read from or write to the same end of the pipe at the same time. Of course there is no risk of corruption from processes using different ends of the pipe at the same time. Also note that, Queues do proper synchronization between processes, at the expense of more complexity. Hence, queues are said to be thread and process safe!

This article is a brief yet concise introduction to multiprocessing in Python programming language.

What is multiprocessing?

Multiprocessing refers to the ability of a system to support more than one processor at the same time. Applications in a multiprocessing system are broken to smaller routines that run independently. The operating system allocates these threads to the processors improving performance of the system.

Why multiprocessing?

Consider a computer system with a single processor. If it is assigned several processes at the same time, it will have to interrupt each task and switch briefly to another, to keep all of the processes going.
This situation is just like a chef working in a kitchen alone. He has to do several tasks like baking, stirring, kneading dough, etc.

So the gist is that: The more tasks you must do at once, the more difficult it gets to keep track of them all, and keeping the timing right becomes more of a challenge.
This is where the concept of multiprocessing arises!

A multiprocessing system can have:

• multiprocessor, i.e. a computer with more than one central processor.
• multi-core processor, i.e. a single computing component with two or more independent actual processing units (called “cores”).

Here, the CPU can easily executes several tasks at once, with each task using its own processor.

It is just like the chef in last situation being assisted by his assistants. Now, they can divide the tasks among themselves and chef doesn’t need to switch between his tasks.

Multiprocessing in Python

In Python, the multiprocessing module includes a very simple and intuitive API for dividing work between multiple processes.
Let us consider a simple example using multiprocessing module:

# importing the multiprocessing module
import multiprocessing

def print_cube(num):
    """
    function to print cube of given num
    """
    print("Cube: {}".format(num * num * num))

def print_square(num):
    """
    function to print square of given num
    """
    print("Square: {}".format(num * num))

if __name__ == "__main__":
    # creating processes
    p1 = multiprocessing.Process(target = print_square, args =(10, ))
    p2 = multiprocessing.Process(target = print_cube, args =(10, ))

    # starting process 1
    p1.start()
    # starting process 2
    p2.start()

    # wait until process 1 is finished
    p1.join()
    # wait until process 2 is finished
    p2.join()

    # both processes finished
    print("Done !")

Square: 100
Cube: 1000
Done!

Let us try to understand the above code:

• To import the multiprocessing module, we do:

  import multiprocessing
  

• To create a process, we create an object of Process class. It takes following arguments:
  • target: the function to be executed by process
  • args: the arguments to be passed to the target function

  Note: Process constructor takes many other arguments also which will be discussed later. In above example, we created 2 processes with different target functions:

  p1 = multiprocessing.Process(target=print_square, args=(10, ))
  p2 = multiprocessing.Process(target=print_cube, args=(10, ))
  

• To start a process, we use start method of Process class.

  p1.start()
  p2.start()
  

• Once the processes start, the current program also keeps on executing. In order to stop execution of current program until a process is complete, we use join method.

  p1.join()
  p2.join()
  

  As a result, the current program will first wait for the completion of p1 and then p2. Once, they are completed, the next statements of current program are executed.

Let us consider another program to understand the concept of different processes running on same python script. In this example below, we print the ID of the processes running the target functions:

# importing the multiprocessing module
import multiprocessing
import os

def worker1():
    # printing process id
    print("ID of process running worker1: {}".format(os.getpid()))

def worker2():
    # printing process id
    print("ID of process running worker2: {}".format(os.getpid()))

if __name__ == "__main__":
    # printing main program process id
    print("ID of main process: {}".format(os.getpid()))

    # creating processes
    p1 = multiprocessing.Process(target = worker1)
    p2 = multiprocessing.Process(target = worker2)

    # starting processes
    p1.start()
    p2.start()

    # process IDs
    print("ID of process p1: {}".format(p1.pid))
    print("ID of process p2: {}".format(p2.pid))

    # wait until processes are finished
    p1.join()
    p2.join()

    # both processes finished
    print("Both processes finished execution !")

    # check if processes are alive
    print("Process p1 is alive: {}".format(p1.is_alive()))
    print("Process p2 is alive: {}".format(p2.is_alive()))

ID of main process: 28628
ID of process running worker1: 29305
ID of process running worker2: 29306
ID of process p1: 29305
ID of process p2: 29306
Both processes finished execution!
Process p1 is alive: False
Process p2 is alive: False

• The main python script has a different process ID and multiprocessing module spawns new processes with different process IDs as we create Process objects p1 and p2. In above program, we use os.getpid() function to get ID of process running the current target function.Notice that it matches with the process IDs of p1 and p2 which we obtain using pid attribute of Process class.
• Each process runs independently and has its own memory space.
• As soon as the execution of target function is finished, the processes get terminated. In above program we used is_alive method of Process class to check if a process is still active or not.

Consider the diagram below to understand how new processes are different from main Python script:

So, this was a brief introduction to multiprocessing in Python.
Find next part here: Multiprocessing in Python | Part-2

References:

• http://learn.parallax.com/tutorials/language/blocklyprop/blocklyprop-functions-and-multicore/bit-about-multicore
• https://docs.python.org/3/library/multiprocessing.html

This article discusses the basics of Logistic Regression and its implementation in Python. Logistic regression is basically a supervised classification algorithm.

In a classification problem, the target variable(or output), y, can take only discrete values for given set of features(or inputs), X.

We can also say that the target variable is categorical. Based on the number of categories, Logistic regression can be classified as:

• binomial: target variable can have only 2 possible types: “0” or “1” which may represent “win” vs “loss”, “pass” vs “fail”, “dead” vs “alive”, etc.
• multinomial: target variable can have 3 or more possible types which are not ordered(i.e. types have no quantitative significance) like “disease A” vs “disease B” vs “disease C”.
• ordinal: it deals with target variables with ordered categories. For example, a test score can be categorized as:“very poor”, “poor”, “good”, “very good”. Here, each category can be given a score like 0, 1, 2, 3.

First of all, we explore the simplest form of Logistic Regression, i.e Binomial Logistic Regression.

Binomial Logistic Regression

Consider an example dataset which maps the number of hours of study with the result of an exam. The result can take only two values, namely passed(1) or failed(0):

Hours(x)	0.50	0.75	1.00	1.25	1.50	1.75	2.00	2.25	2.50	2.75	3.00	3.25	3.50	3.75	4.00	4.25	4.50	4.75	5.00	5.50
Pass(y)	0	0	0	0	0	0	1	0	1	0	1	0	1	0	1	1	1	1	1	1

So, we have

i.e. y is a categorical target variable which can take only two possible type:“0” or “1”.
In order to generalize our model, we assume that:

• The dataset has ‘p’ feature variables and ‘n’ observations.
• The feature matrix is represented as:

Here, denotes the values of feature for observation.
Here, we are keeping the convention of letting = 1. (Keep reading, you will understand the logic in a few moments).

The observation, , can be represented as:

represents the predicted response for observation, i.e. . The formula we use for calculating is called hypothesis.

If you have gone though Linear Regression, you should recall that in Linear Regression, the hypothesis we used for prediction was:

where, are the regression coefficients.
Let regression coefficient matrix/vector, be:

Then, in a more compact form,

The reason for taking = 1 is pretty clear now.

We needed to do a matrix product, but there was no actual multiplied to in original hypothesis formula. So, we defined = 1.

Now, if we try to apply Linear Regression on above problem, we are likely to get continuous values using the hypothesis we discussed above. Also, it does not make sense for to take values larger that 1 or smaller than 0.
So, some modifications are made to the hypothesis for classification:

where,

is called logistic function or the sigmoid function.
Here is a plot showing g(z):

We can infer from above graph that:

• g(z) tends towards 1 as
• g(z) tends towards 0 as
• g(z) is always bounded between 0 and 1

So, now, we can define conditional probabilities for 2 labels(0 and 1) for observation as:

We can write it more compactly as:

Now, we define another term, likelihood of parameters as:

Likelihood is nothing but the probability of data(training examples), given a model and specific parameter values(here, ). It measures the support provided by the data for each possible value of the . We obtain it by multiplying all for given .

And for easier calculations, we take log likelihood:

The cost function for logistic regression is proportional to inverse of likelihood of parameters. Hence, we can obtain an expression for cost function, J using log likelihood equation as:

and our aim is to estimate so that cost function is minimized !!

Using Gradient descent algorithm

Firstly, we take partial derivatives of w.r.t each to derive the stochastic gradient descent rule(we present only the final derived value here):

Here, y and h(x) represent the response vector and predicted response vector(respectively). Also, is the vector representing the observation values for feature.
Now, in order to get min ,

where is called learning rate and needs to be set explicitly.
Let us see the python implementation of above technique on a sample dataset (download it from here):

import csv
import numpy as np
import matplotlib.pyplot as plt
 
 
def loadCSV(filename):
    '''
    function to load dataset
    '''
    with open(filename,"r") as csvfile:
        lines = csv.reader(csvfile)
        dataset = list(lines)
        for i in range(len(dataset)):
            dataset[i] = [float(x) for x in dataset[i]]     
    return np.array(dataset)
 
 
def normalize(X):
    '''
    function to normalize feature matrix, X
    '''
    mins = np.min(X, axis = 0)
    maxs = np.max(X, axis = 0)
    rng = maxs - mins
    norm_X = 1 - ((maxs - X)/rng)
    return norm_X
 
 
def logistic_func(beta, X):
    '''
    logistic(sigmoid) function
    '''
    return 1.0/(1 + np.exp(-np.dot(X, beta.T)))
 
 
def log_gradient(beta, X, y):
    '''
    logistic gradient function
    '''
    first_calc = logistic_func(beta, X) - y.reshape(X.shape[0], -1)
    final_calc = np.dot(first_calc.T, X)
    return final_calc
 
 
def cost_func(beta, X, y):
    '''
    cost function, J
    '''
    log_func_v = logistic_func(beta, X)
    y = np.squeeze(y)
    step1 = y * np.log(log_func_v)
    step2 = (1 - y) * np.log(1 - log_func_v)
    final = -step1 - step2
    return np.mean(final)
 
 
def grad_desc(X, y, beta, lr=.01, converge_change=.001):
    '''
    gradient descent function
    '''
    cost = cost_func(beta, X, y)
    change_cost = 1
    num_iter = 1
     
    while(change_cost > converge_change):
        old_cost = cost
        beta = beta - (lr * log_gradient(beta, X, y))
        cost = cost_func(beta, X, y)
        change_cost = old_cost - cost
        num_iter += 1
     
    return beta, num_iter 
 
 
def pred_values(beta, X):
    '''
    function to predict labels
    '''
    pred_prob = logistic_func(beta, X)
    pred_value = np.where(pred_prob >= .5, 1, 0)
    return np.squeeze(pred_value)
 
 
def plot_reg(X, y, beta):
    '''
    function to plot decision boundary
    '''
    # labelled observations
    x_0 = X[np.where(y == 0.0)]
    x_1 = X[np.where(y == 1.0)]
     
    # plotting points with diff color for diff label
    plt.scatter(x_0[:, 1], x_0[:, 2], c='b', label='y = 0')
    plt.scatter(x_1[:, 1], x_1[:, 2], c='r', label='y = 1')
     
    # plotting decision boundary
    x1 = np.arange(0, 1, 0.1)
    x2 = -(beta[0,0] + beta[0,1]*x1)/beta[0,2]
    plt.plot(x1, x2, c='k', label='reg line')
 
    plt.xlabel('x1')
    plt.ylabel('x2')
    plt.legend()
    plt.show()
     
 
     
if __name__ == "__main__":
    # load the dataset
    dataset = loadCSV('dataset1.csv')
     
    # normalizing feature matrix
    X = normalize(dataset[:, :-1])
     
    # stacking columns wth all ones in feature matrix
    X = np.hstack((np.matrix(np.ones(X.shape[0])).T, X))
 
    # response vector
    y = dataset[:, -1]
 
    # initial beta values
    beta = np.matrix(np.zeros(X.shape[1]))
 
    # beta values after running gradient descent
    beta, num_iter = grad_desc(X, y, beta)
 
    # estimated beta values and number of iterations
    print("Estimated regression coefficients:", beta)
    print("No. of iterations:", num_iter)
 
    # predicted labels
    y_pred = pred_values(beta, X)
     
    # number of correctly predicted labels
    print("Correctly predicted labels:", np.sum(y == y_pred))
     
    # plotting regression line
    plot_reg(X, y, beta)

Estimated regression coefficients: [[  1.70474504  15.04062212 -20.47216021]]
No. of iterations: 2612
Correctly predicted labels: 100

Note: Gradient descent is one of the many way to estimate .
Basically, these are more advanced algorithms which can be easily run in Python once you have defined your cost function and your gradients. These algorithms are:

• BFGS(Broyden–Fletcher–Goldfarb–Shanno algorithm)
• L-BFGS(Like BFGS but uses limited memory)
• Conjugate Gradient

Advantages/disadvantages of using any one of these algorithms over Gradient descent:

• Advantages
  • Don’t need to pick learning rate
  • Often run faster (not always the case)
  • Can numerically approximate gradient for you (doesn’t always work out well)
• Disadvantages
  • More complex
  • More of a black box unless you learn the specifics

Multinomial Logistic Regression

In Multinomial Logistic Regression, the output variable can have more than two possible discrete outputs.

Consider the Digit Dataset. Here, the output variable is the digit value which can take values out of (0, 12, 3, 4, 5, 6, 7, 8, 9).

Given below is the implementation of Multinomial Logisitc Regression using scikit-learn to make predictions on digit dataset.

from sklearn import datasets, linear_model, metrics
  
# load the digit dataset
digits = datasets.load_digits()
  
# defining feature matrix(X) and response vector(y)
X = digits.data
y = digits.target
 
# splitting X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4,
                                                    random_state=1)
  
# create logistic regression object
reg = linear_model.LogisticRegression()
  
# train the model using the training sets
reg.fit(X_train, y_train)
 
# making predictions on the testing set
y_pred = reg.predict(X_test)
  
# comparing actual response values (y_test) with predicted response values (y_pred)
print("Logistic Regression model accuracy(in %):", 
metrics.accuracy_score(y_test, y_pred)*100)

Logistic Regression model accuracy(in %): 95.6884561892

At last, here are some points about Logistic regression to ponder upon:

• Does NOT assume a linear relationship between the dependent variable and the independent variables, but it does assume linear relationship between the logit of the explanatory variables and the response.
• Independent variables can be even the power terms or some other nonlinear transformations of the original independent variables.
• The dependent variable does NOT need to be normally distributed, but it typically assumes a distribution from an exponential family (e.g. binomial, Poisson, multinomial, normal,…); binary logistic regression assume binomial distribution of the response.
• The homogeneity of variance does NOT need to be satisfied.
• Errors need to be independent but NOT normally distributed.
• It uses maximum likelihood estimation (MLE) rather than ordinary least squares (OLS) to estimate the parameters, and thus relies on large-sample approximations.

References:

• http://cs229.stanford.edu/notes/cs229-notes1.pdf
• http://machinelearningmastery.com/logistic-regression-for-machine-learning/
• https://onlinecourses.science.psu.edu/stat504/node/164