tutorial – Indian Pythonista

Facebook Messenger Bot Tutorial

indianpythonista — Mon, 10 Apr 2017 13:14:46 +0000

Learn how to create a Facebook messenger bot using Python in this video tutorial series.

Video 1

Creating a facebook app

A facebook App allows you to retrieve and post data on facebook programmatically via Facebook API.

Facebook API provides the interface and methods to interact with the Facebook App.

This video demonstrates how to create a facebook app.

Video 2

Setting up project

This video demonstrates how to setup the projects folder.

You need to have Python 3 or Python 2 installed. (Python 3 is preferred)
Install virtualenv
```
pip install virtualenv
```
Create a virtual environment
Install:
- flask
- requests
- pymessenger

Video 3

Setting up webhook

This video demonstrates how to setup webhook (callback URL) for your messenger bot.

We use ngrok for exposing local server to internet.

Video 4

Receiving messages

In this video, we write a function to receive messages from FB page as POST HTTP Requests.

Video 5

Create a basic echo bot

This video demonstrates how to add replying capability to our messenger bot.

pymessenger is used for sending replies.

Video 6

Heroku setup

This video demonstrates how to set up your project folder for Heroku deployment.

You need to install:

Heroku CLI
Git CLI
gunicorn
```
pip install gunicorn
```

Video 7

Heroku deployment

This video demonstrates how to deploy your messenger bot on Heroku.

eval in Python

indianpythonista — Sat, 25 Mar 2017 16:55:14 +0000

Note: This article has also featured on geeksforgeeks.org

This article discusses a built-in function in Python, eval.

It is an interesting hack/utility in Python which lets a Python program run Python code within itself.

The eval() method parses the expression passed to it and runs python expression(code) within the program.

The syntax of eval is:

eval(expression, globals=None, locals=None)

expression: this string is parsed and evaluated as a Python expression
globals (optional): a dictionary to specify the available global methods and variables.
locals (optional): another dictionary to specify the available local methods and variables.

Let us explore it with the help of a simple Python program:

from math import *

def secret_function():
    return "Secret key is 1234"

def function_creator():
    # expression to be evaluated
    expr = raw_input("Enter the function(in terms of x):")

    # variable used in expression
    x = int(raw_input("Enter the value of x:"))

    # evaluating expression
    y = eval(expr)

    # printing evaluated result
    print("y = {}".format(y))

if __name__ == "__main__":
    function_creator()

function_creator is a function which evaluates the mathematical functions created by user.

Consider an output:

Enter the function(in terms of x):x*(x+1)*(x+2)
Enter the value of x:3
y = 60

Let us analyze the code a bit:

The above function takes any expression in variable x as input.
Then user has to enter a value of x.
Finally, we evaluate the python expression using eval() built-in function by passing the expr as argument.

Vulnerability issues with eval

Our current version of function_creator has a few vulnerabilities.

The user can easily expose hidden values in the program, or call a dangerous function as eval will execute anything passed to it.

For example, if you input like this:

Enter the function(in terms of x):secret_function()
Enter the value of x:0

You will get output:

y = Secret key is 1234

Also, consider the situation when you have imported os module in your python program. The os module provides portable way to use operating system functionalities like: read or write a file. A single command can delete all files in your system!

Of course in most cases (like desktop programs) the user can’t do any more than they could do by writing their own python script, but in some applications (like web apps, kiosk computers), this could be a risk!

The solution is to restrict eval to only the functions and variables we want to make available.

Making eval safe

eval function comes with the facility of explicitly passing a list of functions or variables that it can access. We need to pass it as argument in the form of a dictionary.

Consider the example below:

from math import *

def secret_function():
    return "Secret key is 1234"

def function_creator():
    # expression to be evaluated
    expr = raw_input("Enter the function(in terms of x):")

    # variable used in expression
    x = int(raw_input("Enter the value of x:"))

    # passing variable x in safe dictionary
    safe_dict['x'] = x

    # evaluating expression
    y = eval(expr, {"__builtins__":None}, safe_dict)

    # printing evaluated result
    print("y = {}".format(y))

if __name__ == "__main__":
    # list of safe methods
    safe_list = ['acos', 'asin', 'atan', 'atan2', 'ceil', 'cos',
                 'cosh', 'degrees', 'e', 'exp', 'fabs', 'floor',
                 'fmod', 'frexp', 'hypot', 'ldexp', 'log', 'log10',
                 'modf', 'pi', 'pow', 'radians', 'sin', 'sinh', 'sqrt',
                 'tan', 'tanh']

    # creating a dictionary of safe methods
    safe_dict = dict([(k, locals().get(k, None)) for k in safe_list])

    function_creator()

Now if we try to run above program like:

Enter the function(in terms of x):secret_function()
Enter the value of x:0

We get output:

NameError: name 'secret_function' is not defined

Let us analyze above code step by step:

First of all, we create a list of methods we want to allow as safe_list.
Next, we create a dictionary of safe methods. In this dictionary, keys are the method names and values are their local namespaces.
```
safe_dict = dict([(k, locals().get(k, None)) for k in safe_list])
```
locals() is a built-in method which returns a dictionary which maps all the methods and variables in local scope with their namespaces.
```
safe_dict['x'] = x
```
Here, we add local variable x to the safe_dict too. No local variable other than x will get identified by eval function.
eval accepts dictionaries of local as well as global variables as arguments. So, in order to ensure that none of the built-in methods is available to eval expression, we pass another dictionary along with safe_dict as well, as shown below:
```
y = eval(expr, {"__builtins__":None}, safe_dict)
```

So, in this way, we have made our eval function safe from any possible hacks!

Uses of eval

eval is not much used due to security reasons as we explored above.

Still, it comes handy in some situations like:

You may want to use it to allow users to enter their own “scriptlets”: small expressions (or even small functions), that can be used to customize the behavior of a complex system.
eval is also sometimes used in applications needing to evaluate math expressions. This is much easier than writing an expression parser.

Linear Regression

indianpythonista — Tue, 14 Mar 2017 18:31:13 +0000

This article discusses the basics of linear regression and its implementation in Python programming language.

Linear regression is a statistical approach for modelling relationship between a dependent variable with a given set of independent variables.

Note: In this article, we refer dependent variables as response and independent variables as features for simplicity.

In order to provide a basic understanding of linear regression, we start with the most basic version of linear regression, i.e. Simple linear regression.

Simple Linear Regression

Simple linear regression is an approach for predicting a response using a single feature.

It is assumed that the two variables are linearly related. Hence, we try to find a linear function that predicts the response value(y) as accurately as possible as a function of the feature or independent variable(x).

Let us consider a dataset where we have a value of response y for every feature x:

For generality, we define:

x as feature vector, i.e ,

y as response vector, i.e

for n observations (in above example, n=10).

A scatter plot of above dataset looks like:-

Now, the task is to find a line which fits best in above scatter plot so that we can predict the response for any new feature values. (i.e a value of x not present in dataset)

This line is called regression line.

The equation of regression line is represented as:

Here,

represents the predicted response value for observation.
and are regression coefficients and represent y-intercept and slope of regression line respectively.

To create our model, we must “learn” or estimate the values of regression coefficients and . And once we’ve estimated these coefficients, we can use the model to predict responses!

In this article, we are going to use the Least Squares technique.

Now consider:

Here, is residual error in observation.
So, our aim is to minimize .

We define the squared error or cost function, J as:

and our task is to find the value of and for which is minimum!

Without going into the mathematical details, we present the result here:

where is the sum of cross-deviations of y and x:

and is the sum of squared deviations of x:

Note: The complete derivation for finding least squares estimates in simple linear regression can be found here.

Given below is the python implementation of above technique on our small dataset:

import numpy as np
import matplotlib.pyplot as plt

def estimate_coef(x, y):
    # number of observations/points
    n = np.size(x)

    # mean of x and y vector
    m_x, m_y = np.mean(x), np.mean(y)

    # calculating cross-deviation and deviation about x
    SS_xy = np.sum(y*x - n*m_y*m_x)
    SS_xx = np.sum(x*x - n*m_x*m_x)

    # calculating regression coefficients
    b_1 = SS_xy / SS_xx
    b_0 = m_y - b_1*m_x

    return(b_0, b_1)

def plot_regression_line(x, y, b):
    # plotting the actual points as scatter plot
    plt.scatter(x, y, color = "m",
               marker = "o", s = 30)

    # predicted response vector
    y_pred = b[0] + b[1]*x

    # plotting the regression line
    plt.plot(x, y_pred, color = "g")

    # putting labels
    plt.xlabel('x')
    plt.ylabel('y')

    # function to show plot
    plt.show()

def main():
    # observations
    x = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
    y = np.array([1, 3, 2, 5, 7, 8, 8, 9, 10, 12])

    # estimating coefficients
    b = estimate_coef(x, y)
    print("Estimated coefficients:\nb_0 = {}  \
          \nb_1 = {}".format(b[0], b[1]))

    # plotting regression line
    plot_regression_line(x, y, b)

if __name__ == "__main__":
    main()

Click here for NumPy tutorial

Output of above piece of code is:

Estimated coefficients:
b_0 = -0.0586206896552
b_1 = 1.45747126437

And graph obtained looks like this:

Multiple linear regression

Multiple linear regression attempts to model the relationship between two or more features and a response by fitting a linear equation to observed data.

Clearly, it is nothing but an extension of Simple linear regression.

Consider a dataset with p features(or independent variables) and one response(or dependent variable).
Also, the dataset contains n rows/observations.

We define:

X (feature matrix) = a matrix of size n X p where denotes the values of feature for observation.

So,

and

y (response vector) = a vector of size n where denotes the value of response for observation.

The regression line for p features is represented as:

where is predicted response value for observation and are the regression coefficients.

Also, we can write:

where represents residual error in observation.

We can generalize our linear model a little bit more by representing feature matrix X as:

So now, the linear model can be expressed in terms of matrices as:

where,

and

Now, we determine estimate of , i.e. using Least Squares method.

As already explained, Least Squares method tends to determine for which is minimized.

We present the result directly here:

where represents the transpose of the matrix while represents the matrix inverse.

Knowing the least square estimates, , the multiple linear regression model can now be estimated as:

where is estimated response vector.

Note: The complete derivation for obtaining least square estimates in multiple linear regression can be found here.

Given below is the implementation of multiple linear regression technique on the Boston house pricing dataset dataset using Scikit-learn.

import matplotlib.pyplot as plt
import numpy as np
from sklearn import datasets, linear_model, metrics

# load the boston dataset
boston = datasets.load_boston(return_X_y=False)

# defining feature matrix(X) and response vector(y)
X = boston.data
y = boston.target

# splitting X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4,
                                                    random_state=1)

# create linear regression object
reg = linear_model.LinearRegression()

# train the model using the training sets
reg.fit(X_train, y_train)

# regression coefficients
print('Coefficients: \n', reg.coef_)

# variance score: 1 means perfect prediction
print('Variance score: {}'.format(reg.score(X_test, y_test)))

# plot for residual error

## setting plot style
plt.style.use('fivethirtyeight')

## plotting residual errors in training data
plt.scatter(reg.predict(X_train), reg.predict(X_train) - y_train,
            color = "green", s = 10, label = 'Train data')

## plotting residual errors in test data
plt.scatter(reg.predict(X_test), reg.predict(X_test) - y_test,
            color = "blue", s = 10, label = 'Test data')

## plotting line for zero residual error
plt.hlines(y = 0, xmin = 0, xmax = 50, linewidth = 2)

## plotting legend
plt.legend(loc = 'upper right')

## plot title
plt.title("Residual errors")

## function to show plot
plt.show()

Click here for Scikit-learn tutorial

Output of above program looks like this:

Coefficients:
[ -8.80740828e-02   6.72507352e-02   5.10280463e-02   2.18879172e+00
-1.72283734e+01   3.62985243e+00   2.13933641e-03  -1.36531300e+00
2.88788067e-01  -1.22618657e-02  -8.36014969e-01   9.53058061e-03
-5.05036163e-01]
Variance score: 0.720898784611

and Residual Error plot looks like this:

In above example, we determine accuracy score using Explained Variance Score.
We define:

where is the estimated target output, the corresponding (correct) target output, and is Variance, the square of the standard deviation.
The best possible score is 1.0, lower values are worse.

Assumptions

Given below are the basic assumptions that a linear regression model makes regarding a dataset on which it is applied:

Linear relationship: Relationship between response and feature variables should be linear. The linearity assumption can be tested using scatter plots. As shown below, 1st figure represents linearly related variables where as variables in 2nd and 3rd figure are most likely non-linear. So, 1st figure will give better predictions using linear regression.
Little or no multi-collinearity: It is assumed that there is little or no multicollinearity in the data. Multicollinearity occurs when the features (or independent variables) are not independent from each other.
Little or no auto-correlation: Another assumption is that there is little or no autocorrelation in the data. Autocorrelation occurs when the residual errors are not independent from each other. You can refer here for more insight into this topic.
Homoscedasticity: Homoscedasticity describes a situation in which the error term (that is, the “noise” or random disturbance in the relationship between the independent variables and the dependent variable) is the same across all values of the independent variables. As shown below, figure 1 has homoscedasticity while figure 2 has heteroscedasticity.

As we reach to the end of this article, we discuss some applications of linear regression below.

Applications:

1. Trend lines: A trend line represents the variation in some quantitative data with passage of time (like GDP, oil prices, etc.). These trends usually follow a linear relationship. Hence, linear regression can be applied to predict future values. However, this method suffers from a lack of scientific validity in cases where other potential changes can affect the data.

2. Economics: Linear regression is the predominant empirical tool in economics. For example, it is used to predict consumption spending, fixed investment spending, inventory investment, purchases of a country’s exports, spending on imports, the demand to hold liquid assets, labor demand, and labor supply.

3. Finance: Capital price asset model uses linear regression to analyze and quantify the systematic risks of an investment.

4. Biology: Linear regression is used to model causal relationships between parameters in biological systems.

References:

Naive Bayes Classifiers

indianpythonista — Fri, 10 Mar 2017 13:51:23 +0000

Note: This article has also featured on geeksforgeeks.org .

This article discusses the theory behind the Naive Bayes classifiers and their implementation.

Naive Bayes classifiers are a collection of classification algorithms based on Bayes’ Theorem. It is not a single algorithm but a family of algorithms where all of them share a common principle, i.e. every pair of features being classified is independent of each other.

To start with, let us consider a dataset.

Consider a fictional dataset that describes the weather conditions for playing a game of golf. Given the weather conditions, each tuple classifies the conditions as fit(“Yes”) or unfit(“No”) for plaing golf.

Here is a screenshot of the tabular representation of our dataset.

The dataset is divided into two parts, namely, feature matrix and the response vector.

Feature matrix contains all the vectors(rows) of dataset in which each vector consists of the value of dependent features. In above dataset, features are ‘Outlook’, ‘Temperature’, ‘Humidity’ and ‘Windy’.
Response vector contains the value of class variable(prediction or output) for each row of feature matrix. In above dataset, the class variable name is ‘Play golf’.

Assumption:

The fundamental Naive Bayes assumption is that each feature makes an:

independent
equal

contribution to the outcome.

With relation to our dataset, this concept can be understood as:

We assume that no pair of features are dependent. For example, the temperature being ‘Hot’ has nothing to do with the humidity or the outlook being ‘Rainy’ has no effect on the winds. Hence, the features are assumed to be independent.
Secondly, each feature is given the same weight(or importance). For example, knowing only temperature and humidity alone can’t predict the outcome accuratey. None of the attributes is irrelevant and assumed to be contributing equally to the outcome.

Note: The assumptions made by Naive Bayes are not generally correct in real-world situations. In-fact, the independence assumption is never correct but often works well in practice.

Now, before moving to the formula for Naive Bayes, it is important to know about Bayes’ theorem.

Bayes’ Theorem

Bayes’ Theorem finds the probability of an event occurring given the probability of another event that has already occurred. Bayes’ theorem is stated mathematically as the following equation:where A and B are events and P(B) ≠ 0.

Basically, we are trying to find probability of event A, given the event B is true. Event B is also termed as evidence.
P(A) is the priori of A (the prior probability, i.e. Probability of event before evidence is seen). The evidence is an attribute value of an unknown instance(here, it is event B).
P(A|B) is a posteriori probability of B, i.e. probability of event after evidence is seen.

Now, with regards to our dataset, we can apply Bayes’ theorem in following way:where, y is class variable and X is a dependent feature vector (of size n) where:Just to clear, an example of a feature vector and corresponding class variable can be: (refer 1st row of dataset)

X = (Rainy, Hot, High, False)
y = No

So basically, P(X|y) here means, the probability of “Not playing golf” given that the weather conditions are “Rainy outlook”, “Temperature is hot”, “high humidity” and “no wind”.

Naive Assumption:

Now, its time to put a naive assumption to the Bayes’ theorem, which is, independence among the features. So now, we split evidence into the independent parts.Now, if any two events A and B are independent, then,P(A,B) = P(A)P(B) Hence, we reach to the result:which can be expressed as: Now, as the denominator remains constant for a given input, we can remove that term:Now, we need to create a classifier model. For this, we find the probability of given set of inputs for all possible values of the class variable y and pick up the output with maximum probability. This can be expressed mathematically as:So, finally, we are left with the task of calculating P(y) and P(x_i | y).Please note that P(y) is also called class probability and P(x_i | y) is called conditional probability.The different naive Bayes classifiers differ mainly by the assumptions they make regarding the distribution of P(x_i | y).

Let us try to apply the above formula manually on our weather dataset. For this, we need to do some precomputations on our dataset.

We need to find P(x_i | y_j) for each x_i ∈ X and y_j ∈ y. All these calculations have been demonstrated in the tables below:

So, in the figure above, we have calculated P(x_i | y_j) for each x_i ∈ X and y_j ∈ y manually in the tables 1-4. For example, probability of playing golf given that the temperature is cool, i.e P(temp. = cool | play golf = Yes) = 3/9.

Also, we need to find class probabilities (P(y)) which has been calculated in the table 5. For example, P(play golf = Yes) = 9/14.

So now, we are done with our pre-computations and the classifier is ready!

Let us test it on a new set of features (let us call it today):

today = (Sunny, Hot, Normal, False)

So, probability of playing golf is given by:

and probability to not play golf is given by:

Since, P(today) is common in both probabilities, we can ignore P(today) and find proportional probabilities as:

and

Now, since

These numbers can be converted into a probability by making the sum equal to 1 (normalization):

and

Since

So, prediction that golf would be played is ‘Yes’.

The method that we discussed above is applicable for discrete data. In case of continuous data, we need to make some assumptions regarding the distribution of values of each feature. The different naive Bayes classifiers differ mainly by the assumptions they make regarding the distribution of P(x_i | y).

Now, we discuss one of such classifiers here.

Gaussian Naive Bayes classifier

In Gaussian Naive Bayes, continuous values associated with each feature are assumed to be distributed according to a Gaussian distribution. A Gaussian distribution is also called Normal distribution. When plotted, it gives a bell shaped curve which is symmetric about the mean of the feature values as shown below:

The likelihood of the features is assumed to be Gaussian, hence, conditional probability is given by:

Now, we look at an implementation of Gaussian Naive Bayes classifier using scikit-learn.

# load the iris dataset
from sklearn.datasets import load_iris
iris = load_iris()

# store the feature matrix (X) and response vector (y)
X = iris.data
y = iris.target

# splitting X and y into training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=1)

# training the model on training set
from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB()
gnb.fit(X_train, y_train)

# making predictions on the testing set
y_pred = gnb.predict(X_test)

# comparing actual response values (y_test) with predicted response values (y_pred)
from sklearn import metrics
print("Gaussian Naive Bayes model accuracy(in %):", metrics.accuracy_score(y_test, y_pred)*100)

Output:

Gaussian Naive Bayes model accuracy(in %): 95.0

Other popular Naive Bayes classifiers are:

Multinomial Naive Bayes: Feature vectors represent the frequencies with which certain events have been generated by a multinomial distribution. This is the event model typically used for document classification.
Bernoulli Naive Bayes: In the multivariate Bernoulli event model, features are independent booleans (binary variables) describing inputs. Like the multinomial model, this model is popular for document classification tasks, where binary term occurrence(i.e. a word occurs in a document or not) features are used rather than term frequencies(i.e. frequency of a word in the document).

As we reach to the end of this article, here are some important points to ponder upon:

In spite of their apparently over-simplified assumptions, naive Bayes classifiers have worked quite well in many real-world situations, famously document classification and spam filtering. They require a small amount of training data to estimate the necessary parameters.
Naive Bayes learners and classifiers can be extremely fast compared to more sophisticated methods. The decoupling of the class conditional feature distributions means that each distribution can be independently estimated as a one dimensional distribution. This in turn helps to alleviate problems stemming from the curse of dimensionality.

References:

Please write comments if you find anything incorrect, or you want to share more information about the topic discussed above.

Thanks for reading!