In order to better understand the nature of probabilistic decisions, consider the following court case of The People v. Collins, 1968. In this case, the robbery victim was unable to identify his assailant. All that the victim could recall was that the assailant was female with a blonde ponytail. In addition, he remembered that she fled the scene in a yellow convertible that was driven by an African American male who had a full beard. The suspect in the case fit the description given by the victim, so the question was “Could the jury be sure, beyond a reasonable doubt, that the woman on trial was the robber?” The evidence against her was as follows: She was blonde and often wore her hair in a ponytail; her codefendant friend was an African American male with a moustache, beard, and a yellow convertible. The attorney for the defense stressed the fact that the victim could not identify this woman as the woman who robbed him, and that therefore there should be reasonable doubt on the part of the jury.

The prosecutor, on the other hand, called an expert in probability theory who testified to the following: The probability of all of the above conditions (being blonde and often having a pony tail and having an African American male friend and his having a full beard, and his owning a yellow convertible) co-occurring when these characteristics are independent was 1 in 12 million. The expert further testified that the combination of characteristics was so unusual that the jury could in fact be certain “beyond a reasonable doubt” that the woman was the robber. The jury returned a verdict of “guilty” (Arkes & Hammond, 1986; Halpern, 1996).

As can be seen in the previous example, the legal system operates on probability and recognizes that we can never be absolutely certain when deciding whether an individual is guilty. Thus, the standard of “beyond a reasonable doubt” was established and jurors base their decisions on probability, whether they realize it or not. Most decisions that we make on a daily basis are, in fact, based on probabilities. Diagnoses made by doctors, verdicts produced by juries, decisions made by business executives regarding expansion and what products to carry, decisions regarding whether individuals are admitted to colleges, and most everyday decisions all involve using probability. In addition, all games of chance (for example, cards, horse racing, the stock market) involve probability.

If you think about it, there is very little in life that is certain. Therefore, most of our decisions are probabilistic and having a better understanding of probability will help you with those decisions. In addition, because probability also plays an important role in science, that is another important reason for us to have an understanding of it.

BASIC PROBABILITY CONCEPTS

Probability is a measure of chance, and we shall propose general rules for calculating the probability of combinations of simple events.

Probability refers to the number of ways a particular outcome (event) can occur divided by the total number of outcomes (events).

The tossing of a coin is a simple example of a large class of games of chance with certain common features. Each game is decided on the results or outcomes of one or more trials, where a trial might be rolling a die, tossing a coin, or drawing a card from a pack. If the outcomes are distinguishable, we say they are mutually exclusive, and if they are the only possible results they are also said to be exhaustive. There may be more than one way of listing the outcomes. If we draw a card from the pack, the outcomes red, black are mutually exclusive and exhaustive, but so are the outcomes Spades, Hearts, Diamonds, and Clubs. The trials are also said to be independent if the result of one trial does not depend on the outcome of any previous trial, or any combination of previous trials.

COUNTING METHODS

Probabilities are often presented or expressed as proportions. Proportions vary between 0.0 and 1.0, where a probability of 0.0 means the event certainly will not occur and a probability of 1.0 means that the event is certain to occur. Thus, any probability between 0.0 and 1.0 represents an event with some degree of uncertainty to it. How much uncertainty depends on the exact probability with which we are dealing. For example, a probability close to 0.0 represents an event that is almost certain not to occur, and a probability close to 1.0 represents an event that is almost certain to occur. On the other hand, a probability of .50 represents maximum uncertainty.

Let’s start with a simplistic example of probability. What is the probability of getting a “head” when tossing a coin? In this example, we have to consider how many ways there are to get a “head” on a coin toss (there is only one way, the coin lands heads up) and how many possible outcomes there are (there are two possible outcomes, either a “head” or a “tail”). So, the probability of a “head” in a coin toss is:

AXIOMATIC APPROACH

Set Theory: A set is a collection of items or events. The items within a set are generally referred to as elements. A set can be an element of another set.

The Universal Set is the set of all possible elements. In probability, the universal set is the set of all possible outcomes of a trial (experiment).

Sample space: In order to avoid continually referring to particular games or experiments, it is useful to employ an abstract representation for a trial and its outcomes. Each distinguishable and indecomposable outcome, or simple event, is regarded as a point in a sample space, S. Thus, for the experiment of drawing a card from a pack the sample space contains 52 points. Every collection of simple events or set of points of S is called an event. The Sample space is an example of a universal set.

Intersection: The intersection of two sets A, B is the set of points of S which belong to both A and B and is an event. Thus the intersection of the sets (HH, TH, HT} and {HT, ΤΤ) is the set containing the single point HT. This event may be called ‘heads on the first coin and tails on the second coin’. It may happen that the two sets have no points in common, that is, their intersection is the empty set. Simply, an intersection of two sets is a set containing elements common to both sets.

Union: This is defined as the set which contains all the points of S which are in either A or B {or both). Thus, the union of the events {HH, TH, ΗΤ) and {HT, TT) , in the present example, is the event ( HH , TH, HT, TT} , which contains every point in the sample space and may reasonably be called ‘the certain event’. In other words a union of sets is a set that contains all unique elements of the sets.

VENN DIAGRAM

Figure 5.1: A Venn Diagram representing the intersection of two sets.

The Venn diagram is a simple graphical tool used to represent set theory computations. In a Venn diagram, the sample space is represented by a rectangle and any event by a circle in this rectangle.

Example

An ice-cream firm, before launching three new flavors, conducts a tasting with the assistance of 60 schoolboys. The findings were summarized as:

32 liked A

24 liked Β

31 liked C

10 liked A and Β

11 liked A and C

14 liked B and C

6 liked A and Β and C.

Since there are only three flavors. A, B, C to consider, the information provided can easily be grasped through a diagram. Can you draw a Venn diagram to represent this relationship?

AXIOMS OF PROBABILITY

For every event, E, in the sample space S we assign a non-negative number, called the probability of Ε denoted by Pr(E), so that the following axioms are satisfied.

(a) For every event E, Pr(E) > 0                                                                                  Non-negativity

(b) For the certain event, Pr(S) = 1.                                                                          Sum of all probabilities =1

(c) If E₁, E₂ are mutually exclusive events Pr(E₁ U E₂) = Pr(E₁) + Pr (E₂)       Additivity

(d) If E₁, E₂ are independent events Pr(E₁ ∩ E₂) = Pr(E₁) x Pr (E₂)                 Multiplication rule

Conditional Probability

Conditional probability measures that probability that an event will occur “given that” another event has occurred. For two events, A and B, the conditional probability of B given that A has occurred is denoted as Pr(B |A).  It is calculated as:

Bayes’ Theorem

Also called Bayes’ Rule or Bayes’ law, it simply relates our current belief based on prior evidence. Sounds vague and mysterious? Not to worry, this will become clearer when we start making deductions based on the probability estimates. For now what you need to remember is that Bayes’ law tell us something about the future based on what we have observed in the past.

The formula:

Can you try and derive this formula using the formula for conditional probability?

Next Lesson we look at the Random Variable and Probability Distributions

When a distribution of scores is very large, it tends to approximate a pattern called a normal distribution. When plotted as a frequency polygon, a normal distribution forms a symmetrical, bell-shaped pattern often called a normal curve (see Figure 5.1). We say that the pattern approximates a normal distribution because a true normal distribution is a theoretical construct not actually observed in the real world.

The normal distribution is a theoretical frequency distribution that has certain special characteristics. First, it is bell-shaped and symmetrical—the right half is a mirror image of the left half. Second, the mean, median, and mode are equal and are located at the center of the distribution. Third, the normal distribution is unimodal—it has only one mode. Fourth, most of the observations are clustered around the center of the distribution, with far fewer observations at the ends, or “tails,” of the distribution. Lastly, when standard deviations are used on the x-axis, the percentage of scores falling between the mean and any point on the x-axis is the same for all normal curves. We will discuss the normal distribution more extensively in later lessons.

Figure 5.1 Normal Curve

Kurtosis

Although we typically think of the normal distribution as being similar to the curve depicted in Figure 5.1, there are variations in the shape of normal distributions. Kurtosis refers to how flat or peaked a normal distribution is. In other words, kurtosis refers to the degree of dispersion among the scores, or whether the distribution is tall and skinny or short and fat. The normal distribution depicted in Figure 5.1 is called mesokurtic—meso means “middle.” Mesokurtic curves have peaks of medium height and the distributions are moderate in breadth. Now look at the two distributions depicted in Figure 5.2.

The normal distribution on the left is leptokurtic—lepto means “thin.” Leptokurtic curves are tall and thin, with only a few scores in the middle of the distribution having a high frequency. Last, see the curve on the right side of Figure 5.2. This is a platykurtic curve—platy means “broad” or “flat.” Platykurtic curves are short and more dispersed (broader). In a platykurtic curve, there are many scores around the middle score that all have a similar frequency.

Figure 5.2 Kurtosis

Positively Skewed Distributions

Most distributions do not approximate a normal or bell-shaped curve. Instead, they are skewed, or lopsided. In a skewed distribution, scores tend to cluster at one end or the other of the x-axis, with the tail of the distribution extending in the opposite direction. In a positively skewed distribution, the peak is to the left of the center point and the tail extends toward the right, or in the positive direction. (See Figure 5.3.)

Notice that what is skewing the distribution, or throwing it off center, are the scores toward the right or positive direction. A few individuals have extremely high scores that pull the distribution in that direction. Notice also what this does to the mean, median, and mode. These three measures do not have the same value, nor are they all located at the center of the distribution as they are in a normal distribution. The mode—the score with the highest frequency—is the high point on the distribution. The median divides the distribution in half. The mean is pulled in the direction of the tail of the distribution; that is, the few extreme scores pull the mean toward them and inflate it.

Negatively Skewed Distributions

The opposite of a positively skewed distribution is a negatively skewed distribution—a distribution in which the peak is to the right of the center point and the tail extends toward the left, or in the negative direction. The term negative refers to the direction of the skew. As can be seen in Figure 5.3, in a negatively skewed distribution, the mean is pulled toward the left by the few extremely low scores in the distribution. As in all distributions, the median divides the distribution in half, and the mode is the most frequently occurring score in the distribution.

Figure 5.3 Skewness

Central Moment

The k^th central moment (or moment about the mean) of a data population is:

Similarly, the kth central moment of a data sample is:

In particular, the second central moment of a population is its variance.

The skewness of a data population is defined by the following formula, where μ₂ and μ₃ are the second and third central moments.

The kurtosis of a univariate population is defined by the following formula, where μ₂ and μ₄ are the second and fourth central moments.

I find statistics rather mischievous. Like lady wisdom in the Book of Proverbs, one endeavours to obtain her riches but the more one obtains it, the more one becomes detached from the realities of everyday living. Statisticians use complex (sometimes even boring) concepts to explain everyday phenomena. This practise is fine if the audience is fellow detached statisticians, but when the audience is laymen, this elitism loses its logic.

The aim of this course is not to teach you to be a statistician (of those we have plenty). The objective is that you become a questioner of human behaviour. That you look at phenomena and ask, how, when, why is that happening. The course goes ahead to introduce a few tools that will help you to answer those questions.

Who?

This course is open to all with a high-school level mathematics and statistics knowledge. I have always found that it is not really the level of exposure that determines success in an endeavour, but the will to do. So the main prerequisite for this course is the will to learn and the desire to ask questions of your society. This course is not for those who want to flaunt their statistical wizardry.

How?

The course shall be delivered through a series of lessons that will be published every Monday, Wednesday and Friday on this blog. We will systematically cover topics covered  in a typical undergraduate introductory statistics class. We shall also use the R language as a tool for statistical computations. THIS IS NOT AN R CLASS! We shall simply present R as a tool for carrying out statistical analyses. Detailed R tutorials can be downloaded here for those who are so inclined.

What?

What’s the point of all this. Well, as mentioned before, I hope that you will be able to ask probing questions of phenomena that you observe. In statistics we call that exploratory statistics. Ultimately, we ask these questions so that we are better able to deal with these phenomena in the future, be better prepared, take advantage, etc. This is known as predictive analytics.

This is an introductory course. The content is designed in a way that the statistical concepts are explained very simply. Nevertheless, where concepts may not be very clear, examples are given to illustrate the same. It is the tutor’s hope that this course will whet the students’ appetite for more and we can consequently get into intermediary and even advanced statistical concepts.

Ready to go? Start here.

Range

The simplest measure of variation is the range—the difference between the lowest and the highest score in a distribution. To find the range, simply subtract the lowest score from the highest score.

Table 4.1 Two – Class Score comparisons

In the example above the range for Class 1 is 100 points, whereas the range for Class 2 is 10 points. Thus, the range provides some information concerning the difference in the spread of the distributions. In this simple measure of variation, however, only the highest and lowest scores enter the calculation, and all other scores are ignored.

Percentile

The n^th percentile of an observation variable is the value that cuts off the first n percent of the data values when it is sorted in ascending order.

> quantile(INCOME, c(.32, .57, .98)) #finds the 32, 57 and 98^th percentiles

Quartile

There are several quartiles of an observation variable. The first quartile, or lower quartile, is the value that cuts off the first 25% of the data when it is sorted in ascending order. The second quartile, or median, is the value that cuts off the first 50%. The third quartile, or upper quartile, is the value that cuts off the first 75%.

quantile(INCOME) #gives the first second and third quartiles

The inter-quartile range is the difference between the third quartile and the first quartile.

Average Deviation and Standard Deviation

More sophisticated measures of variation use all of the scores in the distribution in their calculation. The most commonly used measure of variation is the standard deviation. Most people have heard this term before and may even have calculated a standard deviation if they have taken a statistics class. However, many people who know how to calculate a standard deviation do not really appreciate the information it provides.

To begin, let’s think about what the phrase standard deviation means. Other words that might be substituted for the word standard include average, normal, or usual. The word deviation means to diverge, move away from, or digress. Putting these terms together, we see that the standard deviation means the average movement away from something. But what? It is the average movement away from the center of the distribution—the mean.

The standard deviation, then, is the average distance of all of the scores in the distribution from the mean or central point of the distribution—or, as we shall see shortly, the square root of the average squared deviation from the mean. Think about how you would calculate the average distance of all of the scores from the mean of the distribution. First, you would have to determine how far each score is from the mean; this is the deviation, or difference, score. Then, you would have to average these scores. This is the basic idea behind calculating the standard deviation.

Let’s use these data to calculate the average distance from the mean. We will begin with a calculation that is slightly simpler than the standard deviation, known as the average deviation. The average deviation is essentially what the name implies— the average distance of all of the scores from the mean of the distribution.

X- µ

Then we need to sum the deviation scores. Notice, however, that if we were to sum these scores, they would add to zero. Therefore, we first take the absolute value of the deviation scores (the distance from the mean, irrespective of direction). To calculate the average deviation, we sum the absolute value of each deviation score:

∑|X-µ|

Then we divide by the total number of scores (N) to find the average deviation.

Although the average deviation is fairly easy to compute, it is not as useful as the standard deviation because, as we will see in later modules, the standard deviation is used in many other statistical procedures.

The standard deviation is very similar to the average deviation. The only difference is that rather than taking the absolute value of the deviation scores, we use another method to “get rid of” the negative deviation scores—we square the deviation scores.

The formula for the standard deviation is:

Notice that the formula is similar to that for the average deviation. We determine the deviation scores, square the deviation scores, sum the squared deviation scores, and divide by the number of scores in the distribution. Lastly, we take the square root of that number. Why? Squaring the deviation scores has inflated them. We now need to bring the squared deviation scores back to the same level of measurement as the mean so that the standard deviation is measured on the same scale as the mean.

If, however, you are using sample data to estimate the population standard deviation, then the standard deviation formula must be slightly modified. The modification provides what is called an “unbiased estimator” of the population standard deviation based on sample data. The modified formula is:

s = unbiased estimator of population standard deviation

X = each individual score

= sample mean

N = number of scores in the sample

The main difference is in the denominator—dividing by N – 1 versus N. The reason is that the standard deviation within a small sample may not be representative of the population; that is, there may not be as much variability in the sample as there actually is in the population. We, therefore, divide by N – 1, because dividing by a smaller number increases the standard deviation and thus provides a better estimate of the population standard deviation.

sd(INCOME) #standard deviation of INCOME

var(INCOME) #variance of INCOME

The Variance is the square of the standard deviation

3.1 The Why

Organizing data into tables and graphs can help make a data set more meaningful. These methods, however, do not provide as much information as numerical measures. Descriptive statistics are numerical measures that describe a distribution by providing information on the central tendency of the distribution, the width of the distribution, and the distribution’s shape.

A measure of central tendency characterizes an entire set of data in terms of a single representative number.

Measures of central tendency measure the “middleness” of a distribution of scores in three ways: the mean, median, and mode.

3.2 The What

Mean

The most commonly used measure of central tendency is the mean—the average observation in a set of observations. There are two manifestations (yes, I said it!) of the mean; the arithmetic mean and the geometric mean. We are not interested in the geometric mean at this point but you can look at it here if you are interested. We will stick to the arithmetic mean.

 Arithmetic mean: simply the mean or average when the context is clear, is the sum of a collection of numbers divided by the number of numbers in the collection

We can calculate the mean for our distribution of exam scores (from the previous lesson) by adding all of the scores together and dividing by the total number of scores. Mathematically, this would be:

where

 (pronounced “mu”) represents the symbol for the population mean

 represents the symbol for “the sum of”

X represents the individual scores, and

N represents the number of scores in the distribution

To calculate the mean, then, we sum all of the Xs, or scores, and divide by the total number of scores in the distribution (N). You may have also seen this formula represented as follows:

In this case  represents a sample mean.

One of the main shortcomings of the mean is that the mean is influenced by extreme scores (what we sometimes refer to as outliers).

An outlier is an observation point that is distant from other observations.

Outliers can distort the results by giving an inaccurate representation of the distribution of the population.

Median

Another measure of central tendency, the median, is used in situations in which the mean might not be representative of a distribution. Let’s use a different distribution of scores to demonstrate when it might be appropriate to use the median rather than the mean. Imagine that you are considering taking a job with a small computer company. When you interview for the position, the owner of the company informs you that the mean income for employees at the company is approximately Kshs. 100,000 and that the company has 25 employees. Most people would view this as good news. Having learned in a statistics class that the mean might be influenced by extreme scores, you ask to see the distribution of 25 incomes. The distribution is shown below.

Table 3.1 Employee Salaries Distribution

The mean for this distribution is Kshs. 99,920. Notice that, as claimed, the mean income of company employees is very close to Kshs. 100,000. Notice also, however, that the mean in this case is not very representative of central tendency, or “middleness.” In this distribution, the mean is thrown off center or inflated by one very extreme score of Kshs. 1,800,000 (the income of the company’s owner, needless to say). This extremely high income pulls the mean toward it and thus increases or inflates the mean. Thus, in distributions with one or a few extreme scores (either high or low), the mean will not be a good indicator of central tendency. In such cases, a better measure of central tendency is the median.

The median is the middle score in a distribution after the scores have been arranged from highest to lowest or lowest to highest.

The distribution of incomes in Table 3.1 is already ordered from lowest to highest. To determine the median, we simply have to find the middle score. In this situation, with 25 scores, that would be the 13th score. You can see that the median of the distribution would be an income of Kshs. 27,000, which is far more representative of the central tendency for this distribution of incomes.

Why is the median not as influenced as the mean by extreme scores? Think about the calculation of each of these measures. When calculating the mean, we must add in the atypical income of Kshs. 1,800,000, thus distorting the calculation. When determining the median, however, we do not consider the size of the $1,800,000 income; it is only a score at one end of the distribution whose numerical value does not have to be considered in order to locate the middle score in the distribution. The point to remember is that the median is not affected by extreme scores in a distribution because it is only a positional value. The mean is affected because its value is determined by a calculation that has to include the extreme value.

In distributions with an even number of observations, the median is calculated by averaging the two middle scores. In other words, we determine the middle point between the two middle scores.

Mode

The third measure of central tendency is the mode—the score in a distribution that occurs with the greatest frequency. Sometimes, several scores occur with equal frequency. Thus, a distribution may have two modes (bimodal), three modes (trimodal), or even more. The mode is the only indicator of central tendency that can be used with nominal data. Although it can also be used with ordinal, interval, or ratio data, the mean and median are more reliable indicators of the central tendency of a distribution, and the mode is seldom used.

 3.3 The How

R Code

mydata <- read.table(“testdata.txt”) #import your dataset

#attach(mydata) # In case you want to work with the variables directly

names(mydata) #This shows us all the variable names

mean(INCOME) #If you use the attach() command, you can call variables directly

#otherwise

mean(mydata$INCOME) #Find the mean income

#mean(mydata$INCOME, na.rm=TRUE) #Remove NA values before computation

median(INCOME, na.rm=TRUE) #returns the middle observation

mode(INCOME) #does something weird!

# the function mode( ) in R returns the variable type

temp <- table(as.vector(INCOME)) #The first row of “temp”

#is a sorted list of all unique values in the vector INCOME

#The second row in “temp” counts how many occurrences of each value.

names(temp)[temp == max(temp)]

#this returns the names of the values that have the highest count

#in temp’s second row

# This happens to be the mode!

#R knows you will want to see all the measures of central tendency

summary(INCOME) # So it supplies them all in one command

We will discuss the “Min. 1st Qu.  3rd Qu.    Max. “ in the next class. These are measures of dispersion. We will also look at range, standard deviation, percentiles, skweness and other similar animals. Until next time.

I have thrown this topic here because I am not very sure where it fits. So what is sampling?

Imagine we want to study the effect of the introduction of new parking rates on the taxi business in Nairobi. We may want to know if it has negatively impacted the business or not. So we decide to interview taxi drivers or taxi owners to get their views on the matter.

There we encounter problem number 1; it may not be feasible to interview ALL taxi drivers to get their views, either because we do not have the time or the money or both. The sum total of all the taxi drivers in Nairobi is called the population.

A population is any entire collection of people, animals, plants or things from which we may collect data. It is the entire group we are interested in, which we wish to describe or draw conclusions about.

Obviously, it would be better if we had the views of all taxi drivers in Nairobi in our study. That way we would be extremely confident with our findings. Alas, that is not possible and, therefore, we have to select a smaller group of taxi drivers whose views we will collect. This smaller group we have chosen is what is known as a sample.

A sample is a group selected from a larger group (population). By studying the sample it is hoped to draw valid conclusions about the larger group.

Here we encounter problem number 2; how do we select a smaller group of taxi drivers without appearing to be biased? The best way is if we were able to choose taxi drivers randomly. If we could get a random sample then no one can accuse us of being biased, right?

Ideally, we would like to choose a simple random sample.

A simple random sample is a subset (I will explain this concept of a subset when we start on probability) of a population in which each member of the subset has an equal probability of being chosen. It is meant to be an unbiased representation of the group.

Therein lies problem number 3; it is almost impossible to pick a truly random sample. Maybe the only way to do it is if you had the names of all the taxi drivers, put them in a hat and draw your sample at random. But then you will encounter problems when the members of the sample are too widely spread, or if by some coincidence, all of them belong to one company, or some issue like that. The more likely problem is that we do not have the names of all the taxi drivers in Nairobi. These issues increase the likelihood of a sampling error.

Sampling Error: if the sample does not accurately reflect the population it is supposed to represent. We want to minimize sampling error as much as possible.

It would be easier if we had a method for dividing up the population into manageable units from which we could draw our sample. This is called stratified random sampling. We divide the population into strata (units or divisions) based on some meaningful criteria. In our case, we could divide the sample into geographic regions, e.g. the CBD, Upperhill, Westlands, Harlingham. Then we take a random sample of taxi drivers in that area.

Remember that we are not interested in just the sample, but we want to know the effect of the parking fee increase in the entire taxi business. But we are only getting views for a smaller group (subset) of taxi drivers. Therefore, once we get the results, we have to infer what the population would be.

Statistical inference makes use of information from a sample to draw conclusions (inferences) about the population from which the sample was taken.

Election poll results are an example of statistical inference.

Let’s define two more terms before we proceed:

A parameter is a value, usually unknown (and which we, therefore, are trying to estimate), used to represent a certain population characteristic. The mean, for instance, is a population parameter used to indicate the average value of a particular quantity. In statistics, parameters are represented by Greek letters (for example µ for mean).

A statistic is a quantity calculated from a sample of data. It is used to give information about unknown values in the corresponding population. Statistics are often assigned Roman letters in statistics (for example s for standard deviation).

Back to the lesson: Organizing Data

2.1 The Why

Sometimes we have a small sample to deal with. By just looking at the data we are able to describe the data without having to proceed into various statistical techniques. Often times, though, that is not the case. More often than not we will be working with large data sets. For us to be able to “look” at the data, we would have to arrange it in a meaningful way.

Organizing data will help us to make preliminary descriptions of the data but also, it will give us an indication of the kind of techniques we would need to apply in order to make more sense of the dataset in front of us.

Visualizing the Dataset

Sometimes we want to see the data. Most of us have used excel. When you open a dataset in excel, you have already organized data! Excel is a great tool and it organizes datasets into rows and columns. The columns represent variables (remember lesson 1) while the rows represent observations.

You can also try and view datasets in SPSS, SAS and R. since we are using R in this course, to view a dataset import the dataset and assign it to the dataframe mydata:

>mydata <- read.table(“your_data_set”) #import your dataset

>mydata #view your dataset

RStudio has an even better way of displaying datasets. Can you try importing a dataset in RStudio without using the above code?

Frequency Distributions

You may have a list-like form of data as below:

Table 2.1 Scores of Student in the Class

As we said, we organize data so that meaningful conclusions can be drawn out of it. One way to do that would be to sort the data from lowest to highest or vise versa. Once this is accomplished (see Table 2.2), we can try to condense the data into a frequency distribution—a table in which all of the scores are listed along with the frequency with which each occurs. We can also show a relative frequency distribution, which indicates the proportion of the total observations included in each score. When the relative frequency distribution is multiplied by 100, it is read as a percentage.

Table 2.2 Frequency Distribution

The frequency distribution is a way of presenting data that makes the pattern of the data easier to see. Frequency distributions are great for nominal and ordinal data.

When dealing with Interval and ratio data (especially when the dataset is very large), we group the observations and create a class interval frequency distribution. We can combine individual scores into categories, or intervals, and list them along with the frequency of scores in each interval. In our exam score example, the scores range from 0 to 80—an 80-point range. A rule of thumb when creating class intervals is to have between 10 and 20 categories. A quick method of calculating what the width of the interval should be is to subtract the smallest score from the largest score and then divide by the number of intervals you would like.

Table 2.3 Class Interval Frequency Distribution

Creating Frequency Distributions with R

studentscores <- c(23,45,6,73,23,23,45,50,51,34) # create a vector of the student scores

range(studentscores) #displays the minimum and maximum score

bin = seq(0,80, by=10) #creates a vector sequence

bin # displays the values of our bin vector

studentscores.cut = cut(studentscores, bin, right=FALSE)

studentscores.frequency = table(studentscores.cut) # creates out frequency distribution

cbind(studentscores.frequency) #Displays as a data frame

DATA VISUALIZATION

Frequency distributions can provide valuable information, but sometimes a picture is of greater value. Several types of pictorial representations can be used to represent data. The choice depends on the type of data collected and what the researcher hopes to emphasize or illustrate. The most common type of data visualizations are graphs; pie charts, bar graphs, histograms and frequency polygons (line graphs).

There are new and very powerful data visualization tools. You can research on these tools if you would like to delve a little more into this area. For this class, we shall stick to the basics.

Bar Graphs and Histograms

Bar graphs and histograms are frequently confused. When the data collected are on a nominal scale, or if the variable is a qualitative variable (a categorical variable for which each value represents a discrete category), then a bar graph is most appropriate. A bar graph is a graphical representation of a frequency distribution in which vertical bars are centered above each category along the x-axis and are separated from each other by a space, indicating that the levels of the variable represent distinct, unrelated categories.

If the variable is a quantitative variable (the scores represent a change in quantity), or if the data collected are ordinal, interval, or ratio in scale, then a histogram can be used. A histogram is also a graphical representation of a frequency distribution in which vertical bars are centered above scores on the x-axis, but in a histogram the bars touch each other to indicate that the scores on the variable represent related, increasing values.

//
 

In both a bar graph and a histogram, the height of each bar indicates the frequency for that level of the variable on the x-axis. The spaces between the bars on the bar graph indicate not only the qualitative differences among the categories but also that the order of the values of the variable on the x-axis is arbitrary. In other words, the categories on the x-axis in a bar graph can be placed in any order. The fact that the bars are contiguous in a histogram indicates not only the increasing quantity of the variable but also that the variable has a definite order that cannot be changed.

Pie Chart

Like Bar Graphs, Pie Charts are used to represent categorical variables. While Bar Graphs display the frequencies, Pie charts show the proportions (relative frequencies). Most times, the relative frequencies are represented as percentages.

Frequency Polygons (Line Graphs)

We can also depict the data in a histogram as a frequency polygon—a line graph of the frequencies of individual scores or intervals. Again, scores (or intervals) are shown on the x-axis and frequencies on the y-axis. Once all the frequencies are plotted, the data points are connected. Frequency polygons are appropriate when the variable is quantitative or the data are ordinal, interval, or ratio. In this respect, frequency polygons are similar to histograms. Frequency polygons are especially useful for continuous data (such as age, weight, or time) in which it is theoretically possible for values to fall anywhere along the continuum. For example, an individual can weigh 120.5 pounds or be 35.5 years of age. Histograms are more appropriate when the data are discrete (measured in whole units)—for example, number of college classes taken or number of siblings.

3D Plots

Sometimes we may want to represent complex relationships using graphs. While line graphs are used to show the relationship between 2 variables, we may want to see the relationship between 3 variable. We can graph this using 3D graphing tools. Excel does not have this feature out of the box and it is very difficult to do it directly in R. We do this using package in R.

Packages are collections of R functions, data and compiled code in a well-defined format.

Rcommander and ggplot are touted to be the best plotting packages for R. Installing packages will be part of the assignment for this lesson.

Ensure that you have R and RStudio installed in your PC. For installation instructions go to:

Click here to download R

Click here to download RStudio

Once you have successfully installed R and RStudio, create a folder in a place that is easily accessible and call it r_working_directory. Then go to the desktop and right click on the R icon. Select properties. In the “Start In:” text box put the location of your r_working_directory folder

(e.g. C:\Users\Kevin\Documents\r_working_directory)

Be sure to read the R tutorial. Download R manual. Remember that R has an inbuilt help function that is accessible within the console by either typing ? followed by whatever it is you want to know more about or help( ) function. To test this out, type ?help in the R console.

To start us off, we will work with a simple data set. Sign up here to receive the dataset. Put your extracted data into your r_working_directory. Open RStudio and import the first dataset (testdata.txt) and list all the variable names (column names) then import the second dataset and do the same thing. Getting problems? Try the help function and see where you are going wrong. Try opening the dataset in notepad and look at the data. Then try and open each of them in excel (Hint: you have to import the txt file into excel).

Some of the code can be used as below.

The R Code

#First we get our data

mydata <- read.table(“testdata.txt”)

#?read.table

# Help with the function read.table(). Will also list all the arguments associated with the function

#mydata <- read.csv(“testdata.csv”, sep=”,”, header=TRUE)

# sometimes your data is in csv format and you want to tell R as much

names(mydata) #lists all the variable names

Submit the following:

• the output after the names(mydata) function
• Your excel sheet after importing the data indicating the scale of measurement for each variable

The objective of statistics can be divided into three parts:

• To Describe
• To Explain and
• To Predict

Description begins with careful observation. It involves carefully observing behavior in order to describe it. Description allows us to learn about behavior and when it occurs. Let’s say, for example, that you were interested in the channel-surfing behavior of males and females. Careful observation and description would be needed in order to determine whether or not there were any gender differences in channel-surfing. Description allows us to observe that two events are systematically related to one another. Without description as a first step, predictions cannot be made.

Explanation allows us to identify the causes that determine when and why a behavior occurs. In order to explain a behavior, we need to demonstrate that we can manipulate the factors needed to produce or eliminate the behavior. For example, in our channel-surfing example, if gender predicts channel-surfing, what might cause it? It could be genetic or environmental. Maybe males have less tolerance for commercials and thus channel-surf at a greater rate. Maybe females are more interested. Maybe the attention span of females is greater. Maybe something associated with having a Y chromosome increases channel-surfing, or something associated with having two X chromosomes leads to less channel-surfing. Obviously the possible explanations are numerous and varied. As scientists, we test these possibilities to identify the best explanation of why a behavior occurs.

Prediction allows us to identify the factors that indicate when an event or events will occur. In other words, knowing the level of one variable allows us to predict the approximate level of the other variable.

Variable: An event or behavior that has at least two values

We know that if one variable is present at a certain level, then there is a greater likelihood that the other variable will be present at a certain level. For example, if we observed that males channel-surf with greater frequency than females, we could then make predictions about how often males and females might change channels when given the chance.

1.2 The What

So we want to describe or explain or predict. The question is what? We are interested in phenomena; that is, an event, a characteristic or behavior. We could be interested in the channel surfing behavior of people or the relationship between traffic levels and weather patterns. To do that we have to make observations; to observe the phenomena we want to describe or explain or predict.

An observation is an instance of a variable

This is a good time to talk about a variable. A variable is a phenomena that can take on two or more values. The weights of a group of people in a classroom, for instance, is a variable. Each person (observation) will have their own weight (variable).

Data, therefore, is a set of observations that is arranged in a meaningful way. We shall talk about the arrangement of data later. For now let us focus on the observations themselves.

Data is the focal point of all our statistical activity. How much and what kind of data we have will greatly influence what we can do. It may seem that we are spending a lot of time on what may be considered unnecessary but these are the building blocks.

Characteristics or Properties of Data

These properties include identity, magnitude, equal unit size, and absolute zero. When a measure has the property of identity, objects that are different receive different scores. For example, if members of this class had different heights, they would all receive different measurements. Measurements have the property of magnitude (also called ordinality) when the ordering of the numbers reflects the ordering of the variable. In other words, numbers are assigned in order so that some numbers represent more or less of the variable being measured than others.

Measurements have an equal unit size when a difference of 1 is the same amount throughout the entire scale. For example, the difference between people who are 64 kilos and 65 kilos is the same as the difference between people who are 72 kilos and 73 kilos. The difference in each situation (1 kg) is identical. Notice how this differs from the property of magnitude. Were we to simply line up and rank a group of individuals based on their weight, the scale would have the properties of identity and magnitude, but not equal unit size. Can you think about why this would be so? We would not actually measure people’s weight in kilos, but simply order them in terms of how big they appear, from smallest (the person receiving a score of 1) to biggest (the person receiving the highest score). Thus, our scale would not meet the criteria of equal unit size. In other words, the difference in weight between the two people receiving scores of 1 and 2 might not be the same as the difference in height between the two people receiving scores of 3 and 4.

Lastly, measures have an absolute zero when assigning a score of zero indicates an absence of the variable being measured. For example, bank account balance would have the property of absolute zero because a score of 0 on this measure would mean an individual has no money in the bank. However, a score of 0 is not always equal to the property of absolute zero. As an example, think about the temperature scale. That measurement scale has a score of 0 (the thermometer can read 0 degrees), but does that score indicate an absence of temperature? No, it indicates a very cold temperature. Hence, it does not have the property of absolute zero.

SCALES OF MEASUREMENT

Why are the properties of data important? They are important because data in itself might not be very useful. If we have a bunch of data (please do not use this phrase outside of this class!) we want to do something with it. That “something” is called manipulation. I am not talking about the evil and conniving manipulation of soap opera villains. Manipulation is simply the use of some techniques to convert one or more quantities into another quantity or quantities.

Every January, people are always pledging to join the gym because they have “added weight”. This means they had a starting weight and then gained some more weight to total into the weight they now have (say 65kg + 2kg = 67kg). Conversely, let’s say that a group of men went for a workshop on how to become better men in society. Can we say they have added “manness”? No. The quantity gender cannot be added. Can you explain why that is from the qualities of data above?

As noted previously, the level or scale of measurement depends on the properties of the data. There are four scales of measurement (nominal, ordinal, interval, and ratio), and each of these scales has one or more of the properties described in the previous section. As we will see later on, it is important to establish the scale of measurement of your data in order to determine the appropriate statistical test to use when analyzing the data.

A nominal scale is one in which objects or individuals are broken into categories that have no numerical properties. Nominal scales have the characteristic of identity but lack the other properties. Variables measured on a nominal scale are often referred to as categorical variables because the measuring scale involves dividing the data into categories. However, the categories carry no numerical weight. Some examples of categorical variables, or data measured on a nominal scale, include ethnicity, gender, and political affiliation.

An ordinal scale is one in which objects or individuals are categorized and the categories form a rank order along a continuum. Data measured on an ordinal scale have the properties of identity and magnitude but lack equal unit size and absolute zero. Ordinal data are often referred to as ranked data because the data are ordered from highest to lowest, or biggest to smallest. For example, the number ranks students are given in school based on performance in an exam (number 1, 2, etc.) is an ordinal scale. This variable would carry identity and magnitude because each individual receives a rank (a number) that carries identity, and beyond simple identity it conveys information about order or magnitude (how many students performed better or worse in the class).

An interval scale is one in which the units of measurement (intervals) between the numbers on the scale are all equal in size. When using an interval scale, the properties of identity, magnitude, and equal unit size are met. For example, the temperature scale is an interval scale of measurement. A given temperature carries identity (days with different temperatures receive different scores on the scale), magnitude (cooler days receive lower scores and hotter days receive higher scores), and equal unit size (the difference between 20 and 21 degrees is the same as that between 30 and 31 degrees.) However, the temperature scale does not have an absolute zero. Because of this, we are not able to form ratios based on this scale (for example, 50 degrees is not twice as hot as 25 degrees).

A ratio scale is one in which, in addition to order and equal units of measurement, there is an absolute zero that indicates an absence of the variable being measured. Ratio data have all four properties of measurement—identity, magnitude, equal unit size, and absolute zero. Examples of ratio scales of measurement include weight, time, and height. Each of these scales has identity (individuals who weigh different amounts would receive different scores), magnitude (those who weigh less receive lower scores than those who weigh more), and equal unit size (1 kg is the same weight anywhere along the scale and for any person using the scale). These scales also have an absolute zero, which means a score of zero reflects an absence of that variable. This also means that ratios can be formed. For example, a weight of 100 kg is twice as much as a weight of 50 kg.

Table 1.1: SCALES OF MEASUREMENT

Another means of classifying variables is in terms of whether they are discrete or continuous in nature. Discrete variables usually consist of whole-number units or categories. They are made up of chunks or units that are detached and distinct from one another. A change in value occurs a whole unit at a time, and decimals do not make sense with discrete scales. Most nominal and ordinal data are discrete. For example, gender, political party, and ethnicity are discrete scales. Some interval or ratio data can be discrete. For example, the number of children someone has would be reported as a whole number (discrete data), yet it is also ratio data (you can have a true zero and form ratios).

Continuous variables usually fall along a continuum and allow for fractional amounts. The term continuous means that it “continues” between the whole-number units. Examples of continuous variables are age (22.7 years), height (64.5 inches), and weight (113.25 kg). Most interval and ratio data are continuous in nature.

1.3 The How

Psychiatrists say that we cannot remember the first three years of our lives. Our minds just blot those memories out. For most of us, it is not just these memories that we forget, but where we put our keys, our friends’ birthdays and that all important meeting are just some of the things we regularly forget. Man is by his very nature, inadequate. That is why he built machines!

Al throughout school, you have been trained to memorize concepts. This is not one of those classes. I do not want you to memorize anything. I want you to understand the concept. Anything else can always be found when needed. When you understand how to do it, you can always find the means and the tools to do the job required. Especially in exploratory analysis (you don’t have to know what this is now), the job is in trying to figure out what you want to do, rather the doing itself.

I will try as much as possible to provide reference material that further explains the concepts introduced in this course. Nevertheless, we shall focus on the practical application of the concepts rather than the theoretical.

TOOLS

As mentioned earlier, man built tools to help make his work easier. Therefore, in this class, we shall also make use of statistical tools/packages to help make our work easier. For statistical analysis, we have several options:

1. MS Excel
2. SPSS
3. SAS
4. STATA
5. R

The above 5 tools are the most commonly used tools for statistical analysis. Each one has its own advantages and disadvantages and the choice of which one to use will depend on many factors including expertise of the user, size of task, cost, etc. In this class we shall be using R. There are many reasons for using R but the main reason we are using it in this class is because it is free while all the others have to be purchased.

Please see our R Tutorials. For this lesson, we need to be able to install and run R, create data and import data into R.

Next: Descriptive Statistics I – Organizing Data