Explorando - Un blog de Daniel Villanueva

Pandas and Covid-19

Tue, 21 Apr 2020 08:15:00 +0000

This notebook is an example of data analysis and manipulation with Pandas and has beed created in Google Colab

Enjoy it!

import numpy as np
import pandas as pd

The Data

To get some data I and going to download it from Data Repository by Johns Hopkins CSSE

https://github.com/CSSEGISandData/COVID-19

I first remove the folder where I am goint to store the data so I can re-execute this sentences without any problems …

!rm -rf ./COVID-19

The dataset is avaible in GitHub so I use the git command to get it

!git clone https://github.com/CSSEGISandData/COVID-19.git

Cloning into 'COVID-19'...
remote: Enumerating objects: 14, done.[K
remote: Counting objects: 100% (14/14), done.[K
remote: Compressing objects: 100% (9/9), done.[K
remote: Total 21425 (delta 5), reused 10 (delta 5), pack-reused 21411[K
Receiving objects: 100% (21425/21425), 90.29 MiB | 33.30 MiB/s, done.
Resolving deltas: 100% (11436/11436), done.

Exporing the data

!ls -lt ./COVID-19/csse_covid_19_data/csse_covid_19_daily_reports | head

total 9544
-rw-r--r-- 1 root root 317177 Apr 21 05:16 04-20-2020.csv
-rw-r--r-- 1 root root      0 Apr 21 05:16 README.md
-rw-r--r-- 1 root root 317954 Apr 21 05:16 04-19-2020.csv
-rw-r--r-- 1 root root 315926 Apr 21 05:16 04-18-2020.csv
-rw-r--r-- 1 root root 314848 Apr 21 05:16 04-17-2020.csv
-rw-r--r-- 1 root root 312551 Apr 21 05:16 04-15-2020.csv
-rw-r--r-- 1 root root 314226 Apr 21 05:16 04-16-2020.csv
-rw-r--r-- 1 root root 305548 Apr 21 05:16 04-12-2020.csv
-rw-r--r-- 1 root root 309742 Apr 21 05:16 04-13-2020.csv

Yes!!!
We have data files …

Perfect. Let’s explore the first dataset generated …

first = pd.read_csv("./COVID-19/csse_covid_19_data/csse_covid_19_daily_reports/01-22-2020.csv")

first.head()

	Province/State	Country/Region	Last Update	Confirmed	Deaths	Recovered
0	Anhui	Mainland China	1/22/2020 17:00	1.0	NaN	NaN
1	Beijing	Mainland China	1/22/2020 17:00	14.0	NaN	NaN
2	Chongqing	Mainland China	1/22/2020 17:00	6.0	NaN	NaN
3	Fujian	Mainland China	1/22/2020 17:00	1.0	NaN	NaN
4	Gansu	Mainland China	1/22/2020 17:00	NaN	NaN	NaN

And one of the last ones …

last = pd.read_csv("./COVID-19/csse_covid_19_data/csse_covid_19_daily_reports/04-18-2020.csv")

last.head()

	FIPS	Admin2	Province_State	Country_Region	Last_Update	Lat	Long_	Confirmed	Deaths	Active	Combined_Key
0	45001.0	Abbeville	South Carolina	US	2020-04-18 22:32:47	34.223334	-82.461707	15	0	15	Abbeville, South Carolina, US
1	22001.0	Acadia	Louisiana	US	2020-04-18 22:32:47	30.295065	-92.414197	110	7	103	Acadia, Louisiana, US
2	51001.0	Accomack	Virginia	US	2020-04-18 22:32:47	37.767072	-75.632346	33	0	33	Accomack, Virginia, US
3	16001.0	Ada	Idaho	US	2020-04-18 22:32:47	43.452658	-116.241552	593	9	584	Ada, Idaho, US
4	19001.0	Adair	Iowa	US	2020-04-18 22:32:47	41.330756	-94.471059	1	0	1	Adair, Iowa, US

Can I concatenate both datasets?

last.query("Country_Region == 'Spain'")

	FIPS	Admin2	Province_State	Country_Region	Last_Update	Lat	Long_	Confirmed	Deaths	Recovered	Active	Combined_Key
3025	NaN	NaN	NaN	Spain	2020-04-18 22:32:28	40.463667	-3.74922	191726	20043	74797	96886	Spain

pd.concat((first, last), axis = 0)

	Province/State	Country/Region	Last Update	Confirmed	Deaths	Recovered	FIPS	Admin2	Province_State	Country_Region	Last_Update	Lat	Long_	Active	Combined_Key
0	Anhui	Mainland China	1/22/2020 17:00	1.0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1	Beijing	Mainland China	1/22/2020 17:00	14.0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2	Chongqing	Mainland China	1/22/2020 17:00	6.0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
3	Fujian	Mainland China	1/22/2020 17:00	1.0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
4	Gansu	Mainland China	1/22/2020 17:00	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
3048	NaN	NaN	NaN	418.0	2.0	69.0	NaN	NaN	NaN	West Bank and Gaza	2020-04-18 22:32:28	31.952200	35.233200	347.0	West Bank and Gaza
3049	NaN	NaN	NaN	6.0	0.0	0.0	NaN	NaN	NaN	Western Sahara	2020-04-18 22:32:28	24.215500	-12.885800	6.0	Western Sahara
3050	NaN	NaN	NaN	1.0	0.0	0.0	NaN	NaN	NaN	Yemen	2020-04-18 22:32:28	15.552727	48.516388	1.0	Yemen
3051	NaN	NaN	NaN	57.0	2.0	33.0	NaN	NaN	NaN	Zambia	2020-04-18 22:32:28	-13.133897	27.849332	22.0	Zambia
3052	NaN	NaN	NaN	25.0	3.0	2.0	NaN	NaN	NaN	Zimbabwe	2020-04-18 22:32:28	-19.015438	29.154857	20.0	Zimbabwe

3091 rows × 15 columns

Ups!!! The column names don’t match :-(

Loading the data into Pandas and cleaning it

import glob
import os

files = glob.glob("./COVID-19/csse_covid_19_data/csse_covid_19_daily_reports/*.csv")
files.sort(key=os.path.getmtime)

We are going to:

Create a blank Dataset to store all the data
Load every dataset unifying the column names so we can concatenate it without any problem.
Remove extra blank spaces from the country field
Enrich the information with the date of the data in the correct type

data = pd.DataFrame()
for file in files:  
  df = pd.read_csv(file).rename(columns = {'Province/State' : 'State', 
                        "Country/Region" : 'Country',
                        'Province_State' : 'State', 
                        "Country_Region" : 'Country',
                        'Last Update' : 'Last_Update',
                        'Confirmed' : 'ConfirmedAcum',
                        'Deaths' : 'DeathsAcum',
                        'Recovered' : 'RecoveredAcum'})
  df = df.assign(Date = pd.to_datetime(file[-14:-4], format = '%m-%d-%Y'),
                 Country = df.Country.str.strip())
  data = pd.concat((data, df), axis = 0)

I noticed that the country names were a little messy.
Let’s fix it …

data['Country'] = data.Country.replace({'Bahamas, The' : 'Bahamas',
                         'Congo (Brazzaville)' : 'Congo',
                         'Congo (Kinshasa)' : 'Congo',
                         "Cote d'Ivoire" : "Cote d'Ivoire",
                         "Curacao" : "Curaçao",
                         'Czech Republic' : 'Czech Republic (Czechia)',
                         'Czechia' : 'Czech Republic (Czechia)',
                         'Faroe Islands' : 'Faeroe Islands',
                         'Macau' : 'Macao',
                         'Mainland China' : 'China',
                         'Palestine' : 'State of Palestine',
                         'Reunion' : 'Réunion',
                         'Saint Kitts and Nevis' : 'Saint Kitts & Nevis',
                         'Sao Tome and Principe' : 'Sao Tome & Principe',
                         'US' : 'United States',
                         'Gambia, The' : 'Gambia',
                         'Hong Kong SAR' : 'Hong Kong',
                         'Korea, South' : 'South Korea',
                         'Macao SAR' : 'Macao',
                         'Taiwan*' : 'Taiwan',
                         'Viet Nam' : 'Vietnam',
                         'West Bank and Gaza' : 'State of Palestine'
                         })

I’m going to fill in the null values of the ‘State’ and ‘Admin2’ fields so that I can later group the data correctly

data = data.fillna({'State' : 'NA', 'Admin2' : 'NA'})

Finally I am going to be left alone with the columns that interest me

data = data[['Date', 'Country', 'State', 'Admin2', 'ConfirmedAcum', 'DeathsAcum', 'RecoveredAcum']]

Let’s verify the structure of the dateset …

data.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 98669 entries, 0 to 3080
Data columns (total 7 columns):
 #   Column         Non-Null Count  Dtype         
---  ------         --------------  -----         
 0   Date           98669 non-null  datetime64[ns]
 1   Country        98669 non-null  object        
 2   State          98669 non-null  object        
 3   Admin2         98669 non-null  object        
 4   ConfirmedAcum  98650 non-null  float64       
 5   DeathsAcum     98228 non-null  float64       
 6   RecoveredAcum  98281 non-null  float64       
dtypes: datetime64[ns](1), float64(3), object(3)
memory usage: 6.0+ MB

Wait a set, I think that can be interesting have a column the the active cases. Let’s create it …

data['ActiveAcum'] = data.ConfirmedAcum  - data.DeathsAcum - data.RecoveredAcum

data.query("Country == 'Spain'").sort_values('Date', ascending = False).head()

	Date	Country	State	Admin2	ConfirmedAcum	DeathsAcum	RecoveredAcum	ActiveAcum
3053	2020-04-20	Spain	NA	NA	200210.0	20852.0	80587.0	98771.0
3044	2020-04-19	Spain	NA	NA	198674.0	20453.0	77357.0	100864.0
3025	2020-04-18	Spain	NA	NA	191726.0	20043.0	74797.0	96886.0
3017	2020-04-17	Spain	NA	NA	190839.0	20002.0	74797.0	96040.0
3013	2020-04-16	Spain	NA	NA	184948.0	19315.0	74797.0	90836.0

Perfect :-)

Now, I am going to group and summarize the data because I want to be sure that there is only one row per Date, Country, State and Admin2

data = data.groupby(["Date", "Country", "State", "Admin2"]).agg("sum").reset_index()

data.head()

	Date	Country	State	Admin2	ConfirmedAcum
0	2020-01-22	China	Anhui	NA	1.0
1	2020-01-22	China	Beijing	NA	14.0
2	2020-01-22	China	Chongqing	NA	6.0
3	2020-01-22	China	Fujian	NA	1.0
4	2020-01-22	China	Gansu	NA	0.0

Daily Cases

I am going to enrich the data by creating new columns with the daily cases.

First I create new columns with the cases from the previous day

data = data.sort_values(['State', 'Country', 'Date']).\
            assign(ConfirmedPrevious = data.groupby(['Admin2', 'State', 'Country']).shift(1)["ConfirmedAcum"],
                   DeathsPrevious = data.groupby(['Admin2', 'State', 'Country']).shift(1)["DeathsAcum"],
                   RecoveredPrevious = data.groupby(['Admin2', 'State', 'Country']).shift(1)["RecoveredAcum"],
                   ActivePrevious = data.groupby(['Admin2', 'State', 'Country']).shift(1)["ActiveAcum"],
            ).\
            fillna({ 'ConfirmedPrevious' : 0, 'DeathsPrevious' : 0, 'RecoveredPrevious' : 0 })

data.head()

	Date	Country	State	Admin2	ConfirmedAcum	ActiveAcum	ConfirmedPrevious	ActivePrevious
2599	2020-02-28	Canada	Montreal, QC	NA	1.0	1.0	0.0	NaN
2713	2020-02-29	Canada	Montreal, QC	NA	1.0	1.0	1.0	1.0
2834	2020-03-01	Canada	Montreal, QC	NA	1.0	1.0	1.0	1.0
2961	2020-03-02	Canada	Montreal, QC	NA	1.0	1.0	1.0	1.0
3103	2020-03-03	Canada	Montreal, QC	NA	1.0	1.0	1.0	1.0

After that I am going to assign the new fields subtracting the previous acum cases to the actual acum cases

data = data.assign(Confirmed = data.ConfirmedAcum -  data.ConfirmedPrevious,
            Deaths = data.DeathsAcum - data.DeathsPrevious,
            Recovered = data.RecoveredAcum - data.RecoveredPrevious,
            Active = data.ActiveAcum - data.ActivePrevious
            )

I no longer need the fields I used to make the calculation so I can drop them

data = data.drop(['ConfirmedPrevious', 'DeathsPrevious', 'RecoveredPrevious', 'ActivePrevious'], axis = 1)

Does the data look good?

data.query("Country == 'Spain'")

	Date	Country	State	Admin2	ConfirmedAcum	DeathsAcum	RecoveredAcum	ActiveAcum	Confirmed	Deaths	Recovered	Active
545	2020-02-01	Spain	NA	NA	1.0	0.0	0.0	1.0	1.0	0.0	0.0	NaN
612	2020-02-02	Spain	NA	NA	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0
679	2020-02-03	Spain	NA	NA	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0
749	2020-02-04	Spain	NA	NA	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0
819	2020-02-05	Spain	NA	NA	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0
...	...	...	...	...	...	...	...	...	...	...	...	...
83563	2020-04-16	Spain	NA	NA	184948.0	19315.0	74797.0	90836.0	7304.0	607.0	3944.0	2753.0
86603	2020-04-17	Spain	NA	NA	190839.0	20002.0	74797.0	96040.0	5891.0	687.0	0.0	5204.0
89647	2020-04-18	Spain	NA	NA	191726.0	20043.0	74797.0	96886.0	887.0	41.0	0.0	846.0
92699	2020-04-19	Spain	NA	NA	198674.0	20453.0	77357.0	100864.0	6948.0	410.0	2560.0	3978.0
95770	2020-04-20	Spain	NA	NA	200210.0	20852.0	80587.0	98771.0	1536.0	399.0	3230.0	-2093.0

80 rows × 12 columns

Data By Country

So far, we have data by 3 geographical levels: Country, State and a lower level called Admin2

The problem is that not all countries have this level of information, so I will create a new dataset only with the country level data

data_by_country = data.groupby(["Date", "Country"]).agg("sum").reset_index()
data_by_country = data_by_country.sort_values(['Country', 'Date'])
data_by_country.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 9112 entries, 848 to 3045
Data columns (total 10 columns):
 #   Column         Non-Null Count  Dtype         
---  ------         --------------  -----         
 0   Date           9112 non-null   datetime64[ns]
 1   Country        9112 non-null   object        
 2   ConfirmedAcum  9112 non-null   float64       
 3   DeathsAcum     9112 non-null   float64       
 4   RecoveredAcum  9112 non-null   float64       
 5   ActiveAcum     9112 non-null   float64       
 6   Confirmed      9112 non-null   float64       
 7   Deaths         9112 non-null   float64       
 8   Recovered      9112 non-null   float64       
 9   Active         9112 non-null   float64       
dtypes: datetime64[ns](1), float64(8), object(1)
memory usage: 783.1+ KB

data_by_country.head()

	Date	Country	ConfirmedAcum	ActiveAcum	Confirmed
848	2020-02-24	Afghanistan	1.0	1.0	1.0
886	2020-02-25	Afghanistan	1.0	1.0	0.0
928	2020-02-26	Afghanistan	1.0	1.0	0.0
977	2020-02-27	Afghanistan	1.0	1.0	0.0
1030	2020-02-28	Afghanistan	1.0	1.0	0.0

data_by_country[data_by_country.Country == 'United States']

	Date	Country	ConfirmedAcum	DeathsAcum	RecoveredAcum	ActiveAcum	Confirmed	Deaths	Recovered	Active
7	2020-01-22	United States	1.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0
22	2020-01-23	United States	1.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
33	2020-01-24	United States	2.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0
47	2020-01-25	United States	2.0	0.0	0.0	0.0	1.0	0.0	0.0	0.0
62	2020-01-26	United States	5.0	0.0	0.0	0.0	3.0	0.0	0.0	0.0
...	...	...	...	...	...	...	...	...	...	...
8367	2020-04-16	United States	667801.0	32916.0	54703.0	580182.0	31449.0	4591.0	2607.0	24251.0
8551	2020-04-17	United States	699706.0	36773.0	58545.0	604388.0	31976.0	3857.0	3842.0	24277.0
8735	2020-04-18	United States	732197.0	38664.0	64840.0	628693.0	32491.0	1891.0	6295.0	24305.0
8919	2020-04-19	United States	759086.0	40661.0	70337.0	648088.0	27159.0	1997.0	5497.0	19397.0
9103	2020-04-20	United States	784326.0	42094.0	72329.0	669903.0	25211.0	1433.0	1992.0	21786.0

90 rows × 10 columns

Cases per million inhabitants

We are going to enrich the information with the number of cases per million inhabitants, so we need population data by country.

A small internet search leads me to a page that has population data for 2020:

https://www.worldometers.info/world-population/population-by-country/

It seems that this information is protected to be downloaded automatically so I have no choice but to do it manually and upload the data to a GitHub Repository:

https://github.com/dvillaj/world-population/

I load the data to the Pandas, clean it up and just maintain the field of the population

population = pd.read_excel("https://github.com/dvillaj/world-population/blob/master/data/world-popultation-2020.xlsx?raw=true", sheet_name="Data")

population.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 235 entries, 0 to 234
Data columns (total 11 columns):
 #   Column             Non-Null Count  Dtype  
---  ------             --------------  -----  
 0   Country            235 non-null    object 
 1   Population (2020)  235 non-null    int64  
 2   Yearly Change      235 non-null    float64
 3   Net Change         235 non-null    int64  
 4   Density (P/Km²)    235 non-null    float64
 5   Land Area (Km²)    235 non-null    int64  
 6   Migrants (net)     201 non-null    float64
 7   Fertility Rate     201 non-null    float64
 8   Average Age        201 non-null    float64
 9   Urban Pop %        222 non-null    float64
 10  World Share        235 non-null    float64
dtypes: float64(7), int64(3), object(1)
memory usage: 20.3+ KB

population = population.rename(columns = {
    'Population (2020)' : 'Population',
    'Yearly Change' : 'Yearly_Change',
    'Net Change' : 'Net_Change',
    'Density (P/Km²)' : 'Density',
    'Land Area (Km²)' : 'Land_Area',
    'Migrants (net)' : 'igrants',
    'Fertility Rate' : 'Fertility',
    'Average Age' : 'Mean_Age',
    'Urban Pop %' : 'Urban_Pop',
    'World Share' : 'World_Share'
})

population = population[['Country', 'Population']]

population.head()

	Country	Population
0	Afghanistan	38928346
1	Albania	2877797
2	Algeria	43851044
3	American Samoa	55191
4	Andorra	77265

Now I join the population Dataset with the country data to have the population in this dataset

data_by_country = data_by_country.merge(population, how = 'left', on = 'Country')

data_by_country.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 9112 entries, 0 to 9111
Data columns (total 11 columns):
 #   Column         Non-Null Count  Dtype         
---  ------         --------------  -----         
 0   Date           9112 non-null   datetime64[ns]
 1   Country        9112 non-null   object        
 2   ConfirmedAcum  9112 non-null   float64       
 3   DeathsAcum     9112 non-null   float64       
 4   RecoveredAcum  9112 non-null   float64       
 5   ActiveAcum     9112 non-null   float64       
 6   Confirmed      9112 non-null   float64       
 7   Deaths         9112 non-null   float64       
 8   Recovered      9112 non-null   float64       
 9   Active         9112 non-null   float64       
 10  Population     8783 non-null   float64       
dtypes: datetime64[ns](1), float64(9), object(1)
memory usage: 854.2+ KB

data_by_country.head()

	Date	Country	ConfirmedAcum	ActiveAcum	Confirmed	Population
0	2020-02-24	Afghanistan	1.0	1.0	1.0	38928346.0
1	2020-02-25	Afghanistan	1.0	1.0	0.0	38928346.0
2	2020-02-26	Afghanistan	1.0	1.0	0.0	38928346.0
3	2020-02-27	Afghanistan	1.0	1.0	0.0	38928346.0
4	2020-02-28	Afghanistan	1.0	1.0	0.0	38928346.0

And finally I calculate the number of cases per million inhabitants

data_by_country = data_by_country.assign(ConfirmedAcum_Millon = data_by_country.ConfirmedAcum / data_by_country.Population * 1000000)
data_by_country.head()

	Date	Country	ConfirmedAcum	ActiveAcum	Confirmed	Population	ConfirmedAcum_Millon
0	2020-02-24	Afghanistan	1.0	1.0	1.0	38928346.0	0.025688
1	2020-02-25	Afghanistan	1.0	1.0	0.0	38928346.0	0.025688
2	2020-02-26	Afghanistan	1.0	1.0	0.0	38928346.0	0.025688
3	2020-02-27	Afghanistan	1.0	1.0	0.0	38928346.0	0.025688
4	2020-02-28	Afghanistan	1.0	1.0	0.0	38928346.0	0.025688

Rankins

I’m going to create a dataset of last day’s cases.

The goal is to get a set of rankings that tell me the countries with the most cases

So I need a variable that contains the last date of the dataset

last_day = list(data_by_country.Date.sort_values(ascending = False))[0]
last_day

Timestamp('2020-04-20 00:00:00')

Now I can filter the data by this date

last_day_data = data_by_country[data_by_country.Date == last_day]
last_day_data.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 184 entries, 56 to 9104
Data columns (total 12 columns):
 #   Column                Non-Null Count  Dtype         
---  ------                --------------  -----         
 0   Date                  184 non-null    datetime64[ns]
 1   Country               184 non-null    object        
 2   ConfirmedAcum         184 non-null    float64       
 3   DeathsAcum            184 non-null    float64       
 4   RecoveredAcum         184 non-null    float64       
 5   ActiveAcum            184 non-null    float64       
 6   Confirmed             184 non-null    float64       
 7   Deaths                184 non-null    float64       
 8   Recovered             184 non-null    float64       
 9   Active                184 non-null    float64       
 10  Population            178 non-null    float64       
 11  ConfirmedAcum_Millon  178 non-null    float64       
dtypes: datetime64[ns](1), float64(10), object(1)
memory usage: 18.7+ KB

last_day_data

	Date	Country	ConfirmedAcum	DeathsAcum	RecoveredAcum	ActiveAcum	Confirmed	Deaths	Recovered	Active	Population	ConfirmedAcum_Millon
56	2020-04-20	Afghanistan	1026.0	36.0	135.0	855.0	30.0	3.0	4.0	23.0	38928346.0	26.356116
99	2020-04-20	Albania	584.0	26.0	327.0	231.0	22.0	0.0	13.0	9.0	2877797.0	202.933007
155	2020-04-20	Algeria	2718.0	384.0	1099.0	1235.0	89.0	9.0	52.0	28.0	43851044.0	61.982561
205	2020-04-20	Andorra	717.0	37.0	248.0	432.0	4.0	1.0	13.0	-10.0	77265.0	9279.751505
237	2020-04-20	Angola	24.0	2.0	6.0	16.0	0.0	0.0	0.0	0.0	32866272.0	0.730232
...	...	...	...	...	...	...	...	...	...	...	...	...
9011	2020-04-20	Vietnam	268.0	0.0	214.0	54.0	0.0	0.0	12.0	-12.0	97338579.0	2.753276
9027	2020-04-20	Western Sahara	6.0	0.0	0.0	6.0	0.0	0.0	0.0	0.0	597339.0	10.044548
9038	2020-04-20	Yemen	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	29825964.0	0.033528
9072	2020-04-20	Zambia	65.0	3.0	35.0	27.0	4.0	0.0	2.0	2.0	18383955.0	3.535692
9104	2020-04-20	Zimbabwe	25.0	3.0	2.0	20.0	0.0	0.0	0.0	0.0	14862924.0	1.682038

184 rows × 12 columns

Let’s assign new columns with the most interesting rankins …

last_day_data = last_day_data.assign(
    Rank_ConfirmedAcum = last_day_data.ConfirmedAcum.rank(),
    Rank_Confirmed = last_day_data.Confirmed.rank(),
    Rank_ActiveAcum = last_day_data.ActiveAcum.rank(),
    Rank_Active = last_day_data.Active.rank(),
    Rank_ConfirmedAcum_Millon = last_day_data.ConfirmedAcum_Millon.rank()
)
last_day_data

	Date	Country	ConfirmedAcum	DeathsAcum	RecoveredAcum	ActiveAcum	Confirmed	Deaths	Recovered	Active	Population	ConfirmedAcum_Millon	Rank_ConfirmedAcum	Rank_Confirmed	Rank_ActiveAcum	Rank_Active	Rank_ConfirmedAcum_Millon
56	2020-04-20	Afghanistan	1026.0	36.0	135.0	855.0	30.0	3.0	4.0	23.0	38928346.0	26.356116	105.0	113.0	113.0	132.0	56.0
99	2020-04-20	Albania	584.0	26.0	327.0	231.0	22.0	0.0	13.0	9.0	2877797.0	202.933007	91.0	107.0	81.0	117.5	108.0
155	2020-04-20	Algeria	2718.0	384.0	1099.0	1235.0	89.0	9.0	52.0	28.0	43851044.0	61.982561	129.0	137.0	123.0	135.5	78.0
205	2020-04-20	Andorra	717.0	37.0	248.0	432.0	4.0	1.0	13.0	-10.0	77265.0	9279.751505	98.0	71.0	93.0	31.0	176.0
237	2020-04-20	Angola	24.0	2.0	6.0	16.0	0.0	0.0	0.0	0.0	32866272.0	0.730232	29.5	28.0	27.0	72.0	4.0
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
9011	2020-04-20	Vietnam	268.0	0.0	214.0	54.0	0.0	0.0	12.0	-12.0	97338579.0	2.753276	71.0	28.0	52.5	26.5	19.0
9027	2020-04-20	Western Sahara	6.0	0.0	0.0	6.0	0.0	0.0	0.0	0.0	597339.0	10.044548	6.0	28.0	10.5	72.0	35.0
9038	2020-04-20	Yemen	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	29825964.0	0.033528	1.0	28.0	3.0	72.0	1.0
9072	2020-04-20	Zambia	65.0	3.0	35.0	27.0	4.0	0.0	2.0	2.0	18383955.0	3.535692	45.5	71.0	40.5	100.0	21.0
9104	2020-04-20	Zimbabwe	25.0	3.0	2.0	20.0	0.0	0.0	0.0	0.0	14862924.0	1.682038	31.0	28.0	32.0	72.0	13.0

184 rows × 17 columns

Which countries have the most confirmed cases?

last_day_data.sort_values('Rank_ConfirmedAcum', ascending = False)[['Country', 'ConfirmedAcum']].reset_index(drop = True).head(10)

	Country	ConfirmedAcum
0	United States	784326.0
1	Spain	200210.0
2	Italy	181228.0
3	France	156480.0
4	Germany	147065.0
5	United Kingdom	125856.0
6	Turkey	90980.0
7	China	83817.0
8	Iran	83505.0
9	Russia	47121.0

Which countries have more confirmed cases on the last day?

last_day_data.sort_values('Rank_Confirmed', ascending = False)[['Country', 'Confirmed']].reset_index(drop = True).head(10)

	Country	Confirmed
0	United States	25211.0
1	United Kingdom	4684.0
2	Turkey	4674.0
3	Russia	4268.0
4	France	2383.0
5	Italy	2256.0
6	Brazil	2089.0
7	Canada	2025.0
8	Germany	1881.0
9	Spain	1536.0

Which countries have the most active cases?

last_day_data.sort_values('Rank_ActiveAcum', ascending = False)[['Country', 'ActiveAcum']].reset_index(drop = True).head(10)

	Country	ActiveAcum
0	United States	669903.0
1	United Kingdom	108860.0
2	Italy	108237.0
3	Spain	98771.0
4	France	98152.0
5	Turkey	75410.0
6	Germany	50703.0
7	Russia	43270.0
8	Netherlands	29502.0
9	Belgium	25260.0

Which countries had the most active cases on the last day?

last_day_data.sort_values('Rank_Active', ascending = False)[['Country', 'Active']].reset_index(drop = True).head(10)

	Country	Active
0	United States	21786.0
1	United Kingdom	4219.0
2	Russia	4069.0
3	Turkey	3097.0
4	Brazil	1964.0
5	Belarus	1461.0
6	Singapore	1393.0
7	Belgium	1204.0
8	Canada	1167.0
9	Saudi Arabia	1024.0

Which countries have the most active cases per million inhabitants?

last_day_data.sort_values('Rank_ConfirmedAcum_Millon', ascending = False)[['Country', 'ConfirmedAcum_Millon']].reset_index(drop = True).head(20)

	Country	ConfirmedAcum_Millon
0	San Marino	13615.867496
1	Holy See	11235.955056
2	Andorra	9279.751505
3	Luxembourg	5683.905824
4	Iceland	5195.710974
5	Spain	4282.129198
6	Belgium	3449.896515
7	Switzerland	3228.794972
8	Ireland	3169.841706
9	Italy	2997.395414
10	France	2397.297121
11	Monaco	2395.392692
12	United States	2369.545977
13	Liechtenstein	2124.422996
14	Qatar	2087.778323
15	Portugal	2046.052310
16	Netherlands	1960.213067
17	United Kingdom	1853.931291
18	Germany	1755.288621
19	Austria	1642.721097

What is the total number of cases?

import locale
locale.setlocale(locale.LC_ALL, '')

total = pd.DataFrame(last_day_data[["ConfirmedAcum", "RecoveredAcum", "DeathsAcum"]].aggregate("sum")\
                     .apply(lambda value : locale.format("%d", value, grouping=True)))
total.columns = ["Total Cases"]
total.index = ['Confirmed', 'Deaths', 'Recovered']
display(total.T)

	Confirmed	Deaths	Recovered
Total Cases	2,472,259	645,738	169,986

Saving the clean dataset

Finally I will create an Excel file with the information per country once it is clean and in perfect condition to apply some machine learning algorithms …

data_by_country.to_excel("All_data.xlsx", index = False)

Empresas orientadas a datos

Sun, 28 Feb 2016 16:30:20 +0000

¿Qué significa exactamente que una empresa esté orientada a datos?

En muchas empresas se cree que por generar montones de datos o tener aplicaciones de inteligencia de negocio, con unos cuantos cuadros de mando, ya están orientadas a datos.

Aunque efectivamente estas actividades forman parte de lo que hacen las empresas orientadas a datos, son actividades típicas para mostrar los hechos que están ocurriendo en ese momento o que ya han ocurrido en el pasado, sin explicar su causa, o hacer recomendaciones de qué se debe de hacer a continuación.

En contraste, las organizaciones orientadas a datos realizan análisis con miras al futuro, como utilizar modelos predictivos para optimizar el gasto, reponer una pieza en la cadena de suministro que está a punto de fallar, o parar la fuga de clientes a la competencia, con ciertas estrategias, como por ejemplo , ofrecerles un pequeño descuento justo antes de que se vayan. Realizar estas actividades requiere disponer de ciertas herramientas y habilidades; pero, sobre todo, debe de existir una cultura empresarial que promueva el uso de datos como base en la toma de decisiones.

¿Qué requisitos deben tener las empresas orientadas a datos?

Primer Requisito: ¡Datos!

El primer requisito es el más obvio: la organización debe de recolectar datos. Sin, duda los datos son un requisito clave. La diferencia aquí es que no vale cualquier dato, sino que deben de ser los datos correctos.

En concreto, los datos deben de ser relevantes a la cuestión que se intenta resolver; oportunos, en el sentido de que tiene se tienen que obtener antes de que sea demasiado tarde; completos; coherentes; sin ningún tipo de prejuicio; y lo más importante, tienen que ser de confianza.

Habitualmente los datos siempre requieren un proceso de ‘limpieza’ previo a su uso donde, entre otras actividades, se completan, se corrigen y se identifican los que son incoherentes. Esta tarea no es nada fácil y suele consumir el 80% del tiempo total, por otro 20% del tiempo donde se construyen los modelos, se analiza, se visualiza y se extraen conclusiones de esa información. Una señal que nos indica si estamos o no cumpliendo este requisito, es si en nuestra organización existe o no alguien que es responsable de la calidad de los datos.

Al final, todo es una cuestión de cultura corporativa. Por ejemplo, en las empresas orientadas a datos, los proyectos relacionados con la recolección de datos casi nunca tienen problemas de financiación.

Segundo requisito: Relaciones entre datos

Ya tenemos datos, ¿somos una empresa orientada a datos?

Tener datos relevantes, precisos, limpios y confiables no es suficiente para ser una empresa orientada a datos.

En primer lugar, estos datos tiene que poderse unir a otros datos de la empresa, cuando sea necesario. Esto, básicamente significa que vamos a poder relacionar fácilmente datos de distinta fuente. ¿El código de cliente de la aplicación de reclamaciones es el mismo que el código de cliente de la aplicación que gestiona los pedidos?, ¿se comparten los mismos códigos de producto entre las distintas aplicaciones corporativas?

Además, en la empresa tenemos que contar con un conjunto de herramientas adecuadas para poder buscar, filtrar, agregar y desglosar los datos.

Por ejemplo, si nuestros informes tienen la posibilidad de agregar los datos, podremos ver de un solo vistazo lo que está sucediendo en nuestro negocio y seremos capaces de ver tendencias, o podremos detectar las diferencias que se producen entre los distintos segmentos de clientes.

Pero sin duda, lo más importante es que tiene que existir una cultura corporativa que permita compartir información. Por ejemplo, tendríamos que poder unir los datos de la secuencia de clicks que hace un cliente en nuestra página web, con los datos de otros departamentos de nuestra empresa, como el departamento comercial o el departamento financiero. Será mucho mejor cuanta más información se comparta entre los distintos departamentos de la compañía.

En una organización orientada a datos, se comunican todos los números, independientemente de si ofrecen resultados negativos y, por ejemplo, cuando alguien ofrece una opinión, siempre se acompaña con números y datos.

Más ejemplos, en una organización orientada a datos, se dan mucha importancia a los procesos de gobierno de la información y todo el mundo comparte la misma definición para los distintos términos de negocio. Incluso se sabe perfectamente donde se generar la información, hacia dónde viaja y qué se hace con ella por el camino.

Tercer Requisito: Personas

Hemos cumplido los dos requisitos anteriores, ¿es suficiente? No, todavía no.

Necesitamos personas que utilicen datos, que sean capaces de filtrarlos y agregarlos, pero sobre todo, personas que tengan las habilidades suficientes para diseñar y escoger las métricas adecuadas que determinen la dirección que está tomando la empresa, de forma que se sea fácil monitorizar su progreso, su éxito o, lo antes posible, su fracaso. Además, alguien necesita decidir sobre estas métricas y lguien (posiblemente otra persona) debe de crear los procesos necesarios para proporcionar estos valores.

Por tanto, podemos decir que para que una organización sea orientada a datos, debe de tener personas que realicen las preguntas correctas sobre los datos, personas que tengan las habilidades necesarias para extraer la información correcta, personas que diseñen las métricas precisas, y personas que usen esa información para dirigir los siguientes pasos de la empresa y se den cuenta lo más rápido posible de las desviaciones tan pronto se producen.

Una vez más, la cultura empresarial es clave. En una empresa orientada a datos, cada objetivo, ya sea empresarial o personal de un empleado, se asocia con un indicador que permita establecer rápidamente si se cumple o no.

¿Es tu compañía una empresa orientada a datos?

Importing data from Twitter with python and tweepy

Wed, 21 Oct 2015 09:02:20 +0000

Twitter provides an API that lets you download data from this social network. To do this we will use python and the tweepy library.

The aim is to retrieve tweets related with the word ‘NoSQL’ and store them in a file for later analysis.

The first thing to do is register a new Twitter application via the Twitter Application Management page.

After registering our application, Twitter will give us the keys that we need to access it using its API

consumer_key = 'hRNtRgjzGq4wq3mt3fbuUkQ2c'
consumer_secret = 'yBbXnvNRpm93wvblpG9xhUMFF7w9sgxLfQT8k15Fs3k1RN4pnQ'
access_token_key = '12391902-qHOsgUBIvKuv7DjajXBmdm3SyZH8vgmR3jcpLVnnM'
access_token_secret = '9ViwfNW5FhOLhagaf4qfmDLXfY6qDtGzJ1MmAQM0gN3LK'

The next step is to import the library and login in twitter.

import tweepy

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token_key, access_token_secret)

api = tweepy.API(auth)

Testing the library

To test the library we will retrieve information about a user of twitter, his name, his followers, and so on ..

user = api.get_user('NoSQLDigest')

print "Name:", user.screen_name
print "Description:", user.description
print "Followers count:", user.followers_count
print "Friends' count:", user.friends_count
print "Statues Count [Number of Tweets]: ", user.statuses_count

Name: NoSQLDigest
Description: NoSQL Digest of tweets.
Followers count: 9785
Friends' count: 12
Statues Count [Number of Tweets]:  668120

Retrieving Tweets via a search term

lookup ='NoSQL'

Using the following method we can download tweets quickly, but it depends on the limit set by Twitter

max_tweets = 200
search_results = api.search(q=lookup, lang = 'en', count=max_tweets)

print len(search_results)

from prettytable import PrettyTable

table = PrettyTable(["User", "Fecha", "Texto"])
table.align["User"] = "l"
table.align["Texto"] = "l"

for tweet in search_results[0:10]:
    table.add_row([tweet.user.screen_name, tweet.created_at, tweet.text[:80]])
 
print table

User	Fecha	Texto
retweetjava	2015-10-21 04:00:19	RT @MusicHackFest: #GoGettersNetwork #CodeTalk Apache Hadoop and NoSQL as Analy
NeuvooPhpCA	2015-10-21 04:00:11	CyberCoders is hiring a #Senior #Backend Developer - Ruby, Python, PHP, Agile, S
GujaratiGuy789	2015-10-21 03:57:05	RT @SoftwareJokes: 3 database admins walked into a NoSQL bar. A little while lat
astosyk	2015-10-21 03:53:11	RT @couchbase: Get rapid ramp-up on #NoSQL application development with lessons
tjmickol	2015-10-21 03:40:36	Listening to Sweet Home Wundermude by Spill & Freunde on #kexp #galvanize #z
MooglieTwitimon	2015-10-21 03:38:26	Couchbase CEO on rise of NoSQL - https://t.co/9Zeif7Afw4 https://t.co/yglu7h3WA0
MooglieTwitimon	2015-10-21 03:38:24	Couchbase CEO on rise of NoSQL - https://t.co/9Zeif7Afw4 https://t.co/yglu7h3WA0
bph	2015-10-21 03:19:39	RT @couchbase: The 3 mega-trends that define businesses leading the digital econ
VVagias	2015-10-21 03:13:36	RT @geneolot: It’s jut great Tuesday evening, Bambini Dahlonega, Georgia #BigD
retweetjava	2015-10-21 03:13:35	RT @geneolot: It’s jut great Tuesday evening, Bambini Dahlonega, Georgia #BigD

Retrieving a user’s timeline

timeline_results = api.user_timeline(screen_name = 'NoSQLDigest', count = 1000, include_rts = True)
len(timeline_results)

Retrieving Tweets via a search term using a cursor

This method uses a cursor that skips the restriction of 200 tweets ;-)

c = tweepy.Cursor(api.search, q= lookup).items()
    
search_results = []
while True:
    try:
        tweet = c.next()
        # Insert into db
        search_results.append(tweet)
    except tweepy.TweepError as e:
        print e
        break

print len(search_results)

{"errors":[{"message":"Rate limit exceeded","code":88}]}
2648

This fails too!. In this case is because of a timeout limit :-(.

Retrieving a user’s timeline using a cursor

import sys

c = tweepy.Cursor(api.user_timeline,id='NoSQLDigest').items()    
timeline_results = []
while True:
    try:
        tweet = c.next()
        # Insert into db
        timeline_results.append(tweet)
    except:
        print "Error: ", sys.exc_info()[0]
        break

print len(timeline_results)

Error:  <type 'exceptions.StopIteration'>
3136

Visualizing the content of a Tweet

Twitter returns data in JSON format. Let’s see what a tweet looks like:

import pprintpp

tweet = search_results[0]
pprintpp.pprint(tweet._json)

{
    u'contributors': None,
    u'coordinates': None,
    u'created_at': u'Wed Oct 21 04:00:19 +0000 2015',
    u'entities': {
        u'hashtags': [
            {u'indices': [19, 36], u'text': u'GoGettersNetwork'},
            {u'indices': [37, 46], u'text': u'CodeTalk'},
            {u'indices': [131, 135], u'text': u'IoT'},
            {u'indices': [136, 140], u'text': u'Java'},
            {u'indices': [139, 140], u'text': u'API'},
            {u'indices': [139, 140], u'text': u'Linux'},
        ],
        u'symbols': [],
        u'urls': [
            {
                u'display_url': u'bit.ly/1FAQ7B5',
                u'expanded_url': u'http://bit.ly/1FAQ7B5',
                u'indices': [107, 130],
                u'url': u'https://t.co/z2lcnaHeov',
            },
        ],
        u'user_mentions': [
            {
                u'id': 3243621325,
                u'id_str': u'3243621325',
                u'indices': [3, 17],
                u'name': u'Hackathon News',
                u'screen_name': u'MusicHackFest',
            },
        ],
    },
    u'favorite_count': 0,
    u'favorited': False,
    u'geo': None,
    u'id': 656681395348217856,
    u'id_str': u'656681395348217856',
    u'in_reply_to_screen_name': None,
    u'in_reply_to_status_id': None,
    u'in_reply_to_status_id_str': None,
    u'in_reply_to_user_id': None,
    u'in_reply_to_user_id_str': None,
    u'is_quote_status': False,
    u'lang': u'en',
    u'metadata': {u'iso_language_code': u'en', u'result_type': u'recent'},
    u'place': None,
    u'possibly_sensitive': False,
    u'retweet_count': 1,
    u'retweeted': False,
    u'retweeted_status': {
        u'contributors': None,
        u'coordinates': None,
        u'created_at': u'Tue Oct 20 22:37:58 +0000 2015',
        u'entities': {
            u'hashtags': [
                {u'indices': [0, 17], u'text': u'GoGettersNetwork'},
                {u'indices': [18, 27], u'text': u'CodeTalk'},
                {u'indices': [112, 116], u'text': u'IoT'},
                {u'indices': [117, 122], u'text': u'Java'},
                {u'indices': [123, 127], u'text': u'API'},
                {u'indices': [128, 134], u'text': u'Linux'},
            ],
            u'symbols': [],
            u'urls': [
                {
                    u'display_url': u'bit.ly/1FAQ7B5',
                    u'expanded_url': u'http://bit.ly/1FAQ7B5',
                    u'indices': [88, 111],
                    u'url': u'https://t.co/z2lcnaHeov',
                },
            ],
            u'user_mentions': [],
        },
        u'favorite_count': 0,
        u'favorited': False,
        u'geo': None,
        u'id': 656600272781877248,
        u'id_str': u'656600272781877248',
        u'in_reply_to_screen_name': None,
        u'in_reply_to_status_id': None,
        u'in_reply_to_status_id_str': None,
        u'in_reply_to_user_id': None,
        u'in_reply_to_user_id_str': None,
        u'is_quote_status': False,
        u'lang': u'en',
        u'metadata': {u'iso_language_code': u'en', u'result_type': u'recent'},
        u'place': None,
        u'possibly_sensitive': False,
        u'retweet_count': 1,
        u'retweeted': False,
        u'source': u'<a href="http://ifttt.com" rel="nofollow">IFTTT</a>',
        u'text': u'#GoGettersNetwork #CodeTalk  Apache Hadoop and NoSQL as Analysis Engines for IoT Data ▸ https://t.co/z2lcnaHeov #IoT #Java #API #Linux …',
        u'truncated': False,
        u'user': {
            u'contributors_enabled': False,
            u'created_at': u'Fri Jun 12 20:02:51 +0000 2015',
            u'default_profile': False,
            u'default_profile_image': False,
            u'description': u'Music + Technology + Hackathons = @MusicHackFest',
            u'entities': {u'description': {u'urls': []}},
            u'favourites_count': 105,
            u'follow_request_sent': False,
            u'followers_count': 2143,
            u'following': False,
            u'friends_count': 1388,
            u'geo_enabled': False,
            u'has_extended_profile': False,
            u'id': 3243621325,
            u'id_str': u'3243621325',
            u'is_translation_enabled': False,
            u'is_translator': False,
            u'lang': u'en',
            u'listed_count': 2879,
            u'location': u'Los Angeles, CA',
            u'name': u'Hackathon News',
            u'notifications': False,
            u'profile_background_color': u'000000',
            u'profile_background_image_url': u'http://abs.twimg.com/images/themes/theme1/bg.png',
            u'profile_background_image_url_https': u'https://abs.twimg.com/images/themes/theme1/bg.png',
            u'profile_background_tile': False,
            u'profile_banner_url': u'https://pbs.twimg.com/profile_banners/3243621325/1434140299',
            u'profile_image_url': u'http://pbs.twimg.com/profile_images/609454174841876481/GMtjuot9_normal.jpg',
            u'profile_image_url_https': u'https://pbs.twimg.com/profile_images/609454174841876481/GMtjuot9_normal.jpg',
            u'profile_link_color': u'ABB8C2',
            u'profile_sidebar_border_color': u'000000',
            u'profile_sidebar_fill_color': u'000000',
            u'profile_text_color': u'000000',
            u'profile_use_background_image': False,
            u'protected': False,
            u'screen_name': u'MusicHackFest',
            u'statuses_count': 301851,
            u'time_zone': u'Pacific Time (US & Canada)',
            u'url': None,
            u'utc_offset': -25200,
            u'verified': False,
        },
    },
    u'source': u'<a href="http://not.yet/" rel="nofollow">final one kk</a>',
    u'text': u'RT @MusicHackFest: #GoGettersNetwork #CodeTalk  Apache Hadoop and NoSQL as Analysis Engines for IoT Data ▸ https://t.co/z2lcnaHeov #IoT #Ja…',
    u'truncated': False,
    u'user': {
        u'contributors_enabled': False,
        u'created_at': u'Thu Dec 25 02:07:45 +0000 2014',
        u'default_profile': False,
        u'default_profile_image': False,
        u'description': u"Hey, I retweet #Java related tweets. Follow us and maybe you'll learn something new! Questions/concerns? Contact @jdf221 or @Jordanb844",
        u'entities': {u'description': {u'urls': []}},
        u'favourites_count': 62458,
        u'follow_request_sent': False,
        u'followers_count': 2819,
        u'following': False,
        u'friends_count': 19,
        u'geo_enabled': False,
        u'has_extended_profile': False,
        u'id': 2942356560,
        u'id_str': u'2942356560',
        u'is_translation_enabled': False,
        u'is_translator': False,
        u'lang': u'en',
        u'listed_count': 2696,
        u'location': u'',
        u'name': u'Java',
        u'notifications': False,
        u'profile_background_color': u'022330',
        u'profile_background_image_url': u'http://abs.twimg.com/images/themes/theme15/bg.png',
        u'profile_background_image_url_https': u'https://abs.twimg.com/images/themes/theme15/bg.png',
        u'profile_background_tile': False,
        u'profile_image_url': u'http://pbs.twimg.com/profile_images/547942714675712000/Kr9dDPXJ_normal.png',
        u'profile_image_url_https': u'https://pbs.twimg.com/profile_images/547942714675712000/Kr9dDPXJ_normal.png',
        u'profile_link_color': u'0084B4',
        u'profile_sidebar_border_color': u'A8C7F7',
        u'profile_sidebar_fill_color': u'C0DFEC',
        u'profile_text_color': u'333333',
        u'profile_use_background_image': True,
        u'protected': False,
        u'screen_name': u'retweetjava',
        u'statuses_count': 164272,
        u'time_zone': None,
        u'url': None,
        u'utc_offset': None,
        u'verified': False,
    },
}

Parsing the result

The aim of the following functions to simplify the information we keep on file because Twitter provide too much information

def parse_user(usr):
    user = {}  
    user["created_at"] = usr['created_at']
    user["description"] = usr['description']
    user["favourites_count"] = usr['favourites_count']
    user["followers_count"] = usr['followers_count']
    user["friends_count"] = usr['friends_count']
    user["geo_enabled"] = usr['geo_enabled']
    user["_id"] = usr['id']
    user["id_str"] =usr['id_str']
    user["name"] = usr['name']
    user["screen_name"] = usr['screen_name']
    user["statuses_count"] = usr['statuses_count']
    user["profile_image_url"] = usr['profile_image_url']
    if usr['time_zone'] <> None:
        user["time_zone"] = usr['time_zone']
    
    return user

def parse_tweet(t):
    tweet = {}
    tweet['created_at'] = t['created_at']
    #for ht in tweet.entities.hashtags:
    #    print ht.text

    tweet['entities'] = []
    for k in t['entities']['hashtags']:
        tweet['entities'].append(k['text'])
  
    tweet['user_mentions'] = []
    for k in t['entities']['user_mentions']:
        k.pop("indices", None)
        tweet['user_mentions'].append(k)

    tweet['favorite_count'] =  t['favorite_count']

    if t['geo'] <> None:
        tweet['geo'] = t['geo']

    tweet['_id'] = t['id']
    tweet['id_str'] = t['id_str']  

    tweet['lang'] = t['lang']
    tweet['retweet_count'] = t['retweet_count']
    tweet['source'] = t['source']
    tweet['text'] = t['text']
    tweet['user'] = parse_user(t['user'])

    if 'retweeted_status' in t.keys():
        tweet['retweeted_status'] = parse_tweet(t['retweeted_status'])
        
    return tweet

We parse the content we’ve previously downloaded …

tweets = []

for tweet in search_results:
    tweets.append(parse_tweet(tweet._json))

Now the content also in JSON format but much simpler

pprintpp.pprint(tweets[0])

{
    '_id': 656681395348217856,
    'created_at': u'Wed Oct 21 04:00:19 +0000 2015',
    'entities': [
        u'GoGettersNetwork',
        u'CodeTalk',
        u'IoT',
        u'Java',
        u'API',
        u'Linux',
    ],
    'favorite_count': 0,
    'id_str': u'656681395348217856',
    'lang': u'en',
    'retweet_count': 1,
    'retweeted_status': {
        '_id': 656600272781877248,
        'created_at': u'Tue Oct 20 22:37:58 +0000 2015',
        'entities': [
            u'GoGettersNetwork',
            u'CodeTalk',
            u'IoT',
            u'Java',
            u'API',
            u'Linux',
        ],
        'favorite_count': 0,
        'id_str': u'656600272781877248',
        'lang': u'en',
        'retweet_count': 1,
        'source': u'<a href="http://ifttt.com" rel="nofollow">IFTTT</a>',
        'text': u'#GoGettersNetwork #CodeTalk  Apache Hadoop and NoSQL as Analysis Engines for IoT Data ▸ https://t.co/z2lcnaHeov #IoT #Java #API #Linux …',
        'user': {
            '_id': 3243621325,
            'created_at': u'Fri Jun 12 20:02:51 +0000 2015',
            'description': u'Music + Technology + Hackathons = @MusicHackFest',
            'favourites_count': 105,
            'followers_count': 2143,
            'friends_count': 1388,
            'geo_enabled': False,
            'id_str': u'3243621325',
            'name': u'Hackathon News',
            'profile_image_url': u'http://pbs.twimg.com/profile_images/609454174841876481/GMtjuot9_normal.jpg',
            'screen_name': u'MusicHackFest',
            'statuses_count': 301851,
            'time_zone': u'Pacific Time (US & Canada)',
        },
        'user_mentions': [],
    },
    'source': u'<a href="http://not.yet/" rel="nofollow">final one kk</a>',
    'text': u'RT @MusicHackFest: #GoGettersNetwork #CodeTalk  Apache Hadoop and NoSQL as Analysis Engines for IoT Data ▸ https://t.co/z2lcnaHeov #IoT #Ja…',
    'user': {
        '_id': 2942356560,
        'created_at': u'Thu Dec 25 02:07:45 +0000 2014',
        'description': u"Hey, I retweet #Java related tweets. Follow us and maybe you'll learn something new! Questions/concerns? Contact @jdf221 or @Jordanb844",
        'favourites_count': 62458,
        'followers_count': 2819,
        'friends_count': 19,
        'geo_enabled': False,
        'id_str': u'2942356560',
        'name': u'Java',
        'profile_image_url': u'http://pbs.twimg.com/profile_images/547942714675712000/Kr9dDPXJ_normal.png',
        'screen_name': u'retweetjava',
        'statuses_count': 164272,
    },
    'user_mentions': [
        {
            u'id': 3243621325,
            u'id_str': u'3243621325',
            u'name': u'Hackathon News',
            u'screen_name': u'MusicHackFest',
        },
    ],
}

Saving the data in a file

Finally we will record the data downloaded to a file, so that we can analyze later

import json
    
with open('./data/tweets.json',"w") as file:
    for t in tweets:
        r = json.dumps(t)
        file.write(r)
        file.write("\n")

print "Number of tweets ... ", len(tweets)

Number of tweets ...  3136

tweets = []
for tweet in timeline_results:
    tweets.append(parse_tweet(tweet._json))
    
with open('./data/timeline.json',"w") as file:
    for t in tweets:
        r = json.dumps(t)
        file.write(r)
        file.write("\n")
        
print "Number of tweets ... ", len(tweets)

Number of tweets ...  3136

Hope this helps!

Visualizing distributions of data

Tue, 15 Sep 2015 11:21:29 +0000

This notebook demonstrates different approaches to graphically representing distributions of data, specifically focusing on the tools provided by the seaborn package.

%matplotlib inline

import numpy as np
from numpy.random import randn
import pandas as pd
from scipy import stats
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_palette("deep", desat=.6)
sns.set_context(rc={"figure.figsize": (8, 4)})
sns.set_style("white")
np.random.seed(9221999)

Basic visualization with histograms

The most basic and common way of representing a distributions is with a histogram. We can do this directly through the hist function that is part of matplotlib.

data = randn(75)
plt.hist(data);

By default, hist separates the data into 10 bins of equal widths and plots the number of observations in each bin. Thus, the main parameter is the number of bins, which we can change.

The more bins you have, the more sensitive you will be to high-frequency patterns in the distribution. But, sometimes those high-frequency patterns will be noise. Often you want to try different values until you think you have best captured what you see in the data.

plt.hist(data, 6, color=sns.desaturate("indianred", .75));

The normed argument can also be useful if you want to compare two distributions that do not have the same number of observations. Note also that bins can be a sequence of where each bin starts.

data1 = stats.poisson(2).rvs(100)
data2 = stats.poisson(5).rvs(120)
max_data = np.r_[data1, data2].max()
bins = np.linspace(0, max_data, max_data + 1)
plt.hist(data1, bins, normed=True, color="#6495ED", alpha=.5)
plt.hist(data2, bins, normed=True, color="#F08080", alpha=.5);

The hist function has quite a few other options, which you can explore in its docstring. Here we’ll just highlight one more that can be useful when plotting many observations (such as following a resampling procedure).

x = stats.gamma(3).rvs(5000)
plt.hist(x, 70, histtype="stepfilled", alpha=.7);

You can also represent a joint distribution with the histogram method, using a hexbin plot. This is similar to a histogram, except instead of coding the number of observations in each bin with a position on one of the axes, it uses a color-mapping to give the plot three quantitative dimensions.

In seaborn, you can draw a hexbin plot using the jointplot function and setting kind to "hex". This will also plot the marginal distribution of each variable on the sides of the plot using a histrogram:

y = stats.gamma(5).rvs(5000)
with sns.axes_style("white"):
    sns.jointplot(x, y, kind="hex");

Estimating the density of the observations: `kdeplot` and `rugplot`

A superior, if more computationally intensive, approach to estimating a distribution is known as a kernel density estimate, or KDE. To motivate the KDE, let’s first think about rug plots. A rug plot is a very simple, but also perfectly legitimate, way of representing a distribution. To create one, simply draw a vertical line at each observed data point. Here, the height is totally arbitrary.

sns.set_palette("hls", 1)
data = randn(30)
sns.rugplot(data)
plt.ylim(0, 1);

You can see where the density of the distribution is by how dense the tick-marks are. Before talking about kernel density plots, let’s connect the rug plot to the histogram. The connection here is very direct: a histogram just creates bins along the range of the data and then draws a bar with height equal to the number of ticks in each bin

plt.hist(data, alpha=.3)
sns.rugplot(data);

A kernel density plot is also a transformation from the tick marks to a height-encoded measure of density. However, the transformaiton is a bit more complicated. Instead of binning each tick mark, we will instead represent each tick with a gaussian basis function.

# Draw the rug and set up the x-axis space
sns.rugplot(data);
xx = np.linspace(-4, 4, 100)

# Compute the bandwidth of the kernel using a rule-of-thumb
bandwidth = ((4 * data.std() ** 5) / (3 * len(data))) ** .2
bandwidth = len(data) ** (-1. / 5)

# We'll save the basis functions for the next step
kernels = []

# Plot each basis function
for d in data:
    
    # Make the basis function as a gaussian PDF
    kernel = stats.norm(d, bandwidth).pdf(xx)
    kernels.append(kernel)
    
    # Scale for plotting
    kernel /= kernel.max()
    kernel *= .4
    plt.plot(xx, kernel, "#888888", alpha=.5)
plt.ylim(0, 1);

We then estimate the distribution that our samples came from from by summing these basis functions (and normalizing so, as a proper density, the function integrates to 1).

There is also a function in the scipy.stats module that will perform a kernel density estimate (it actually returns an object that can be called on some values to return the density). We see that plotting the values from this object give us basically the same results as summing the gaussian basis functions.

# Set up the plots
f, (ax1, ax2) = plt.subplots(2, 1, sharex=True)
c1, c2 = sns.color_palette("husl", 3)[:2]

# Plot the summed basis functions
summed_kde = np.sum(kernels, axis=0)
ax1.plot(xx, summed_kde, c=c1)
sns.rugplot(data, c=c1, ax=ax1)
ax1.set_yticks([])
ax1.set_title("summed basis functions")

# Use scipy to get the density estimate
scipy_kde = stats.gaussian_kde(data)(xx)
ax2.plot(xx, scipy_kde, c=c2)
sns.rugplot(data, c=c2, ax=ax2)
ax2.set_yticks([])
ax2.set_title("scipy gaussian_kde")
f.tight_layout()

The seaborn package has a high-level function for plotting a kernel density estimate in one quick step, along with some additional nice features, such as shading in the density.

sns.kdeplot(data, shade=True);

Much like how the bins parameter controls the fit of the histogram to the data, you can adjust the bandwidth (bw) of the kernel to make the densisty estimate more or less sensitive to high-frequency structure.

pal = sns.blend_palette([sns.desaturate("royalblue", 0), "royalblue"], 5)
bws = [.1, .25, .5, 1, 2]

for bw, c in zip(bws, pal):
    sns.kdeplot(data, bw=bw, color=c, lw=1.8, label=bw)

plt.legend(title="kernel bandwidth")
sns.rugplot(data, color="#333333");

Although the gaussian kernel is the most common, and probably the most useful, there are a variety of kernels you can use to fit the density estimate. The kernel you choose generally has less influence on the resulting estimate thean the bandwidth size, but there may be situations where you want to experiment with different choices.

kernels = ["biw", "cos", "epa", "gau", "tri", "triw"]
pal = sns.color_palette("hls", len(kernels))
for k, c in zip(kernels, pal):
    sns.kdeplot(data, kernel=k, color=c, label=k)
plt.legend();

The cut and clip parameters allows you to control how far outside the range of the data the estimate extends: cut influences only the range of the support, while clip affects the fitting of the KDE as well.

with sns.color_palette("Set2"):
    f, (ax1, ax2) = plt.subplots(2, 1, figsize=(8, 8), sharex=True)
    for cut in [4, 3, 2]:
        sns.kdeplot(data, cut=cut, label=cut, lw=cut * 1.5, ax=ax1)

    for clip in [1, 2, 3]:
        sns.kdeplot(data, clip=(-clip, clip), label=clip, ax=ax2);

As in the case of the histogram, plotting shaded density plots on top of each other can be a good way to ask whether two samples are from the same distribution. This also implements a simple classification operation. For a given x value with an unknown label, you should conclude it was drawn from the distribution with a greater density at that value.

A legend can be helpful when overlaying several densities. This is done either by providing a label keyword to kdeplot, which explicitly assigns a value to the data, or by inferring the name if you pass an object (like a Pandas Series) with a name attribute.

f, (ax1, ax2) = plt.subplots(2, 1, sharex=True, figsize=(8, 6))
c1, c2, c3 = sns.color_palette("Set1", 3)

dist1, dist2, dist3 = stats.norm(0, 1).rvs((3, 100))
dist3 = pd.Series(dist3 + 2, name="dist3")

sns.kdeplot(dist1, shade=True, color=c1, ax=ax1)
sns.kdeplot(dist2, shade=True, color=c2, label="dist2", ax=ax1)

sns.kdeplot(dist1, shade=True, color=c2, ax=ax2)
sns.kdeplot(dist3, shade=True, color=c3, ax=ax2);

If you want to plot the density along the y axis, use the vertical keyword.

plt.figure(figsize=(4, 7))
data = stats.norm(0, 1).rvs((3, 100)) + np.arange(3)[:, None]

with sns.color_palette("Set2"):
    for d, label in zip(data, list("ABC")):
        sns.kdeplot(d, vertical=True, shade=True, label=label)

You can also use kdeplot to estimate the cumulative distribution function (CDF) of the population from your data. This plot will tell you what proportion of the distribution falls at smaller values than a given point on the x axis:

with sns.color_palette("Set1"):
    for d, label in zip(data, list("ABC")):
        sns.kdeplot(d, cumulative=True, label=label)

Multivariate density estimation with `kdeplot`

You can also use the kernel density method with multidimensional data. For visualization, we are mostly concerned with plotting joint bivariate distributions. As with the hexbin plot, we will color-encode the density estimate over a 2D space. The kdeplot function tries to infer whether it should draw a univariate or bivariate plot based on the type and shape of the data argument. If using a 2d array or a DataFrame, the array is assumed to be shaped (n_units, n_variables).

data = np.random.multivariate_normal([0, 0], [[1, 2], [2, 20]], size=1000)
data = pd.DataFrame(data, columns=["X", "Y"])
mpl.rc("figure", figsize=(6, 6))

sns.kdeplot(data);

You can also pass in two vectors as the first positional arguments, which will also draw a bivariate density.

sns.kdeplot(data.X, data.Y, shade=True);

As in the univariate case, you can also set the kernel bandwidth and choose different points at which to clip the data or cut the estimate. However, only the gaussian kernel is availible for the multidimensional kde.

When specifying the colormap, the multivariate kdeplot accepts a special token where colormaps that end with _d are plotted in a way that maintains the overall color palette but that uses darker colors at one extreme so the contour lines are fully visible.

sns.kdeplot(data, bw="silverman", gridsize=50, cut=2, clip=(-11, 11), cmap="BuGn_d");

sns.kdeplot(data, bw=1, clip=[(-4, 4), (-11, 11)], cmap="PuRd_d");

sns.kdeplot(data.values, shade=True, bw=(.5, 1), cmap="Purples");

Bivariate and univariate plots using `jointplot`:

The jointplot function allows you to simultaneously visualize the joint distribution of two variables and the marginal distribution of each. Above, we showed how to use it to draw a hexbin plot. You can also use it to draw a kernel density estimate:

with sns.axes_style("white"):
    sns.jointplot("X", "Y", data, kind="kde");

Combining plot styles: `distplot`

Each of these styles has advantages and disadvantages. Fortunately, it is easy to combine multiple styles using the distplot function in seaborn. distplot provides one interface for plotting histograms, kernel density plots, rug plots, and plotting fitted probability distributions.

By default, you’ll get a kernel density over a histogram. Unlike the default matplotlib hist function, distplot tries to use a good number of bins for the dataset you have, although all of the options for specifying bins in hist can be used.

sns.set_palette("hls")
mpl.rc("figure", figsize=(8, 4))
data = randn(200)
sns.distplot(data);

hist, kde, and rug are boolean arguments to turn those features on and off.

sns.distplot(data, rug=True, hist=False);

You can also pass a distribution family from scipy.stats, and distplot will fit the parameters using maximum likelihood and plot the resulting function.

sns.distplot(data, kde=False, fit=stats.norm);

To control any of the underlying plots, pass keyword arguments to the [plot]_kws argument.

sns.distplot(data,
             kde_kws={"color": "seagreen", "lw": 3, "label": "KDE"},
             hist_kws={"histtype": "stepfilled", "color": "slategray"});

You can also draw the distribution vertically, if for example you wanted to plot marginal distributions on a scatterplot (as in the jointplot function):

plt.figure(figsize=(4, 7))
sns.distplot(data, color="dodgerblue", vertical=True);

If the data has a name attribute (e.g. it is a pandas Series), the name will become the label for the dimension on which the distributio is plotted, unless you use axlabel=False. You can also provide a string, which will override this behavior and label nameless data.

sns.distplot(pd.Series(data, name="score"), color="mediumpurple");

Comparing distributions: `boxplot` and `violinplot`

sns.set(rc={"figure.figsize": (6, 6)})
sns.set_style("white")

Frequently, you will want to compare two or more distributions. Although above we showed one method to do this above, it’s generally better to plot them separately but in the way that allows for easy comparisons.

The traditional approach in this case is to use a boxplot. There is a boxplot function in matplotlib we could use…

data = [randn(100), randn(100) + 1]
plt.boxplot(data);

…but, it is quite ugly by default. To get more aesthetically pleasing plots, use the seaborn.boxplot function:

sns.boxplot(data);

The default rules for a boxplot are that the box encompasses the inter-quartile range with the median marked. The “whiskers” extend to 1.5 * IQR past the closest quartile, and any observations outside this range are marked as outliers.

This is quite a mouthfull though, and the outliers can be distracting, so you can just make the whiskers extend all the way out. Let’s also tweak the aesthetics a bit.

sns.boxplot(data, names=["group1", "group1"], whis=np.inf, color="PaleGreen");

If fits better with the plot you are drawing, the boxplot can be horiztonal. This just uses the matplotlib parameter, which is awkwardly named vert:

sns.boxplot(data, names=["group1", "group1"], linewidth=2, widths=.5, color="skyblue", vert=False);

In some cases, you may want to plot repeated-measures data. In this case, a subtle effect that is consistent across subjects can be masked and look non-consequential.

To show such an effect, use the join_rm argument.

pre = randn(25)
post = pre + np.random.rand(25)
sns.boxplot([pre, post], names=["pre", "post"], color="coral", join_rm=True);

The boxplot is more informative than a bar plot, but it still compresses the a distribution to about five points. Just as the kernel density plot is a modern alternative to the histogram, we can use our computing power to bring more information using a kernel density estimate to these comparative plots.

These plots are known as “violin” (apparently, sometimes “viola”) plots. They essentially combine a boxplot with a kernel density estimate.

Let’s create a toy case that demonstrates why we might prefer the increased information in the violin plot.

d1 = stats.norm(0, 5).rvs(100)
d2 = np.concatenate([stats.gamma(4).rvs(50),
                     -1 * stats.gamma(4).rvs(50)])
data = pd.DataFrame(dict(d1=d1, d2=d2))

First, draw a boxplot. Note that the color argument can take anything that can be used as a palette in addition to any single valid matplotlib color, and that the function is Pandas-aware and will try to label the axes appropriately.

sns.boxplot(data, color="pastel", widths=.5);

Based on this plot, it looks like we basically have two samples from the same distribution.

But, let’s just see what the violin plot looks like:

sns.violinplot(data, color="pastel");

Woah! Now it looks like the distribution on the left is roughly normal, but the distribution on the right is bimodal with peaks at $+/-$ 5.

It may be rare to run into such data, but more information doesn’t hurt even in non-pathological cases, and might catch problems that otherwise could slip through.

(Of course, if you looked at each distribution with a histogram/KDE plot as above, you might have caught this before making any comparisons.)

Both the boxplot and violin functions can take a Pandas Series object as the data and an object that can be used to perform a groupby on the data to group it into the boxes/violins.

y = np.random.randn(200)
g = np.random.choice(list("abcdef"), 200)
for i, l in enumerate("abcdef"):
    y[g == l] += i // 2
df = pd.DataFrame(dict(score=y, group=g))
sns.boxplot(df.score, df.group);

Much like the kdeplot, you can tune the bandwidth of the kernel used to fit the density estimate in the violin.

sns.violinplot(df.score, df.group, color="Paired", bw=1);

If using a groupby, the default is to plot the boxes or violins in the sorted order of the group labels. You can override this, though:

order = list("cbafed")
sns.boxplot(df.score, df.group, order=order, color="PuBuGn_d");

The violin plot by default plots the median, along with the 25th and 75th percentile – the same information we get from the boxplot. There are, however, other options. You might want to plot each observation (similar to what we do with a rug plot under a KDE). There are two ways to accomplish this.

data = pd.melt(data.ix[:50], value_name="y", var_name="group")

f, (ax_l, ax_r) = plt.subplots(1, 2)
sns.violinplot(data.y, data.group, "points", positions=[1, 2], color="RdBu", ax=ax_l)
sns.violinplot(data.y, data.group, "stick", positions=[3, 4], color="PRGn", ax=ax_r)
plt.tight_layout()

Of course, you can plot repeated-measures data with the violin as well.

pre = randn(20)
data = pd.DataFrame(dict(pre=pre, post=pre + 1 + randn(20)), columns=["pre", "post"])
sns.violinplot(data, inner="points", join_rm=True, color="RdGy_r");

Using a palette colorscheme can be particularly useful if you have many bins.

mpl.rc("figure", figsize=(9, 6))

data = randn(50, 8) + np.random.uniform(3, 5, 8)
sns.boxplot(data);

Chose the color scheme carefully! The above is good for categorigal bins, but perhaps there is some ordering to the grouping variable:

data.sort(axis=1)
sns.boxplot(data, widths=.8, color="cubehelix");

Different kinds of relationships lend themselves to different kinds of color palettes:

data = data * -1 + data.mean()
sns.violinplot(data, color="coolwarm_r", lw=2);

Apache Spark en 10 minutos

Tue, 15 Sep 2015 11:21:29 +0000

Hola, os dejo una guía paso a paso para poder probar Apache Spark a partir de la instalación de una máquina virtual en vuestro equipo. Los requisitos mínimos son los siguientes:

Un equipo con más de 2Gb de memoria
Sistema operativo Windows, Linux o Mac
VirtualBox
Una máquina virual con todo el software necesario ya configurado.
Ganas de explorar lo desconocido ;-)

Ahí vamos… El primer paso es descargar la aplicación VirtualBox (https:/www.virtualbox.org), en el caso de que no se tenga previamente instalada en el equipo.

Una vez descargado el fichero, se debe de instalar VirtualBox en el equipo y ejecutarlo (necesitareis permisos de administrador).

Una vez en VirtualBox, hay que importar el fichero sparkvm.ova, desde la opción “Importar servicio virtualizado” del menú Archivo:

Cuando finalice la importación, se tiene pulsar el botón “Iniciar” para ejecutar la máquina virtual y probar Spark.

Con la opción “Inicio sin pantalla” es suficiente, ya que la forma de probar Spark es a través del navegador, y no necesitamos tener la consola activa.

Se abre un navegador y se accede a la dirección http://localhost:8001, donde está desplegado Jupyter sobre la máquina virtual que acabamos de instalar.

Jupyter es un software donde se pueden editar y ejecutar Notebooks de IPython o python interactivo. Un Notebook es un documento que contiene un conjunto de celdas. Cada celda puede tener texto, en formato MarkDown, o en código Python que se puede ejecutar de forma individual. Cuando se ejecuta el código contenido en una celda, el resultado se muestra en el propio notebook, justo debajo de la celda. Por ejemplo si introducimos en una celda el código python [print 1+ 1] y lo ejecutamos, el resultado será …

Para probar Spark tenemos a descargar el notebook spark_tutorial_student.ipynb que se encuentra disponible en el repositorio mooc-setup de GitHub. Para descargar este fichero en nuestro disco lo más fácil es a través de la opción de “Guardar enlace como …”

Una vez Jupyter, ya podemos cargar este notobook pulsando el botón upload …

Y volvemos a pulsar el botón Upload, está vez en color azul:

Solo nos queda hacer doble click sobre el notebook para que aparezca una nueva ventana:

Para seguir el tutorial de Spark se deben de ejecutar cada una de las celdas de este notebook, leyendo atentamente los textos explicativos. Para ejecutar una celda, primero hay que seleccionarla haciendo click sobre ella … (la celda se muestra con un recuadro verde)

Y pulsar el botón de ejecución …

Hasta que lleguemos al final… También se pueden ejecutar otros notebooks de este repositorio, aunque la mayoría requiere que introduzcamos código en python. En especial es muy interesante hacer los notebook ‘lab0_student.ipynb’ y ‘lab1_word_count_student.ipynb’. El resto es un poco más complicado pero os animo a que lo intentéis. Por último, comentaros que para cerrar la máquina virtual simplemente se deberá ejecutar la opción “Apagar” o pulsar Control-F

Espero que lo hayáis encontrado interesante…