Justin Williams | Personal Website

Query Historical Weather Data with API

2021-04-16T00:00:00-07:00

It has been a while since I last posted a blog to the site here. This has been due to a combination of factors including:

Finishing my dissertation/defense :tada:
Starting a new job with the Los Angeles Dodgers :baseball:
General lack of motivation to write :pen:

2020 was an abnormal year for many of us, but things have started to become more normal both personally and in the world around as vaccine rollout here in Los Angeles is now open to all adults 16+. With this return to normalcy, I have found a renewed desire to pick up the virtual pen and continue to write new posts.

My post today is on how you can use an API to query historical weather data, given a location and date of interest. The API I have found particularly useful for this is the one provided by Visual Crossing Weather Data. Users can sign-up through their website to create a free account which comes with a unique API key. This free tier allows you to run 1,000 requests per day. Their service is particularly helpful in that it allows you to query historical dates, which are not available at the free tier for services like OpenWeather API.

Example

Below is an example showing how to retrieve the weather for Los Angeles, CA on August 9th, 1991 at 10am local time using Python.

import requests, json

base_url= 'https://weather.visualcrossing.com/VisualCrossingWebServices/rest/services/timeline/'
query_key='?key=' + YOUR_API_KEY_HERE
query_params='&include=current&unitGroup=us'
query_loc='Los Angeles, CA'
query_date='1991-08-09T10:00:00'

url=base_url + query_loc + '/' + query_date + query_key + query_params

weather_conditions=response.json()['currentConditions']

print(f"Temp: {weather_conditions['temp']} Humidity: {weather_conditions['humidity']}")

Note that above we set the parameter &include=current so that we return a single record for the current time requested. By default the request will return records for every hour, which can end up using more requests then you need. We also set &unitGroup=us to return values in US units, e.g., will return temperature in degrees Fahrenheit. Above we specified the location as {City, State} but it will also handle other partial address forms such as {Street Address, City, State}, US Zip Codes, or Lat/Lon coordinates.

For more details on the available parameters and query structure see their documentation of the Timeline Weather API, as it is flexible to adapt to many different use cases.

By using this API, you can reliably and consistently query weather information to include as features in a data analytics projects. Or you can use it to query the weather on your birthday from any location.

GitHub Welcome Feature

2020-07-13T00:00:00-07:00

Today, I wanted to write a brief blog about a new release from GitHub that allows users to create personal welcome messages automatically. This new feature was released as a “secret” feature in a recent GitHub update.

Unlocking the Secret

To create the welcome message you must create a repository with the same name as your GitHub username. For instance my username is williazo, so to unlock this feature I would simply create a new repository with the name williazo. Note that as soon as you give the repository the appropriate name, a note will appear identifying that you have uncovered the secret. Ensure that your repository is set to public and initialize it with a README.md. The README.md serves as the template for creating the welcome message and comes with some initial suggestions for things to includes such as projects you are currently working on, contact info, etc. From there though the choice is yours about what and how you want to include relavent information. Below is a screenshot of the landing page for my GitHub account with the new welcome message.

My personal GitHub landing page.

You can see that this provides you an opportunity to explain a little bit more about yourself and your goals rather than strictly listing your repositories. You will also notice that the welcome README.md supports GitHub emojis¹.

Inspiration

To get the creative juices flowing I have also included several examples below of welcome pages to serve as inspiration. Links to each of the respective GitHub pages is included in the captions. There is also a great directory at https://github.com/abhisheknaiidu/awesome-github-profile-readme dedicated to showcasing these welcome pages.

Examples of interesting GitHub welcome pages.
Example 1: Available at https://github.com/BLKKKBVSIK with animation.
Available at https://github.com/vanshkapoor.

For a complete list of the supported emojis see https://gist.github.com/rxaviers/7360908. ↩

vscode-Intro for Python

2020-06-18T00:00:00-07:00

I am back on a roll now with blogging and I can feel the momentum. This week I wanted to spend some time going over an introduction to one of my favorite new discoveres, Visual Studio Code.

Visual Studio Code, referred to as vscode throughout the remaining post, in its most simple sense is a text editor, but it is a text editor on steroids. This software was developed by the Microsoft Team and has become extremely popular within the software develoment community. The basic install version of vscode comes pre-installed with several usefull features such as IntelliSense that provides users with comfortable highlighting and autocomplete, built in Git compatibility for distributed version control, and an expansive catalogue of extensions. These extensions range from highly functional to purely aesthetic.

My personal favorite aesthetic extension is Color Highligher that allows you to preview colors entered as Hex, e.g., “#F00123”, values or rgb, e.g., “rgb(80, 20, 80)”, functions within the editor rather than after running the code. See the screen below from vscode that shows the preview color for each of these cases, red and purple, in the text editor as boxes before the text.

While it has many of these abilities to help make code more readable and programmer friendly, it become widely accepted among developers due to its performance with languages such as JavaScript, CSS, and other web development essentials. Today, in addition to web development vscode is also gaining increasing popularity among data scientists due to its Pyton compatibility.

vscode-Python

We can add the Python extension from the vscode marketplace using their extension here. This extension allows you to select which Python interpretor, Python 2 or Python 3, you want to run based on what you have installed locally on your system. One of my favorite things though about working with Python in vscode has been the integration it provides for working with Jupyter notebooks.

Integration with Jupyter Notebooks

As someone who was introduced into statistical programming primarily using R, I have struggled everytime I have tried to switch to working in Python because I had become so dependent on the RStudio IDE with windows to display the local environment, generate plots, etc. However, on the occasions when I have worked on projects in Python I have tried to work using Jupyter notebooks.

Jupyter was developed as a project to allow for interactive computing across different programming languages such as R, Julia, Python, and others. Jupyter notebooks are web applications that allow you to simultanously include live code in your desired language alongside markdown blocks that would allow you to write text and equations. By allowing the code to be live, anyone who was reading the notebook could instantly run the code themselves to check that they were receiving the same output or download the notebook and expand/edit the code easily wihout having to copy and paste. In addition, Jupyter notebooks automatically render¹ within GitHub as you can see from this notebook which I used when writing documentation for my pacakge microbiomeDASim to simulate longitudinal differential abundance.

While I enjoyed all of these features about Jupyter notebooks, I still had challenges transitioning to working in them full-time as I found it difficult and clunky to open notebooks from the terminal which seemed to regularly throw error messages whenever I updated Python. I was so grateful to find that once I started using vscode that it natively supported working Jupyter notebooks. For any file with extension .ipynb vscode will automatically open the Jupyter notebook editor. Below I have included a short GIF that shows the vscode interface for Jupyter notebooks. In this GIF I can click a markdown block, enter text, run Python chunks, and even view local variables.

The full Jupyter notebook used in the GIF can be downloaded here for those interested in trying to play around with some of the features. Overall, I have enjoyed discovering many of the features within vscode that make programming slighlty easier, especially when working in Python.

Additional Notes

There are extensions for working in R with vscode. From my brief experience it works really well for handling R Markdown, i.e., files with .rmd extensions, but lacks many of the qualities that make working in RStudio seamless such as viewing the data environment and plots. I need to spend some more time playing around with writing R code using vscode-R, and I will refrain full judgement until that point. As an alternative, you can also change the language selected in the .ipynb file to support R allowing you to write Jupyter notebooks in R.
For sharing Jupyter notebooks it can be helpful to transport the code to the Google Collab platform as they will allow you to run notebooks and create a simple link for other users to access the code rather than installing Jupyter and its dependencies locally.

While the notebooks are rendered they cannot run code live when displayed within a GitHub page as they are not running using the Jupyter server. ↩

Comparing R Graphic Packages - ggplot2 vs. plotly

2020-06-11T00:00:00-07:00

After starting this blogging section of my personal website I set a lofty goal to try to have a new blog post each week. However, reality set in quickly and this is now my first post in nearly 5 months. Despite having more time available at home due to the COVID-19 quarantine and safer at home mandates in Los Angeles County¹, I continually found ways to avoid writing new posts.

That was until today. I recently found myself motivated to write this new post as I have been working with different graphics packages in R as part of my dissertation work. In particular, I wanted to compare two popular R packages for creating graphics: ggplot2 and plotly.

Motivating Example

To demonstrate some of the abilities of these two packages I will use my own R package, rwindow.baseball, to pull several baseball statistics and create relevant graphics for attendance figures for the Los Angeles Dodgers. I first start by installing the rwindow.baseball package.

# install.packages("devtools") #uncomment if devtools has not been installed previously
devtools::install_github("williazo/rwindow.baseball")

Now I will pull² attendance figures for the Los Angeles Dodgers from the 2015-2019.

lad_dt <- tm_standings_schedule("LAD", start_year = 2015, end_year = 2019)

From this data we may be particularly interested to see attendance patterns over time with respect to the following objectives.

Objectives

Graph home attendance for each Dodger game from 2015-2019
Estimate trend lines for each year
Identify attendance on July 4th as special date of interest

In order to achieve these three objectives we will first pre-process our data before passing it off to our respective graphic packages. I use piping features from the dplyr package to process the data which is also a component of the tidyverse collection.

library(tidyverse)
lad_dt <- tm_standings_schedule("LAD", start_year = 2015, end_year = 2019)
lad_dt <- lad_dt%>%
  filter(home_gm==1)%>% #only including home games
  group_by(Year)%>%
  mutate(home_gm_num=row_number())%>%
  ungroup()
lad_dt$july_forth <- ifelse(grepl("Jul 4", lad_dt$Date), 1, 0)

Our data is now ready to be used to generate our graphics of interest.

`ggplot2`

This is the most common graphics package outside of base R graphics. It is the result of the work of one of the most well-known contributors to R, Hadley Wickham, and a key component of the increasingly popular tidyverse collection. ggplot2 has made it incredibly easy and flexible for users to create high quality graphics using a consistent syntax. For a gentle introduction to some of the features and capabilities of ggplot2 I would point users to Chapter 3 of Hadley’s “R for Data Science” book available for free at https://r4ds.had.co.nz/data-visualisation.html that outlines the grammar of graphics as envisioned by its creator.

We achieve our objectives using ggplot2 using the following code

library(ggplot2)
ggplot(data=lad_dt, aes(x=home_gm_num, y=Attendance, group=Year, color=Tm))+
  geom_point()+
  geom_line()+
  facet_wrap(~Year)+
  xlab("Home Game #")+
  scale_x_continuous(breaks=seq(1, 81, 10))+
  ylab("Total Attendance")+
  stat_smooth(method="loess", se=FALSE, color="black", linetype=2)+
  scale_color_manual(name="Team", values="#005A9C")+
  geom_vline(data=subset(lad_dt, july_forth==1), aes(xintercept=home_gm_num), col="red", linetype=3)+
  theme_bw()+
  guides(color=FALSE)

From this graph we can see that the home attendance has been consistently been high over the past 5 years. There was also a notable increase in fans during later games in the 2017 season and high variability in the beginning of the season. We also see that the Dodgers have had a home July 4th game each season. During these games 2016 was the only case where attendance was relatively low while all other years attendance on July 4th was near the maximum.

`plotly`

As someone who has used R frequently for academic research and professional internships, I have defaulted to using ggplot2 almost religiously when generating graphics. It wasn’t until the beginning of this year when I was looking for options to create 3d visualizations in R that I stumbled upon plotly. The developers at plotly have written this code for several different software systems including Python, R, and JavaScript. My go-to resource as I have started learning about plotly has been their R reference materials available at https://plotly.com/r/ which contains examples and documentation to help guide beginners.

To compare the graphic packages, I will attempt to achieve the same objectives as described in the Motivating Example using the processed data. First, we need to generate the smoothed LOESS estimates manually for plotly as it does not have an analogous stat_smooth() built-in function.

library(plotly)
attnd_loess <- loess(Attendance~home_gm_num+Year, data=lad_dt)
lad_dt_loess <- data.frame(lad_dt[, c("home_gm_num", "Year", "Tm")],
                           pred_attnd=attnd_loess_pred)

We then graph the results using plotly syntax as follows

plot_ly(data=lad_dt, x=~home_gm_num, y=~Attendance,
             linetype=~Year, split=~Year,
             mode='lines+markers', type='scatter',
             hovertemplate=~paste('<i>Home Game #</i>: %{x}',
                            '<br><i>Attendance</i>: %{y:,.0f}',
                            '<br>Date:', Date,
                            '<br>Current Streak', Streak))%>%
  add_lines(x=~home_gm_num, y=predict(attnd_loess), split=~Year,
            name='LOESS',
            hovertemplate=~paste('<i>Home Game #</i>: %{x}',
                                '<br><i>Smoothed Attendance</i>: %{y:,.0f}',
                                '<br>Date:', Date,
                                '<br>Current Streak', Streak))%>%
  add_trace(data=subset(lad_dt, july_forth==1),
            x=~home_gm_num, y=~Attendance, split=~Year,
            mode="markers", type="scatter", name="July 4th",
            marker=list(color='rgba(16, 78, 139, 1)',
                        line=list(color='rgba(16, 78, 139, .8)')),
            hovertemplate=~paste('<i>Home Game #</i>: %{x}',
                                 '<br><i>Attendance</i>: %{y:,.0f}',
                                 '<br>Year:', Year,
                                 '<br>Current Streak', Streak))%>%
  layout(xaxis=list(title='Home Game #', dtick=10, tick0=1),
         yaxis=list(title='Attendance'))

We now have an .html figure showing the same attendance figures as we graphed in ggplot2. Initially this graphic appears more dense with all features and years displayed simultaneously. However, we can click, or double-click, on a specific year, LOESS trend, and July 4th point to systematically look at trends of interest. Alternatively, we can examine only LOESS curves or only July 4th attendance figures. This figure also allows users to manually edit the “hovertemplate” to control the text that is displayed as you move over points or trend lines. This feature allows us to add additional information such as Date and Current Streak to motivate additional exploration of the attendance figures.

Key Differences

Having used both packages to generate graphics that achieve our objective, below I have included a table highlighting some of the key differences between these two graphic packages in R.

	`ggplot2`	`plotly`
Filetype	.jpeg, .pdf, .png, etc.	.html
Interactive?	No	Yes
R Documentation	+++	+
Extensions	Extensive	Limited
Software Availability	R	R, Python, JavaScript
Sharing	Globally	Locally³

The main difference between graphics constructed using these packages within R is the corresponding filetype. ggplot2 creates static images with the extension of choice while plotly creates dynamic .html images which are web responsive. This difference allows for plotly to have interactive figures.

Another key difference is the available documentation. As ggplot2 has been developed for over 10 years it has an extremely large amount of documentation and help available when searching online, while plotly has more limited documentation as it has been gaining popularity in the R community more recently.

Further, in the example above we saw how simple extensions that are built into the ggplot2 framework can easily be incorporated into our figures. Adding trends lines was a simple addition of the stat_smooth() function while points had to be manually estimated in plotly.

As mentioned earlier the plotly package is available natively in multiple software languages including R, Python, and JavaScript, which can be especially helpful when moving between these languages.

Finally, sharing figures generated from these two packages is significantly different as static images created by ggplot2 are easily shared globally while .html files produced by plotly are available only locally by default.

Recommendation

In conclusion to this post, I would like to give users my two-cents about which package to choose when debating how to create R graphics. My default is to still to use ggplot2 as the static figures they create are easy to save and pass along to colleagues, incorporate into reports, and can be read uniformly. However, when the goal is to present 3d graphics or materials that allow the user to investigate within the graphic manually then I reach for plotly. An area where I tend to explore more between these graphics packages is generating spatial figures as plotly seems to offer more many more options for creating these maps as shown in https://plotly.com/r/#maps. For researchers working across multiple software systems, such as Python and R, I would also suggest focusing efforts on plotly as this will allow for consistency when moving between languages.

For those interested, the complete R code used to generate the images and data is downloadable here

For those interested the following is a link to the state of California COVID-19 dashboard to track cases here. ↩
Note that this data is pulled from baseball-reference.com team home pages. ↩
Since plotly creates .html files these can be included in local presentations only by default but can be hosted on web otherwise to share globally, e.g., on GitHub repository. ↩

From the Gym, to the Crag, to the Ground

2020-01-22T00:00:00-08:00

It might have taken slightly longer than a week, but I am here to officially kick off this blogging adventure. After my bold proclamation in last weeks welcoming post to try and write these once a week, I have to admit that I went into a slight panic. Immediately I regretted having uttered this statement, let alone placing it online where the evidence is eternal. I thought about removing it and scrapping this whole idea. What sort of topics would I feel passionate about enough to warrant long form discussion? How would I be able to present things in a compelling manner?

But despite this self-doubt I decided to stick to my gut, and hoped that magically an idea would fall into my lap. As chance would have it, such an idea did indeed fall into my lap, or more accurately I found the idea while lying on the ground staring up at rocks.

Looking up at Pixie Boulder in Joshua Tree National Park (JTNP).

One of my hobbies is rock climbing, specifically bouldering. My passion for climbing has waxed and waned depending on my time availability, climbing partners, and general motivation. Recently, I have become reinvigorated to spend more time practicing climbing.

I have only ever climbed in indoor gyms, and had always been curious to try and take these skills outdoors. However, whenever I thought about this idea I became anxious that I would make a fool of myself in front of other more experienced climbers. I have never been a particularly strong climber, maxing out at V3¹ boulders indoors at Hollywood Boulders, so I knew that I would have limited options in terms of boulders within my abilities outdoors. Luckily though, living in Los Angeles we have easy access to a mecca for rock climbing, Joshua Tree National Park. So this last week we packed up the car with climbing shoes, crash pad, and an optimistic outlook to try and tackle my first outdoor bouldering session.

Joshua Tree: Day 1

When we arrived in Joshua Tree we had to stop by Nomad Adventures to pick up some last minute necessities for climbing and camping. They were super friendly and helpful as we were shopping around, even providing a list of their recommended boulders to check out while we were in the park. In particular they mentioned an area of the park called Barker Dam, which had some of the most classic boulders including Gunsmoke. This area is also usually less crowded then some of the other nearby climbing locations such as Hidden Valley.

Barker Dam

With this suggestion we made it over to Barker Dam, and after a short approach hike I was able to officially try my first outdoor boulder, Gunsmoke. This is a V3 boulder, which is the level I am used to climbing indoors at the gym, and the employee at Nomad Adventures also mentioned that it was an easier V3 with most of the difficulty coming from the length of the boulder. There were a few other climbers working the Gunsmoke boulder when we arrived bringing out all of my original anxieties about failing in front of these strangers. I eventually hopped on the boulder and immediately found out that outdoor bouldering really was harder than the gym. Connecting even two moves felt like a Herculean effort, and this problem spanned over 100 feet!

After falling a couple of times I began to strike up a conversation with the other climber working the problem. We began to exchange beta², which was extremely helpful for me to gain someone else’s perspective on how to tackle this problem. It also assuaged some of my biggest concerns about bouldering outdoors with other climbers. I felt so much more comfortable failing without worrying about being judged.

I continued to work on Gunsmoke for a little while longer making incremental improvements with each attempt. We then moved over to another boulder in the Barker Dam area called The Chube, which was graded as a V2. Again, all of the moves on this problem felt incredibly hard but it was exciting nonetheless to work on it. After several unsuccessful attempts with many quizzical looks at the rock trying to find any semblance of a foothold, we eventually decided to call it a day and explore the rest of the park.

Although the first day of climbing did not result in any successful attempts, I left feeling proud that I was able to conquer the fear of failure. I also realized that the next day I would need to lower my sights and try working on problems that were much easier.

Snapshot of the 1st day of bouldering in Barker Dam at JTNP.

Joshua Tree: Day 2

We spent the night under the stars at the Indian Cove Campground which was truly an amazing experience, despite the chilly nighttime temperatures. After deciding to grab breakfast in town rather than making a sketchy Huevos Rancheros backpacking meal, we were once again back in the park looking to find a problem that I could finish. Browsing through a climbing app on my phone during breakfast I found an interesting problem that was one of the easiest grades and seemed like the best option for success. So we headed into the park to the Planet X area which had the problem that had caught my eye, Dragon Skin.

Planet X

Finding the Planet X boulders was slightly more challenging as there was no dedicated parking lot, but rather a paved pullout with room for about 3 cars. Since we had arrived directly after breakfast there was still parking available, and only one other group that was just unpacking there gear as we arrived. There are many boulders in this area and as we walked down the trail I assumed that we would end up at a different boulder than the other group, but to my surprise we both ended up starting on the exact same problem.

I could clearly tell that this group was more experienced as they made easy work of the Dragon Skin problem. Their warm-up problem would be my main problem, but I was happy to watch them complete it and know where each of the moves were supposed to be. After they had all dismounted, which ended up being a harder process with this boulder as the back side was difficult to descend, I walked up to the start holds and began to make my way up.

Climbing Dragon Skin at Planet X in JTNP.

I made it! Standing at the top of the boulder I was so happy! I had finally done my first outdoor boulder! It was an amazing feeling and one that I would not soon forget.

I tried another problem to the side of the Dragon Skin problem, only to be shut down immediately in a repeat of the experience at Barker Dam the previous day. But this failure rolled off my back and I basked in my earlier success.

We spent the rest of the day enjoying some of the other hikes and sights around Joshua Tree before heading back to LA.

Final Thoughts

Although my first time climbing outdoors was a humbling experience, it gave me motivation to continue to work hard and come back to top out these boulders. I am excited to improve and return to finish some of these boulders that felt impossible during this first session.

The Chube, V2, Barker Dam, JTNP.

Boulders are rated on the V-scale in the United States which generally ranges from VB to V15 with higher values corresponding to more difficult problems. For a full description of the grading system see the following page from 99Boulders: https://www.99boulders.com/bouldering-grades. ↩
Beta is a term used by climbers to describe the moves/steps needed to complete the boulder problem. ↩

Blogging?…Blogging!

2020-01-14T00:00:00-08:00

Hello! Thank you for visiting my website. As I began to explore the structure available, and learned more about how to actually format my material using Jekyll, I have toyed with the idea of starting a blog aspect as part of this content. This would be a collection of posts for my thoughts on stats, sports, life experiences, traveling, cooking, and who knows what else.

As any great idea starts there is much promise at the beginning and yet much unknown about execution. In this first month or so I will try to post weekly and then we will see where it goes from there. So buckle in for the journey and let’s see where this journey takes me.