Dan Ivanovich

Introducing the EA Dashboard!

Sat, 20 Jun 2020 00:00:00 +0000

What is the EA Dashboard?

Throughout this year, I have been working on a project for Energize Andover. I call it the “Energize Andover Dashboard,” or “EA Dashboard” for short. It is a web application that allows its users to customize a dashboard that displays various types of town data.

On the front end, the dashboard is composed of two essential components: the layout and the widgets. Widgets are unique graphs (or other forms of displaying data) for a specific subset of town data, such as a line graph of the total electricity used by Andover High School at the end of every day in the past week. Although I intend on creating many types of widgets, the majority will be similar to the previous example: interactive charts that update with new data. The layout, on the other hand is the arrangement and sizes of these widgets on the dashboard. Widgets have different aspect ratios (some are square, some are rectangular, etc.) and thus the layout will determine what widgets are displayed, and where on the dashboard those widgets lie.

Application Structure

I have done my best to design the application so that the server sustains the lightest loads possible, generating the graphs, layout, and more in the client’s browser and minimizing interactions between the clients and the server as much as possible.

On the server side, a Flask application hosts a web server. This server retrieves data from our BACnet API using the requests package, but this data-retrieval process will eventually be replaced by one that uses MySQL to query the server that Ayush is currently building.

This server takes the data it gathers and makes it accessible by the clients and also generates a configuration JSON file that details the requirements of each widget that may appear on the dashboard (such as the “aspect ratio” of the grid element’s size, what subset of the data it will be using, the chart type and option overrides, etc.). Thus, each client’s browser handles much of the computation and rendering of the data. Bulma, a CSS framework, is used to style the site and create much of the layout. FontAwesome icons are used throughout the site to make the UI more intuitive, and jQuery is used for most of the JavaScript functionality and interactivity. Gridster.js, along with lots of my own code that I wrote to correct its flaws (the library is no longer being developed), is used to create an intuitive layout configuration system that involves draggable, stretchable rectangles on a 4x4 grid. This layout is stored in the client’s browser-specific LocalStorage, to avoid the need for a separate database for this project, along with which widgets are to be populated in which of the grid’s boxes. This information is later pulled from the LocalStorage and is used to generate the containers and layout for the dashboard’s widgets on the main page. The JavaScript libraries are then used to pull the data, pull the requirements for each widget, and gather all the information needed to generate a widget. Finally, Plotly.js is used to generate clean, interactive charts for the widgets in each grid element. Overall, this structure was designed to reduce the load on the server and to be fully functional without a server-side database to store configuration information or the expiration of cookie-based data storage.

A diagram showing the structure of the EA Dashboard

The Present

So far, I have done much of the front-end work for the dashboard. I have built a full layout customization system by implementing gridster.js and writing a good amount of custom JavaScript to fix its flaws and mold it to my purposes. I have set up the integration with LocalStorage, designed a clean and intuitive interface, and have had the layout apply to the dashboard page. I have also configured and set up Plotly.js with some test widgets and experimented with various layouts to test the connection between layouts and widgets. I have also set up most of the data collection process for the server, but I have not made a connection between the server’s data and the front-end Plotly.js.

The Future

I will do my best to finish this project by the end of the summer. The next step for me is to complete the data-collecting code on the server side and run thorough testing. When that is complete, I will work on connecting the data from the backend to Plotly.js.

Scripting Python Package Maintenance

Sat, 18 Jan 2020 00:00:00 +0000

Python’s accessible and vast library of user-made packages is arguably one of its most powerful features. When developing a project with Python, it is often not long before you find your project dependent on a large list of libraries, and specific versions of those libraries, at that. Communicating such a long list of libraries to potential users of your own code base (or even a Python package of your own) thus could have been an arduous process, were it not for requirements.txt. By generating a requirements.txt file and distributing it with your project, you can make it far, far easier for other users or developers to install the correct versions of the packages required for your code to execute properly. In this guide, I’ll break down how to generate a requirements.txt and share two particularly useful scripts to automate working with them.

Generating `requirements.txt`

To generate a requirements.txt file for your project, open your project’s directory in the terminal. Ensure that you have installed the currently-required packages in your current Python environment (virtual environment, system-wide installation, or otherwise) using pip. Then, simply execute the following command:

pip freeze > requirements.txt

Here, you could replace requirements.txt with the path to the file you want to generate, but conventionally the generated file is named requirements.txt and placed within the root directory of the project. If you open the file, you should find something in this format:

cycler==0.10.0            # via matplotlib
kiwisolver==1.2.0         # via matplotlib
matplotlib==3.2.1
numpy==1.18.5
pandas==1.0.4
pyparsing==2.4.7          # via matplotlib
python-dateutil==2.8.1    # via matplotlib, pandas
pytz==2020.1              # via pandas
scipy==1.4.1              # via seaborn
seaborn==0.10.1
six==1.15.0               # via cycler, python-dateutil

Note how the specific versions of the packages are listed, but keep in mind that version numbers could also be listed using >=, <=, and other operators.

Installing All Required Packages

To install the specified versions of all required packages, once again open your project’s directory in the terminal and ensure that you have the proper Python environment opened. Then, run the following command:

pip install -r requirements.txt

This will install the proper version of all packages listed in requirements.txt, all in one command. Isn’t that convenient?

Updating Packages

Python packages are updated quite frequently. As anyone who frequently uses requirements.txt on GitHub can tell you, frequent updates are often required to fix security flaws in many widely-used packages. However, as the version of each package is explicitly defined in requirements.txt, updating is a multi-step process.

To update all of your project’s packages, open your project’s directory in the terminal and ensure that you are inside the proper Python environment. Then, execute the following command:

pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U

This will fetch and install any available updates to the project’s packeges, along with any new package dependencies. However, you may notice that requirements.txt remained untouched. Double check that your code still executes properly and, if it does, regenerate the requirements file by repeating the command from above:

pip freeze > requirements.txt

You should now be able to see the new version numbers reflected in the file.

Automating These Processes

You can use the scripts that I have published to this GitHub repository to simplify the processes of installing packages from requirements.txt and updating your packages. The following shell scripts are included:

install_packages.sh

#! /bin/bash

pip install -r requirements.txt

Usage: ./install_packages.sh - output of installation from requirements.txt will be shown. update_all_packages.sh

#! /bin/bash
pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1  | xargs -n1 pip install -U

# Prompt
read -p "Would you like to update requirements.txt? (Y/N)" -n 1 -r
echo # move to new line
if [[ ! $REPLY =~ ^[Yy]$ ]]
then
    exit 1
fi

pip freeze > requirements.txt

Usage: ./update_all_packages.sh - enter Y or N after the updating has been completed if you would like to update the versions of your requirements.txt file.

Implementing these scripts into your workflow will greatly simplify your Python package updates and installs. I hope you find them as useful as I do!

OpenCV and AHS Heatmap | Autonomous Room Detection in Floor Plans With OpenCV (Part 3)

Sat, 28 Dec 2019 00:00:00 +0000

This is part 3 of a series of blog posts about the computer vision behind the AHS Heatmap. Read part one here; Read part two here

In this final part of this series, I will explain how OpenCV is used in combination with the files and data obtained from the steps described in parts one and two in order to generate the final heatmap.

Processing the Generated PNG With OpenCV

By analyzing the filename of the original SVG, the Python was able to detect the floor of the school that it is currently generating a map for. The first digit of room numbers in AHS indicate their floor (for example, room 203 is on the second floor). Thus, the program can use the floor number to filter the pandas dataframe associating the addresses of AHS’s sensors and their room numbers. The program then iterates through this cut-down dataset and compares it with the dataset of text boxes and coordinates obtained in part two. If a sensor’s room is not labeled on the floor plan (that is to say, if a room number from the sensor dataframe is not found in the PDF dataframe), the room is skipped. If not, the temperature or CO2 level of the sensor is pulled, a proper color between blue, green, and red is chosen, and the following process is performed to detect the shape of the room:

Detecting Room Shapes

To demonstrate this process, let us say that the program is currently looking for room 217 in the following floor plan (listed as “level 3” because the field house, a part of the first floor, was separately considered as “level 1” by the floor plan’s creators):

First the PNG of the floor plan is loaded into OpenCV. Then a series of image processing and computer vision techniques are employed to detect the shape of the room, as follows:

First, the image is binarized. This means that it is converted from standard 3-channel RGB color codes, with 3 values between 0 and 255 defining a color, into an image with 1-channel color codes, dictated by a single value between 0 and 255. This results in what is more commonly known as a “gray-scale” effect, as 0 represents black, 255 represents white, and everything else determining a certain shade of intermediate gray. Binarization makes the detection of objects (in our case, rectangles) much easier in the future.

Next, the coordinates of the room’s label’s text box, filtered out from the dataframe obtained by the PDF extraction process shown in part 2, are used. As it is known that the room labels, however they are placed, will be within the room, we can begin a flood fill algorithm (think of the paint bucket feature from MS Paint) from one pixel outside the border of the textbox. This will now be guaranteed to result in the room surrounding the text being filled with a neutral gray, exactly between white and black to minimize its chances of being grouped in with the black and white pixels around it when the image is processed further.

The flood filled image is now binarized again, converting the intermediate gray that filled the room with to white and all other colors to black (a 0-tolerance binarization). As you can see, this has isolated the shape of the room surrounding the text box and has made it much easier to spot the room in the image. However, the shape of the room number’s text still remains and posed a problem to future steps.

Thus, using the coordinates, width, and height of the text box that were extracted from the PDF, a white rectangle is drawn over the former location of the text box using OpenCV. As you can see, this process fills in the hole and does so consistently without impacting the rest of the shape.

Now, the shape of the room has been isolated very well. It is now possible to quickly detect the outline of the shape, known as the “contour” in OpenCV:

Here, I have displayed the contour (blue) that my OpenCV-based computer vision code detected over the original floor plan.

Although I could have used the countours generated by this process to describe the shape of my rooms on the heatmap, I decided to opt for a consistent, rectangular style that matches the style of the rooms in the building and the floor plan. Thus, the bounding rectangle of the countour is calculated.

Here, the bounding rectangle (green) of the contour (still blue) is shown.

This bounding box is returned by the function that performs all of this analysis as the detected shape of the room.

The Code

Here is the code that performs the OpenCV component of the process described (located in openCV_tools.py):

def get_room_max_contour(room_text_coords):
    bottom_left = (room_text_coords[0], room_text_coords[1])
    top_right = (room_text_coords[2], room_text_coords[3])

    background_color = (255, 255, 255)  # Runs much, much, much faster if we don't try to detect the dominant colors
    lower_bound = np.array(np.clip([value - 5 for value in background_color], 0, 255))
    upper_bound = np.array(np.clip([value + 5 for value in background_color], 0, 255))
    binary = cv2.inRange(img, lower_bound, upper_bound)

    # cv2.imshow('binary 1', binary)
    # cv2.waitKey(0)
    # cv2.destroyAllWindows()

    replacement_color = 128

    cv2.floodFill(binary, None, bottom_left, replacement_color)
    binary = cv2.inRange(binary, replacement_color, replacement_color)

    # cv2.imshow('binary 2', binary)
    # cv2.waitKey(0)
    # cv2.destroyAllWindows()

    cv2.rectangle(binary, bottom_left, top_right, 255, -1)  # Fill in the room-number-shaped hole for shape recognition

    # cv2.imshow('binary 3', binary)
    # cv2.waitKey(0)
    # cv2.destroyAllWindows()

    ret, thresh = cv2.threshold(binary, 127, 255, 0)
    contours, hierarchy = cv2.findContours(thresh, 1, 2)
    contour = max(contours, key=cv2.contourArea) # grab largest contours

    return contour

Drawing Rooms on the SVG

The svgwrite package, taking advantage of SVG’s “programmablity,” is utilized to add custom SVG rectangles that take the size of the contours’ bounding boxes.

The Code

contour = get_room_contour(room, self.media_box, self.text_and_coords)

if contour is not None:
    path_text = "M"
    for i in range(len(contour)):
        x, y = contour[i][0]
        # Invert y axis (OpenCV measures in the opposite direction)
        y = self.media_box[3] - y
        svg_coords = get_svg_coords((x, y), self.view_box, self.media_box)
        path_text += '{0} {1} '.format(svg_coords[0], svg_coords[1])

    dwg = svgwrite.Drawing(temp_path)
    dwg.add(dwg.path(d=path_text, fill=color_hex_code, opacity=opacity, id="room-rect-{0}".format(room), onmouseover="showRoomData(this, {0}, '{1}')".format(value, units), onmouseout="hideRoomData(this)"))

Notice that I add custom onmouseover and onmouseout properties to the SVG rectangle. This was my sneaky way of simplifying the creation of the interactivity behind the heatmap (when you hover over a room, a popup appears with the room number and an exact reading of its temperature or CO2 level). showRoomData() and hideRoomData() are JavaScript functions that manage this functionality, and onmouseover and onmouseout make detecting when a certain room is hovered over far easier.

The Completed `fill_room` Function

The complete code for the fill_room() function is shown below. Notice that it includes code to manage duplicating, saving, and merging the generated SVG rectangles (each saved into its own file) into a _filled_rooms_temperature.svg or _filled_rooms_co2.svg file, which are the files displayed by the web interface.

def fill_room(self, room, color_hex_code, opacity, value, units, is_temperature):
    temp_path = self.svg_path[0:-4] + '_temp_rect.svg'
    output_path = self.svg_path[0:-4] + '_filled_rooms_{0}.svg'.format('temperature' if is_temperature else 'co2')

    if not os.path.exists(output_path):
        shutil.copy(self.svg_path, output_path)

    contour = get_room_contour(room, self.media_box, self.text_and_coords)

    if contour is not None:
        path_text = "M"
        for i in range(len(contour)):
            x, y = contour[i][0]
            # Invert y axis (OpenCV measures in the opposite direction)
            y = self.media_box[3] - y
            svg_coords = get_svg_coords((x, y), self.view_box, self.media_box)
            path_text += '{0} {1} '.format(svg_coords[0], svg_coords[1])

        dwg = svgwrite.Drawing(temp_path)
        dwg.add(dwg.path(d=path_text, fill=color_hex_code, opacity=opacity, id="room-rect-{0}".format(room), onmouseover="showRoomData(this, {0}, '{1}')".format(value, units), onmouseout="hideRoomData(this)"))

        dwg.save()  # Save the path to a temporary file

        # Merge the files
        floor_plan = st.fromfile(output_path)
        second_svg = st.fromfile(temp_path)
        floor_plan.append(second_svg)
        floor_plan.save(output_path)
        os.remove(temp_path)

Thus, I could now programmatically fill in the room number of my choice with the color of my choice. The output of calling fill_room() 4 times with dummy colors of my choosing is shown below:

Putting it All Together

By connecting the completed fill_room() function with live sensor data and code that determines the proper color for a room based off of its temperature or CO2 concentration, the heatmaps seen in the final product are generated.

The driver code for filling an entire floor’s heatmap:

def fill_from_data(self, data, is_temperature_value):
    value = data['temperature'] if is_temperature_value else data['co2']
    units = data['temperature units'] if is_temperature_value else data['co2 units']

    if not math.isnan(value) and units is not None and units != '':
        color = self.get_value_color(value, is_temperature_value)
        self.fill_room(data['room'], color, FILLED_PATH_OPACITY, value, units, is_temperature_value)

Ending Notes

Thank you very much for reading this series! This specific aspect of the AHS Heatmap took me the most time to design and develop and involved a great deal of learning on my part. I hope that this explanation may serve others well and save them time in the future. Best of luck with your OpenCV adventures, and Happy New Year!

OpenCV and AHS Heatmap | Working with SVGs (Part 2)

Sat, 12 Oct 2019 00:00:00 +0000

This is part 2 of a series of blog posts about the computer vision behind the AHS Heatmap. Read part one here

When I was experimenting with SVGs and OpenCV, I realized that I would have to convert my floor plans to PNGs to perform computer-vision-based analysis to detect rooms. This CV-based room-detection algorithm was not something I realized early into the project - it was maybe my third or fourth idea, but it certainly was my best and most-thought-out one.

Problem 1: Extracting Room Number Text Box Coordinates

In order to detect the shape of the room around a room number, I would first have to determine where that room number was in the converted PNG. PNGs, unlike SVGs, do not have particular “selectable” text elements - PNGs are simply pixels that make no distinction between lines, shapes, text, or anything else. The SVG floor plan I received, however, stored its room numbers in a rather unapproachable manner, placing them in a seemingly-random order throughout the file on a character-by-character basis. This would make it very difficult to distinguish programmatically which character corresponded to which digit of which room number in which location, so I found myself faced with the challenge of devising a separate method of determining which rooms were on a map, then determining where on the map their room numbers were.

I decided to brainstorm other types of files that distinguish the text contained within them. PDFs came to mind, and luckily Inkscape’s command-line tool can convert SVGs not only to PNGs, but also to PDFs!

I will not go into the details of the PDF text-box-coordinate-extracting algorithm in this post, as there is already a post on my blog dedicated to it. However, I will explain how I used Inkscape to perform the conversions: I used Python’s built-in os package to call os.system(), which executes commands. My SVG conversion code is below:

from pathlib import Path
from sys import platform
import os


def svg_to_png(svg_path, png_path, dpi):
    # Delete the file if it exists, as inkscape won't overwrite
    remove_file(png_path)

    # run the command to convert the svg, adding '> /dev/null' on the end to silence the output by storing it into null
    options = '--without-gui --export-area-page --export-background="#ffffff"'

    if platform == "linux" or platform == "linux2":
        # run command to convert the svg, adding '> /dev/null' on the end to silence the output by storing it into null
        os.system('inkscape %s "%s" --export-dpi=%s --export-png="%s"  > /dev/null' % (options, svg_path, dpi, png_path))
    else:
        os.system(
            'inkscape %s "%s" --export-dpi=%s --export-png="%s"' % (options, svg_path, dpi, png_path))

    wait_for_creation(png_path)


def svg_to_pdf(svg_path, pdf_path):
    # Delete the file if it exists, as inkscape won't overwrite
    remove_file(pdf_path)

    options = '--without-gui --export-area-page'

    if platform == "linux" or platform == "linux2":
        # run command to convert the svg, adding '> /dev/null' on the end to silence the output by storing it into null
        os.system('inkscape %s "%s" --export-pdf="%s" > /dev/null' % (options, svg_path, pdf_path))
    else:
        os.system('inkscape %s "%s" --export-pdf="%s"' % (options, svg_path, pdf_path))

    wait_for_creation(pdf_path)


def remove_file(path):
    try:
        os.remove(path)
    except OSError:
        pass


def wait_for_creation(path):
    while not Path(path).is_file():
        continue  # Wait until it's completed

Something worth noticing is my use of the --export-dpi flag in the SVG to PNG conversion command. By specifying the proper DPI, I could result in PDFs and PNGs of matching pixel dimensions. Although not necessary, this was very useful to using OpenCV to detect rooms, as I did not have to scale the coordinates extracted from the PDF for the location of the text boxes to match up across files. Thus, I could simply use the coordinates of the PDF as coordinates on the PNG, and the extracted text box positions were guaranteed to match up with those of the image OpenCV was analyzing (this was essential to the floodfill algorithm described in part 3).

Problem 2: SVG to PNG Coordinates

Although both SVGs and PNGs can use pixel coordinate systems, the coordinates of a point on a floor plan’s SVG would be nowhere near the coordinates of the same point on the converted PNG.

Here, "on screen" means in relation to the converted PNG.

This was a serious problem because it meant that any room coordinates detected by OpenCV would be unusable when overlaying the colored rectangles on the filled SVG (see part 3).

Eventually, I discovered two important things:

When creating SVG elements, the y axes of PNGs and SVGs start at different points (one starts from the bottom of the screen, the other starts from the top of the screen)
One can calculate what SVG coordinate a particular PNG coordinate corresponds to by forming a ratio that uses the SVG’s view box dimensions.

By combining these two discoveries, I wrote the following Python code that converts PNG coordinates to SVG coordinates:

def get_svg_coords(png_coords, svg_view_box, pdf_media_box):
    x_coord = png_coords[0] * svg_view_box[2] / pdf_media_box[2]
    y_coord = svg_view_box[3] - png_coords[1] * svg_view_box[3] / pdf_media_box[3]
    coords = [x_coord, y_coord]
    return tuple(coords)

OpenCV and AHS Heatmap | Automatically Finding Rooms in an Image (Part 1)

Sat, 14 Sep 2019 00:00:00 +0000

This is part 1 of a series of blog posts about the computer vision behind the AHS Heatmap.

As I was designing the AHS Heatmap, one of my main goals was to create software that was not bound to any specific floor map. That is to say, I did not want the software to be designed for a specific set of SVG floor plans or rely on a manually-labeled list of room coordinates to know which rooms were on the current floor, which color to fill them with, and where to place the colored rectangle. By designing it in this way, the program would have the potential to be applied to all of Andover’s public schools, not solely Andover High School. Implementing this, however, proved to be quite complicated.

Throughout this post, I will introduce the tools behind this process and the file format that will be worked with.

Input Format

AHS Heatmap takes in a folder of SVG floor maps. These equate to enormous vector images (those of AHS are roughly 11,000 by 8,500 pixels). SVGs are quite difficult to work with programmatically - although their status as essentially “programmed” images makes them great to work with directly or within websites, the vast difference between the content of an SVG and a standard image format, such as a PNG or a JPG, makes SVGs unusable by many image-processing tools and libraries.

A sample floor plan from AHS is shown below:

OpenCV

OpenCV is a massively popular computer vision library with implementations in many languages, including Python. It is free-to-use and very well-documented, so I decided to employ it for the heatmap’s computer vision.

Converting SVGs to PNGs

The heatmap depends on Inkscape a free vector graphic design tool. Inkscape uniquely has a command-line interface that allows the usage of Inkscape to convert SVGs to PNG images (alongside many other formats) with a great degree of flexibility. This worked out very well for the project the ability to utilize Inkscape’s conversion functionality without GUI meant that the heatmap could be deployed to SSH-only machines, such as the Google Compute Engine virtual machine upon which it is currently hosted.

Converting SVGs to PDFs

Although the reasoning behind my need for this functionality may not be clear just yet, it will be explained in part 2 of this series. It turns out that Inkscape’s CLI is also capable of performing this kind of conversion.

AHS Heatmap Presentation Recap

Thu, 08 Aug 2019 00:00:00 +0000

Earlier today, I presented to town officials and leaders in energy alongside fellow members of Energize Andover. Specifically, I presented a year-long project of mine, the AHS Heatmap. I gave a brief overview of the process of creating the heatmap, one of its essential underlying algorithms, and shared a demo. Afterwards, I answered questions and discussed the future of the project and what it could be used for. As the heatmap was designed flexibly, there is the potential for this codebase to be applied to all of Andover Public Schools, not solely Andover High School. This, however, will require the installation of many more sensors into other school buildings and the expansion of our BACnet API (Andover High School has far more accessible sensors inside its building than any other public school). This is the direction that I would love to take this project, but, as of now, it will continue to be employed for AHS alone. Overall, this presentation was a great experience and is yet another motivator to continue my work for Energize Andover. I am looking forward to completing projects of a similar scale after concluding this one.

This is the slideshow that I presented:

Building Energy Boot Camp 2019 - Day 6

Mon, 05 Aug 2019 14:00:00 +0000

Today, I spend the day working on the AHS Heatmap, improving its ability to fill rooms, reconfiguring the VM that hosts it, and installing an SSL Certificate for HTTPS support. I successfully managed to make the program draw the room’s contours (the outlines of the rooms) instead of the bounding box (the smallest rectangle containing the contour). The following is a demonstration of the difference between the terms:

This is the contour of the room:

It is the outline of the room. It can be irregularly shaped. It is a more precise representation of the room, but is much harder to work with for calculations.

The following is the bounding box of said contour:

It is the smallest rectangular shape that contains the contour. It is a less precise approximation of the room, but its rectangular shape makes it much easier to work with in calculations.

What Changed?

My code used to exclusively use the bounding box to both color rooms and perform client-side calculations with JavaScript (to move the information box that displays the room number and sensor reading). This meant that the application would not work with irregularly shaped rooms (rooms that were not rectangular) without drawing an overly-large rectangle. Now, it uses both the bounding box and the contour in combination. The contour is used to color the rooms, meaning the application can now work with rooms of any shape. The bounding box, however, still is used to calculate the desired position of the information box, as SVG paths are very difficult to work with in these calculations without such approximations. Although this very slightly increased the time required to generate a map (as contours are more difficult to draw), it greatly increased the accuracy of the map, allowing it to avoid overlapping colored areas and to fill the non-rectangular rooms in the building, such as the library (room 270 in the image below).

A map of the second floor of the building. Notice the non-rectangular shapes used to color the rooms.

SSL Certificate and HTTPS

After deploying my updates to the virtual machine (VM) that hosts the map on heatmap.energizeandover.com, I began to look into ways to secure my site and protect its users with HTTPS security. To do this, I needed to secure an SSL License and install it into Nginx, the web server the VM uses. I ended up securing a free license from Let’s Encrypt, a non-profit certificate authority providing. I did this by using certbot, a command line tool that can automatically secure, renew, and install an SSL License to a variety of web servers, Nginx included. Finally, I tested the renewal of the license and eventually scheduled a daily check for an expiring license, so certbot will automatically renew the license for me 30 days before it expires. I tested my license, and secured an A+ rating from a widely-used SSL testing site!

SSL report by Qualys SSL Labs.

At the end of the day, the heatmap was in a much better state than it was before the bootcamp meeting.

Building Energy Boot Camp 2019 - Day 5

Fri, 02 Aug 2019 14:00:00 +0000

Today, I returned my focus to the matplotlib project we were assigned on Day 2 of the boot camp. I made tremendous progress, and nearly finished the entire project. I successfully implemented a multi-threaded data-retrieval system. Most notably, the threading class is completely and safely killable, something I have found terribly difficult to achieve with Python in the past. See Killable Threads - Background and Killable Threads - The Code for a detailed explanation of how I achieved this. I also learned about many of matplotlib’s features that I was previously unaware of, including the abilities to set custom tick marks along the x and y axes, scroll the graph without deleting off-screen points, and bind function to various user-inputted events in the opened window. By using these features, I was able to add much more functionality to the graph, such as panning and zooming with the mouse wheel. Throughout the rest of this post, I will describe in detail the process behind these additions.

Killable Threads - Background

In Python, there is no way to safely “kill” an instance of the threading.Thread class. That is to say, there is no way to stop the Thread’s execution of its target function without running the risk of leaving open critical resources that must be closed. You could always terminate the entire program by using sys.exit(), but, for the aforementioned reason, this is a very dangerous practice, particularly so in my case, as I sought to save all the data collected by the threaded function to a CSV file (a tabular format). However, I did not always want to wait for said function to finish whenever a user requested the termination of the program, as the process of requesting and storing sensor data typically takes anywhere between 10 and 15 seconds. When this long delay is added to the time required to output all stored data to a file, I was facing a shutdown time long enough to bore users into forcibly killing my program as a whole, which both be risky for their computers and my program’s chances of outputting its data to a file. With this in mind, I set about modifying my threaded function, continuous_update, to make it quickly and safely terminable. Here’s how I achieved this goal:

Before I explain the implementation of my killable system, I will first explain the general structure of the Line Graph portion of my boot camp code (which is available on GitHub!) and the relationship between its files. My code currently follows this file structure:

Building-Energy-Bootcamp-2019
├── BuildingEnergyAPI       # Python package containing the API used to get data from our sensors
├── csv                     # Contains csv files with the data needed to request from a sensor
├── LineGraph               # Python package containing files specific to the Line Graph
│   ├── app.py              # The main file that runs the Line Graph
│   └── update_thread.py    # The file containing the code defining the DataUpdateThread class
├── pylive                  # Python package my modified version of pylive, used to plot data 
│   └── pylive.py           # The file containing the code related to initializing and updating the graph
└── ...                     # etc. (Other directories and files unrelated to the Line Graph)

To run the Line Graph, the file LineGraph/app.py is executed from the Building-Energy-Bootcamp-2019 (root) directory. This file imports DataUpdateThread, the class that asynchronously manages the data-retrieval process, from LineGraph/update_thread.py, and instantiates a copy of the class, providing the pandas DataFrame (in short, a table of data) used by the entire project to the class. It also manages the initialization and updates of the line graph, using methods from the file pylive/pylive.py. This entire process and relationships can be summarized by the following 3 code snippets:

from LineGraph.update_thread import DataUpdateThread
from pylive.pylive import live_plotter_init, live_plotter_update, has_been_closed

...

update_thread = DataUpdateThread(data_df, csv_path, output_path)
update_thread.begin_update_thread(update_interval)

...

live_plotter_init(data_df, ...)

Although the vast majority of the code in app.py has been omitted in these snippets, enough may be derived about the relationship between app.py, update_thread.py, and pylive.py. The important thing to notice about this code is that the same DataFrame instance is being passed to pylive.py’s methods and to the DataUpdateThread’s constructor. Following the rules of object oriented design and references to objects, this means that both the update thread and the plotter are referencing the same table, with the same data. Thus, when the DataUpdateThread instance requests new data and appends it to the end of the table, the plotter’s reference to the data table sees the same changes. To maintain this connection and communication, both update_thread.py and pylive.py are entirely dependent on app.py. The two files have no direct means of communication, but rather must rely on methods in app.py to relay any important information, such as the user’s request to close the graph, initiated by their press of the X button. Keep this in mind as I explain how I made my DataUpdateThreads terminable.

Killable Threads - The Code

As the update_thread.py and pylive.py files are completely blind to each other, I designed both to be centered around management by app.py. The first changes that I made were to the DataUpdateThread class, starting with its constructor:

class DataUpdateThread:
    def __init__(self, df, csv_path, output_path):
        self.data_df = df
        self.csv_path = csv_path
        self.output_path = output_path
        self.thread = None
        self.stopped = False
        self.fully_stopped = False

Note the two fields stopped, and fully stopped. They are both booleans, and default to False. stopped represents whether or not a call to stop the thread’s execution has been made, and fully_stopped is set to True when the thread has been stopped and the information in data_df has been saved to output_path, signifying that a completely safe shutdown is ready. I also defined a stop(self) method and a is_fully_stopped(self) method, to allow app.py to remotely trigger the shutdown process and to provide an accessor that informs app.py when it can safely stop.

Next, I turned to continuous_update, a method that repeatedly calls get_new_values (which requests new data and appends it as a row at the bottom of the data_df shared with pylive.py) until a stop is requested.

def continuous_update(self, repeat_interval):
    while True:
        if self.stopped or has_been_closed():
            break
        self.get_new_values()
        if self.stopped or has_been_closed():
            break
        time.sleep(repeat_interval)

    self.save_data()
    self.fully_stopped = True

Before and after it begins the process of requesting new data, this method first checks if a stop has been requested or if the pylive window has been closed (using has_been_closed()). If either of these conditions of met, it exits its infinite loop and saves its data before flagging the thread as fully_stopped.

Finally, to allow the interruption of the 10-15 second long method get_new_values, I used a similar strategy:

def get_new_values(self):
    df = pd.read_csv(self.csv_path)
    retrieved_data = []
    
    for index, row in df.iterrows():
        value, units = get_value(row['Facility'], row['Meter'], live=True)
    
        value = float(value) if isinstance(value, numbers.Number) else ''
        retrieved_data.append(round(value, 2))
    
        if self.stopped or has_been_closed():
    
            return
    retrieved_data.append(dt.datetime.now())
    
    # Add to the end of the DF
    self.data_df.loc[len(self.data_df)] = retrieved_data

After receiving data from a sensor, the method checks to see if a stop has been requested by either app.py or the user. If that is the case, it exits the function without bothering to request data from the remaining sensors, or appending the incomplete data to the end of the dataframe. This greatly reduced the shutdown times, as users now would have to wait, in the worst case scenario, for only one piece of data to be received instead of a piece of data from each sensor.

Lastly, I defined the proper_shutdown method in app.py, which always ensures a safe shutdown of the program and thread.

def proper_shutdown():
    print("Shutting down, please wait...")
    update_thread.stop()
    while not update_thread.is_fully_stopped():
        time.sleep(0.5)
    print("Fully shut down")
    sys.exit()

This method takes advantage of my new additions to the DataUpdateThread class to ensure the thread has safely stopped and has saved its data before ending the program. By utilizing the atexit library, I made sure that proper_shutdown() is run before the program terminates for any reason. To communicate a shutdown to app.py when the graph is closed, I used matplotlib’s event bindings to call a method that initiates proper_shutdown, window_closed:

close_bind = fig.canvas.mpl_connect('close_event', window_closed)

And with that, my killable threads were done!

Graph Interactivity Improvements

Today, I also modified pylive.py to make my graph more complete and interactive. First, I made the view scroll with the graph by setting the limits of both the x and y axes to always fit the 10 most recent points of every line, which allowed me to plot the entirety of the data_df DataFrame. To make off-screen data visible, I programmed in the ability to scroll with the mouse wheel and planned on utilizing matplotlib’s prebuilt panning system. However, I quickly realized that the bound-setting code I had added for scrolling would always send the panned graph back to its original position. I fixed this by binding new methods to mouse events and adding a new boolean variable, panning_allowed, that stops the code from adjusting the limits of the visible portion of the x and y axes while the user’s mouse is inside the graph. This made my graph very interactive and easy-to-use.

Screenshots

New Line Graph (Default, Auto-Scrolling View)

New Line Graph (Panning and Zooming Demonstration)

Code

app.py

import os
import sys
import time
import atexit
import numbers
import pandas as pd
import datetime as dt
from LineGraph.update_thread import DataUpdateThread
from BuildingEnergyAPI.building_data_requests_external import get_value
from pylive.pylive import live_plotter_init, live_plotter_update, has_been_closed

# Open dataframe
csv_path = os.path.join('csv', 'ahs_power.csv')
output_path = os.path.join('LineGraph', 'line_graph_out.csv')
df = pd.read_csv(csv_path)


def get_readings():
    values = []

    for row_num in range(num_lines):
        value, units = get_value(df.loc[row_num]['Facility'], df.loc[row_num]['Meter'], live=True)
        value = float(value) if isinstance(value, numbers.Number) else ''
        values.append(round(value, 2))

    return values


# Pull Labels
labels = [row['Label'] for index, row in df.iterrows()]
columns = labels + ['Time']

# How many seconds between updates
update_interval = 2

# How many lines to use (number of rows in df)
num_lines = len(df.index)

# Setup data storing dataframe
initial_values = get_readings()
initial_values.append(dt.datetime.now())
data_df = pd.DataFrame([initial_values], columns=columns)
data_df['Time'] = pd.to_datetime(data_df['Time'])  # Index by datetime

update_thread = DataUpdateThread(data_df, csv_path, output_path)
update_thread.begin_update_thread(update_interval)


def proper_shutdown():
    print("Shutting down, please wait...")
    update_thread.stop()
    while not update_thread.is_fully_stopped():
        time.sleep(0.5)
    print("Fully shut down")
    sys.exit()


try:
    # Properly shutdown when sys.exit() is called
    atexit.register(proper_shutdown)

    lines = [None for line_num in range(num_lines)]

    color_options = ['b', 'g', 'r', 'c', 'm', 'y', 'k', 'w']

    # Cycle through every color in the order shown in color_options
    formats = ['{0}-o'.format(color_options[format_num % len(color_options)]) for format_num in range(num_lines)]

    live_plotter_init(data_df, lines, formats, [item.replace('(kW)', '') for item in labels],
                      title="AHS Power Usage (Realtime)", xlabel='Elapsed Time (Hours:Minutes:Seconds)',
                      ylabel='Power (kW)')

    while True:
        live_plotter_update(data_df, lines)
        if has_been_closed():
            sys.exit()

except KeyboardInterrupt:
    proper_shutdown()

update_thread.py

import threading
import numbers
import time
import os
import sys
import pandas as pd
import datetime as dt
from pylive.pylive import has_been_closed
from BuildingEnergyAPI.building_data_requests_external import get_value


class DataUpdateThread:
    def __init__(self, df, csv_path, output_path):
        self.data_df = df
        self.csv_path = csv_path
        self.output_path = output_path
        self.thread = None
        self.stopped = False
        self.fully_stopped = False

    def get_new_values(self):
        df = pd.read_csv(self.csv_path)

        retrieved_data = []

        for index, row in df.iterrows():
            value, units = get_value(row['Facility'], row['Meter'], live=True)

            value = float(value) if isinstance(value, numbers.Number) else ''
            retrieved_data.append(round(value, 2))

            if self.stopped or has_been_closed():
                return

        retrieved_data.append(dt.datetime.now())

        # Add to the end of the DF
        self.data_df.loc[len(self.data_df)] = retrieved_data

    def continuous_update(self, repeat_interval):
        while True:
            if self.stopped or has_been_closed():
                break
            self.get_new_values()
            if self.stopped or has_been_closed():
                break
            time.sleep(repeat_interval)

        self.save_data()
        self.fully_stopped = True

    def begin_update_thread(self, repeat_interval):
        self.thread = threading.Thread(target=self.continuous_update, args=(repeat_interval,))
        self.thread.start()

    def save_data(self):
        # if file does not exist write to file with header
        if not os.path.isfile(self.output_path):
            self.data_df.to_csv(self.output_path, header=True)
        else:  # else it exists so append without writing the header
            self.data_df.to_csv(self.output_path, mode='a', header=False)

    def stop(self):
        self.stopped = True

    def is_fully_stopped(self):
        return self.fully_stopped

pylive.py

import sys
import numpy as np
import datetime as dt
import matplotlib
import matplotlib.pyplot as plt

# use ggplot style for more sophisticated visuals
plt.style.use('ggplot')

is_closed = False
panning_allowed = False


def enable_panning(event):
    global panning_allowed
    panning_allowed = True


def disable_panning(event):
    global panning_allowed
    panning_allowed = False


def has_been_closed():
    return is_closed


def window_closed(event):
    global is_closed
    is_closed = True


# Converts a numpy.datetime64 to a datetime.datetime object by converting dt64 to UTC time (for later use)
def datetime64_to_datetime(dt64):
    return dt.datetime.utcfromtimestamp((dt64 - np.datetime64('1970-01-01T00:00:00Z')) / np.timedelta64(1, 's'))


# Gets a list of x (time) and y (sensor reading) coordinates for the index-th column of data_df
# Also returns the labels for the x ticks (strings in HH:MM:SS) format
def get_coordinate_lists(data_df, index):
    time_list = data_df['Time'].tolist()
    value_list = data_df.iloc[:, index].tolist()

    time_list = [datetime64_to_datetime(time) for time in time_list]

    # Convert time_list to timedeltas, representing the time between each element of time_list and time_list[0]
    time_list = list(map(lambda time: time - time_list[0], time_list))

    # Convert the timedeltas to seconds
    time_list_seconds = list(map(lambda timedelta: round(timedelta.total_seconds()), time_list))

    # Convert the timedeltas to HH:MM:SS format
    time_list_strings = list(map(lambda timedelta: "%.2d:%.2d:%.2d" % (
        int(timedelta.seconds / 3600), (timedelta.seconds // 60) % 60, timedelta.seconds % 60), time_list))

    return time_list_seconds, value_list, time_list_strings


def live_plotter_init(data_df, lines, formats, labels, xlabel='X Label', ylabel='Y Label', title='Title'):
    plt.ion()
    fig = plt.figure(figsize=(13, 9))
    ax = fig.add_subplot(111)

    # Set window title
    gcf = plt.gcf()
    gcf.canvas.set_window_title(title)

    # Event bindings
    close_bind = fig.canvas.mpl_connect('close_event', window_closed)
    enter_bind = fig.canvas.mpl_connect('axes_enter_event', enable_panning)
    exit_bind = fig.canvas.mpl_connect('axes_leave_event', disable_panning)

    # Setup mouse wheel zooming
    def zoom_factory(ax, base_scale=2.):
        def zoom_fun(event):
            # get the current x and y limits
            cur_xlim = ax.get_xlim()
            cur_ylim = ax.get_ylim()
            cur_xrange = (cur_xlim[1] - cur_xlim[0]) * .5
            cur_yrange = (cur_ylim[1] - cur_ylim[0]) * .5
            xdata = event.xdata  # get event x location
            ydata = event.ydata  # get event y location
            if event.button == 'up':
                # deal with zoom in
                scale_factor = 1 / base_scale
            elif event.button == 'down':
                # deal with zoom out
                scale_factor = base_scale
            else:
                # deal with something that should never happen
                scale_factor = 1
                print
                event.button
            # set new limits
            ax.set_xlim([xdata - cur_xrange * scale_factor,
                         xdata + cur_xrange * scale_factor])
            ax.set_ylim([ydata - cur_yrange * scale_factor,
                         ydata + cur_yrange * scale_factor])
            plt.draw()  # force re-draw

        fi = ax.get_figure()  # get the figure of interest
        # attach the call back
        fi.canvas.mpl_connect('scroll_event', zoom_fun)

        # return the function
        return zoom_fun

    zoom = zoom_factory(ax)

    # Plot initial data
    for index in range(len(lines)):
        x_vec, y_vec, skip = get_coordinate_lists(data_df, index)
        lines[index] = ax.plot(x_vec, y_vec, formats[index], alpha=0.8, label=labels[index])

    ax.legend(loc='upper right')
    plt.xlabel(xlabel)
    plt.ylabel(ylabel)
    plt.title(title)
    plt.gcf().subplots_adjust(bottom=0.15)
    plt.show()


# Update the line graph
def live_plotter_update(data_df, lines, pause_time=0.01, max_points_to_show=10):
    # All x_vec and y_vec lists, used to set the bounds of the graph
    x_vecs = []
    y_vecs = []
    last_x_vec = None  # Store the last x_vec, in full (not only the last max_points_to_show points), for time labeling
    time_list_strings = None

    for index in range(len(lines)):
        x_vec, y_vec, list_strings = get_coordinate_lists(data_df, index)

        lines[index][0].set_data(x_vec, y_vec)

        # Add to x_vecs and y_vecs
        x_vecs.append(x_vec[-max_points_to_show:])
        y_vecs.append(y_vec[-max_points_to_show:])

        # Override time_list_strings
        time_list_strings = list_strings

        # Override last_x_vec, so the time labels are properly applied to all points, not just those visible
        last_x_vec = x_vec

        if has_been_closed():
            return  # Exit program early if closed

    # Do not adjust bounds if panning because it will send them back to the original view
    if not panning_allowed:
        # Adjust the bounds to fit all the lines on the screen and only show at most max_points_to_show at once

        # Find the smallest and largest x values (in the last max_points_to_show of each x_vec in x_vecs)
        smallest_x = np.min(x_vecs)
        largest_x = np.max(x_vecs)

        # Find the smallest and largest y values (in the last max_points_to_show of each y_vec in y_vecs)
        smallest_y = np.min(y_vecs)
        largest_y = np.max(y_vecs)

        # Update the x axis to use time_list_strings instead of values in seconds for easier reading (HH:MM:SS format)
        plt.xticks(last_x_vec, time_list_strings, rotation=-45, ha="left", rotation_mode="anchor")

        # Adjust the bounds to be a fraction of the standard deviation past the max and min points, to keep space
        # between the points and the borders
        plt.xlim(smallest_x - np.std(np.asarray(x_vecs).astype(np.float32)) / 3,
                 largest_x + np.std(np.asarray(x_vecs).astype(np.float32)) / 3)
        plt.ylim(smallest_y - np.std(np.asarray(y_vecs).astype(np.float32)) / 2,
                 largest_y + np.std(np.asarray(y_vecs).astype(np.float32)) / 2)

    if has_been_closed():
        return  # Exit program early if closed

    plt.pause(pause_time)

Building Energy Boot Camp 2019 - Day 4

Thu, 01 Aug 2019 14:00:00 +0000

Today, we spent the entirety of the boot camp session working on the project we were assigned yesterday. I managed to finish the gallons per day (GPD) calculations and output the number of houses with GPDs higher than 300, 500, and 1000. I also wrote code to calculate the GPD of each street in Andover and write this, along with the GPD data for each house, into .csv files and Excel spreadsheets. However, as this code dealt with such a vast amount of data, I quickly noticed its long run time. Thus, I spent much of the work time optimizing my code. First, I made sure to drop any and all unnecessary columns, to greatly increase the total amount of cells in the pandas DataFrame. I also discovered cython, a tool that compiles Python code into C code, which runs many times faster than Python. Using cython, I managed to greatly decrease run times, but was still unable to bring them under 80 seconds. I have included the code and its output in the Code and Output section.

Images

The output spreadsheet displaying GPD per household (censored for privacy). Note the different sheets, Household GPD and Street GPD.

The output spreadsheet displaying GPD per street (censored for privacy).

Code and Output

import os
import time
import sqlalchemy
import numpy as np
import pandas as pd
import datetime as dt

# Used to record elapsed time
start_time = time.time()

# Define unit constants
CUBE_IN_PER_GALLON = 231
CUBE_IN_PER_CUBIC_FT = 1728

# The default length of time between 'prior_date' and 'current_date' if incomplete data is given
DEFAULT_DAYS_BETWEEN = 90

# The path to the sqlite file
db_path = os.path.join('SQLite', 'student.sqlite')
engine = sqlalchemy.create_engine('sqlite:///' + db_path)  # Read the sqlite file into sqlalchemy

# Read the Water table into a dataframe
df_water = pd.read_sql_table('Water', engine, parse_dates=True)

# Delete unnecessary columns
df_water = df_water.drop(
    columns=['service_id', 'account_number', 'service', 'meter_number', 'transaction_date', 'transaction_type', 'units',
             'description', 'id'])

# Convert the 'current_reading' column into floats
df_water['current_reading'] = df_water['current_reading'].astype(float)

# Delete rows with 0 as a value for 'current_reading'
df_water = df_water[df_water['current_reading'] > 0]


# Convert the 'prior_date' and 'current_date' into datetime or None
def convert_to_datetime(s):
    if s is None:
        return None

    try:
        return dt.datetime.strptime(s, '%Y-%m-%d %H:%M:%S')
    except ValueError:
        return None


df_water['prior_date'] = df_water['prior_date'].apply(convert_to_datetime)
df_water['current_date'] = df_water['current_date'].apply(convert_to_datetime)

# Create an empty column, filled with 0s by default, to store gpd for the time period
# Data will be added to the column as it is calculated
gpd_list = [0 for counter in range(len(df_water.index))]
df_water['GPD'] = gpd_list


# Converts a numpy.datetime64 to a datetime.datetime object by converting dt64 to UTC time (for later use)
def datetime64_to_datetime(dt64):
    return dt.datetime.utcfromtimestamp((dt64 - np.datetime64('1970-01-01T00:00:00Z')) / np.timedelta64(1, 's'))


# Convert values from cubic ft to gallons
df_water['current_reading'] = df_water['current_reading'].apply(lambda x: x * CUBE_IN_PER_CUBIC_FT / CUBE_IN_PER_GALLON)

# Create a dataframe to store the average GPD for each house, but leave it blank
df_base_dictionary = {'address_street_number': [], 'address_street_name': [], 'GPD': []}
df_gpd = pd.DataFrame(df_base_dictionary)

# Mark the 'GPD' column as floats
df_gpd['GPD'] = df_gpd['GPD'].astype(float)

print('Building table of average gallons used per day (average GPD)...')

# Iterate through every unique street
street_names = df_water['address_street_name'].unique()
for street in street_names:
    # Get the list of all house numbers on that street
    house_numbers = df_water[df_water['address_street_name'] == street]['address_street_number'].unique()

    # Create a row for each house
    for house in house_numbers:
        df_gpd.loc[len(df_gpd)] = [house, street, 0]

print("Done!\n")
print("Calculating all average GPDs...")

# Go through every house and calculate the GPD
for index, row in df_gpd.iterrows():
    # Grab the portion of the df_water that has to do with this house
    df = df_water[df_water['address_street_name'] == row['address_street_name']]
    df = df[df['address_street_number'] == row['address_street_number']]
    df = df.sort_values(by='prior_date')
    df = df.reset_index(drop=True)

    for indx in range(1, len(df.index) - 1):
        df = df.drop(index=indx)  # Drop all but the first and last rows

    df = df.reset_index(drop=True)

    difference_in_gallons = df.iloc[1, :]['current_reading'] - df.iloc[0, :]['current_reading']

    start_date = df.iloc[0, :]['current_date']
    end_date = df.iloc[1, :]['current_date']

    # If the start_date is valid, convert it into a datetime
    if not pd.isnull(start_date):
        start_date = datetime64_to_datetime(start_date)
    else:
        # No start_date, use 90 days after the prior_date
        start_date = datetime64_to_datetime(df.iloc[0, :]['prior_date']) + dt.timedelta(days=90)

    # If the end_date is valid, convert it into a datetime
    if not pd.isnull(end_date):
        end_date = datetime64_to_datetime(end_date)
    else:
        # No end date, use 90 days after the prior_date
        end_date = datetime64_to_datetime(df.iloc[1, :]['prior_date']) + dt.timedelta(days=90)

    days_between = (end_date - start_date).days
    gallons_per_day = difference_in_gallons / days_between
    df_gpd.loc[index, 'GPD'] = gallons_per_day

print("Done!\n")

print("Sorting table by street name, then by house number")

df_gpd = df_gpd.sort_values(by=['address_street_name', 'address_street_number'])

print("Done!\n")

print("Creating table of street GPDs...")
# Create a dataframe for each street
street_gpd_base_dict = {'street': [], 'GPD': []}
street_gpd = pd.DataFrame(street_gpd_base_dict)

for street in street_names:
    house_data = df_gpd[df_gpd['address_street_name'] == street]
    gpd = house_data['GPD'].sum()

    street_gpd.loc[len(street_gpd)] = [street, gpd]

street_gpd = street_gpd.sort_values(by=['street'])

print("Done!\n")

elapsed_time = time.time() - start_time

# Round elapsed time to 2 decimal places
print("Elapsed time: {0} seconds".format(round(elapsed_time * 100) / 100))


# Returns the number of houses with more than min_gpd gpd
def count_by_min_gpd(min_gpd):
    temp_df = df_gpd[df_gpd['GPD'] > min_gpd]
    return len(temp_df.index)


print('\n{0} households have an average GPD greater than 300'.format(count_by_min_gpd(300)))
print('{0} households have an average GPD greater than 500'.format(count_by_min_gpd(500)))
print('{0} households have an average GPD greater than 1000\n'.format(count_by_min_gpd(1000)))

print("Saving GPD data to 'household_gpd.csv'...")
df_gpd.to_csv('household_gpd.csv', index=False)
print('Done!\n')

print("Saving street GPD data to 'streets.csv'...")
street_gpd.to_csv('streets.csv', index=False)
print('Done!\n')

print("Saving all GPD data to 'gpd.xlsx'...")
# Write to Excel sheet
with pd.ExcelWriter('gpd.xlsx') as writer:
    df_gpd.to_excel(writer, index=False, sheet_name='Household GPD')
    street_gpd.to_excel(writer, index=False, sheet_name='Street GPD')
print("Done!")

main.py, the program that loads the database's into pandas and performs the GPD calculations

Building table of average gallons used per day (average GPD)...
Done!

Calculating all average GPDs...
Done!

Sorting table by street name, then by house number
Done!

Creating table of street GPDs...
Done!

Elapsed time: 86.3 seconds

403 households have an average GPD greater than 300
94 households have an average GPD greater than 500
14 households have an average GPD greater than 1000

Saving GPD data to 'household_gpd.csv'...
Done!

Saving street GPD data to 'streets.csv'...
Done!

Saving all GPD data to 'gpd.xlsx'...
Done!

A sample output from main.py

import os
from distutils.core import setup
from Cython.Build import cythonize

setup(ext_modules=cythonize(os.path.join('SQLite', 'main.py')))

cython-setup.py, which is used by Cython to run main.py.
I run the code by running

python3.6 cython-setup.py build_ext --inplace

in the terminal and then calling

import SQLite.main

in a python terminal

Building Energy Boot Camp 2019 - Day 3

Wed, 31 Jul 2019 14:00:00 +0000

Today, we learned about SQLite and the SQLite Database Browser. We were given a database containing several tables full of town data of every type, from census results to quarterly water usage. After learning about how to utilize the sqlalchemy and pandas packages together to load tables from the database as dataframes. Armed with this knowledge, we were assigned a new project, in which we must calculate how many houses use, on average, more than 300 gallons of water per day. The difficulty in this project lies in the calculation of the gallons per day, as only the total amount of water used, in cubic feet, is stored in the database and assigned to a certain date range. This means that to calculate how many gallons a house uses per day, all of the rows corresponding to that house must be found (ranging from 2 to 6 rows), the total number of days accounted for must be calculated using the two date columns, prior_date and current_date, and this total, along with the converted difference between the largest and smallest measurements, will give the average GPD (gallons per day).

Unfortunately, I was unable to finish the assignment today, and will do my best to finish it tomorrow.

Dan Ivanovich

Introducing the EA Dashboard!

What is the EA Dashboard?

Application Structure

The Present

The Future

Scripting Python Package Maintenance

Generating requirements.txt

Installing All Required Packages

Updating Packages

Automating These Processes

OpenCV and AHS Heatmap | Autonomous Room Detection in Floor Plans With OpenCV (Part 3)

Processing the Generated PNG With OpenCV

Detecting Room Shapes

The Code

Drawing Rooms on the SVG

The Code

The Completed fill_room Function

Putting it All Together

Ending Notes

OpenCV and AHS Heatmap | Working with SVGs (Part 2)

Problem 1: Extracting Room Number Text Box Coordinates

Problem 2: SVG to PNG Coordinates

OpenCV and AHS Heatmap | Automatically Finding Rooms in an Image (Part 1)

Input Format

OpenCV

Converting SVGs to PNGs

Converting SVGs to PDFs

AHS Heatmap Presentation Recap

Building Energy Boot Camp 2019 - Day 6

What Changed?

SSL Certificate and HTTPS

Building Energy Boot Camp 2019 - Day 5

Killable Threads - Background

Killable Threads - The Code

Graph Interactivity Improvements

Screenshots

Code

Building Energy Boot Camp 2019 - Day 4

Images

Code and Output

Building Energy Boot Camp 2019 - Day 3

Generating `requirements.txt`

The Completed `fill_room` Function