I am pretty open with my enthusiasm for open-source software (OSS) and, as a result, often receive questions from others on how they can get started contributing to projects. I’ve received enough questions to notice some trends in where people are starting from and what kind of problems they’re having. As such, I wanted to write this article to help anyone who is wanting to get started contributing to OSS projects, because getting started is the hardest part! To begin, let’s briefly talk about the advantages of contributing to OSS.

Contribute to the Community

One of the main reasons I like contributing to OSS projects is because I generally like being helpful. I enjoy fixing bugs that trip people up or adding features that help people get their work done more efficiently.

I also like contributing specifically to my field. As a PhD student and possibly future scientist, I have a vested interest in the quality of the software in my field–specifically, structural bioinformatics. I use several tools in this field and often find areas that can be improved, both for myself and others. As an example, consider this minor documentation change I added to the Biopython documentation.

This is admittedly a very small change, however I updated documentation that was incorrect because on two separate occasions, I landed on this page while trying to do something and got errors by following the provided example. In the future when I (or anyone else) arrive at this page, they will not have to go through the trouble of debugging the examples.

Develop or Hone new Skills

This is one of the primary direct benefits of contributing to OSS projects–the chance to work on significant projects with technologies you’re interested in working with. This can be fun for you and it can also help provide “proof” that you know how to work with certain technologies even though your work life might not provide you with exposure to them.

One personal example, doing this PR adding support for some operators to the autowrap project gave me the chance to learn a lot about Cython and work with a small bit of C++. Another example from the Biopython ecosystem is this PR updating the documentation-deployment process. This work was one of my first exposures to working with CI/CD systems, a topic I feel I excel in now. Being able to deliver this feature on a project that impacts many people was a very valuable and fun learning experience.

Develop a Name for Yourself

Another benefit to contributing to OSS projects is making a name for yourself in a certain area. For example, many of the frequent contributors to the CPython project become Python fellows or contributing members. This designation signals a kind of proven competence. Anyone who regularly contributes to CPython must be either an expert in Python, C, or both, right?

Being known as a helpful presence in some software community will garner goodwill for yourself, and that goodwill can go a long way.

Setting up your GitHub Profile

Now that we know why we want to contribute to open-source software, lets finally get to how. The very first thing you need to do, if you haven’t already, is set up your GitHub account. You can also additionally set up a GitLab account if you know the project you want to which you want to contribute is hosted there.

There’s really no beating the official documentation from GitHub on setting up your account, so I wont’ write out all the steps here. Follow the steps in that link so you can start contributing.

If you’re new to Git, feel free to check out my professional git series (part 1, part 2, part 3), which should teach you all you’ll need to know about working with Git and contributing to OSS.

Picking a Project

Perhaps the most important step is the first: picking a project to contribute to. This is where I see a lot of aspiring OSS contributors quit before they even start, but it’s not their fault! Often, the advice to people just finishing learning the basics is to find an OSS project in that language and “just start contributing!” Personally, I find this advice unhelpful because it can send inexperienced learners towards complex projects they’re likely not ready for.

As an example, when I first learned Python, someone suggested that I start contributing to Django. Django is a web development framework for Python and it’s incredibly popular, but it was way over my head at the time as I had just finished an introductory level book on Python development and had absolutely no experience. I didn’t even know how to use Django, let alone contribute to it. My inability to even figure out how to get Django running locally put me off OSS development for a long time. My point here is, don’t just browse GitHub and start trying to make sense of the most popular project written in your language of choice, find a project that really interests you.

I believe the trick to being a frequent contributor to a project is caring enough about it to stick with it. You will likely never make meaningful contributions to a project in just a few hours of work, you will likely spend days or weeks on fixing bugs and adding features; and that’s after you’ve spent a bunch of time figuring out how the software works as is. So, you need to find a project that you are interested in enough to work on for extended periods of time.

Caring about a project requires one of two things: investment in the problem being solved by the project, or an interest in the work itself. To illustrate the first point, consider Biopython. Most of the contributors to that project are bioinformatics researchers. For the second point, consider the fact that many contributors to open source programming languages (like CPython, Rust, and more) are language enthusiasts that contribute because they like the kinds of challenges language design presents.

Since finding a project to contribute to is so important, take your time doing it. There’s no reason you have to decide today what project you want to contribute to. Browse different topics in GitHub and see what sparks your interest the most.

Figuring out the Development Workflow

Once you’ve found a project to contribute to, the next step is figuring out how the project’s community builds their software. If you want to contribute to their work, you have to respect their process. Often, this will be explained in a file called CONTRIBUTING.md or something similar.

The kinds of things you want to look out for is how the community prefers to receive pull requests. For some, maintainers want all pull requests to be associated with a corresponding GitHub issue. For example, this pull request to the OBOFoundry home page removes a page from their website, but I only made this PR because one of the maintainers created issue 2590 in the “Issues” tab asking for that page to be removed.

The most common workflow I’ve encountered involves forking the main repository into your own profile, creating a branch based on some issue, making your changes, writing tests for the changes, and making a pull request to the main repo. Development workflows frequently include a CI pipeline with tests and checks your changes have to pass.

More often than not, the final step in the development workflow is a review process. This typically includes a project maintainer looking over your changes and making sure you’ve solved the problem and that your code code is in line with project’s standards. This is usually a back-and-forth process, especially for your first contribution, in which a reviewer may ask you for changes multiple times. Don’t get discouraged or take these reviews personally! This is someone else’s project after all, you are a guest (though a helpful one!).

Project’s sometimes have guidelines for tagging reviewers (mentioning their GitHub username so they’re notified that you’re awaiting their review) so your work can move forward. Don’t get impatient and tag reviewers constantly, but don’t be too shy to bump your PR if the guidelines allow for it.

The main takeaway here is to set yourself up to be a productive member of the project community by adhering to their process and not just diving in headfirst writing code and making changes that no one wants. Remember, software engineering is a collaborative process, even (or especially!) in open source development.

Conclusion

Contributing to OSS can be beneficial to you, project maintainers, and the general public alike. It can also be a lot of fun! Remember to carefully choose the projects you want to contribute to and respect the development process. Feel free to reach out if you have any questions.

The post How to Start Contributing to Open Source Software first appeared on Erik's Code Space.

NOTE: At erikscode.space, book reviews do not contain affiliate links or paid advertisements. They are not written at the request of one of the authors or publishers, paid or otherwise. There are no incentives that would sway this review in a positive or negative direction. This review is based purely on the article's author's personal opinion.

Background

Naming Things

Phil Karlton once said

There are only two hard things in Computer Science: cache invalidation and naming things.

This is a fairly famous quote among software professionals and it refers to the difficulty in naming variables, functions, classes, projects, teams, and pretty much anything else requiring a moniker in software development. Naming things well is an incredibly important skill because it affects the way we think about the problems we’re solving and the code we’re writing.

Surprisingly, there is comparatively very little written about naming things in computer science outside of academic papers and (probably) diatribes in company Slack threads. Tom Benner addresses this gap in the literature by writing a brief and enjoyable book on the topic.

The Author

I had not heard of Tom Benner before but while researching this article I found his website, which led me to his GitHub showcasing a few highly-starred repositories and some non-trivial contributions to the Kubernetes repository. Between that and his LinkedIn profile, I think he has the credentials to be considered someone worth reading.

Ironically, if I’m reading it right, he added the current-context command to kubectl config. I say this is ironic because I never liked the word “context” for what this tool does and I’m about to recommend his book about naming things.

Overview

This is the first book I’ve ever read that tackles the topic of naming various things in day-to-day programming. The book intentionally avoids topics like case or naming projects or teams, narrowing the scope of the book to variables, functions, and classes.

The book identifies four attributes of a name that should be considered when naming things: understandability, conciseness, consistency, and distinguishability. The book starts by introducing the challenges in naming things, then defines the four name attributes, and ends by providing examples for applying the attributes.

Review: Great!

I really liked this book. I read it on my iPad so I’m not exactly sure how many “pages” it is, but I read most of it on a 2-hour flight and finished it up the next morning in about an hour, so it’s a very quick read compared to other books in our field.

General Notes

I think tech books often present their ideas dogmatically and with little room for nuance. However, I think that this topic necessitates a bit of dogmatism, and the author does a pretty good job of being prescriptive when necessary but leaving wiggle room when appropriate.

There was only one thing I disagreed with in the book. In the chapter about conciseness, the book advocates not including “how” a method is accomplished in its name and favors configuration arguments instead of multiple method names. The example given includes a CSV processor that can either process a CSV serially or in parallel processes. Specifically, the example given is:

# Bad
csv_processor.process_in_parallel(rows)
csv_processor.process_in_serial(rows)

# Good
csv_processor.process(rows, in_parallel=True)
csv_processor.process(rows, in_parallel=False)

I personally disagree with this specific example because I believe the implementation leads to a method that is too big (the body having code for both kinds of processing) or a method that is named inaccurately (it’s called process but it actually determines “how” to process the CSV and then calls the appropriate private method, presumably). In my opinion, I think the solution to this particular example is to have a CSVProcessor parent class and two child classes called SerialCSVProcessor and ParallelCSVProcessor that each implement a process method.

This is a super minor disagreement though, and the author has quite a few more years of experience than I do, so I would likely defer to their judgement if they were on my team. In fact, I will look for areas where this rule can apply and see if I still disagree with its sentiment.

Required Reading for Newly Minted Developers

I believe this book would be most helpful to a developer in their first year on the job. The concepts are neither too abstract nor too technical, and a new developer has enough experience to understand and appreciate the content of the book. I also think most seasoned developers would agree with the majority of the content, so it would be helpful for new developers to get on the same page with their more senior colleagues.

Conclusion

Pros: Very useful information concisely delivered.

Cons: None really

Overall: The first book on the topic at this level of accessibility. It’s useful to senior and junior developers alike (although moreso for the latter). Also, the references were quite interesting if you want a deeper dive on the topic.

The post [Review] Naming Things: The Hardest Problem in Software Engineering first appeared on Erik's Code Space.

Originally posted here on dev.to

First, the C Function

To demonstrate, we’re going to write a program in C to find the factorial of a number. If you don’t remember factorials from high school, here’s an example:

4! (read four factorial) = 4 * 3 * 2 * 1

That is what our C program is going to do. Fire up a text editor and lets crank this function out:

long factorial(int user_input) {
  long return_val = 1;
  if (user_input <= 0) {
    return -1;
  else {
    for (long i = 1; i <= user_input; i++) {
      return_val *= i;
    }
  }
  return return_val;
}

int main() {
  return 0;
}

We are defining a function called “factorial” which will return a “long.” We’re using long instead of int because factorial functions can return some pretty big numbers.

Next, we’re declaring and initializing return_val which we’ll use to return the value of the calculation.

Now, the if statement is ensuring the number passed in by the user is positive, and if not, to return the value of -1. We’re returning -1 because later, when we wrap this function in Python, we’re going to know that getting -1 back from the C function probably means there was bad input.

If the number returned is greater than 0, we enter our loop in which we use an iterator, i, and multiply our return_val variable by it until i is equal to the number passed in by the user. Basically, this loop is saying:
n! = 1 * 2 * 3 * 4 ... * n

The final part, with the int main() is to appease the C compiler when we turn this into a .so file. I may be mistaken, but I’m pretty sure this part is necessary even though it doesn’t do anything. If anyone knows any better, please feel free to mention so.

The Pre-Python Part

Now that our C is written we have a couple things to do before we write the Python bit. First, save the .c file. I called mine cfactorial.c. Now, we have to turn this into a “shared object” file. In Linux, the command to do so is this:

$ cc -fPIC -shared -o cfactorial.so cfactorial.c

This particular command will make a cfactorial.so out of my cfactorial.c file. Now, to the actual Python

The Python Part

Almost done! Fire up that text editor again and lets script out some Python. First, we need to import the ctypes module. Then, if you’re anything like me, you’ll want to put the absolute path of the .so file into its own variable. So the top of my pyfactorial.py looks like this:

from ctypes import *

so_file = '/home/ewhiting/cstuff/cfactorial.so'

The next thing we want to do is create our cdll object out of our previously created .so file. So, after the so_file variable assignment, put:

cfactorial = CDLL(so_file)

Now, technically at this point you can start messing with calling the C function in the Python script by running python in the command line but lets be a little responsible first. Before we play with it some more, lets wrap our C function in a Python function. After creating the cfactorial variable, create the following function:

def factorial(num):
  c_return = cfactorial.factorial(num)
  if (c_return != -1):
    return c_return
  else:
    return "C Function failed, check inputs"

Save this file as pyfactorial.py. Altogether, it should look like this:

from ctypes import *

so_file = '/home/ewhiting/cstuff/cfactorial.so'
cfactorial = CDLL(so_file)

def factorial(num):
  c_return = cfactorial.factorial(num)
  if (c_return != -1):
    return c_return
  else:
    return "C Function failed, check inputs"

Note, the way to call functions inside the imported C shared object file is by saying <CDLL Object>.<function name from C code>(<parameter>). Easy!

So basically, any time we want to use that C function within Python, we call the factorial function which will run the C function with the parameter passed in by the user and evaluate the result. If the C function returns -1 (remember we put that in there), the Python script knows that there was a problem. Otherwise, it will return the number. Lets try it out! Fire up your terminal and start python

>>> import pyfactorial as pf
>>> pf.factorial(5)
120
>>> pf.factorial(10)
3628800
>>> pf.factorial(-4)
'C Function failed, check inputs'

Ta-da!! That’s the basic idea behind using C functions in Python. This is definitely a tool worth having. Apply all your other programmerly knowledge to making awesome functions and features, and let me know if you have any questions.

For my of my writing on C, check out my series on pointers starting with a gentle introduction. If you want reasons to not ever write C in your life, check out my article about why you shouldn’t learn C.

The post How to Use C Functions in Python first appeared on Erik's Code Space.

Defining Software Architecture

It is difficult to find a helpful and exact definition for software architecture that accurately communicates its importance and prevalence. One simple definition of software architecture might be “the structure of a software system.” But that definition lacks the gravitas the concept deserves.

More colloquially, you will often hear people define software architecture as “the stuff that’s expensive to change later.” Such a definition, while true, doesn’t really tell us much about what it is. For example, the programming language in which a system is written is expensive to change, but so is the application’s project management software (for example, moving from internal documents to a professional project management system like Jira is very expensive). These are both expensive things to change, but only the programming language is a facet of software architecture.

In truth, software architecture means different things to different people and there is no universal definition that all developers will agree on. To some, software architecture is similar to class design in that it models the relationships between classes throughout the entire system.

Others extend the concept to include not only class relationships, but also the infrastructure upon which the system runs such as the type of database a system might use or the type of web server.

An official definition for software architecture exists in a document from the International Organization for Standardization (ISO) in conjunction with the International Electrotechnical Commission (IEC) and the Institute of Electrical and Electronics Engineers (IEEE) called ISO/IEC/IEEE 42010:2022 – Systems and software engineering – Architecture Description. This standardization document describes software architecture as:

[The] fundamental concepts or properties of an entity in its environment and governing principles for the realization and evolution of this entity and its related life cycle processes

ISO/IEC/IEEE 42010:2022 – Systems and software engineering – Architecture Description

“Environment” in that definition is later defined in the same document as:

[The] context of surrounding things, conditions, or influences upon an entity

ISO/IEC/IEEE 42010:2022 – Systems and software engineering – Architecture Description

Personally, I like this definition and I think it serves the purpose of defining architecture well enough for newcomers to the topic. For the purposes of the rest of this article, however, we will only learn about two facets of software architecture: architectural patterns and components, and reusability. These are the most significant and impactful parts of software architecture as well as the easiest to explain. Let’s go over what each one means.

Architectural Patterns and Components

Architectural components are layers of functionality within a system that are responsible for a specific system-wide task like accessing a datastore or routing application requests. Architectural patterns refer to the ways in which these different components work together to achieve the system’s purpose.

The idea of architectural patterns is somewhat similar to that of design patterns; like design patterns, architectural patterns are made up of components accomplishing specific tasks that are pieced together to accomplish broader tasks in the most effective way possible. Another way in which architectural patterns are like design patterns is that there are many well-documented patterns to choose from, each with their own strengths and weaknesses. However, unlike design patterns, architectural patterns have system-wide impacts.

As an example, consider the factory method design pattern which creates instances of objects based on a given context. This pattern’s components work together to accomplish a very useful task, but outside of that task, it has little—if any—impact throughout the system. While architectural patterns are also made up of multiple components, each with a specific responsibility, the scope of that responsibility is much larger and less specific than those of a design pattern’s responsibility.

It’s useful to think of the components of an architectural pattern as self-contained collections of code, interfaces, and possibly infrastructure. This is a level of abstraction higher than the idea of components in a design pattern because design pattern components only consist of code. As an example of an architectural component, consider the model-view-controller (MVC) pattern. In this architectural pattern, the model component (more often referred to as the model layer) consists of not only class definitions, but also the underlying datastore—like a database—of the application. It’s important to understand this idea of components at the architectural level because components are what make up architectural patterns and influence their strengths and weaknesses.

Reusability

On the topic of software architecture, reusability refers to a component’s suitability for handling a specific system-wide task. To use the MVC architecture example again, the architectural component called the controller layer is responsible for handling application routing tasks. Most of the time, MVC is used in web applications, so the controller layer is responsible for handling HTTP/HTTPS requests from browsers, sending them to the appropriate code hosted on a web server, and sending the response back to the browser. Once the controller layer is built, we don’t have to write it again. This means that no matter how many features we have, the controller layer is reusable for handling all the application routing needs of that feature. This saves the development team weeks of time by sparing them the task of writing HTTP routing code every time they want to introduce a new feature into a system.

By now we should be starting to understand what the concept of software architecture is getting at. Now, let’s talk about why software architecture is worth knowing about.

Why Architecture Matters

Software architecture is the “big picture” of a software system and therefore permeates every aspect of the code. Consider an obvious metaphor: the architecture of a building. The building’s architecture dictates everything about the building that makes it useful; how many people it can hold, where the emergency exits are, how to move from one area to another, the functions of different parts of the building, how future contractors might make upgrades to the building, even how the building affects the environment around it.

Additionally, a building’s architect has to make decisions about what attributes of the building are most important, which are nice to have, and which are not important at all. For example, an architect may be told to make a building with the primary goal of keeping operational costs low, even if the building isn’t very aesthetically pleasing. In this case, the architect will decide that, even though floor-to-ceiling windows on every floor would make for a very nice-looking building, it would be terrible for operational costs because the sun will heat up the building during the day and the people inside will have to turn up the air conditioner. In this case, the architect will have to opt for smaller windows throughout the building and perhaps even exterior walls of concrete to absorb the heat.

Likewise, software architecture dictates everything about the system that makes it useful, and software architects have to make decisions that sacrifice one thing for another. In software architecture, these “things” that are sacrificed are called quality attributes, and the decisions about which quality attributes to prioritize are called tradeoffs. Let’s talk about both of these concepts, starting with quality attributes.

Quality Attributes

Quality attributes are the non-functional attributes of a software system. In this case, an attribute is non-functional in that it doesn’t do anything. For example, suppose we’re building a calculator app. Being able to do addition is an example of a functional attribute. On the other hand, how quickly the calculator can produce the results of adding two numbers together is a non-functional attribute. Saying the calculator is fast or slow is a comment on one of the calculator’s quality attributes. Aside from the power and modernity of the physical infrastructure upon which a project runs, nothing impacts a system’s quality attributes more than its architecture. Let’s define some of the most important quality attributes, what they are, and how architectural decisions affect them.

Reliability

Reliability is a quality attribute that defines a system’s ability to perform its tasks under some given conditions for some given amount of time. For example, suppose we’re building a web server upon which someone to host their website. Say one of our users has their homepage, index.html, on our server. We expect that any time someone in the world sends and HTTP request to our server requesting that index.html file, our server will properly send the page back to that person. Occasionally, due to any number of reasons, our server will fail to send the requested index.html page and will instead send the user a 404 – Not Found response code. This is considered a failure.

The probability of our server correctly returning the requested index.html page at any given time under some given condition is a measure of reliability. For example, suppose we send one thousand requests for the index.html page over the span of fifteen minutes and our server correctly sends back the index.html page 991 times, the reliability of our server with at one thousand requests over fifteen minutes is thus 99.1%.

One way in which software architecture affects reliability in this case is how we decide to handle request failures on the server. When the server receives a request for index.html and for some reason cannot find it, instead of sending back a failure 404 message to the user, we could tell the server to try again up to five times. A very simplified version of how this might look in the code is as follows:

def handle_request(file_name, retries=0):
  try:
    resource = open(file_name)
    # File was found, send it to the requester
    send(resource)
  except:
    # File not found, try again
    retries += 1
    if retries < 5:
      print(f"Resource not found, retry number {retries}")
      handle_request(file_name, retries)
    else:
      # Couldn't find file after 5 attempts,
      # send 404 error message
      send(404)

In the code above, we wrote a function that handles a request for a resource like index.html and sends it back to the user (note, the send method is not defined in this code, this is just for example purposes). We put the open method inside of a try block and use the except block to handle any failure to find the resource. This will help alleviate any random failures that might happen by simply telling the server to try again. However, we also prevent any infinite loops from happening by telling the server to quit trying to find the requested resource after five tries. If the max number of retries is reached, the server sends the 404 response code. Now if we were to test the server again with a thousand requests for index.html, we may see improvements; perhaps the server eventually sends the requested resource 999 times, increasing our reliability measure to 99.9%.

Availability

Availability is closely related to reliability in that it measures the probability that a system will be available to perform its task at any given time. This measure is directly impacted by the system’s reliability, but also takes things like maintenance downtime into account. Availability over some period of time is measured as that period of time, minus the amount of downtime and then divided by the period of time being measured. For example, if we measure daily availability and find that the system is unavailable for about five minutes every day, the availability of the system is 1440 (the number of minutes in a day) minus 5 (the number of minutes the system is unavailable), all divided by 1440, or 99.65%.

You may occasionally hear people refer to their system availability in terms of nines. The system in our previous example has “two nines” of availability because the number 99.65 has two nines in it. These numbers are often used in service level agreements (SLAs), contracts that system developers have with their customers agreeing on how much availability they can expect from the system. Five nines (99.999%) of availability is considered very high availability as it allows for five minutes and 15 seconds of downtime a year.

Software architecture impacts the system’s availability by defining how system maintenance is conducted. For example, if an application’s database goes offline for some reason, can the application still be accessed while we figure out how to get the database back up and running, or does the whole system have to come offline? If we wrote our components to be reliant on the database, the whole application will likely be unavailable while the database is down. If it takes us five minutes and 15 seconds to get the application back online, we can’t have any more downtime for the rest of the year if we want maintain five nines availability.

Scalability

Scalability refers to how a system’s performance and cost increases and decreases with demand. For example, imagine our system has a baseline number of 500 users at any given time, but for some reason, one day we have two-thousand users for a couple of hours. In order for the system to perform identically for all users, we’ve programmed it to spin up a duplicate server every additional 500 users to handle the extra demand. Also, we’ve programmed the system to spin down those servers as the spike in demand tapers off. This means that both cost and performance increase and decrease based on the demands placed on the system.

Scalable systems are created by writing modular code. As a senior engineer or the architect on your team, you enforce code modularity by ensuring that the components that rely on infrastructure can handle that infrastructure being duplicated. For example, hardcoding the IP address of our servers in the code would not be conducive to scalability since new servers would have different IP addresses.

Security

Security is a quality attribute that refers to a system’s ability to protect data from unauthorized access, prevent users from accessing a higher level of privilege in the system than they need, and much more. Security is a broad topic and there’s a ton of information to know about it, so this section will cover only a very small subset of how it applies to software architecture.

When you’re architecting a system, you have to think about how data will be stored and if it’s allowed to travel networks unencrypted. For example, when transmitting data from a web server to a user’s browser, the data is accessible to people (network administrators, people using packet sniffers, and so on). In fact, it’s good practice to assume that any data traveling over a network is being read by nefarious people. We might not care who sees some data if it’s something innocuous like a person’s shoe size, but often we transmit data like names or credit card numbers over a network and we definitely don’t want people reading that, so we encrypt it.

Encryption, like security, is also a broad topic but I wanted to briefly define what it is in case you’ve not heard of it before. Basically, encryption is the process of scrambling data so that it’s not human-readable, and the only way to unscramble it is with a special decoder called an encryption key. Generally speaking, only the machine receiving encrypted data has the appropriate encryption key, so even if that data is captured by nefarious people, it is no good to them because they can’t read what it says.

The system’s data encryption policy is an important architectural decision because data is much bigger when it’s encrypted than when not. As a consequence, encrypted data takes slightly longer to travel from point a to point b in a network. It’s not a huge difference, but as your system grows, the amount of data flowing through the system will impact the system’s performance. Figuring out what data needs to be encrypted is just one small facet of security minded software architecture.

Maintainability

The last quality attribute we’ll discuss is maintainability. A system’s maintainability is a qualitative measure of how easily developers can add features, fix bugs, or tweak performance. Maintainability isn’t something we can measure with a formula; it is more of a general feeling about how easy the system is to work with from a developer’s perspective. Many of the topics you learn as you become a more senior developer are techniques for writing maintainable code. For example, making smart decisions about class design makes future work easier to do because intuitive class design lowers cognitive complexity. Likewise, using design patterns where appropriate along with naming their components descriptively helps developers navigate the system easily, reducing the time needed to add a new feature or fix an overlooked bug.

Software architecture impacts the maintainability of the system because every architectural decision influences how code must be written. As an example, a microservices architecture pattern is useful for separating business concerns into individually deployable components. This means that, as a developer, if I get tasked with writing a feature for the marketing team, I know exactly which component I need to work on. However, if I’m working in a system built on the MVC architectural pattern, I may have to work with several components to deliver the required feature.

Architectural Tradeoffs

Now that know what quality attributes are and have seen some examples, let’s talk about architectural tradeoffs. As a senior developer or software architect, you’ll often be faced with building features that have competing priorities. Consider the example we used when talking about reliability. Adding the try … except blocks to the server’s code allowed us to increase the server’s reliability, but it most likely will make the server a little slower. This is because exception handling is very slow when exceptions are raised, meaning that every now and then, a user will have a slightly slower experience than if we got rid of the server’s exception handling altogether.

This blog's article on exception handling in Python

Deciding which is more important, the speed of the request response or the reliability of the server, is an architectural tradeoff you will have to make. This decision will be influenced by many factors such as the kind of application you’re building, the needs of the users you’re building it for, and much more. There are no universally correct answers to architectural tradeoff questions, they always depend on the context in which you’re working.

One of the tradeoffs you’ll have to think about quite often is security. As we discussed in the quality attribute section, something like encryption can have a significant impact on your system’s performance. As an architect, you’ll have to decide what needs to be encrypted and what doesn’t. If you’re working in the defense industry, you’ll likely have to encrypt every bit of data that travels over a network, causing your system to perform slowly. However, if you’re writing web-based browser games that don’t require a login, you likely don’t need to encrypt anything. The kinds of tradeoffs you make will depend on the industry in which you’re working and the type of system you’re building.

One thing that will impact your architectural decisions and tradeoffs is the due date of your projects. For example, if you have a year to build a back-office application for a small company, you will likely have maintainability as a high priority so that you can troubleshoot problems easily and add new features to the system upon request. Conversely, if you’ve been tasked with building the same system in a month, you will not take the time to consider maintainability. This is because building maintainable systems requires extra care and planning, time you don’t have with such a quick due date.

Yet another decision about architecture you’ll have to make is whether to build a component yourself or use a third-party component. Third party components, sometimes called commercial off the shelf (COTS) software, are components built by other organizations that you usually have to pay for. COTS software is supposed to require very little work for a developer, and sometimes includes 24-hour support (that is, you can call someone for help if the COTS software isn’t doing what you want it to). One example of a homegrown vs. COTS solution is the decision on how to monitor your system’s performance. Quite often, it’s necessary to watch your system’s performance such as CPU or RAM usage, network latency, and much more. Some teams find it best to write their own software for monitoring system performance while others buy performance monitoring software from other vendors. The decision on which way to go will depend on a lot of things such as staffing (you need more developers to build and maintain such software) and cost (some performance monitoring applications are very expensive), and whether or not the COTS software can easily service your needs.

Finally, perhaps the most important consideration when making architectural tradeoffs is cost. You and/or your company have a finite amount of money set aside to build whatever system you’re working on. How you use that money to accomplish the goals of the system often includes making several architectural tradeoffs. For example, do you want to pay for a database management system like Microsoft SQL Server so that you can have 24-hour support ready to help you if something goes wrong, or do you prefer using an open-source option like MySQL or Postgres for free, but which do not have a helpdesk for you to call? The answer to that question will depend on many factors. For example, are you a lone developer for the company you’re building this system for, and the company needs the application to be running all the time? If so, you might want to opt for the SQL Server option since you may need help one day. Conversely, are you building an open-source project for people to share recipes with each other? If so, your budget is probably small, and your users won’t lose millions of dollars if your database stops working. In such a case, self-hosting a free database system like MySQL should be fine.

The responsibility for making the decisions on these tradeoffs will fall on you more often as you become more experienced in whatever tech community you’re a part of, be it an open-source organization or a company. Understanding the overall goals your system and how your organization plans to use and support it will be an important step towards making good architectural tradeoffs.

Conclusion

This article introduced an important concept in your path to becoming an experienced and professional-grade programmer: software architecture. Software architecture can be defined in a number of ways, but it’s ultimately the process of developing components and fitting them together in the most efficient way relevant to our system’s goals. We learned about those components and how they form architectural patterns. Then, we learned about quality attributes and how they’re impacted by software architecture. Learning about quality attributes is a prerequisite to learning about architectural tradeoffs, the decisions you must make when developing systems. Hopefully now you have an understanding for what software architecture is and why it’s important.

The post An Introduction to Software Architecture first appeared on Erik's Code Space.

Fundamentals of Debugging

As a developer, you will spend a lot of time debugging. In fact, several sources estimate that as little as 35% or as much as 75% of a developer’s time is spent debugging code that’s already been written rather than writing new code (here’s a pretty good source that puts that number between 35% and 50%). Since so much of our time as developers will be spent debugging, we should put effort into building this skill. As with all skills, you will get better at debugging the more you do it. There are, however, some fundamental techniques to troubleshooting and debugging software that you should know and build upon. This section will introduce those fundamentals, starting with confirming the bug.

Confirm the Bug

An important part of debugging code is understanding the broader context and knowing what the code is intended to do. This is important because your first step in fixing a bug is confirming that it truly is a bug. For example, consider a bug report that says, “the area_of_cube function isn’t returning the proper values unless you pass it 6.” In order to start debugging, we first decide to try the function out. We load up a console and try a few values:

>>> from formulas import area_of_cube
>>> area_of_cube(4)
96
>>> area_of_cube(5)
150
>>> area_of_cube(6)
216

It looks right, so we go and double check the formula for finding the surface area of a cube and find that it is the length of an edge squared and then multiplied by six. That means that the numbers we got back from our console test are correct. However, we remember that the bug report says the function only works when you pass it six. Now we wonder if the user reporting the bug is confusing area with volume, since the formula for the volume of a cube is the edge length raised to the third power (or cubed). This means that a cube with edges of length 6 will have both a surface area and volume of 216, and we think the user reporting this bug meant to use the volume function and erroneously reported the area function as having a bug.

At this point, we have to track down the user reporting the bug and see if they really meant to use the area function when they were reporting the bug or if they meant to use the volume function. This is what confirming the bug is about, making sure a reported bug truly is a bug. This is an important step in debugging because it could potentially save you from fixing code that isn’t actually broken. This example may seem silly, but you will be surprised how often reported bugs in the software are actually due to misuse.

Reproduce the Bug

Once you’ve confirmed a bug report is genuine, the next step in fixing a bug is reproducing it. Many times, bug reports will say something isn’t working but when you go to check it out yourself, it seems like it’s working fine. In order to properly diagnose the problem, you have to first see the problem happening. As an example, imagine you get a bug report from a coworker that says something like “The greet_user function isn’t working. The function doesn’t print anything to the console when called.” You start your console to test it out and see no problem:

>>> from greeters import greet_user
>>> greet_user('Erik')
Good evening, Erik

In the above console session, you imported the reportedly buggy greet_user method and called it by passing Erik as an input parameter. The function then printed “Good evening, Erik” as expected. You haven’t yet reproduced the bug, so now you have to go look at the code and see if you can figure out what’s going on. You open up the greeters.py file and look for the greet_user method and see this:

import datetime

def greet_user(name):
  now = datetime.datetime.now()
  hour = now.hour
  if hour < 12:
    print(f'Good morning, {name}')
  elif hour > 12 and hour < 18:
    print(f'Good afternoon, {name}')
  elif hour > 18:
    print(f'Good evening, {name}')

As you look over this code, you see that the method extracts the hour of the day and prints an appropriate greeting: “good morning” if the hour is before 12, “good evening” if it’s after 12 but before 18, and “good evening” if it’s after 18. The bug report said the function isn’t printing anything, how could that be possible?

Upon further inspection, you notice that the method’s if statement only provides instructions for what to do if the hour variable is 0 to 11, 13 to 17, or 19 to 23. If hour is 12 or 18, the code does nothing! This means there’s 2 hours of the day in which this code will not print anything to the console. The person who reported the bug must have been trying to use it during one of those times. In order to test your theory, you decide to manually set the hour variable to 12 to see if that reproduces the bug:

def greet_user(name):
  now = datetime.datetime.now()
  # hour = now.hour
  hour = 12
  if hour < 12:
    print(f'Good morning, {name}')
    . . .

Now, you rerun the same commands from your console session earlier:

>>> from greeters import greet_user
>>> greet_user('Erik')

But this time, you don’t see any output. You’ve successfully reproduced the bug and you now know why the bug is happening. In order to fix this bug, we simply have to update the conditionals to use greater-than-or-equal-to comparisons:

import datetime

def greet_user(name):
  now = datetime.datetime.now()
  # hour = now.hour
  hour = 12
  if hour < 12:
    print(f'Good morning, {name}')
  elif hour >= 12 and hour < 18:
    print(f'Good afternoon, {name}')
  elif hour >= 18:
    print(f'Good evening, {name}')

Now, with your now = 12 line still active, you can see that the code now works. This is why reproducing the problem is important, some of the trickiest bugs only show up under specific conditions.

Challenge Assumptions

Another technique for debugging especially tricky bugs is challenging your assumptions. Sometimes, a bug makes no sense and no matter how many things we test, we cannot seem to figure out why a bug is happening. When this happens, the next step is to ask ourselves if the things we generally take for granted still apply. This means making sure environment variables are what you think they are, verifying any authentication functions are still working, making sure you’re using versions of software that support what you’re trying to do, and so on.

As an example, imagine you receive a bug report saying there’s something wrong with the length_check method; it’s throwing an error any time someone tries to use it. You find the method in your codebase to see it’s defined as the following:

def length_checker(max, values):
  if (count := len(values)) > max:
    print(f'Cannot have more than {max} values, you have {count}')

This method takes two inputs, max and values, and prints an error if the number of items in the values parameter is above the max. Everything seems to look fine to you, so you start a console session to test the method out:

>>> from checkers import length_checker
>>> length_checker(1, [2,2])
Cannot have more than 1 values, you have 2
>>> length_checker(2, [1,2])
>>>

In the above example, you call length_checker and pass it a max value of one and a list with two items. Doing so causes the function to print an error to the console as expected. In the next line, we call length_checker with a max value of two and again pass it a two-item list. This time, nothing is printed to the console, as we expected. This function seems to be working perfectly, why did someone report a bug?

It’s time to challenge our assumptions. The error report came from someone else on a different computer, perhaps there’s some difference between their computer and our own that is causing this function to fail. You decide to start by checking the Python version of your computer and the bug reporter’s. You run python --version in your command terminal and see that you’re currently using Python 3.10. The bug reporter runs the same command and lets you know they’re using Python 3.7.

Could the error be because the bug reporter is on an older version of Python than you? You take a closer look at the function and realize that it uses the walrus operator (the := syntax). You vaguely remember reading about this being added to Python a few years ago, so you look up when it was added and find out that the walrus operator wasn’t part of Python until Python 3.8! No wonder the function isn’t working for the bug reporter, that version of Python doesn’t even know what to do with the := syntax. To fix this bug, you simply ask the reporter to update their Python version to 3.8 or higher, and the problem is solved.

Challenging your assumptions is one of the easiest steps to forget when it comes to advanced troubleshooting and debugging. Whenever you find yourself working on a bug that doesn’t make sense, ask yourself if you’ve tested literally every facet of the program—from the operating system to the software versions, and every line of code up to and including the error. Doing so will often show you that one of your assumptions is wrong and that’s why you’re encountering the bug.

Now that we’ve talked about the fundamentals of debugging, it’s time to talk about a built-in Python tool that makes troubleshooting easier: the debugger.

PDB: The Python Debugger

The Python debugger (called PDB) is a tool built into the Python programming language that allows us to stop a script at run time so we can check the value of variables at that given moment in the script’s execution. In this section we’ll learn how to set breakpoints, check the value of variables, and step into functions and methods. We’ll start with setting breakpoints.

Setting Breakpoints and Checking Values

When using a debugger, a breakpoint refers to a line in the code upon which we want to pause execution of the program. Before we see how to set a breakpoint with PDB, let’s write a script in which to use the debugger:

def add(x, y):
  sum = x + y
  return sum

a = 1
b = 2
c = add(a, b)
print(f"c is {c}")

This code is very simple. First, we define a method called add which takes two parameters and returns their sum. Then, we create two variables, a and b, and a third c which is the return value of add when passed a and b variables. Finally, we print the value of c to the console.

Now, lets set our first breakpoint to see how using the debugger works. Just above the line a = 1, write breakpoint(), and then run the script. You should see the following output in the console:

> path/to/script.py(6)<module>()
-> a = 1
(Pdb)

There are three things to note in the output above. First, the line preceded with the > symbol is telling us what file we’re in (your output may vary depending on what you named this file and where you saved it). The next line, -> a = 1, shows us where in the Python script we’ve stopped. Notice that this is the line just below where we set our breakpoint. The line a = 1 hasn’t run yet, we’ve paused the script just before its execution. Finally, you see (Pdb) and a prompt. We can type debugger commands in this prompt.

NOTE: You don’t also write import pdb; pdb.set_trace() instead of breakpoint(), but this is considered the “old way”. In fact, if you’re using Python 3.7 or below (which you shouldn’t because they’re all deprecated), you’ll have to use this. When I first wrote this article, I was using the set_trace way exlusively, so if you see one in the article, it’s because I missed it and you can easily just drop in breakpoint().

The first command we’ll run is next, type next into the console and press enter. You should see the following output:

> path/to/script.py(7)<module>()
-> b = 2
(Pdb)

Notice that we’re on the next line of the script. The next command executes the line in which we were paused and then pauses on the next line. So, when we typed next just now, Python ran the line a = 1 and stopped on the line you see in the console now, b = 2.

One of the main benefits of using the debugger is seeing the value of variables at a given time. Since we’re paused just after the a = 1 line, we can type the variable name a into the debugger console and see what the value is:

(Pdb) print(a)
1

This is useful for when we want to confirm that a variable holds the value we think it does. If we want, we can also change the value of a here as well, check it out:

(Pdb) a = 5
(Pdb) print(a)
5

Manually setting variable values at runtime like this is helpful for experimenting when debugging; sometimes you want to see how different values will cause a script to behave.

NOTE: If you have a variable that happens to share a name with a debugger command, you can prefix the variable name with an exclamation mark (!) to have the word interpreted as the script variable instead of the debugger command.

Now, let’s stop pausing the script and allow it to finish running. We do this with the continue command which will let the program run until it reaches the next breakpoint or until it’s done executing. Run continue and you should see the following output:

(Pdb) continue
c is 7

Notice two things here. First, the output says c is 7. This is because when we paused the script, we set the value of a to 5, so when we passed a and b to the add function, we actually passed 5 and 2 instead of 1 and 2 like the script appears to say. Also notice that your console is probably back to your normal command prompt instead of the debugger. This is because the script has finished executing.

Now that we know how to set breakpoints, move around in a script, check the value of a variable at a given time, and change the value of a variable, lets look at stepping into functions.

Stepping Into Functions

When we set a breakpoint in a script, the debugger does not leave the scope of where it’s pausing the script. As an example, move your breakpoint from the previous section to just over the c = add(a, b) line. Now, run the script, you should see the following output:

> path/to/script.py(8)<module>()
-> c = add(a, b)
(Pdb)

Notice that the debugger is paused just before the add function is called. Type next and hit enter in the debugger console, you should see the following output:

> path/to/script.py(9)<module>()
-> print(f"c is {c}")
(Pdb)

Notice that the debugger paused at the next line after the function call instead of going into the add function itself like we might expect. This is because the default behavior of the debugger is to step over function calls so that you, as the developer, can stay inside of the context in which you’re debugging. If you want to step into a function, you have to use the step command. Run the script again, once you get here:

> path/to/script.py(8)<module>()
-> c = add(a, b)
(Pdb)

Type step and then press enter. Now, you’ll be in the add function and you should see the following in your console:

--Call--
> path/to/script.py(1)add()
-> def add(x, y):
(Pdb)

Notice a couple of new things about the debugger’s output now. First, at the top of the output is a line that says --Call--, this lets us know that we’ve left the context in which we were originally debugging because a function or method call was made. Also notice that the line containing the file location no longer says <module> but says add(). Again, this lets you know that you are currently debugging a function inside of the script.

Now you know how to use the debugger to move around a script, check and change values of variables, and step into functions. These operations exist in virtually every programming language debugger out there, so the commands you learned in this chapter will likely be the same or similar for any other debugger you might use in the future.

Conclusion

In this article you learned the foundational skills of debugging software. We started by learning about three important steps to debugging: confirming the bug, reproducing it, and challenging your assumptions. Then, we learned about PDB, Python’s built-in debugging tool. We learned how to pause a script at run time so we can check the values of variables, then we learned how to control a script’s execution with the next and continue commands. Finally, we learned how to step into functions while debugging with the step command. All of these skills will serve you well in your future work as a programmer and will go a long way in advanced troubleshooting.

If you liked this article about debugging, you may like my primer on TDD with Python.

The post How to Debug Code (with Python Examples) first appeared on Erik's Code Space.

NOTE: At erikscode.space, book reviews do not contain affiliate links or paid advertisements. They are not written at the request of one of the authors or publishers, paid or otherwise. There are no incentives that would sway this review in a positive or negative direction. This review is based purely on the article's author's personal opinion.

Background

The Concept of Technical Debt

Technical debt is a concept that many–if not most–software professionals are familiar with. In this book specifically, technical debt is defined as:

design or implementation constructs that are expedient in the short term but that set up a technical context that can make a future change more costly or impossible

Managing Technical Debt: Reducing Friction in Software Development. Kruchten P., Nord R., Ozkaya I.

This is a concise and approachable definition of the concept upon which most software professionals would agree. Technical debt pervades software development, it is simply a fact of life for anyone involved in building software. However, despite the foreboding “debt” in its name, it’s not something we universally avoid and reject. Technical debt is something we manage and use intelligently to build the best software with the resources we have.

The first written reference to technical debt as an analogy that I’m aware of is in Ward Cunningham’s paper The WyCash Portfolio Management System. Another foundational source is Frederick P. Brooks’ paper, No Silver Bullet — Essence and Accident in Software Engineering, which conceptualizes the challenges inherent in software development.

The Authors

Phillipe Kruchten is a professional software engineer with a few decades of experience. He is also a professor emeritus at The University of British Columbia. Kruchten has published several peer-reviewed papers and is an editor of both Journal of Systems and Software, and IEEE Software. Another notable accolade is that Kruchten was one of the main developers of the Rational Unified Process (RUP).

Robert Nord is a principal member of technical staff at the Software Engineering Institute (SEI). He is an experienced researcher and prolific author of peer-reviewed publications.

Ipek Ozkaya is a technical director at the SEI and is the editor-in-chief of IEEE Software. She has a long history of impactful involvement in the IEEE and has also authored several peer-reviewed papers. Like Nord, her experience appears to be primarily academic and research-based.

Overview

Managing Technical Debt, as its name implies, is a book about the concept of technical debt in software development. The authors spend this short book talking about how to manage technical debt by presenting three companies:

derived … from actual companies that we authors have interacted with, but we abstracted many characteristics and details for confidentiality reasons, and in some cases we combined characteristics from two similar organizations into a single example.

The book is thirteen chapters long and broken into four parts:

1. Exploring the Technical Debt Landscape
2. Analyzing Technical Debt
3. Deciding What Technical Debt to Fix
4. Managing Technical Debt Tactically and Strategically

Review: Mostly Good!

Overall, the book is well-written and contains a lot of good information for software professionals of all disciplines. I believe readers should have at least a few years of experience in software in order to fully absorb the value of this book as one needs to have experience working inside of a team and company for some time to understand the true nature of technical debt.

General Notes

Typically, I am wary of software engineering books written by career professors and researchers. Decades of research experience are certainly nothing to scoff at, but I believe industry experience is a prerequisite for having valuable information to share with software engineers. Conducting research on software development process phenomena requires being on the outside looking in. Because of this, it’s easy to make principled statements that are naïve to the subtle effects of things like business contexts, office politics, or team dynamics.

Having said that, this book remains relevant to industry practitioners throughout. There is one important thing to note if you are an experienced developer about to read this book for the first time. This book isn’t about preventing technical debt, it’s about managing it once you know it’s there. Dr. Ozkaya made this distinction in episode 481 of IEEE Software Engineering Radio. Once I understood this, I reread some of the passages I highlighted as “naïve to industry practices” and appreciated them more.

One example of such a passage was in chapter 5: Technical Debt and the Source Code. The authors introduce a company whose business goal is to create evolvable software but is constantly hamstrung by poor quality in the source code. The listed cause of this is “Inexperienced team members create conditions for the occurrence of technical debt.”

To me, the obvious way to prevent such a problem is improving code review processes. “if inexperienced team members are merging bad code into main, the review process is obviously broken” I thought to myself. However, the goal of this chapter is not to talk about preventing this from happening, it’s about managing the debt that already exists. The authors do a great job of sticking to this goal by suggesting ways to measure and track this problem and its symptoms in support of the “create an easy-to-evolve product” business goal.

Not as Helpful for SaaS Developers

One weakness I did find with the book is that, as a developer in a medium sized SaaS company, I thought a lot of the content was tailored to developers in big companies, particularly where the software was not a product being sold.

There are a lot of suggestions on analyzing and communicating a project’s technical debt and quantifying its impacts. However, this documentation effort isn’t the kind of thing small to medium sized companies typically make time for. In my experience, it’s enough for a senior engineer to say something like “we need to take some time to clean some of this up” to get management sign off on technical debt cleanup efforts.

Working in SaaS, there is lots of technical debt to contend with, and many parts of this book are helpful. However, if you are like me in that the software you work on is actually being sold to customers, you can probably get a way with skipping chapters 8, 9, and 13 and you can just skim over parts of the book in which you see this figure:

Conclusion

Pros: Excellent examples, thoughtful approaches to a tricky problem, superb sources

Cons: Seemingly tailored to a large company context. May be a bit too “academic” at times for purely “industry” people.

Overall: Quick and useful read for mid-level software professionals and managers. Possibly a good starting point for software engineering researchers looking to conduct research in the areas of technical debt, development methodologies, or the intersections of software development and project management.

Check out my post 5 Great Programming Books Rarely Mentioned in “Great Programming Books” Articles

The post [Review] Managing Technical Debt: Reducing Friction in Software Development first appeared on Erik's Code Space.

What are Exceptions?

Exceptions can be thought of as unplanned events in the execution of a program that disrupt that execution. When a runtime error occurs, a specific exception is raised. If an exception is raised and the program does not have any code defining how to handle that exception, the exception is said to be uncaught, and it will terminate execution of the program. Exceptions are not unique to Python; every programming language has exceptions and a means of handling them. You’ve actually most likely seen lots of exceptions already, but just for clarity, let’s raise a few exceptions on purpose.

First, in a Python console, try to divide any number by zero and press enter. You should see something similar to the following:

>>> print(3 / 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero

When we tried to divide by zero, Python raised an exception. In this case, the exception raised was ZeroDivisionError. ZeroDivisionError is the name of the exception raised when a program tries to divide by zero. There are other kinds of exceptions as well, let’s raise another. In the Python console, try to add a number to a string:

>>> print('hi' + 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate str (not "int") to str

This time, Python raised a TypeError. TypeError is the kind of exception raised when a program tries to do something to an object that the object’s class doesn’t support. Notice also that there’s a message along with the exception: can only concatenate str (not “in”) to str. Exceptions can, and often do, have different messages for different ways in which they can be raised. For example, let’s raise another TypeError, this time by trying to divide a number by a string:

>>> print(20 / 'hi')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for /: 'int' and 'str'

Again, Python raises a TypeError in this situation, but notice that the message is different. The message says unsupported operand type(s) for /: ‘int’ and ‘str’. This will be relevant later when we write our own exceptions.

Now that we have an intuition about what exceptions are, lets learn how to handle them.

Handling Exceptions

An exception will terminate the execution of a program if left unhandled. For example, consider the following script:

def divide(x, y):
  return x / y

print(divide(4, 2))
print(divide(2, 0))
print(divide(9, 3))

In this script, we have a method called divide that takes two parameters, divides them, and returns the result. Next, we have three lines calling the divide function and printing the result. Notice that the second line passes zero as a second parameter; let’s see what happens when we try to run this script:

2.0
Traceback (most recent call last):
  File "path/to/script.py", line 5, in <module>
    print(divide(2, 0))
  File "path/to/script.py", line 2, in divide
    return x / y
ZeroDivisionError: division by zero

Notice that the program printed 2.0 to the console, meaning the first call to the divide function worked. Then, the program raised an exception, ZeroDivisionError, on the second line because we tried to divide by zero. Notice also that the program did not run the final line, print(divide(9, 3)). This is because the ZeroDivisionError was an uncaught exception. In order to make the program continue running, we have to write code specifically for handling that exception. We do this in Python with try except blocks.

Using try and except

In Python, we use the keywords try and except to handle exceptions. The try block contains the operation that may raise an exception, and the except block contains the code that defines what we want the program to do when an exception is raised. As an example, let’s rewrite the divide function from earlier to handle ZeroDivisionError exceptions:

def divide(x, y):
  try:
    return x / y
  except ZeroDivisionError:
     return "Cannot divide by zero"

Notice that the line return x / y is now in a try block. This tells Python to run the code in this block until the last line, or until an exception is raised. If an exception is raised, Python checks for an except block that matches the type of exception. In our case, we wrote except ZeroDivisionError, so if a ZeroDivisionError is raised, Python will run the code inside of the except block. To see this is action, run the script with the updated divide function. You should see the following output in the console:

2.0
Cannot divide by zero
3.0

This time, when the script got to the line that said print(divide(2, 0)), the program did not print the exception message to the console, it instead ran the code in the except block, which we can see in the console when it printed Cannot divide by zero. Notice also that even though an exception was raised, the program continued execution, as we can see by the 3.0 printed to the console when Python ran the last line of our script. This is an example of how exception handling is useful, we can anticipate problems and handle them without terminating the program.

Note: When a program encounters an error or otherwise fails in some way but does not terminate its execution, it is said to fail gracefully.

Handling Other Types of Exceptions

The divide function can now fail gracefully when a ZeroDivisionError exception is raised, but what about other exceptions? What if someone tries to pass a string to the function? As an example, update the last lines in the previous script to look like the following:

print(divide(4, 2))
print(divide(2, 0))
print(divide('hi', 2))
print(divide(9, 3))

If you try to run the script now, you’ll see the following output:

2.0
Cannot divide by zero
Traceback (most recent call last):
  File "path/to/script.py", line 14, in <module>
    print(divide('hi', 2))
  File "path/to/script.py", line 7, in divide
    return x / y
TypeError: unsupported operand type(s) for /: 'str' and 'int'

Notice this time that Python raised a TypeError when we tried to divide a string with an integer. We never wrote any code to handle this exception, so Python terminated the program once this exception was raised. If we want to add code to handle this exception, we have two options.

First, we could add another except block to the divide function for that specific exception. The function would then look like the following:

def divide(x, y):
  try:
    return x / y
  except ZeroDivisionError:
     return "Cannot divide by zero"
  except TypeError:
    return "Can only divide number types"

This would handle both ZeroDivisionError and TypeError exceptions. But there are yet more exceptions, we cannot anticipate them all. Instead of writing an except block for each exception type that could possibly beraised, we can write an except block that catches all exceptions besides the ZeroDivisionError one. This approach would have the function looking like this:

def divide(x, y):
  try:
    return x / y
  except ZeroDivisionError:
     return "Cannot divide by zero"
  except:
    return "An error occurred"

With the divide function written this way, there is no exception that can be raised which would terminate execution of our program. We have an except block dedicated specifically to catching ZeroDivisionError exceptions and informing the user they cannot divide by zero, and we have an except block that catches any other kind of exception that could possibly be raised and simply alerts the user that an error occurred. These are the basics of handling exceptions in Python but there’s one more thing to learn about, finally blocks.

Using finally

When a line of code raises an exception, Python immediately leaves that line and goes to the except block (or terminates the program) meaning that nothing under the line raising the exception gets executed. To get an idea why this is a problem, consider the following script:

def add(x, y):
  try:
    print(x + y)
    print("Add function completed")
  except:
    print("An error occurred")

add(2, 4)
add('hi', 9)

Here, we’ve defined an add function that tries to add its parameters together and print the result to the console, then prints that it has completed. An except block catches any exceptions and prints that an error has occurred. Under the function we call the function twice, once with numbers and once with a string and a number, forcing an exception. Let’s see what happens:

6
Add function completed
An error occurred

Notice that when the exception was raised, we didn’t see Add function completed printed to the console. This is because once the error was raised, the Python interpreter left the try block and went to the except block, leaving the print statement unexecuted. If we want to execute code regardless of if an error is raised or not, we have to use a finally block. The finally keyword is used like so:

def add(x, y):
  try:
    print(x + y)
  except:
    print("An error occurred")
  finally:
    print("Add function completed")

We’ve added a finally block and put the line printing Add function completed in it. This will ensure that this line runs every time the function is called, regardless of whether there is a caught exception. Run the script again and you should see the following output:

6
Add function completed
An error occurred
Add function completed

This time, the message Add function completed prints to the console every time. This is how finally blocks work.

Now we know how to handle exceptions with try, except, and finally. There is one other thing you should know about exception handling before we move on to customized exceptions. Handling exceptions is computationally slow. This is for a variety of reasons that are much too technical to be relevant to this chapter. The main point is that using exception handling to handle errors may be slower than other techniques (like if statements). Exception handling is best used when we don’t want errors to terminate program execution. Otherwise, it’s often better to use if statements or to just let the program execution terminate. With that caution out of the way, lets talk about customized exceptions.  

Customized Exceptions

In the previous sections, we raised two different types of exceptions, ZeroDivisionError and TypeError. These exception classes are just two of many other built-in exception classes in Python. Other exception classes include ImportError, ModuleNotFoundError, KeyError, and many more. In addition to these built-in exception classes, we can also make our own custom exception classes. Customizing exception classes is useful in large projects because it aids in debugging, and it helps you define how errors are handled in your system.

Creating a Custom Exception Class

To create a customized exception, we simply write a class that inherits Python’s Exception class. Write the following:

class MyError(Exception):
  pass

As you can see, writing a custom exception is just a matter of making a subclass of Exception. Now, to raise this exception, we use the raise keyword. Consider the following script:

for i in range(1, 5):
  print(i)
  if i == 3:
    raise MyError

In this script, we make a simple loop that prints its iteration number but raises our customized MyError exception if the number is 3. If you run this script, you should see the following output:

1
2
3
Traceback (most recent call last):
  File "path/to/script.py", line 8, in <module>
    raise MyError
__main__.MyError

Notice that in the output, we see that our custom exception was raised. Suppose we did not know about this script but found a bug report saying “some method is throwing a MyError exception.” Since this is a custom (that is, non-built-in) exception class, we can search our codebase for any line that says raise MyError and figure out what’s going on. When developing large and complex software systems, saving time by searching for customized exceptions is very helpful.

Editing the Exception Message

We can further customize our exception classes by overriding their constructor methods and defining what message they print to the console. For example, suppose we are writing a human resources application for tracking employee salaries. We might have an Employee class like the following:

class Employee:
  def __init__(self, name, salary):
    self.name = name
    self.salary = salary

Here we have a basic Employee class that takes a name and salary attribute in its constructor. Now suppose we wanted to ensure that an employee’s salary is between 20 thousand and 500 thousand and we want to not only raise an exception when someone attempts to make an Employee object with a salary attribute outside of that range, we also want to log such an event to a database. First, we have to create a custom exception, we’ll call it SalaryError. Then, we’ll override its constructor to print why an exception was raised and to log the exception to a database. Check it out:

class SalaryError(Exception):
  def __init__(self, salary):
    self.message = f"Salary must be between 20k and 500k, you put {salary}"
    print("Logging the following to the database:")
    print(f"Attempted to create employee with salary {salary}")
    super().__init__(self.message)

In this code, we create a new class called SalaryError and inherit the Exception class, creating a custom exception. Then, we override the constructor by telling it to expect an input called salary. Then, we set the exception message to inform that a salary attribute must be between 20,000 and 500,000 and put what the attempted salary was.

Next, we print that we are logging the error to the database (in this example, we don’t actually log anything to a database since setting up the necessary connections for that would take away from the focus of this article). Finally, we use super() to fill out the base Exception class’s information so that the class will behave as a standard Python exception.

In order to implement this exception, we’ll tweak the Employee constructor to raise the exception if the salary attribute falls out of the desired range:

class Employee:
  def __init__(self, name, salary):
    if salary < 20000 or salary > 500000:
      raise SalaryError(salary)
    self.name = name
    self.salary = salary

Here, we’ve rewritten the Employee constructor to check if the salary range is within the 20 to 500 thousand range. If it is, we continue with constructing the object; if not, we raise the SalaryError exception. To see how this works, write the following code:

bob = Employee('Bob', 19000)

This code tries to instantiate an Employee object with a salary of 19,000, just below the required range. If you try to run this code, you should see the following output in the console:

Logging the following to the database:
Attempted to create employee with salary 19000
Traceback (most recent call last):
  File "path/to/script.py", line 16, in <module>
    bob = Employee('Bob', 19000)
  File "path/to/script.py", line 11, in __init__
    raise SalaryError(salary)
__main__.SalaryError: Salary must be between 20k and 500k, you put 19000

We can see here that the Employee constructor raised the SalaryError exception and printed out the appropriate error message. It also alerts the user that the error is being logged to the database.

This is the benefit of writing custom exceptions. Not only do we help our future debugging efforts by making exceptions searchable, we also can craft descriptive and helpful error messages. Additionally, custom exceptions let us specify the behavior of our program when encountering errors, allowing us to do things like report errors to a database.

Conclusion

In this article, we improved our debugging and quality assurance skills by learning about exceptions and exception handling in Python. We started by learning about handling exceptions, allowing us to define how the occurrence of certain errors affect the behavior of our programs. Then, we learned how to write our own custom exceptions and how doing so can improve our future troubleshooting efforts. These skills come in handy especially as your software system grows in complexity and usage, since this typically leads to interesting program states that need to be specifically handled.

For more Python-specific information on exceptions, check out chapter 8 in the Python docs.

If you’re trying to get better at Python, try one of my other language-specific tutorials like the delegation/decorator pattern series starting here: Delegate and Decorate in Python: Part 1 – The Delegation Pattern.

The post Python Exception Handling and Customization first appeared on Erik's Code Space.

What is Test-Driven Development?

To put it simply, test-driven development, or TDD, is a programming style in which a component’s tests are the primary mechanism guiding its development–this is often accomplished by writing tests before writing feature code. This is in contrast to the way software is traditionally built, where a feature is built end-to-end and then testing is added later as an afterthought.

There are several reasons for taking a test-first approach to building software. Most importantly, it forces you to think before you start writing code. Pondering how you might test something you’ve not written yet will necessarily make you imagine how you intend to build it, what you want it to do, and how other parts of the code might interact with it. Writing tests first also helps flush out unanswered questions you hadn’t yet thought about (for example, should a math function gracefully handle bad datatype inputs or fail loudly? What kind of data should a function expect as input? And so on).

One practical reason for writing tests before application code is that after a while, you have a fairly comprehensive test suite that reports on the functionality of most—if not all—of your system. This is a lifesaver as your projects get more complex and the possibility of a minor change in one part of the code breaking a feature in another area increases.

Now you may be wondering what it means to “write” a test. In the context of TDD, the word test is almost always shorthand for unit test—a program that checks the functionality of one small piece of functionality (a unit). If that sounds like it means we write code to test code, it’s because we do! In Python, unit tests are classes and methods that we build in order to verify the performance of other classes and methods that we build. Let’s first learn how to write unit tests before diving in to the TDD workflow.

Getting Started with Unit Testing in Python

As their name suggests, unit tests test one unit of code—that could be a function’s return value, the attributes of a class, or even a single unit of functionality like the ability to login to a website. We don’t have to get hung up on the semantics right now, let’s just start writing. Create a file called add_fucntion.py and write this function in it:

def add(x, y):
  return x + y

The above function simply takes two arguments and returns their sum (adds them together). Now, suppose we wanted to test this function to make sure it works. We might open an interactive Python shell and make sure it returns the values we expect based on the values we pass to it:

>>> from add_function import add
>>> add(5, 5)
10
>>> add(4, -2)
2
>>> add(True, "hello")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "path/to/add_function.py", line 2, in add
    return x + y

In the above session, we imported the add function from add_function. We passed 5 and 5 to the function to see if we got 10 as expected; we pass a 4 and a -2 to make sure the function handles signed and unsigned integers appropriately; and finally, we pass the Boolean True and string hello to the function to make sure it throws an error. All inputs behave as expected, so we can be pretty sure the add function behaves as expected.

This is fine for this function since it’s small and simple, but what if we have much larger functions or functions that change a lot? Are we going to manually test every function in a Python shell like this? As you might expect, the answer is no!

This is where unit testing comes in, we’ll write a program that will run all these different lines and more. Python has two popular testing frameworks: pytest and unittest. The unittest module is part of the standard library, meaning we don’t have to do anything special to install it, so we’ll use it for the examples in this chapter.

To get started with unittest, make a file in the same directory as add_function.py and call it test_add_function.py. Inside this new file we first need to import the unittest module and the file with the add function in it. To do that, make the top of the test_add_function.py file look like this:

import unittest

from add_function import add

Now, with the add function and unittest module imported, it’s time to write our first test. With unittest, we start by first defining a test class. Test classes are usually a collection of tests for one conceptual level of functionality; in this case, the add function. Let’s call ours AddTest:

class AddTest(unittest.TestCase):

Notice that the test class inherits the TestCase class from unittest. This gives us access to the assert methods we’ll use in a few minutes. Next, we’ll define a test method. Test methods are what make up test classes and we usually write one for each of the different ways our code might be used. For example, the most likely use of our add function is probably to add positive numbers, so let’s write a test method for that:

class AddTest(unittest.TestCase):
  def test_positive_addition(self):

In the above code, all we did was add the test method test_positive_addition to the AddTest test class. Now we’re ready to write the actual assertions. In this case, we’ll pass the add function two numbers, save its return value in a variable called actual, and compare that with a variable called expected (the number that should be returned). Then, we’ll use the assertEqual method from unittest to test that the two values are the same. Here’s how your entire test_add_funciton.py file should like by now:

import unittest

from add_function import add
class AddTest(unittest.TestCase):
  def test_positive_addition(self):
    expected = 30
    actual = add(10, 20)
    self.assertEqual(actual, expected)

Notice inside of the test method test_positive_addition we create two variables, expected with the value of 30, and actual with the return value of add when passed 10 and 20. Finally, we call the assertEqual class method and pass it actual and expected. As you might guess from its name, this method checks if the two values passed to it are equal.

You’ve now written your first unit test! It’s time to actually run the test. Open a terminal and, from whatever folder your add_function.py and test_add_function.py files are in, run the following command:

python -m unittest

This runs the unittest program which will find any file that starts with the word test and runs the assertions inside. You should see output similar to the following:

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

You might not see it, but the single dot at the top of the terminal output represents a test method that passed. Had the test failed, you would have seen an F. This segues nicely into an important point: you should always make sure your tests can fail. We actually never saw our test fail so how can we be 100% certain that our unit test is working?

To make sure our unit test is working the way we expect, let’s change the expected variable to 40 and see what happens. Make this change in your code:

expected = 40

And run python -m unittest in your console once again. You should see something like the following output:

F

==================================================================

FAIL: test_positive_addition (test_add_function.AddTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/path/to/test_add_function.py", line 9, in test_positive_addition
    self.assertEqual(actual, expected)

AssertionError: 30 != 40
----------------------------------------------------------------------

Ran 1 test in 0.001s
FAILED (failures=1)

This is very interesting output, not only do we see that our test is testing the right thing (which we can see by the FAILED message when we tried to see if 20 + 10 could equal 40), but we also see what it looks like when a test fails. Notice that the output includes the file name and test method that failed as well as the line number of the assertion that caused the failure. This is super helpful when running big test suites (collections of automated tests like this one) with hundreds or thousands of tests.

Notice too that the failure message also tells us what failed. Specifically, it tells us that 30 != 40, which is exactly what we put into the test to force a failure. Go ahead and change the expected variable back to 30.

Now that we know how to write unit tests, let’s write a couple more. First, let’s test the method’s ability to handle negative numbers. We’ll also write this test specifically to fail. Check it out:

def test_negative_addition(self):
    expected = 0
    actual = add(10, -20)
    self.assertEqual(actual, expected)

Here we’ve defined another test method under the AddTest test class. This time, we are using negative numbers to see if the add function handles numbers lower than zero. We’ll test this by passing 10 and -20 to the add function, but instead of setting the expected variable to -10, we set it to 0. We do this because we want to make sure the unit test we just wrote is testing the right thing, and the best way to do that is by forcing an assertion failure. Once again, go ahead and run python -m unittest in your terminal, you should see something like the following:

F.

==================================================================
FAIL: test_negative_addition (test_add_function.AddTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "path/to/test_add_function.py", line 14, in test_negative_addition
    self.assertEqual(actual, expected)

AssertionError: -10 != 0
----------------------------------------------------------------------

Ran 2 tests in 0.001s
FAILED (failures=1)

 Notice this time that the first line of output has a . followed by an F. This means that one of the test methods failed while another passed. As we hoped, the rest of the output tells us that our test_negative_addition method failed because -10 does not equal 0. Go ahead and change the expected variable to -10 so that the tests will pass.

Let’s write a slightly different test now. In the previous examples, we asserted two values were equal. This is of course an important kind of test to run but that’s not the only thing functions ever do. Remember earlier we tested what would happen by calling the add function by passing it True and the string hello and we saw that it raised a TypeError? We can actually test for that behavior too, check it out:

def test_bad_datatypes(self):
    with self.assertRaises(TypeError):
      add(True, "hello")

In the above test method, we start by using the with keyword in combination with the unittest assert method assertRaises and pass it TypeError. Finally, inside the with block, we call the add function and pass True and the string hello. We have to use the with block because otherwise the method will error (even though it’s the error we’re testing for).

If you run this, it should pass. For the sake of time, I did not show you how to make this test intentionally fail. I leave that as an exercise to the reader.

Aside from assertTrue and assertRaises, unittest has many other assertion methods such as assertAlmostEqual for approximating, assertIn for checking the presence of a single element in a list, assertFalse for checking things you expect to be untrue, and many more.

Now that we’ve learned the basics of writing unit tests in Python with unittest, let’s talk about the flow of TDD.

The TDD Flow

As I mentioned at the beginning of this article, test-driven development often consists of writing tests before writing actual code, but that’s just the beginning. The TDD workflow follows a three-phase cycle: write a failing test, write just enough code to make that test pass, refactor. You repeat this cycle until the code is doing what it’s meant to do, is thoroughly tested, and is as well designed as it can be. To understand the TDD workflow, we’ll start with analyzing the requirements and then repeating the cycle a couple of times.

The Requirements

Imagine we are building a cash register application for our favorite coffee shop. This application currently consists of a Drink class, representing the many different drinks the coffee shop serves, an Order class that records a customer’s orders, and an OrderItem class representing an individual item in an order. These three simple classes are in a file called coffee_cash_register.py and look like this:

class Drink:
  def __init__(self, name, price):
    self.name = name
    self.price = price

class OrderItem:
  def __init__(self, drink):
    self.drink = drink

class Order:
  def __init__(self, items):
    self.items = items

  def add_item(self, item):
    self.items.append(item)

The above class hierarchy consists of a Drink class which contains just a constructor and name and price attributes, an OrderItem class which simply contains a drink attribute, and an Order class that contains a list of OrderItem instances and a method for adding more OrderItem objects to that list.

We now want to build a class called CashRegister that serves as an way to access the Order class and allows an employee to create orders and build receipts. Since we are doing this the TDD way, our first step is to write a failing test for the CashRegister class.

Write a Failing Test

To write our first test for CashRegister, we first have to think about what kind of attributes and features the CashRegister class should have. Let’s start by thinking about what a real-world cash register does. It allows an employee to build an order by adding items a customer wants to purchase or removing items if one was accidentally added to the order. It should also be able to calculate the total price for the entire order, so the employee knows how much to charge the customer.

This thought process has revealed three things the CashRegister class must be able to do, and thus three test methods. Create a file called test_cash_register.py and write the following code:

import unittest

from coffee_cash_register import *

class CashRegisterTest(unittest.TestCase):
  def test_add_item_to_order(self):
    cr = CashRegister()

  def test_remove_item_from_order(self):
    cr = CashRegister()

  def test_calculate_total(self):
    cr = CashRegister()

Now remember, you want to write just enough of the test that the test should fail. We currently have three test methods, each creating a new instance of the CashRegister class—a class we haven’t even created yet. Run python -m unittest in the directory you’ve created these files and see that you have three failures with messages like name CashRegister is not defined. We’ve successfully completed the first part of our TDD cycle, writing a failing test.

Get the Test to Pass

Now that we have our test class and three test methods created and failing, it’s time to get them to pass. Since our tests failed because CashRegister wasn’t defined, let’s go ahead and create that class. In coffee_cash_register.py, add the following class:

class CashRegister:
  def __init__(self):
    pass

Here, we’ve created the CashRegister class with a very basic constructor. If you run the tests again, you’ll see that they pass. Believe it or not, that means we’re actually done with the second part of the TDD cycle: getting the test to pass. Let’s move on to the next step.

Refactor

Before we return to the start of the TDD cycle, we need to refactor the code we’ve written. This includes both the test code and the application code; but right at this moment in our particular example, the only thing that needs refactoring is our test code. Notice how we instantiated a CashRegister object at the beginning of each test method. In order to save us from retyping the same line for every test method, let’s just make cr a class attribute:

class CashRegisterTest(unittest.TestCase):
  cr = CashRegister()

 def test_add_item_to_order(self):
    pass

  def test_remove_item_from_order(self):
    pass

  def test_calculate_total(self):
    pass

Notice that we defined an attribute for the test class called cr and instantiated it with CashRegister(). Now we don’t have to write this line at the beginning of every test method. Notice also that we replaced the line inside each test method with pass, this is simply because since we no longer need to instantiate a CashRegister object, there is nothing to write in the test methods. If we leave them empty, our subsequent test runs will fail.

Now, let’s repeat the TDD cycle a couple more times to really get a feel for how this works.

Repeat

We’ve now completed one iteration of the TDD cycle. The next step is to write another failing test. Actually, we’ll just add to one of our existing tests and get it to fail. Let’s write the test code for the test method test_add_item_to_order. Again, we have to start by thinking about how this might work. We decide that adding an item to an order will most likely be done by creating a method in CashRegister that takes an instance of Drink, turns it into an OrderItem, and adds it to an Order object’s items attribute. We write the test method as such:

def test_add_item_to_order(self):
    drink = Drink("Latte", 3.49)
    self.cr.add_item_to_order(drink)  

Now, if we run the tests, we should see a failure from this test method because the CashRegister object has no attribute add_item_to_order. The next step in the cycle is to get the test to pass, which right now means building that method for the CashRegister class. We might start by writing a method in CashRegister that would look something like:

def add_item_to_order(self, drink):
    item = OrderItem(drink)

But now that we’ve written this line and are wondering how to add it to an order, we realize that neither the method nor the class actually has an Order object to add the OrderItem to. Now we have to think about how we plan to instantiate the idea of an Order into our CashRegister app. Do we make it a class attribute of the CashRegister class? Do we make CashRegister inherit Order? Neither of these approaches sound quite right since a cash register in real life is mostly independent from the orders it builds. We decide instead to pass an instance of Order to the add_item_to_order method. Check it out:

def add_item_to_order(self, drink, order):
    item = OrderItem(drink)

In the method above, we’ve updated the argument list to take in an order parameter—an instance of the Order class. This will allow us to add the OrderItem instance to an actual Order instance. We also need to update our test code to account for the new method parameter:

def test_add_item_to_order(self):
    drink = Drink("Latte", 3.49)
    order = Order([])
    self.cr.add_item_to_order(drink, order)

Now that we’ve instantiated an empty Order object, we can call the add_item_to_order method without breaking anything. This means we’ve completed the “get test to pass” portion of the TDD cycle once again! The next step should be to refactor the code but, in this case, there is not really much to refactor, so we start the cycle over once again.

We’re now starting iteration three of the TDD cycle and once again, our first task is to get the test to fail. The best way to accomplish that at this point is to write our first assertion. Since the add_item_to_order method is supposed to add an Item object to an Order object’s items attribute, and since we are instantiating an Order object with an empty items attribute in our test, we can assert that this method is working as intended by checking that the length of order.items is greater than 0. Check it out:

def test_add_item_to_order(self):
    drink = Drink("Latte", 3.49)
    order = Order([])
    self.cr.add_item_to_order(drink, order)
    self.assertTrue(len(order.items) > 0)

Notice the line we added, self.assertTrue(len(order.items) > 0). We figure if the add_item_to_order method is working like it should, the order.items list will have one item in it. If we run this test, it should fail with a message like False is not True. This fails because, in our application code, we never add the OrderItem object to the Order object’s items attribute. We’re again done with the first part of the TDD cycle. Let’s complete the second part of the TDD cycle by fixing the add_item_to_order method:

def add_item_to_order(self, drink, order):
    item = OrderItem(drink)
    order.add_item(item)

We make this test pass by adding a call to the Order object’s add_item method which adds the item variable to the order variable’s items attribute. If you run the tests again, you should see that this one now passes. It’s time for the refactor stage.

Now that we think about it, the test we wrote is a little weak. The length of the order variable’s items attribute is only a side effect of what we really wanted to do which was add a LineItem containing the drink variable to the order variable’s items attribute. By only testing the length of items, the add_item_to_order method could add anything at all to the items attribute and our test would say that it’s passing. For the refactor stage, let’s be a bit more explicit about what we’re testing:

def test_add_item_to_order(self):
    drink = Drink("Latte", 3.49)
    order = Order([])
    self.cr.add_item_to_order(drink, order)
    item = order.items[0]
    self.assertIsInstance(item, OrderItem)
    self.assertEqual(item.drink, drink)

We’ve rewritten the test method to be more explicit. First, we extract the first object out of the order variable’s items attribute after calling the add_item_to_order method. We expect that the item should be an OrderItem object, so we use the unittest assertIsInstance method to make sure it is. Then, we use the assertEqual method to make sure that the item variable’s drink attribute is the Drink instance we created earlier in the test. This is a much more explicit test of the functionality we want because not only will it tell us if the method incorrectly created the OrderItem, it will let us know if we somehow accidentally appended the wrong kind of object to the Order instance’s items list.

It looks like this part of the class is pretty thoroughly tested now and we’re confident that the CashRegister class’s add_item_to_order method works as expected. You’ve now completed your first TDD workflow! Practice your understanding of this flow by repeating the steps for the other two test methods. I’ll give you a hint, to write a failing test for test_remove_item_from_order, you’ll probably have to call a method you haven’t written yet.

Conclusion

This article gently introduces the world of software testing by exploring the concept of test-driven development (TDD). We started by defining what TDD is and why it’s helpful. We highlighted the fact that TDD not only helps us think about the software we build, but it also has the side effect of leaving us with a collection of tests that we can always check our software against. Then, we took a quick detour to talk about how to use Python’s unittest testing framework so we could learn what it means to “write” tests. We did this so we could practice the TDD workflow with an example. In our example we ran through the TDD cycle three times to fully build out one thorough test method for our CashRegister class.

To learn more about TDD, I suggest you start trying to use the flow in your daily programming work. At first, you will likely be less productive than you are used to, but as you practice more, you may be more productive than you’ve ever been! If you are primarily a Python programmer, I suggest you install pytest and learn how to use that instead of unittest. The pytest package is much more popular in Python projects, we started with unittest so we could skip the tedium of installing a new package.

The post Test-Driven Development with Python: a Primer first appeared on Erik's Code Space.

Introduction to Joining Tables

In SQL, we use JOIN in our queries to create result sets made up of multiple tables. For example, it’s common to have countries and states tables when capturing address information, so an address record may look something like this:

Here, the numbers in the country_id column of the addresses table refer to the IDs of their respective country in the countries table, which may look like this:

Now, say we want to write a query that displays a full address, something like 123 Main St, Houston, TX United States. Since the street, city, and state names are recorded in the addresses table, and the country name is recorded in the countries table, we have to join the two tables to get the result set that we want. The relevant SQL would probably look something like this:

SELECT
    a.street_address,
    a.city,
    a.state_or_province,
    c.country_name
FROM
    addresses a
JOIN
    countries c
ON
    a.country_id = c.id

The SQL code above shows us something like the following results:

Notice here that we have values from the street_address, city, and state_or_province columns of the addresses table, and values from the country_name column of the countries table. We were able to join these two tables together by matching the country_id value in addresses with the id column of countries.

There are a few kinds of joins in SQL, each one behaving slightly differently. This article aims to help you understand the difference between the different kinds of joins in SQL.

Download and Build the Sample Database

If you’d like to follow along with this article, go ahead and clone the code to build the example database . To build the database in pgAdmin, first run the command in create_database.sql then make sure to change connections to the newly created employee_database database and run the commands in build_and_seed_tables.sql.

You should now have a database with four tables: addresses, countries, departments, and employees. Let’s quickly discuss the data model before moving on to the actual point of this article.

The addresses and countries tables are just like the ones in the examples above; each address record references a country record. Then, we have an employees table, each record of which references an address record as well as a department record. The department records are pretty simple; the departments table simply has an id and name column and refers to what department an employee works.

Ok, now on to actual joins!

Right vs. Left Joins

Let’s start by talking about the difference left and right joins. It’s helpful to think about all the tables in a SQL query as a horizontal list of tables. The leftmost table is the table in the FROM clause, the next table to the right of the leftmost table is whatever table we are joining. So for example, if we write the query

SELECT * FROM employees e JOIN addresses a ON e.address_id = a.id

Then the employees table is the leftmost table while the addresses table is the next one to the right, kind of like this:

If we take it a step further and join the countries table to the addresses, the next table to the right would be countries. In other words, the following SQL query:

SELECT *
FROM employees e
JOIN addresses a ON e.address_id = a.id
JOIN countries c ON a.country_id = c.id

can be visualized as the following:

Notice again that whenever a table is joined to another, the already-existing table (for lack of a better word) is the left table and the joining table is the right table. So in our example above, our base table is employees so it’s on the left; then we joined addresses to it, so it’s to the right of employees. Finally, we then joined countries to addresses, putting addresses to the right of countries and thus the rightmost table.

This visualization is helpful in understanding how left and right joins work. For example, query all of the employees records and LEFT JOIN the addresses table and see what happens. The query:

SELECT e.first_name, a.street_address, a.city
FROM employees e
LEFT JOIN addresses a ON e.address_id = a.id

The results:

"Bob" "123 Main St" "Houston"
"Jordan" "220 C 30" "Izamal"
"Nicole" "3030 Burgos" "Xalapa"
"Erik" "1010 2nd St" "Omaha"
"Lina" "1010 2nd St" "Omaha"
"Sandra", NULL, NULL

Notice here that all records for employees, the leftmost table, appear as results in this query, including a record with no associated addresses record. The records in the rightmost table, addresses, only appear if they can be joined to the leftmost table. There are records in addresses that cannot be joined to employees because no employees record has a corresponding address_id.

Conversely, if we use RIGHT JOIN to join the addresses table, we will see results that include all of the addresses table–the rightmost table–despite not being able to be joined to the employees table. For example, the query:

SELECT e.first_name, a.street_address, a.city
FROM employees e
RIGHT JOIN addresses a ON e.address_id = a.id

The results:

"Bob" "123 Main St" "Houston"
"Jordan" "220 C 30" "Izamal"
"Nicole" "3030 Burgos" "Xalapa"
"Erik" "1010 2nd St" "Omaha"
"Lina" "1010 2nd St" "Omaha"
null "558 Maple Ave" "Toronto"
null "99 Jackson Rd" "Flin Flon"
null "821 Carol" "Nuuk"

Notice this time that we have three addresses records in the result set that have null values in their respective addresses column. Also notice that the employee record with no associated address record is not in this result set. This is because by doing a RIGHT JOIN, we essentially tell the database to give us all results of the rightmost table, and join only the leftmost table’s records if they are associated with the rightmost.

Conversely, we can switch which table is right and left by referring to the addresses table in the FROM clause and then joining the employees table. For example, let’s SELECT from addresses and LEFT JOIN the employees table. The query:

SELECT e.first_name, a.street_address, a.city
FROM addresses a
LEFT JOIN employees e ON e.address_id = a.id

The result:

"Bob" "123 Main St" "Houston"
"Jordan" "220 C 30" "Izamal"
"Nicole" "3030 Burgos" "Xalapa"
"Erik" "1010 2nd St" "Omaha"
"Lina" "1010 2nd St" "Omaha"
null "558 Maple Ave" "Toronto"
null "99 Jackson Rd" "Flin Flon"
null "821 Carol" "Nuuk"

The results when we LEFT JOIN the employees table to the addresses table are the same as when we RIGHT JOIN the addresses table to the employees table. That’s because in this case, the leftmost table is addresses so by left joining the employees table to it, we are telling the database to give us all the addresses records and then join the employees table records if they exist.

Now, let’s take it a step further and see what happens when we chain JOIN commands together. First, let’s SELECT FROM the addresses table and RIGHT JOIN the countries table so we can see how many of the countries records are not associated to addresses. The query:

SELECT
    a.street_address,
    a.city,
    a.state_or_province,
    c.country_name
FROM addresses a
RIGHT JOIN countries c
ON a.country_id = c.id

The results:

"1010 2nd St"	"Omaha"	"NE"	"United States"
"123 Main St"	"Houston"	"TX"	"United States"
"3030 Burgos"	"Xalapa"	"Ver"	"Mexico"
"220 C 30"	"Izamal"	"Yuc"	"Mexico"
"99 Jackson Rd"	"Flin Flon"	"MB"	"Canada"
"558 Maple Ave"	"Toronto"	"ON"	"Canada"
"821 Carol"	"Nuuk"	"SQ"	"Greenland"
NULL	NULL	NULL	"Iceland"

As we can see from the result set of our previous query, the only country in our database that isn’t related to an address record is Iceland.

Knowing this, let’s see what happens when we SELECT FROM the employees table, LEFT JOIN the addresses table to it, and then RIGHT JOIN the countries table to that. Do you think we’ll see all the countries? Let’s find out! The query:

SELECT
    e.first_name,
    e.last_name,
    a.street_address,
    a.city,
    a.state_or_province,
    c.country_name
FROM employees e
LEFT JOIN addresses a
ON e.address_id = a.id
RIGHT JOIN countries c
ON a.country_id = c.id

The results:

"Lina"	"Mazin"	"1010 2nd St"	"Omaha"	"NE"	"United States"
"Erik"	"Whiting"	"1010 2nd St"	"Omaha"	"NE"	"United States"
"Bob"	"Robertson"	"123 Main St"	"Houston"	"TX"	"United States"
"Nicole"	"Nicholson"	"3030 Burgos"	"Xalapa"	"Ver"	"Mexico"
"Jordan"	"Mays"	"220 C 30"	"Izamal"	"Yuc"	"Mexico"
NULL	NULL	NULL	NULL	NULL	"Canada"
NULL	NULL	NULL	NULL	NULL	"Greenland"
NULL	NULL	NULL	NULL	NULL	"Iceland"

These are interesting results that show us something about the way RIGHT and LEFT joins behave. Notice that we only get 5 rows of results that include records from the addresses table even though there are 7 records in that table. This is because there are only 5 addresses that are associated to employees records, and when we LEFT JOINed the addresses table, we told the database that we only want records from the addresses table if they’re associated with an employee record. However, when we RIGHT JOIN the countries table, we tell the database we want all records from the countries table, even if there’s no associated addresses record in the result set.

Now that we’ve talked all about the difference between RIGHT and LEFT, let’s talk about INNER and OUTER joins.

Inner vs. Outer Joins

The difference between INNER and OUTER joins are very similar to the differences between LEFT and RIGHT joins. However, in this case, it’s more helpful to think of tables as Venn diagrams. You know, the ones where two circles meet in the middle and the overlapping section is some shared attribute between the circles?

We still want to think of tables being joined from left to right, but with INNER joins, we have to think about how records might overlap.

NOTE: The default JOIN operation in Postgres (and every other RDBMS I've ever worked with) is the INNER JOIN. That means if you write something like SELECT * FROM A JOIN B ON A.b_id = B.id, it is implied that you mean to do an inner join and the SQL SELECT * FROM A INNER JOIN B ON A.b_id = B.id is exactly the same thing.

Let’s see how INNER JOIN works when we join addresses to employees. The query:

SELECT e.first_name, a.street_address
FROM employees e
INNER JOIN addresses a
ON e.address_id = a.id

The results:

"Bob"	"123 Main St"
"Jordan"	"220 C 30"
"Nicole"	"3030 Burgos"
"Erik"	"1010 2nd St"
"Lina"	"1010 2nd St"

Notice this time that we have five results, despite there being six employees records in the database. This is unlike the LEFT JOIN from the previous section where the employee record with no associated address record was still in the result set. In this way, INNER JOIN means we only want results from the leftmost table if the rightmost table can be joined to them. That’s why you often see Venn diagrams used to explain joins. In this case, INNER JOIN is this Venn diagram:

So, the main difference between LEFT and INNER joining that we’ve seen so far is that if you want all records of the leftmost table–whether they have associated records in the rightmost table or not–you want to use a LEFT join. If you only want records from the leftmost table if a record from the right most table can be joined to it, you want to use INNER JOIN.

So what about OUTER? The OUTER JOIN is a special case because even though it seems antimonious with INNER JOIN, OUTER joins need to to be specified with either RIGHT, LEFT, or FULL. For example, the following SQL will result in a syntax error:

-- Doesn't work!
SELECT e.first_name
FROM employees e
OUTER JOIN addresses a
ON e.address_id = a.id

We have to specify if we want the leftmost or right most table to be outer-joined. Let’s see how LEFT OUTER JOIN behaves. The SQL:

SELECT
    e.first_name,
    a.street_address
FROM employees e
LEFT OUTER JOIN addresses a
ON e.address_id = a.id

The results:

"Bob"	"123 Main St"
"Jordan"	"220 C 30"
"Nicole"	"3030 Burgos"
"Erik"	"1010 2nd St"
"Lina"	"1010 2nd St"
"Sandra"	NULL

Here we specified that we want the leftmost table to be outer-joined and therefore got all results from employees regardless of whether a record from the addresses table can be joined to it or not.

The RIGHT OUTER JOIN on the other hand will include all records from the rightmost table regardless of whether a record from the leftmost table can be joined to it. The SQL:

SELECT
    e.first_name,
    a.street_address
FROM employees e
RIGHT OUTER JOIN addresses a
ON e.address_id = a.id

The results:

"Bob"	"123 Main St"
"Jordan"	"220 C 30"
"Nicole"	"3030 Burgos"
"Erik"	"1010 2nd St"
"Lina"	"1010 2nd St"
NULL	"558 Maple Ave"
NULL	"99 Jackson Rd"
NULL	"821 Carol"

See here that all the addresses records were included in the result set, even if there was no employees record that could be joined to it. Notice also that the value from employees that has no associated addresses record is not included in the result set.

Did you notice that the results for LEFT OUTER JOIN and RIGHT OUTER JOIN are the exact same results we get from LEFT JOIN and RIGHT JOIN respectively? If so, good on you! LEFT JOIN and LEFT OUTER JOIN are exactly the same thing; same with RIGHT JOIN and RIGHT OUTER JOIN. Whenever you use RIGHT or LEFT in your JOIN, the OUTER is implied and you actually don’t have to write it (though some people do because they say it adds clarity).

There’s one more kind of OUTER join: FULL. The FULL OUTER JOIN will include all results of both the left and rightmost tables, regardless of whether they can be joined to each other. Check it out, the SQL:

SELECT
    e.first_name,
    a.street_address
FROM employees e
FULL OUTER JOIN addresses a
ON e.address_id = a.id

The results:

"Bob"	"123 Main St"
"Jordan"	"220 C 30"
"Nicole"	"3030 Burgos"
"Erik"	"1010 2nd St"
"Lina"	"1010 2nd St"
"Sandra"	NULL
NULL	"558 Maple Ave"
NULL	"99 Jackson Rd"
NULL	"821 Carol"

See how this time we have all records from each table. The leftmost table, employees, includes even the record with no associated addresses record. Likewise, the result set includes three addresses records with no associated employees record. That’s because the FULL join is both a LEFT and RIGHT join put together.

Conclusion

Nearly everyone struggles with the different kinds of joins when they’re first learning SQL, so let’s review what we learned:

• LEFT, RIGHT, and FULL joins are the same as LEFT OUTER, RIGHT OUTER, and FULL OUTER joins
  • LEFT will include all records from the leftmost table, even if there are no records in the rightmost table that can join to them
  • RIGHT will include all records from the rightmost table, even if there are no records in the leftmost table that can join to them
  • FULL includes records from both the rightmost and leftmost tables, even if the records from one table have no joining records in the other
• INNER JOIN is the default join
  • INNER will only include records where both the leftmost and rightmost tables have associated records
  • Use this when you want only full results; records from one table with no associated records in the other table will not show up in the result set

Now that you know how to use joins, try out some of them in the example database using the departments column. As always, feel free to tweet me at @erikwhiting4 or send me an email at [email protected] if you have any questions. Good luck!

The post Understanding INNER, OUTER, LEFT, and RIGHT Joins in SQL first appeared on Erik's Code Space.

Inspiration for this Article

In late 2021, our team decided the daily standups were taking too long. If you don’t know, a daily standup is an Agile ceremony consisting of each member briefly stating what they worked on the day before, what they’ll be working on today, and anything that’s blocking them from getting their work done. Any necessary subsequent conversations are supposed to take place offline.

Our team was was doing great except when it came time to talk about blockers. We would have full-blown conversations, taking deep dives into the code, and often solving the problem for the developer that was stuck.

This was great and everything, but our team consisted of two backend developers, two frontend developers, a tester, and a product owner. Often, only one or two people were interested in the troubleshooting or could even contribute to figuring the problem out. Also, standup isn’t the time for debugging, so we decided to make a concerted effort to keep our standups on task.

However, as a newcomer to the company, I was learning so much from these deep dive sessions; they were instrumental in building familiarity with the codebase, learning how the company “did things”, and picking up new debugging and troubleshooting techniques. I didn’t want to miss out on these invaluable learning sessions, so I proposed a new weekly meeting: the team Rubber Ducking session.

The team rubber ducking session was an instant hit and is still going on 8 months later with no indication of stopping. The rest of this article will talk about some things we learned about making these sessions helpful to everyone.

General Format

The format of a rubber ducking session is pretty simple. Everyone logs in digitally (or meets in a room if that’s your thing) and the facilitator asks if anyone has anything they’d like to rubber duck.

If someone has something, they bring it up. They start by explaining the problem they’re trying to solve, the general approach they’re taking, and why it’s not working. Example:

Facilitator: Does anyone have something they’d like to rubber duck with the team today?
Erik: Yes! I would like some help on this thing I’ve been fighting with.
F: Ok, Erik, go ahead.
E: Ok, so I’m trying to create a pruning feature in the database that copies some data into a table with a temporary name, deleting the original table, and then renaming the temporarily-named table to the original table’s name. I tried doing this by adding the functionality to a rake task, which seems like the right way to go. But my problem is that some of the DDL statements can’t be rolled back in case of an error, I’m not sure what to do.

At this point, I (Erik) would share my screen so everyone can see my code. Then, I ask everyone else if they have any questions and listen to any ideas they might have. In many cases, we’ll start implementing changes right away.

After a while, either the problem is solved, or there’s a pretty obvious path forward that I should run with later after the session. At that point, the facilitator asks if anyone else has anything. If so, you repeat the process. If not, the meeting ends and everyone gets goes back to what they were doing.

Improving Developer Productivity

It seems pretty simple right off the bat, but don’t let that make you think it’s not an incredibly helpful meeting. One obvious way these meetings improve developer productivity is by getting someone unstuck on whatever problem they’re working on.

Another way rubber ducking sessions help with developer productivity is by exposing developers to parts of the code they’re not yet familiar with. This knowledge sharing helps developers see the bigger picture of a codebase and the current sprint. It’s also helpful to watch people fix problems in case you end up confronting a similar problem later; you’ve already seen how to fix it!

Detailed Description of a Rubber Ducking Session

Here’s how a typical rubber ducking session might work in real life. Say our meeting is scheduled for 1pm, right after lunch. I have a problem I’ve been struggling with and want to show the whole team.

I go to the meeting room or log into the zoom/google meeting/teams/whatever. The facilitator asks if anyone wants to rubber duck anything. No one says anything, so I bring up my problem.

I give a quick synopsis of what I’m trying to do and why. Then, I will say what I’ve tried to do so far and talk about what is failing. I will give my own opinions about why my approach isn’t working, but then I’ll let someone else give their opinions.

In many cases, another developer will speak up, asking if they can see this file or that, ask a few questions, and so on. As you collectively explore the problem, at some point, someone will say something that clicks for you. Either they’ve seen a similar problem or they frame the problem in a way that you hadn’t thought about before.

At this point, the rubber ducking session is over for you. You’ve gotten beyond your main roadblock and either have an answer or a new thing to try. It’s time to give someone else the chance to present their problems so the team (and you!) can help them through it.

Dos and Don’ts of Rubber Ducking

If you do decide to introduce a recurring rubber ducking meeting in your org, you should do it in whatever way works best for your team. Having said that though, there are some things I learned about facilitating this meeting that I think are probably universal:

Do invite QA, designers, and product owners. In my experience, people on these teams are interested to see what you do and they can add nuanced opinions that you wouldn’t get from a room full of developers.

Do make the session the same time every week but don’t make the meeting mandatory. The meeting should happen on a schedule, but attendees should be free to come and go as their work permits. This should not be a mandatory meeting!

Do invite people from other teams but don’t prioritize their problems over your own team’s. When we started this, a few developers from unrelated teams asked if they could join. You should let them because it’s nice to have ideas from people with different experiences. However, you must prioritize your team’s issues over another’s.

Do invite staff-level developers or architects. Whatever you call them, the developers in your organization whose knowledge and responsibility transcends a single team. If you can get them to fit your rubber ducking session into their busy schedules, your team will benefit immensely.

Don’t let the meeting get too big. I recently learned that a similar meeting used to be commonplace in the company, but the list of attendees included more than 50 people. I think it’s important for the rubber ducking session to be somewhat of a little personal and informal. With more than 50 attendees, it’s likely many of them aren’t paying attention or are wasting their time by attending.

Room for Improvement?

Like I mentioned earlier, each team will require a slightly unique approach to the rubber ducking session. You’re more than welcome to use my description as a general framework, and I invite you to let me know what you’ve tried. Tell me your own dos and don’ts, your own experiences. Let’s learn from each other!

The post Improve Knowledge Sharing and Productivity with Rubber Ducking first appeared on Erik's Code Space.