Aaron Lelevier

How to calculate effective top tube length

2024-03-30T00:00:00+00:00

Summary

Don’t just rely on mountain bike reach! Effective top tube(ETT) is also an important number for bike size fit. Recently, I was looking at Airdrop Bikes and became interested in the Airdrop Filter, but sizes were limited. This blog is my research on figuring out ETT and if an in stock size would fit.

I have ridden a Commencal Meta 29 with a reach of 460 and seat tube angle(STA) of 76.5.

I have ridden a Chromag Stylus with a reach of 465 and STA of 76.0.

Question, will an Airdrop Filter with a reach of 475 and STA of 78.2 fit? (This is size large and medium is out of stock).

Geometry Numbers

How to calculate

% Functions
DegreesToRadian = fun(Deg) -> math:pi()/180 * Deg end.
Hypotenuse = fun(B, Beta) -> B / math:sin(DegreesToRadian(Beta)) end.

% Parameters
Stack = 614.0.
HeightLow = 600.0.
STALow = 78.1.
HeightHigh = 700.0.
STAHigh = 78.7.
Reach = 475.0.

% seat tube angle(STA) difference
STADiff = abs(STALow - STAHigh).
% percentage difference between high and low STA
PercentHeightDiff = abs(HeightLow - Stack) / abs(HeightLow - HeightHigh).
% Effective STA
ESTA = STADiff * PercentHeightDiff + STALow.

% Answer - effective top tube(ETT)
ETT = Hypotenuse(Stack, ESTA) * math:cos(DegreesToRadian(ESTA)) + Reach.

Conclusion

The Airdrop Filter ETT is shorter than expected. Comparing to a reach of 460 or 465, the 475 reach Filter has the shortest ETT of 603. This is due to the steep STA.

What reading Distributed Algorithms by Nancy Lynch cover-to-cover meant to me

2023-06-19T00:00:00+00:00

Concurrency has always been of great personal interest to me. I purchased Java Concurrency in Practice by Goetz, the book with the trains on it, and would just look at the cover and think of when to start reading it…

Background

I remember reading the Erlang finite state machine (FSM) statem module docs and seeing notation that looked like TLA+. I owned Distributed Algorithms (DA) by Nancy Lynch for 3 years, and I read the first two chapters, and could understand the concept of an FSM for leader election in a ring, but I couldn’t understand the model notation or explanations of theorems, lemma, and so on. I use some concurrency in Python programming with threads, but mostly for fan-out-and-back.

Recently

It is now June 2023. I was driving my kids to school, and the school year was ending, and I told my kids.

“I read Distributed Algorithms by Nancy Lynch and it took me from September (2022) to May (2023). The same as the school year for you guys. It took me a while, but I read the whole book.”

The difference

I later asked myself, what was the difference? How was I able to understand the book and also read it in its entirety? I think that the difference was learning to think on my own at a new level, and loooking up each concept whether it was math with set theory or graph theory, and being able to connect the concepts. It required doing this because the author assumes that you know these concepts. Also, trying to think in the shoes of the author, and what she was trying to tell me.

I have been fortunate to have had a mentor for software engineering early in my career for about a year, but only one. After this time it has been largely through books and reading the learnings and thoughts of smart people that have helped me grow. It takes time to mentor somebody, so it is a special thing when it happens. Anyone can pick up a book though. The cost is relatively small, and it just takes time, and it is like listening to a senior engineer tell you everything they know. A truly special experience.

Here are some of the concepts that the author assumed knowledge of. These lead to leisurly journeys on Google just looking up the concepts from multiple sources, re-reading them, writing them down in my own words, and trying to connect the dots.

infimum, supremum, irreflexive partial ordering
agreement, k-agreement, approximate agreement, 1-valent, bi-valent, uni-valent
tree, spanning tree, forest, spanning forest, minimum spanning tree, strongly and weakly connected graphs
singleton set, multiset, superset, poweset
set union, difference, disjunction, symmetric difference
function domain, codomain, image, range, injective, surjective

Impressions

The DA book is an academic book with so many concepts. If you are in the field of computer science or math, I highly recommend this book to you. I can say that I understood what safety and liveness were, but this book adds so many rich concepts to their understanding. For example, it explains low-level and high-level fairness in the context of mutual exclusion, where low-level is concerned with the user steps, and high-level is concerned with granting of the resource. High-level fairness can be guaranteed in some cases and not low-level fairness.

Other important concepts are:

atomic objects and serialization points
composition of models, where each model is a component
simulation relation of a model A to B where a high-level model can be made to show low-level model properties using successive refinement
impossiblity results - e.g. asynchronous agreement with fault tolerance
upper and lower complexity for both time and messages

After

After reading DA, I re-read Lamport Time Clocks and the meaning was different. The LamportTime algorithm is covered in DA.

I tried learning TLA+ before reading DA, and I couldn’t get to the point of writing my own TLA+ models. After reading DA, I now get it and can write models from DA in TLA+, which was one of my dreams. Here is the mutual exclusion Bakery algorithm for example: Bakery.tla. I wanted to write this model spec in particular because we run a bakery at my work, and I just liked the name of the algorithm, but the bakery at work is for AWS AMI’s so a little different :)

I am going to continue learning distributed systems. Whatever I learn can be distilled down and used for concrete cases in my AWS programming work.

I found this MIT course schedule for distributed systems with a lot of links to research papers. This may be a good next step. I may also read Reliable and Secure Distributed Programming by Cachin.

I have been careful to not tell myself that “I finished the DA book”. I plan to keep referring back to it. Instead I tell myself, “I read the DA book”.

The Fearless Mind by Dr. Craig Manning

2023-03-18T00:00:00+00:00

This blog is a summary of key points from The Fearless Mind by Dr. Craig Manning. This book helped me in my daily personal and professional life, and I hope that sharing the following concepts helps you as well.

The Fearless Mind

The Fearless Mind is a state of being where one is mentally positive, confident, and focused on the task at hand.

Components of a Fearless Mind

First, being confident is key. A confident person believes in their skills and does not let others affect their efforts. Confidence is increased by learning and developing core skills which improves performance on tasks.

A task is an activity required in competition, for example a backswing in tennis.

Focus on the present improves performance by not thinking about past or future outcomes and instead focusing on tasks. This requires concentration.

Focus and concentration require decision making. There are two ways to improve decision making. One is through block practice, the practice of core skills in isolation. The other is random-practice, which is practice that simulates competition.

It is necessary to take calculated risk because it improves focus, and can make the difference when other elements in competition are equal.

Positive thinking and self-talk is a key element because competition has highs and lows, so it is important to remain positive and not let swings affect performance.

General context of the book

Dr. Manning replicated other top school’s physical training programs for his tennis team, but hit a plateau. He wanted to further improve his team’s performance, which led to him focus on the mental aspect of competiion. What he learned became The Fearless Mind.

How did I find out about The Fearless Mind book

I found out about this book through watching the YouTube film 8600FT by Canyon Bicycles. Dr. Manning is in the movie, and Braydon Bringhurst, the star of the film, who is attempting to climb 8600FT of the Whole Enchilada trail in Moab, was one of Dr. Manning’s students. Bringhurst uses the above strategies openly in the film, especially the positive self-talk strategy.

How I came to read the first half of Distributed Algorithms by Nancy Lynch

2023-02-02T00:00:00+00:00

This blog is my story of how I came to read half of Distributed Algorithms by Nancy Lynch. This is a personal story, but also has ideas for how to learn and get unstuck.

The beginning

I first learned of the Distributed Algorithms book from watching the video Distributed Systems Theory for Practical Engineers. In this video, the speaker references 4 books:

Distributed Algorithms by Nancy Lynch
Reliable and Secure Distributed Programming by Christian Cachin
Replication Theory and Practice by Bernadette Charon1.Bost
Guide to Reliable Distributed Systems by Kenneth Birman

Lets refer to these books by number 1-4. The speaker calls book 1 “the bible”. Then says, “if you are going to read any book, read book 2. I went online and picked up the first two books.

Getting stuck

Next followed a series of attempts to read book 1 with each time resulting in the concepts not sinking in:

Try to read book 1
Try to read book 2 before book 1
Learn Erlang by reading Programming Erlang by Joe Armstrong
Create an Erlang project with concurrency named dta
Use Erlang gen_statem behavior
Learned about TLA+ by this video
Learn more about TLA+ by doing the video series
Try to read book 1 again and covert book models to TLA specs

Then came motivation and a good environment

None of the above attempts worked as far as enabling me to start reading book 1 and understand it. I did learn more concepts, but I was not quite there. Next came a series of things that pulled and pushed me in the right direction:

I had a soft tissue injury in my back and needed to take a break from mountain biking, so I had time to do other things
I started physical therapy for my back which made me optimistic
My manager told me “Aaron, you can code all day. What you need to do is learn the way things work”
I stopped drinking a beer or wine with dinner. Even this casual drinking was slowing me down and causing me to lose motivation
My youngest started school, so the mornings were now quiet
TLA+ motivated me, and I knew that I needed more backgound knowledge
I wanted to grow at work while working on my current projects

Then I could understand

September 4, 2023, which happened to be around the same time that my youngest started school, I started reading book 1. I took the time to understand the concepts and research what I didn’t know. I’ve been reading the book in the morning. I am a morning person, and it is a nice start to the day to get the new knowledge and consider things differently.

Now

It is now February 2, 2023, and I am on page 425 of 827 pages, so I am half way.

Conclusion

I think the difference was having enough reasons, enjoying what you are doing, and taking the time to learn.

What Next

I plan to continue reading book 1. I may take a break to write a TLA spec from a model in the book. I am thinking the Peterson2P algorithm, but I am going to wait and see. I don’t want to lose momentum.

Thank you to the authors

I am thankful to the authors for writing these books and sharing them with us. I have had three people in my career who I consider mentors. I learned a lot from them. When you read someone’s book, if it is the right book, it is like having a senior mentor teaching you.

Learning Golang and Network Programming

2022-03-06T00:00:00+00:00

This blog contains notes on mostly NTP Learning in conjunction with reading the ntpc.go module in this Github repo vladimirvivien/go-networking.

Intro

I have been starting to learn Golang and wanted to start learning why Golang is good for Network Programming. After searching, I found this repo vladimirvivien/go-networking. Here are my notes from reading the ntpc.go and ntps.go modules, mostly the former, in this repo.

ntpc.go

First, the Packt book is no longer available, so I thought to read ntpc.go.

Learning about Go, this module starts with:

var host string
flag.StringVar(&host, "e", "us.pool.ntp.org:123", "NTP host")
flag.Parse()

Here flag.StringVar is intialized to the address of a pointer, with the default value of the 3rd arg, func StringVar(p *string, name string, value string, usage string)

Usage:

// uses default
go run ntpc.go

// uses ntp server from ntps.go
go run ntps.go
go run ntpc.go -e localhost:1123

Golang net pkg

The net pkg is used to create a UDP object, establish a connect then send a request and response. I looked up the us.pool.ntp.org IP information to compare to the Golang objects:

$ nslookup us.pool.ntp.org
Server:		192.168.1.1
Address:	192.168.1.1#53

Non-authoritative answer:
Name:	us.pool.ntp.org
Address: 70.35.196.28
Name:	us.pool.ntp.org
Address: 168.61.215.74
Name:	us.pool.ntp.org
Address: 216.229.4.69
Name:	us.pool.ntp.org

Address: 66.151.147.38

The first Server address is our private network DNS address with th

192.168.0.0/16 Address for private networks (intranets). Such addresses never appear on the public Internet.

References

https://timetoolsltd.com/ntp/ntp-client/

How Import to an IAM Role into Cloudformation

2022-01-30T00:00:00+00:00

This is a blog about importing an existing IAM Role into Cloudformation(CFN). An IAM Role is one of the supported resource types that can be imported into CFN. The use case for doing this is that you may have resources that were created via the AWS Console, and you want to move them to version control.

Summary

Here are the docs on how importing resources work in AWS: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import-new-stack.html#resource-import-new-stack-cli

And here are the docs for an example of how to import a resource: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-import.html

I created an example Github repo with working code that follows this process using the boto3. Please refer to this repo for example templates on the process. There is also a Jupyter notebook with documentation on how the created CFN template and IAM Role resource in AWS are updated at different steps of the process: https://github.com/aaronlelevier/import-iam-to-cfn

At the end of this test, I tried deleting the Role via the CFN stack, and the Role in AWS was not deleted and had to be deleted outside of the CFN stack.

One Month of TLA+ Learning Adventure

2022-01-23T00:00:00+00:00

In this blog, I share my experience from learning TLA+ for one month. This blog does not teach TLA+. Instead, I link to resources for those who are interested in learning TLA+. All resources are free as of this blog.

What is TLA+

TLA+ is a specification language, where you write a specification(spec) of a finite state machine(FSM). You denfine the inputs(CONSTANTS), mutable state(VARIABLES), and transitions from one state to another(action expressions). The TLC model checker is then used to run the spec and will investigate all possible actions for all states. One key point is that you write “what can happen”, not “how it happens”. This is why TLA+ is a specification language and not a programming language.

TLA+ is created by Leslie Lamport.

How did I find out about TLA+

I discussed this in my previous blog 2021 EOY Blog - The adventure of learning TLA+.

Where I started

I started learning TLA+ by watching The TLA+ Video Course. Here are 3 highlights that really grabbed me.

The Introduction to TLA+ video. Here Leslie Lamport says, “now is not the time to be modest”, and he goes on to say what he has achieved and why you should listen to him and learn TLA+. He also states how AWS has used TLA+ since 2013 to prove their internal services, and how AWS senior management believes in TLA+.
The Paxos Commit video at 12:30 in the video, Leslie Lamport shows slide 126, and he says, “I don’t know how to write a clearer precise description of this step of the algorithm.” At first, I could not understand this code, but then going back and rereading it later, I understood it, and thought, “wow this is the most concise and elegant way to state the behaivor of this algorithm in TLA+.”
The final video of The TLA+ Video Course. Leslie Lamport closes the video with this quote, “As you go forward, remember to take the time to stop and think. I hope what you’ve learned here will help you do that.”

Where am I at now

I am currently reading the Specifying Systems TLA+ book, and am on page 80. I think that watching the videos first has made the book easier to understand.

I have written my 2nd TLA+ spec RTOP.tla.

RTOP explained

RTOP stands for R-to-P and is a spec that shows a mapping of many-to-many P to R and the inverse.

CONSTANT defines a set of P and a set of R. Everything in TLA+ is a set. These are inputs to the spec.

VARIABLE declares the mutable state of the spec. ptor and rtop are mappings from “P to R” and “R to P”, which is the inverse.

TypeOK declares our type invariants, which states what are the allowed types of our mutable state. Here we define ptor to be a mapping for the domain P, where the value always a subset of R. The inverse is true for rtop.

Init initializes the spec. Here we initialize the ptor mapping for all p in the set of P to a value in the subset of R. rtop is initialized to the inverse.

PAddR(p, r) defines a single Action where r is added to the mapping for a given p.

Next declares that in each transition(step) of the FSM that there should exist some p and some r such that r can be added to p’s mapping.

Spec declares what is run by the TLC model checker. We first initialize our state with Init and then we run the Next Action where all state variables vars are subject to change. The []Next means that Next runs continuously until all states are reached.

THEOREM states that our Spec requires []TypeOK to always be true, so in all states, Init and Next.

How can I run this code

RTOP.tla can be run using the TLA+ Toolbox.

What I learned

Leslie Lamport is 100% correct in the fact that TLA+ will help you think about problems differently. He talks about how creating abstractions is difficult, and you have to learn it over time. TLA+ forces you to think about the abstraction and at what level you want to create it. There are tradeoffs to coarse of fine grained abstractions.

When going back to a programming language, thinking about the design separately from the implementation has helped. The design must be correct, or the implementation has to try and make up for it and will struggle. Getting the design right is a must, which then simplifies the implementation. TLA+ is the design, and that is why already I can tell that TLA+ is super valuable.

What’s Next

I plan to keep reading Specifying Systems. I also want to continue reading Distributed Algorithms by Nancy Lynch at the same time and implement the algorithms from the book in TLA+ to get more practice.

TLA+ References

TLA+ Website

TLA+ Cheatsheet

TLA+ syntax

Unofficial Learn TLA+ website

2021 EOY Blog

2022-01-08T00:00:00+00:00

The year of the learning adventure.

The adventure of reading TCP Illustrated

I started reading TCP Illustrated out of curiosity. I got to page 100 and thought, “this is pretty good.” Then I got to page 200 and thought, “wow, I can’t believe that I made it to page 200.” Then, I had to decide to read the whole book or not. The book is 922 pages. It was a challenge, and I had to pace myself to read 40-50 pages a week, but I did it. The benefits from reading this book weren’t really planned, but I started seeing them right away.

The adventure of learning TLA+

I somehow stumbled upon TLA+ through watching a series of YouTube videos. I first watched Trust but Isolate The future of software which talked about the language Isabelle for formal methods. I did some research and saw a few languages mentioned and some pros/cons of different languages, one of which was TLA+, so I decided to next check this language.

I then watched the video SREcon20 Americas - Weeks of Debugging Can Save You Hours of TLA+ which in my mind connected programming to a specification language for formal methods and was the perfect next step to learning TLA+.

From here, I researched TLA+ and found the website and watched the TLA+ intro videos. I watched all videos, and am now checking out the Examples and TLA+ Specifying Systems book.

Some other highlights

Tried out Github Actions. They work nicely with AWS OIDC auth.

Coded with my son and daughter for the first time. This was quite funny. My daughter was giggling because she was having so much fun. My son hung in there for a while, but then got bored and said, “this is boring. How do you turn this off.”

What’s next for 2022

I am continuing to learn TLA+. I’m currently reading the TLA+ Specifying Systems book. I also started reading Distributed Algorithms by Nancy Lynch again. This has new meaning after watching the TLA+ videos. I plan to read both at the same time.

I still want to learn Golang, but I don’t want to pivot just yet.

Thoughts on the InfoQ talk Netflix Networking Beating the Speed of Light with Intelligent Request Routing

2021-10-31T00:00:00+00:00

This blog is about my thoughts on the InfoQ talk: Netflix Networking: Beating the Speed of Light with Intelligent Request Routing₁. This is a top notch talk. Even if you are not into networking, you can learn something from the methodologies discussed in this talk. After listening to this talk, I was like “wow!”, and so I just want to share some thoughts on the talk as they relate to myself and other learning.

Summary

The talk is 38 minutes long including Q&A, so let me first summarize the talk. The talk is about improving Netflix streaming speeds using networking by intelligently either connecting to the Cloud or a CDN endpoint. This is done by using an agent installed on Netflix clients to measure and compare DNS Resolver response times, and then this data is aggregated and analyzed, and DNS is updated to direct the client to either the Cloud or CDN, whichever is faster.

Thoughts

I listened to this talk while driving, and did not see all the diagrams, so my summary may be overly vague for certain points. With that said, what really impressed me about this talk is that the speakers used their knowledge of networking and the empirical process to make hypotheses, prototype, analyze results, and then repeat.

I am currently reading the book TCP Illustrated₂ and am about two thirds done. Having some of the networking knowledge from the book, I was impressed how they use Anycast addresses₃ networking for the CDN to provide the best service based upon location.

They also discussed how they have about 200k DNS Resolver₄ IPs that they store when they build the DNS map that is used for the intelligent routing, and this data pipeline refreshes every 10 minutes. This was a pretty interesting statistic after reading the DNS chapter in TCP Illustrated₂, also because Internet routes change over time, so the mapping must be updated on some interval.

Building in a failure mode to the service. This was a particularly interesting point that the service will fall back to the default if it fails. They noted how this is critical for the service to be resilient and make the change on such a large scale.

Beyond Networking

The part in the talk about time to integrate their service and make it to production was also super interesting. They did experiments in production to analyze traffic before their service went live, then did Canary tests, A/B tests, and progressive rollouts.

Conclusion

I recommend this talk. Even the Q&A was really good.

References

[1] - Netflix Networking: Beating the Speed of Light with Intelligent Request Routing

[2] - TCP Illustrated

[3] - Anycast Address Networking

[4] - DNS Resolver

What I Learned from 1 Year of AWS Custom Resource

2021-10-24T00:00:00+00:00

The title of this article could have also been, “Using AWS Custom Resource’s.” or maybe “AWS Custom Resource Gotchas and how to avoid them.” Either way, the goal of this article is to talk about usage and gotchas. With that said, this article is meant to be in addition to the AWS Official Documentation, linked below the References. I will highlight a few important aspects of AWS Custom Resources with the way that they work.

What is an ACR

An AWS Custom Resource₁ (ACR) is a mechanism in AWS Cloudformation (CFN) that allows an arbitray payload to be sent to a compute function.

There are two ACR types:

AWS Lambda - sync - must always return a response (pass or fail)
AWS SQS Queue - async

This article covers AWS Lambda as this is the common case.

ACRs have a Request₂ and Response₃ structure.

The first thing you learn

The first major lesson of AWS Lambda ACRs is that they must always return a Response. If an ACR errors and fails to send a Response, the CloudFormation (CFN) parent stack invoking the ACR will spin for 1 hour, and eventually timeout, and fail.

The ACR Error Kernel

The ACR always sending a Response could be descibed as the “Error Kernel”, which is a concept that Joe Armstrong₁₁ introduces in his talk “The Do’s and Don’ts of Error Handling”₁₂

How to always send a Response

In order to achieve always sending a Response, you should wrap the invocation of your code in a top level try/catch where all errors be caught and send a ACR Response of fail

You could wrap your code in the following:

try:
  f(event)
  success
catch:
  fail

Long Responses

There may be some cases where a long response is generated by your code. You must truncate your response to a max size of 4096 bytes per the current docs, or else the CFN stack will fail with “Custom Resource Response Too Long”, and your CFN stack deploy will fail with no valuable error message from the custom resource.

Success case

In the success case, ACR Response data can be returned, which can be an arbitrary single level key/value map.

This data is then referenceable using the CFN intrisic function: ``Fn::Getatt`₈

Fail case

In the failure case, an Error Reason and Error Message can be sent.

This will result in a ROLLBACK by your parent CFN stack.

ACR Request

An ACR Request consists of the ACR endpoint, e.g. the AWS Lambda ARN, optional key/value arguments, and metadata.

The Lambda ARN

This payload will be sent as an Event to the Lambda ARN.

Optional Key/Value arguments

These arguments will appear as top level key/value arguments in the ResourceProperties section of the Lambda Event.

The value of these arguments can be a nested data structure.

Metadata

AWS metadata is also sent in the ACR Request. This will include the parent CFN Stack Id, PhysicalResourceId, and so on.

The PhysicalResourceId

This value can be updated in order to trigger certain desired behavior. First CFN will process a RequestType of UPDATE, based on the new PhysicalResourceId in the Event. Then CFN sends an Event with a RequestType of DELETE with the old PhysicalResourceId and previous Event. This is sent after the UPDATE has succeded during the UPDATE_COMPLETE_CLEANUP_IN_PROGRESS phase.

ACR RequestType

There are 3 request types, and these match based on what the parent CFN stack is doing:

CREATE
UPDATE
DELETE

Deploy it

An ACR, like any other AWS Resource can be deployed with CFN₄, AWS SAM₅, or some other deployment method.

Replace an ACR

An ACR can NOT be replaced via tearing down and rebuilding if it is referenced in another CFN Stack, regardless of referencing the ACR directly or as variable using an SSM Parameter₉ or CFN ImportValue₁₀. You must solve for this.

Rollback

Rollback by default will use the same AWS Lambda code. This will be a problem if the Lambda suffers a bad update, e.g. a coding error, as the ACR will fail, then on rollback, if the same code path is used, rollback will also fail. You must solve for this as well.

Some strategies for this could be to use a stable version of the Lambda on rollback, or to always roll forward, and so on.

Finally

Okay, now that I know everything about ACRs, can I go write some code?

Yes!

Thank you.

References

[1] - AWS Custom Resource

[2] - AWS Custom resource request objects

[3] - AWS Custom resource response objects

[4] - AWS::CloudFormation::CustomResource

[5] - AWS Serverless Application Model (SAM)

[6] - AWS::Lambda::Function

[7] - AWS::Lambda::LayerVersion

[8] - Fn::GetAtt

[9] - AWS::SSM::Parameter

[10] - Fn::ImportValue

[11] - Joe Armstrong

[12] - The Do’s and Don’ts of Error Handling

How to create Golang code snippets in VSCode

2021-09-06T00:00:00+00:00

Here is how to create Golang code snippets in VSCode.

Steps

Go to Preferences > User Snippets
Select New Global Snippets File
Name the file go.json
Start with existing snippets like: go.json

Now validate

Test one of the existing snippets
Add a new snippet
Test new snippet

Value Objects

2021-07-10T00:00:00+00:00

Hello! This blog discusses the pattern of value objects as I use them, why they are useful, and some real world use cases.

Hello, my name is Aaron Lelevier, and I hope that you enjoy the blog and come away with a new tool (design pattern) in your tool chest.

What is a Value Object

Let’s start with a definition. A value object (VO) is an object that stores some data that is logically grouped together. It can have data attributes but not behavior.

The VO can represent an object in the business domain, concept, or rule(s).

The VO should be properly constructed, so it recieves all arguments when it is initialized and does not call separate methods at a later time to mutate itself.

The VO should be read only. It can potentially make API reads to load data, possibly lazy load, and this should be idempotent.

The VO can read by other objects that understand its type (interface).

The VO should not have methods for API write calls or writing to a database. After all, it is a value.

Proper Construction

The Proper Construction concept is powerful because it allows an object to be immediately useful upon initialization. I have worked in codebases where objects have to call 2 or 3 methods before they have all of their data. Proper Construction removes that.

Some programming languages, use immutable objects, for example Erlang. In these languages the new object needs to be assigned to a different variable. Immutability removes the mental overhead of needing to confirm if an object has been mutated, or worse reassigned.

Value Objects are for Reading

The VO can be a business object, a concept, rule, etc… but it should be a noun. The VO can then be read by other objects in the system, like those that do API writes.

Program to an interface not an implementation

The VO should have a Type (Interface). This will allow creation of an object with the same interface in different circumstances that are then understood by other objects in the system.

Value Objects and Eliminators

An Eliminator, from the book The Little Typer, is a function that takes a value as an argument. An Eliminator then must understand the value’s type. This pattern scales well if the VO’s complies to an interface, and then the Eliminator can be programmed to work with that interface.

CQRS

VO’s and Eliminators also natrually reinforce CQRS or Command Query Resource Separation, which is the pattern of separating code for reading and writing. The VO’s should only read data and the Eliminators, not all but some, are then used for API writes.

Real World use case for Value Objects

Here is a real world example. Let’s say we have a HR system that is responsible for storing employee information.

The first time an employee is hired, the VO in the system requires all their information to be properly constructed. The VO’s type is Employee. The Empolyee VO is then passed to an Eliminator that handles database writes, and the new employee is added to the HR system. We have followed CQRS.

Then, the employee is granted access AWS. We will need their info to grant them access in AWS. Since all of their information is in the HR system, we only need a unique identifier, like their work email, and we can construct an Employee VO of the same type by querying the database. The Employee VO is then used to create as a User in AWS and is provisioned via Cloudformation, which handles writing and updating resources in AWS. We have followed CQRS again, and this time the API writes were done by Cloudformation.

Conclusion

Thank you for reading. I hope that you can find the value object pattern as useful as I have. Cheers!

One Year of Hobby Coding Erlang

2020-05-17T00:00:00+00:00

I am Celebrating 1 year of hobby coding Erlang as of May 15, 2020. I was talking with my wife and a couple of things stood out about reaching this milestone. There’s some important things that I learned, and some of these have been outside of programming. There are also some bridges that I have yet to cross and next steps that I’d like to list.

How Erlang compares to other languages that I’ve surveyed

The language that I code in every day is Python, but I try to make a point to learn other languages. Besides SQL, which is the first language that I learned, and used at work for 3 years before getting into Python, I have used a few other languages, but I have never crossed the 1 year mark.

Java

I used for 10 months with Android, and eventually quit when I realized that I’d have to do UI design if I wanted to build an app. I quit around the time that Android announced support for Kotlin. I think what first got me into Java was reading Test Driven Development by Kent Beck, which is in Java, and then coding up the examples, and I wanted to then learn the language.

Swift

Prior to learning Java, I spent 8 months using Swift. I mostly did online courses, and never really made anything other than the demo apps, and one simple SpriteKit game that my daughter was the only one that played. It was about a bunny that hopped to collect coins, and I purchased the images from GraphicRiver.

C++

My wife laughed when I told her about this one. I spent 2 weeks learning C++. I bought the book Accelerated C++, and was driven to learn C++ because machine learning frameworks use C++ under the hood, and the developer API is in Python, so I thought that I should learn it, but, it only lasted 2 weeks.

Objective-C

This predates all the above languages, and was after I learned Python. I think that I spent 2 months with Objective-C, and did one book, but forgot all of it.

Erlang

Now Erlang. I had bought Joe Armstrong’s book, Programming Erlang 2nd ed. because I wanted to learn Erlang for a while, but then held onto the book for 6 months and didn’t read it. Then in May of 2019, I learned about Joe Armstrong’s unexpected passing, and felt compelled to read his book.

The book is really good. It covers the whole language and Erlang OTP. One of my favorite parts of the book is in the chapter on Maps. I tried to code up the examples, and the code wouldn’t compile. I started googling to see what I did wrong, and I guess Joe had included experimental syntax in the book, that didn’t get accepted by the Erlang language board. I could feel Joe’s personality when I read the book.

So then what happened?

More than just one Erlang book

In July of 2019, I had the pleasure of attending Code Elixir LDN 2019. This was the first time that I had been exposed to the Erlang/Elixir community.

Some of the ideas were very different. At the conference, I listened to some of the best talks that I have ever heard, fresh with ideas that were new to me.

I attended the happy hour after and got to hang out with the speakers. Then the founder of Erlang Solutions, Francesco Cesarini, approached me and asked me how I liked the conference. I told him about my Erlang journey, and how I was half way through the Programming Erlang book. Francesco said:

That’s good. You’re reading Joe’s book. After your done with his book, you should read my book.

I listened to what he said, but let me get to that in a minute.

So in November/December 2019, I finished the Programming Erlang book and thought that I was ready to build software using Erlang. I started a project using Erlang Cowboy, a “Small, fast, modern HTTP server for Erlang/OTP”, and got stuck, so I contacted the creator of Cowboy, Loïc Hoguin, for consulting. I booked 1 hour, and thought that it would be worth it if I could make this software idea that I had. We ended up working together for 30 minutes, and it was pretty obvious to me that I wasn’t ready. We exchanged a few emails after that, and I told Loïc that I wasn’t sure if I needed the other 30 minutes of consulting because I needed to learn Erlang first. He told me:

Glad you’re still enjoying Erlang! It is indeed pretty pretty big.

After that, I quit the software project and started reading Francesco’s book Erlang Programming. This time taking my time to let the language and concepts soak in, and not force myself to learn it with the goal of building something, as I did with the first book.

Where am I at today

As of today, I’m on pg. 290, ch-12 of Erlang Programming, and have gotten back to Erlang OTP. I am taking my time to learn more about Erlang, distributed programming and functional programming.

I have mostly been coding gen_server’s at this point, and some one or two gen_statem. I am about to add supervisor’s to the mix.

I started reading the book Distributed Algorithms by Nancy Lynch in conjunction with Erlang Programming by Francesco Cesarini. Today, I finished coding the Leader Election LCR algorithm in Erlang. Here is the commit tag, so I know what happened later!

I am also reading The Litter Typer. This book is about dependent types. It was recommended by one of the speakers at Code Elixir LDN 2019. I’m enjoying it so far, and it should be valuable to go along with the other learning.

I am learning more about functional programming (FP). After using Erlang OTP more, and some of the behaviors, I plan to spend more time on FP.

What has been really cool

Erlang and FP in general makes you think differently.

I was explaining Erlang to a friend and relating it to Python and Object Oriented Programming (OOP), and I said:

In Erlang each process is like an object instance, but those objects are all running processes, and they communicate concurrently.

If that’s not cool, I don’t know what is.

When coding in Erlang, you have to think about concurrency, and if something is timing related. I have written code with tests, where the test needs to wait for a “done” message from a certain process before proceeding. What you end up with is a partially synchronous system, which makes up most systems.

Erlang has all of the primitives for a highly concurrent fault tolerant system, and being able to build stuff with these primitives is a lot of fun.

What’s next

Finish Francesco Cesarini’s book, Erlang Programming.

Fully learn Erlang OTP and all the behaviors.

Write a simple Erlang App using OTP.

I want to learn how to use the Erlang debugger, and not just log statements.

Ourside of Erlang, I want to continue to read the Distributed Algorithms’s book. I first put “finish”, but this is a big book!

For FP, I want to learn more concepts and about functional design patterns.

Conclusion

I am definitely not where I thought that I would be. I thought that I would be building full fledged Erlang applications by now, but I am okay with that. I am still having fun, and don’t plan to on changing hobby languages quite yet. For a minute, the thought entered my mind, “how about trying Haskell.” But, then I thought about all of the time that I’ve spent with Erlang learning the primitives, and it’s better to stick with it.

A RDBMS for AWS using Django and Erlang

2019-12-01T00:00:00+00:00

I have a dream for a tech stack where Python and Erlang live together. I feel like I took the first step towards that today. Here is an overview of a code spike for creating a Postgres DB with data from AWS and reading from the same Postgres DB using Erlang.

The Stack

The stack consists of technologies that I currently use or have used in the past. It’s familiar. So far, the stack consists of:

Python3
Django
PostgerSQL
Erlang

Let’s see it

This is best demostrated by examples, so I’m going to refer to the Github Repo’s and READMEs.

Django/Python

Here is the Django/Python README: https://github.com/aaronlelevier/djangoaws.

This code can be used to create a PostgreSQL database, migrate the schema, and populate it with data from AWS EC2.

Erlang

Then here is example code for connecting to the same database setup and populated by Django, but reading from it using Erlang: https://github.com/aaronlelevier/motivatedcoder

Conclusion

This is a code spike. The code could be extend for more AWS resources, or connected to other parts of tech stack.

Code Elixir LDN 2019

2019-07-21T00:00:00+00:00

Interesting thing. Do you know that if you go to the conference website, this blog uses the same slug? Here’s the link. Or you can trust that the link is the same as the href text below. https://codesync.global/conferences/code-elixir-ldn-2019/

Code Elixir London

This blog post is from my conference notes. I took 10 pages of notes. The notes however got soaked by rain inside my backpack because it was raining so hard the next day and I cycled to Richmond Park. They have since dried up pretty good though. Here are notes. They are not in exact cronological order of the speeches because when I dried out the pages the order was messed up.

Annotated notes may be part 2. For now it was just a lot to copy 10 pages of notes, fixing spelling and typos errors and so on.

Conference Notes

Here we go :)

Keynote

understand the team level

review project - understand level

flash cards - improves recall

pair programming (for 3 hours) others can view but don’t interupt

teaching - ask for understanding first, listen, then add, not the other way around

flashcards

why? options

positive reinforcement

pair programming - be writing code or code challenge, to understand how to help

mentor - focus on teaching concepts

Performant String Processing in Elixir - work at Castle

Erlang Term Storage (ets) - like Redis for Erlang. read/writes in constant time. Replace map with ets:table

streams - read data as a stream and only split 1x

iolists - can call print 1x instead of per line. List of strings vs. list of references

Thought Process

confirm problem
set goal
create benchmark
iterate on changes
consider tradeoffs

can’t pass a ets:table between processes

can pass a map between processes

binary compile pattern - for pattern matching strings

Unix Interface Design Patterns - book

dtrace - to fully trace system calls

nible csv - csv paring library

Elixir for IoT by James

everything that is a physical IoT thing is a process

have a durable message store

John Hughes - Lambda Days talk Feb 2019 - also IoT related

communication is all using binary

Gattling - load testing for Users

Mocks - testing devices - use mocks for send / receive to IoT device

Basher

Akka - other async language

The Little Typer - book

Nerves - for embedded devices

IoT <—> Cloud <—> Demand Response Predictor

1984 - by George Orwell - book

Type Classes and Programming Vocabulary - Erlang Solutions, Michael

productive / inspiring - did a venn diagram of the two potentially overlapping but said lets think of then not overlapping

Vocabulary

Semigroupoid
Contravarient
Monad
Comonad
Traversible
Monoid
endofactors
functor
Applicative

Type Class - is the interface

Functor - an iterable that a func can be applied to. Should return the same number of items. Traversible

Brookly Zelentak @ espede Elixir

Words

GenServer
Supervisor
Process
Actor
Enum
Stream
Task
GenStage

design patterns

put all impure functions in a module called effects

learn the concepts and words then apply them

Resisting object Oriented Programming - works at MailChimp

hegemony - vocab word - means someone dominating - can be physical or by concept

Gang of Four - design patterns

LU Cache - least recently used Cache

Factories - there are faults

Hold My State

always initialize with a default state

Calling

process calculates and returns a value each time before accepting another message
provides psuedo back pressure - if process has to answer back before accepting more work

Casting

send messages + don’t wait for reply
request values whenever
kill process + clan up if don’t respond after 10 sec for example

Workers

(I missed this part of the talk)

Event Sourcing in Elixir

Everything that occurs is recorded as a Domain Event and saved in Event Store

Process can subscribe to the Event Store

Read Stores - can be created to organize data for use by API, UI, etc…

Language for Producing Types

Coq
Agda

Design

Pure Func - maintainable, can run in parallel

Keyword List - Elixir type, Erlang record type?

Types

umbrella application - allow for plug + play, dependency injection, ubiquitous language, releases, delete ability

Why not umbrella? - if project is small, start with monolith, avoid circular dependencies until business domain is known

Interface programming

Struct for create - structs give compile time guarantee, type checking

Interface	Business Logic
impure functions	pure functions
create ->
	<- struct
struct ->	-0

GenServer

Phoenix

Ecto

Debuggging

UI Testing with Hound - by Vanessa Lee from Interactive Vacations

to do Selenium tests using Elixir

target elements, xpath

use execute scripts - to call Javascript to input text in to a form

take screenshot, move mouse, keystroke

Property Based Testing - book

Can do this testing for checking various different inputs to the same form, func etc…

The Alchemists Code

3-7 years to become a senior engineer

Painting Analogy

work in teams
use templates - frameworks

Phoenix

Absinthe

Separate a Project by Context

Contexts are like a microservice, have their own domain and should be isolated from other Context

Can draw a dependency graph between models

model is aware of another module by importing, sharing a struct, then that’s a dependency

context - could also be though by directory

Should be NO cross context communication

Service -> Model

Service -> IO -> Model

Pyramid, from top down could be thought of:

Service
IO
Model

Example path pattern to use:

mylib/*/(model|io|service)/*

Model - Actors, pure functions

IO - for impure functions, calling anything external

Service - all business logic

Domain Modeling Made Functional - book

Restart GenServer w/ new State

state -> perform -> new state

move impure functions to boundry. This is the:

interface
most public functions
1st function called that calls other functions

purity

types

organization

architecture - event based architecture - all requests sent to broker and distributed from there

aka Brokered approach

confident code

unit test
integration test (don’t mock db)

libs

credo
formatter
dialyzer

smoke tests - selenium

test coverage

property test

debug

tracing

discuss concepts

what - question
why
how

Error Free Elixir

design code w/ error handling in minde

source of errors

users
other services
infrastructure
developers

let it crash

error tuples

Heuristics

reasonable defaults
declarative operations
don’t fight the real word

think idempotent - if get cancel command 2x let the 2nd one pass because the goal to cancel has been achieved so it’s not an error

handle the error correctly for what it is

understand if it is an error, why did it happen

can we recover from the error

A Philosophy of Software Design - John Ousterhout - book

The Art of Destroying Software - Greg Young - talk

Understand real use cases of the application in order to reduce error handling + understand what reall is an error or not

Build a CLI in Elixir

Why CLI

light weight
can utilize OS
infrastructure level features
reproduceability

run app, output data to JSON, can then run JSON as an event command, etc…

Developer Experience - robust, speedy, visually intuitive

Option Parser - Elixir argument parsing module

use colored text in terminal

whne debugging - can reference same script + call with arguments reference secrets for environment variables in prod

don’t reinvent language - use standard commands

stdout / stderror - be able to easily customize logging

Exit Codes - don’t use 0/1 - have an error code per failure type

fail fast

lightening talks

exercism.io - structured series of programming exercises

community - social platform for sharing thinking

stackshare.io - share software stack

portmidi - Elixir midi music player that uses LiveView for SVG UI

umbrella application with a single port

don’t do it in test
initialize at compile time

PubSub in Postres

LISTEN channel;
NOTIFY chanel;

Can have an Elixir process listening for Postgres Notifications

Ecto-sandbox - use to proxy to the database when locally testing a frontend app

creates a separate DB for testing
data API requests can be sent form App to DB

ex_machina

natterbox
red matter

kubernetes

Telemetry
Prometheus

closing keynote

The River by Bruce Tate from grox.io

author

programming phoenix
dsesigning elixir otp systems

organization

lifecycle - get failover for free; turn off + on again

concurrency - modeling real life objects

Platformatec, Bleacher Report, Dockyard - Chris, Le Tote, Erlang Solutions

Erlang Ecosystem Foundation - way to get involved

Garden Bot - IoT Elixir bot for watering garden

Texas Whitewater - by Steve Daniel - book

Stewardship Made Practical - by Stewart ? - talk

Outside Magazine - has a contest for the best places to live Outdoors

Regents Canal - places to go

“Elixir is done” - Jose Valim

Syntax is stable

libs

LiveView
Scene
Nerves

vote with your values, dollars, time, companionship

Speaking with the people

I am kind of bad with names. I started talking to someone about cycling. He recommended that I go to Richmond Park the following day. That’s what I ended up doing, and it was amazing. There’s deer with giant antlers, 2 lakes. It was great.

Everyone was so nice. Herb, it was great to talk with you, your friend, James, and the others.

We talked about the how the infix operator isn’t efficient for appending to the tail of a list. It is also the case for removing items from the tail of a list and the complexity is N square.

Everyone was so kind and welcoming.

Advice I got

Francesco, it was great to meet you. Thank you for some road map advice to consider after I’m done with Programming Erlang 2nd Ed., Joe Arstrong’s book.

Erlang Programming: A Concurrent Approach to Software Development
Defining Scalability for OTP

First learn concurrency, a process can only change it’s own state

Next, processes are an event handler, finite state machine. Client server, supervisor, worker, and so on

Glamorous Toolkit - editor

Purely functional data structures

Conclusion

The conference was a very unique oppurtunity and I appreciate it a lot. I hope to go to Code Mesh SF in March 2020. Or maybe the RabbitMQ conference.

Erlang and Python Codejam

2019-07-03T00:00:00+00:00

If you can read this Erlang code jam, the syntax is kind of insane. Just think about it. This is how to manipulate primitive types in Erlang. But not just that. After that is a Python codejam. Python is my main language. It will be fun to compare.

Here are list, tuple, and zip two lists in both languages. But, first, a short background.

My Story

I’m new to Erlang. I’m on page 110 of Joe Armstrong’s book, Programming in Erlang, 2nd.

Erlang is an interesting language. The first time that I tried to learn Erlang, about 1 year ago, a good friend said, “wait until you see pattern matching.” I was reading this exact same book, and was to about page 70, and I still hadn’t seen pattern matching.

My interests changed, and I gave up reading the book. I had gotten so stuck on Tensorflow Extended (TFX) and had paused any project coding because I was so frustrated. 2 months ago, I started reading the book after hearing in May 2019 of Joe Armstrong’s untimely death.

TFX had promised me an end-to-end pipeline. There are about 9 steps. When I got two thirds of the way through to the transform step, I got so frustrated with the objects involed and their api. Whatever I was trying to do wasn’t working. I gave up.

In May, I stated reading “Erlang Programming 2nd Ed.” by Joe Armstrong. So, if I do the math, in 60 days I have read 110 pages, so 2 pages a day. Ouch. Okay, that’s kind of bad. Either way, here is my progress. Maybe I only got serious in the last month, and I’m being hard on myself. Enjoy.

Python and Erlang

Let’s do Erlang first. I’ll build myself up for Python. Python is my main language, so I’ll show that last.

Erlang

Map, List, Tuple, Zip Code Example

And from there, I could get back to where I eventually was.

Python

Map, List, Tuple, Zip Code Example. Here is the Python codejam. Enjoy.

Wow, that was a world tour of Python. The slicing here involved lists, where the Erlang slicing used Binaries. Both were slicing of data structures.

Verdict

Okay, hands on the keyboard verdict. Both were amazing! Okay, until next time. Thanks for reading.

Learn How to Code with a Code Challenge

2019-06-15T00:00:00+00:00

The theme this week is to do a code challenge in a new language. For this week I will suggest a code challenge and solve it. I challenge you the reader to attempt the same challenge, or make another one up and solve it in a language that you are trying to get better at.

Background

For me, I have had success by reading books on computer languages or frameworks, and then implementing them in a personal project. This has helped solidify the concepts, syntax of a language, standard library knowledge, and so on.

Set the Scene

Code challenges are like code katas, or working out. You have to flex that muscle to stay fit. And push yourself to new limits in order to grow. That’s the goal of the code challenge.

Code Challenge

Here is the challenge to solve. This is an actual problem of mine that I wanted to solve in code. I have a book that I’m reading. I want to know what the date is when I will finish reading this book if I read N number of pages per day. There are a few things that go into this situation that I considered:

how many pages are in the book
what page am I on
how many pages will I read per day
what is the current date
do I have any vacation days where I won’t be reading
how may days a week am I resting and not reading

(Optional)

Optional challenge to test your code

Optional challenge to document your code

My Code Challenge Attempt

Given the above details, I attempted this using Erlang. I hadn’t done date arithmetic before, so I would be learning something new. The above code challenge is an actual real problem for me that I wanted to solve. I am reading Programming Erlang 2nd Edition. Here is my data for the above questions:

= "how many pages are in the book"
= "what page am I on"
= "how many pages will I read per day"
{2019, 6, 15} = "what is the current date"
= "how many vacation days when I won't be reading"
= "how may days a week am I resting and not reading"

Here is my code in Erlang:

Conclusion

I enjoyed the code challenge. It gave me a goal to test my current understanding where I wasn’t purely copying book code with small tweaks. I now feel like I understand the language more and am ready to continue reading next week.

If you are an Erlang developer, I know that the code is really bad. It’s part of the learning process. Don’t worry if your code challenge isn’t pretty. The goal is to solve the problem, and code can always be made prettier or more concise.

Code Challenge Solutions

If you solved the code challenge in another language, please send me a pull request for this blog with a gist of the code and I can see about adding it to the blog. Thanks.

Thank you for reading :)

Test Driven Development with Python

2019-06-09T00:00:00+00:00

For this blog, I’m going to code up an example project in Python with unit tests. I’ll discuss unit testing while building doing the project. This blog could have been a precursor to Unit Testing Concepts with Python, which discusses more concepts then application. Okay, I hope that you enjoy. Let’s get started.

Project Overview

For this project, let’s create in Python objects AWS IAM Roles and Policies. Policies can be inline to a single Role, so only that Role can be use that Policy, or a Policy can be attached to a Role, and then the Policy can be also used with other Roles. Let’s start with these initial objects. We can add more detail later.

Design first before writing code and tests

Okay, at this point, I thought that I was ready to start coding up an initial file and tests. Then, I looked up the AWS IAM documentation and some IAM Roles to see what properties they define to make sure that this blog is as accurate as possible uses the same naming.

I first thought that we will need a single parent class for the Policy and Role objects. Then I looked at the ARN of a single Role. A lot of data is present here. Here is an example ARN and a description of the data it contains:

Example:

arn:aws:iam::123456789:role/rds-monitoring-role

It contains this data:

Service - iam
AccountId - 123456789
ResourceType - role
Role Name - rds-monitoring-role

At this point, a lot more objects have come forward. Policy and Role are both ResourceType’s. There is also an Account and a Service.

At this point, I have written 2 designs. Here are the 2 designs. The first is in blue pen, the second in pencil. I forgot my rule, always diagram in pencil!

We can use them in the future. But first, let’s make sure that our objects can satisfy a simple requirement. They should be able to generate an ARN from the composite data that we have identified.

Our first test

Let’s ignore the design for now and write our first test. We don’t know how the production (prod) code will be implemented. We do know what it should be able to do though. We want a Role to be able to tell us its ARN.

All of this code lives in python-unit-testing-examples. Follow the Project Setup instructions in the README to install the project and code along with this blog.

Here’s the test:

import unittest
from iam import Role

class RoleTests(unittest.TestCase):

    def test_arn(self):
        name = 'rds-monitoring-role'
        role = Role(name=name)

        ret = role.arn

        assert ret == 'arn:aws:iam::123456789:role/rds-monitoring-role'

Then run py.test from the project home directory to run the test suite.

There’s no prod code written at this point. Running the above test only with no prod code, will result in this error:

Traceback:
tests/test_iam.py:3: in <module>
    from iam import Role
E   ImportError: cannot import name 'Role' from 'iam' (/Users/aaron/Documents/github/testing_example/iam/__init__.py)
=================================== 1 error in 0.15 seconds ===================================

Let’s add some initial prod code and see what happens. Let’s add this code:

class Role:
    @property
    def arn(self):
        pass

And re-run py.test. This results in the error:

    def test_arn(self):
        name = 'rds-monitoring-role'
>       role = Role(name=name)
E       TypeError: Role() takes no arguments

tests/test_iam.py:10: TypeError
================================== 1 failed in 0.16 seconds ===================================

This is because we did correctly add a Role class in the iam/__init__.py file. But, classes have a default empty constructor of: def __init__(self):pass that accepts no arguments, but our test passed in a name argument. Let’s add a constructor to accept the name. Our goal of the test here is to test the ARN. Let’s update our code to:

class Role:
    def __init__(self, name):
        self.name = name

    @property
    def arn(self):
        pass

Re-run py.test and we now get the error:

self = <test_iam.RoleTests testMethod=test_arn>

    def test_arn(self):
        name = 'rds-monitoring-role'
        role = Role(name=name)

        ret = role.arn

>       assert ret == 'arn:aws:iam::123456789:role/rds-monitoring-role'
E       AssertionError: assert None == 'arn:aws:iam::123456789:role/rds-monitoring-role'

tests/test_iam.py:14: AssertionError
================================== 1 failed in 0.09 seconds ===================================

This is good! All of our previous test fails were due to test setup errors. This is an important concept.

Test Setup Errors

A test setup error is different then a prod code error. This is an error like we just saw. Maybe it’s a missing import or an undefined variable in the test code. Either way, it’s a problem with the test code and can be solved by setting up the test correctly and only changing test code.

To fix the first test setup error, we did what is known as method stubbing.

Method Stubbing

Method stubbing is important. We do this so we can call our prod code correctly and know that everything is connected, but not returning the value that we want to test for. For compiled languages like Java, this has to be done, or else the code won’t compile. Methods must be stubbed in order to run initial tests before implementing the method body.

Python is more forgiving because of dynamic typing, but it is just as important to stub methods when testing. This tells us that we correctly called or prod code from the test.

After our two test setup errors were fixed, we got what is known as a graceful test failure.

Graceful Test Failure

A graceful test failure is when the test finally fails on an assertion. We have set up the test and called our prod code correctly, and now the only thing wrong is that the value that we are supposed to get back. Now we can implement the prod method.

Let’s now define an initial implementation for the prod method.

Initial Implementation

There are a lot of things wrong with this implementation, but let’s ignore that for a second and see if this allows our test to pass. Here’s the updated method:

class Role:
    def __init__(self, name):
        self.name = name

    @property
    def arn(self):
        return f'arn:aws:iam::123456789:role/{self.name}'

Run py.test and now we get:

tests/test_iam.py .                                                                     [100%]

================================== 1 passed in 0.05 seconds ===================================

Commit! Our tests have now passed. We don’t have to git commit, that’s just a joke. Or test have passed though for the first time. We succeeded here in a couple of ways:

we used the self.name attribute defined on the class
we failed first. By failing first, when the test passed later, we know it’s the code that we added that caused it to pass
we failed fast, a favorite tech cliche! The change was quick, and we got to see the test pass

There’s still a lot to be desired though. Thinking about our initial design and the objects that we identified, and the method body of the arn property, here are some thoughts:

AccountId shouldn’t be hardcoded in a single method body for Role.arn. What about Policy.arn when we implement that.
same for role. This is hardcoded. We know that this is the ResourceType
same for iam. This is hardcoded and this is the Service

Before going on to the next section, I did git commit and git tag at this point for the example code repo. You can check out to the git tag to this point by running:

git checkout first-test-pass

Improving the Implementation

Our test is already defined at this point. We can make prod code changes to improve the implementation. This will make the prod code better.

From our design, we might consider our class as a Resource, and it should have a required resource_type attribute, equal to 'role' in this case. We can use the Python abc module to require a property on a class. Let’s update the code:

import abc

class AbstractResource(abc.ABC):

    @abc.abstractproperty
    def resource_type(self):
        pass


class Role(AbstractResource):
    def __init__(self, name):
        self.name = name

    @property
    def arn(self):
        return f'arn:aws:iam::123456789:role/{self.name}'

We run py.test and get this error:

self = <test_iam.RoleTests testMethod=test_arn>

    def test_arn(self):
        name = 'rds-monitoring-role'
>       role = Role(name=name)
E       TypeError: Can't instantiate abstract class Role with abstract methods resource_type

tests/test_iam.py:10: TypeError
================================== 1 failed in 0.07 seconds ===================================

This is the error that we want. We are using the test feedback loop here.

Test Feedback Loop

The test feedback loop means using the development workflow of changing code, run tests, make changes, test, write new test, prod code change, etc… It is:

the feedback in between writing test and prod code that let’s us know that the code is doing what we want

The test feedback loop exists in manual testing!. It is more time consuming to do manual testing though, so if this can be done as much as possible through automated testing, and unit testing because these are the cheapest tests to run, then this is the goal.

Let’s update the prod code:

import abc

class AbstractResource(abc.ABC):
    @abc.abstractproperty
    def resource_type(self):
        pass


class Role(AbstractResource):
    def __init__(self, name):
        self.name = name

    @property
    def resource_type(self):
        return 'role'

    @property
    def arn(self):
        return f'arn:aws:iam::123456789:{self.resource_type}/{self.name}'

The test should now pass again.

Move out the rest of the data

Let’s move out the rest of the data in the method that probably doesn’t belong there. We have lots of choices on how to add configuration to our project. It will vary in complexity. Here’s a way to keep it hardcoded in code, but improve the variable scoping:

import abc

ACCOUNT_ID = '123456789'


class AbstractResource(abc.ABC):
    @abc.abstractproperty
    def resource_type(self):
        pass


class IamResourceMixin:
    service = 'iam'


class Role(IamResourceMixin, AbstractResource):
    def __init__(self, name):
        self.name = name

    @property
    def resource_type(self):
        return 'role'

    @property
    def arn(self):
        return f'arn:aws:{self.service}::{ACCOUNT_ID}:{self.resource_type}/{self.name}'

Here we have made ACCOUNT_ID a global. This could also be loaded as an environment variable or dynamically, but the point is that it’s scope is larger than the initial method.

We created a mixin class IamResourceMixin. The service is defined here. This pattern is the composition pattern.

Composition Pattern

The composition pattern combines pieces of functionality in order to make a fully functional class. The classes used to build the concete class may not have all of the functionality to exist on their own, and this is okay. Piecies of functionality can be defined separately, then shared by the classes that need them.

This pattern is used a lot in django-rest-framework to define API endpoints as classes that support different HTTP request types, where a single HTTP request type is a Mixin.

Maybe change the implementation to this

We could further change our working code example to:

import abc

ACCOUNT_ID = '123456789'


class AbstractResource(abc.ABC):
    @abc.abstractproperty
    def resource_type(self):
        pass


class IamResourceMixin:
    service = 'iam'


class AbstractIamResource(IamResourceMixin, AbstractResource):
    pass


class Role(AbstractIamResource):
    def __init__(self, name):
        self.name = name

    @property
    def resource_type(self):
        return 'role'

    @property
    def arn(self):
        return f'arn:aws:{self.service}::{ACCOUNT_ID}:{self.resource_type}/{self.name}'

Here, we put the Mixin and Abstract class in a single class, and Role inherits from that class. This might be a nice change, so when Policy is added, it can inherit from a single class, instead of the IamResourceMixin and AbstractResource class, which Policy will need both, since it also defines an 'iam' service with a resource_type.

I did a code commit here. To git checkout to this point, run:

git checkout first-mixin-added

Challenge

Okay, now, here’s the challenge! Implement a Policy class with a test for it’s arn attribute. The arn should equal:

'arn:aws:iam::123456789:policy/DenyIAMAccess'

How did that go?

Policy class with tests and refactored

I implemented the Policy class and also refactored. To see the new code, run:

git checkout add-policy-class

There’s been lots of code blocks, so at this point, I did a few changes at once. First, the test code:

class PolicyTests(unittest.TestCase):

    def test_arn(self):
        name = 'DenyIAMAccess'
        role = Policy(name=name)

        ret = role.arn

        assert ret == 'arn:aws:iam::123456789:policy/DenyIAMAccess'

Very similar to the RoleTests.test_arn method.

For the prod code:

import abc

ACCOUNT_ID = '123456789'


class AbstractResource(abc.ABC):
    @abc.abstractproperty
    def service(self):
        pass

    @abc.abstractproperty
    def resource_type(self):
        pass

    @property
    def arn(self):
        return f'arn:aws:{self.service}::{ACCOUNT_ID}:{self.resource_type}/{self.name}'


class AbstractIamResource(AbstractResource):
    def __init__(self, name):
        self.name = name

    @property
    def service(self):
        return 'iam'


class Role(AbstractIamResource):
    @property
    def resource_type(self):
        return 'role'


class Policy(AbstractIamResource):
    @property
    def resource_type(self):
        return 'policy'

So, I had the PolicyTests code in place. I got it to pass by basically copying the Role class and renaming the resource_type. There was a fair amount of duplication though. I started moving things around, and came up with the above code. I think this is better for a few reasons:

the arn property exists on the AbstractResource class. This is better. All AWS Resources have ARNs, so it makes sense for this to be defined on AbstractResource
service is now an abstract property. This is better because Services should have to define what kind of Service that they are.

Abstract is an important concept here.

Abstract and Concrete Classes

Abstract classes can’t be instantiated from. They are meant to be inherited from. In the above example, the arn property is being accessed, and this property references a resource_type and name attribute. These are defined on the concrete classes, Role and Policy.

If we were to instantiate a AbstractIamResource and access it’s arn we would get an error.

Conclusion

This blog may have been better named:

scenic stroll through testing and code design

I kind of took some side trips away from just testing, but I hope that you enjoyed it and they made sense in the context of testing and added value.

In this blog we learned how to:

write test code first
test setup error
test stub
graceful test fail
test feedback loop
composition pattern
code challenge
abstract class
concrete class

We didn’t quite get to inline vs. attach Policies. There are more testing concepts to cover next as well. I will extend upon this example and cover more concepts in the next blog.

Unit Testing Concepts with Python

2019-06-01T00:00:00+00:00

In this blog I am going to discuss some concepts of Unit Testing. I will give some background about my experience and how I got started on unit testing. Then some thoughts and concepts on unit testing in order to best apply it. I will then show a code example in Python and discuss different unit testing concepts used.

Introduction

Let me start out by introducing myself and the person that got me started on testing. Hi, my name is Aaron Lelevier. I’ve been using Python for 6 years, 5 years professionally. Prior to that and with some overlap, I did MsSQL for 3 years.

The person that got me started on unit testing was Toran Billups. This was 4 years ago. I was hired on at Big Sky Technologies as a backend Python Django Developer on a team of 3, myself, Scott Newcomer, who did frontend EmberJs, and Toran Billups, who’s background is in Javascript and Test Driven Development (TDD), as Tech Lead. I didn’t know this at the time, but Toran had only agreed to come on the team as Tech Lead if the team embraced testing.

Toran made a big impact on me. I’d like to share some of his quotes, and give examples with how they apply.

Quotes from Toran

Let me start off with a funnier quote. We had been into our project for about 3 months. I was asking Toran about his thoughts on testing for some concept or topic. Then, Toran says to me:

“You know, discussing testing is like discussing religion.”

This is pretty interesting. I have come across this thought a lot. Some people are against testing and they will fight it. Other people are on the reverse side, and are testing pedantics, and brutally enforce every little thing be tested. Our team was growing while I was at Big Sky, and we wanted to keep our culture of TDD, and people that we discussed this with said “good luck”. It is hard to keep testing a first priority.

Another quote from Toran was:

“Test first or test last. I don’t know about you, but once I’m done writing my code, I don’t want to write tests. I want to go and do the next thing, that’s why I write my tests first.”

This is true, but also for other reasons. After a feature is considered working, writing tests for it seems laborious. The other reason to test first though, and the real reason I would argue, is to see if the functionality currently exists. If you can write a test that something should occur, and it doesn’t, then when you write the production (prod) code for this, and the test passes. Then you know that it was the code you added that made it work, whereas before it didn’t. The testing cycle proves that.

Is this a TDD blog?

No, it is not. I bring it up as part of the previous quotes, and from some of what Toran taught me. Test First has its place. It is not the only way to test though.

Sometimes what you are doing is too unknown, so writing tests for something you’ve never done before is too slow, or simply doesn’t work. An example of this is learning a new programming language or framework. It’s much easier to learn by writing code to do the thing then to write tests. For machine learning, this is also true. Some parts of machine learning like data transformations are good places for heavy testing and even TDD. Making a model with layers and iterating on that model may not be.

There is the concept of Defect Driven Testing (DDT). This is when a defect is discovered, then a test can be written to capture that defect (or bug), so it won’t resurface. Here, the testing scenario isn’t known ahead of time, or else the bug wouldn’t have existed, so TDD doesn’t apply.

Okay, so what is the goal of testing?

Goal of Testing

This is best said by this quote from Toran:

“You don’t know if something works unless you have a test that says it does.”

This could be an automated test, whether it’s a unit or integration test. It could be a manual test by someone on the QA team or it could be your customers testing in Prod by using your product. Ideally, it’s some sort of automated test though because these are the cheapest test to run, and we don’t want our customers to be the first to test and discover bugs, then have to report them and wait for them to be fixed!

Let’s start with an overview of unit testing and see a simple application of it in Python for this first blog.

Unit testing overview

What is a unit test?

A unit test is a test for a single piece of functionality in your code. It would be a test for a single function that tests a single code path. If this function, for example, has an if/else block, you would want a unit test for the “if” code path in the function, and a separate unit test for the “else” code path in the function.

Unit tests are cheap. They are the quickest tests to write and fastests tests to run. They won’t prove that your whole application works. They prove that pieces of the application work in isolation with certain inputs and then verify the outputs.

Here is the famous Test Pyramid.

Unit Tests

Unit tests are at the bottom. This should make up the bulk of your test suite. At Google, they use this ratio, and unit tests make up 70% of their total tests. I heard about this from a Google I/O talk.

Integration Tests

These tests start to connect functions and bodies of code together. They don’t stand up the whole application and test though. They are above unit tests for individual pieces and below whole application automated testing.

UI Tests (Functional Tests)

UI Tests, or Functional Tests if there is no UI, are automated tests of the whole application. This would be Selenium for automated UI testing of a web site, Appium for mobile apps, and so on.

Level of Tests as a Concept Vary by Situation

This is a very good talk on how to test infrastructure.

In this talk, the speaker discusses how infrastructure can only be fully tested by deploying it. In this testing paradigm, Unit Tests are then single pieces of infrastructure deployed in isolation with the minimal number of dependencies. Integration Tests starts to connect infrastructure, and Functional Tests test the infrastructure as a whole.

Unit test in Python

Here is the prod and unit test code from one of my Python project on converting Labelme annotations to COCO dataset annotations project called mlearning. Let’s take a look at this code and discuss it.

There are two unit tests for the example class and it’s single method. One tests the scenario that all image files in a directory are ordered by name. The other tests the scenario that they aren’t ordered by name and what the outputs of the method would then be.

I’m going to discuss a couple of concepts and mechanics from this example code.

Concepts

Unit tests are calling prod code to make assertions about the inputs and outputs. A simple example would be if a function multiplies an input value by two, then the unit test would assert if 1 goes in, 2 comes out, if 3 goes in, 6 is returned, etc…

Inputs may vary. Sometimes functions have defaults or varying number of arguments. In this case, you would want multiple tests to assert the behavior of that function. The above code is an example of this. The prod code only has a single argument, but two situations are being tested.

Mechanics of Unit Testing in Python

If you go to the Github repo linked in the description of the gist, or you may be able to derive most of this information from the gist. Here are a few things going on.

The Test Runner

To run unit tests, Python’s unittest test runner needs to be able to discover them.

I like to put my tests in a directory called “tests” at the project directory level. This is a known pattern. django-rest-framework uses this pattern. Another pattern is to put the tests in the same directory as the prod code file and append “_test” to the name. TensorFlow Extended uses this pattern. I prefer the former because it separates test code from prod code.

The test runner will find each Python file that starts with “test_”. I like to name the test file names with the “test_” prefix, so they are discovered by the test runner, and then use the same name as the prod test file. This makes them easier to keep track of.

The test class should subclass unittest.TestCase. This is because there are setUp and tearDown hooks in the class that can be used.

Each test method in the class should start with “test_”. The test runner will call these test methods and report success or failure. Test classes are just Python classes. They can have other methods for test helpers, etc… and then these would not start with “test_” because they are a helper, not a test method.

Pytest

The example code is using pytest. I prefer the test syntax of pytest to the standard library unittest methods. Also, pytest gives nice red/green colored highlighting on test output. There is a table in the unittest documentation for the different assertion methods. The testing semantics are the same, just with different syntax.

In the example code, the Python assert keyword can be used to assert if something is truthy or falsy. Assert if equal or not equal, etc…

Test Class Names

I like to name the test class the same as the prod class, and then add “Tests” to the end. This makes the test method class easy to search for and identify which prod code the test class relates to.

Test Method Names

Test methods have to start with “test_”. For their actual name, I like to have the name of the prod method that I’m testing in the test method name. This makes it easier when searching for tests for a given method.

A co-worker of mine, Vince Grato, said to me once, I use rspec for naming tests. He was calling me out because I had named my test methods based on the prod methods and then with a suffix like “true” or “false”, which would be the output that they test for, but the test method name didn’t say anything about what the test actually did. Over time, and now currently, this is how I name my tests. I still like to have the prod method name in the test method name, and then append a description of what the test is doing.

Another co-worker of mine, Bill Heaton, said something similar. He said that when he wants to know how some code or library works, he first runs the test suite, and then reads the method names output by the test run to see what the code is doing. This follows along with what Vince said.

Format of the Test Method Body

For the test method body, I like to follow the format of:

Test setup - linespace - code being tested - linespace - assertions

This is the format of the test method bodies of the example test code above. This makes it easier to follow and separate initial setup that the prod code needs before being called. Then actual code being called, and finally the assertions for that code.

I believe that I got this from Kent Beck’s book, TDD by Example, but I am not sure. Either way linespaces can be used to denote meaning in code, and this is a good place for them.

Test Method Ordering

I prefer to put the test methods in the same order as the prod methods. This makes it easier to relate the prod code to the test code if they follow the same method declaration ordering.

The above code example is only for a single prod method, but if there were a second method, then the test method would be ordered below the current test methods.

Test Code Doesn’t Look Like Prod Code

When I worked with Toran, he said this to me one time:

“Where are all the assertions here. There’s all kinds of missing assertions.”

Toran is really nice. I asked him for constructive feedback one time when we were having reviews, and he only told me positive things, but I could also glean some things that I could improve on even though he was so positive.

That said, the above quote means that I had missed a lot in the test coverage. This stemmed from me thinking that code looks like code and test code looks like prod code. It’s object oriented, has arguments, etc…

The test in question was asserting return values of an API endpoint. The API endpoint was returning 10-15 fields with data, but my test was only testing that 2-3 came back. The prod code could have half of it’s fields removed for example. Maybe it’s a cut/paste error, and it’s not seen in code review. If so, the test would never catch this because it doesn’t assert the full behavior of the API endpoint. This leads to another quote:

“Test the things that you care about.”

We can’t test everything. There is a limit. But, we should do our best to test the things that we care about. And, if this API endpoint is something that we care about, we should fully test it’s behavior.

Conclusion

I wrote out 1 ½ pages of bullet points on testing over the last few weeks as I thought about different aspects of testing and things that I’d like to discuss. The above is ⅕ of those bullet points. This is a really good start though I think. Thank you for reading. I hope that you enjoyed and this blog added value for you.

Next blog I will discuss more mechanics for testing, and I will show code examples for different testing concepts and build upon what I discussed in this blog.

Any feedback is more than welcome. Drop me an email or comment below.

Thank you.

My Current Scenic Stroll Through TFX

2019-04-30T00:00:00+00:00

This is a blog is about my current an ongoing stroll through the TFX library.

Why TFX?

The opening line of the TFX README says it all. It says:

TensorFlow Extended (TFX) is a Google-production-scale machine learning platform based on TensorFlow. It provides a configuration framework to express ML pipelines consisting of TFX components.

This is so compelling. The README also says, it’s a:

framework for building end-to-end ML Pipelines

So, I plan to discuss and document what I’ve seen with TFX so far.

Ok, let’s go.

My Goal

My goal and strategy when doing this scenic stroll is to build my own TFX Pipeline for the MNIST dataset and ML models. The example TFX Pipeline uses the Chicaco Taxi Cab dataset and CSV flat files. By building a slightly more complicated Pipeline, it is forcing me to learn the TFX Framework.

The Start

When I first started, I though, I’ll just plug in MNIST data at the correct file path, and everything will just work. But, it didn’t…

I next thought, okay, let me tweak a little bit of code. I started to read the source code, and try to do this, but I didn’t understand the source code.

Apache Airflow, then Apache Beam, what are these things? These are the first 3rd party libraries that you run into when reading the TFX source.

Apache Airflow

Airflow is for creating workflow pipelines as directed acyclic graphs.

I did a separate blog post of Airflow here detailing learnings.

Apache Beam

Beam is for implementing batch or streaming jobs with a single programming model that uses MapReduce with windows and triggers.

For Beam, the core-overview in the official documentation was really good.

This YouTube video, A Whirlwind Overview of Apache Beam was also really good for helping to understand Beam.

Then coding along and doing the core transforms helped make them real before trying to do Beam transforms in TFX. Repo link here: aaronlelevier/apache_beam_examples

Okay, here’s the steps that I’ve gone through of TFX so far.

Details on TFX Pipeline Steps 1-4

I have been tweaking my work flow for trying to run each Pipeline Airflow Operator.

The first thing that helped me was to symlink the Airflow pipeline file mnist_pipeline_simple.py to ~/airflow/dags/mnist/mnist_pipeline_simple.py where Airflow expects to find it.

For debugging Airflow, the steps in a blog post that I did, for “What I know about Apache Airflow so Far”, in this section have worked well: Iterate on developing a DAG in Airflow

For refreshing the local DBs, I created a helper script: refresh_mnist_pipeline.sh This way any changes to the Pipeline wont cause a fail due to stale Database schema, data, etc..

ExamplesGen

Here, going through the Apache Beam core-transforms was important to know what’s possible.

CoGroupByKey is needed here to combine 2 files in the case of MNIST, where the image data and labels are in separage Gzip files. CoGroupByKey is used to combine 2 or more Beam PCollection’s, then the single PCollection can be output to a TFRecord with beam.io.WriteToTFRecord.

StatisticsGen

This step is for running Tensorflow Data Validation (TFDV) on the dataset to generate statistics.

In order to do this and visualize the statistics, Bazel must be installed. On Mac OSX, I first installed Bazel using Homebrew, but that didn’t work, so I installed from source, and it worked. I had created an issue with tensorflow/data-validation but I then closed it once I realized the issue.

This TFDV example Jupyter Notebook was super helpful for figuring out how to call TFDV.

SchemaGen

This step uses TFDV to generate a Schema, which is the features, labels, and their types for a given dataset.

ExampleValidator

This step then uses TFDV to then compare the statistics generated by StatisticsGen and the Schema generated by SchemaGen. TFDV will look for any anomalies between the train and eval dataset splits, and write them to a file.

TFDV can also be used to fix these anomalies.

There’s 4 more steps, then Tensorflow Serving can be used to serve the model.

I will report back soon!

Thanks for reading :)

What I know about Apache Airflow so Far

2019-04-07T00:00:00+00:00

This is a blog recording what I know about Apache Airflow so far, and a few lessons learned. This blog is in no means exhuastive on all Airflow can do. Maybe the main point of interest for the reader is the workflow section on how to iterate on adding tasks and testing them.

Details

I started Apache Airflow with the Quickstart and Tutorial and everything seemed pretty straight forward, but when I tried to do some of my own DAG ideas and learn some other concepts in Airflow, I got stuck. My DAGs weren’t registering, and I wasn’t sure what was wrong. My DAGs were taking too long to run, too long of a feedback loop. Some of it was because they were in the wrong directory, or had Python Errors.

I learned to eventually follow this workflow. Okay, here’s it is, but first some vocabulary.

Airflow Vocabulary

Some Airflow vocabulary and concepts that will be used in this blog.

DAG - directed acyclic graph - in Airflow, a description of the work to take place

Operator - a class that acts as a template for a Task. This can be a BashOperator, PythonOperator, etc…

Task - an instance of an Operator

Task instance - a task instance is a Task in a DAG that was run at a certain point in time with a given configuration

Okay, workflow time!

My Airflow Workflow

First, Airflow DAGS need to be registered in the AIRFLOW_HOME directory. This is set as an environment variable. By convention, export AIRFLOW_HOME=~/airflow. Other things in this dir are:

airflow.cfg   airflow.db    dags          logs          unittests.cfg

But, the DAGs that the developer works on are in a Project directory somewhere. To isolate my Project dir and where Airflow resources live and get my DAGs discovered, I symlinked them from the Project dir to the Airflow dir.

Next, to test a DAG, starting airflow scheduler and running the full DAG isn’t ideal. The feedback loop is too long. It could take 5 minutes for a DAG to run, and it will run all steps. Here are the steps that I learned to use to iterate on developing a DAG in Airflow:

Iterate on developing a DAG in Airflow

First, call it as a Python script to see if there’s any errors:

$ python my_dag.py

Second, try seeing if the DAG is registered:

$ airflow list_dags

Third, output the Tasks for a DAG. This lets you know what Tasks are configured for the DAG

$ airflow list_tasks my_dag

Then, a Task can be tested in isolation. A date param is required.

$ airflow test my_dag my_task 2019-04-07

Depending on the situation, and if we’re ready to run the DAG, then the DAG can be run by starting these two processes:

# terminal 1 - DAGs are turned on an run here
airflow webserver

# terminal 2 - crawls the directory structure to discover and register DAGs
airflow scheduler

When running a full DAG, I was looking in the $AIRFLOW_HOME/logs to check output. Logs are by DAG name, task, and date. It’s reachable, but it can be easier found in the Airflow webserver UI.

Python Debugging

I couldn’t get pdb to work when running the full DAG. This is because the Tasks are run as subprocesses, so they won’t hit the debugger, but when running airflow test though, pdb works!

PythonOperator.python_callable

Python functions can be tested outside of Airflow, and this is ideal. I unit tested a couple, and in a real application, I would do this extensively and outside of Airflow. This is also what Matt Davis recommends in the video and notes linked below.

Here’s how to Hide example DAGs

This can be done in $AIRFLOW_HOME/airflow.cfg by setting: load_examples = False

Some Airflow Features

PythonOperator - are used to call a Python function. They can be passed positional arguments with op_args, keyword arguments with op_kwargs, and Airflow context, such as the Task instance, and datetime info with provide_context=True

BranchPythonOperator - are used for doing conditional branching in a DAG

SubDagOperator - can be used for setting repeatable steps for Tasks at different points in a DAG. Tensorflow Extended uses these for common steps in each Task to first check the cache, and then decide to execute if needed.

XCom - is for passing key/value primitive data from one Task to another. This can be:

the return value of a PythonOperator
any value using xcom_push and xcom_pull

As a note, this can be used for a Task to get a reference to the previouse Task instance that just run in it’s context: context['task_instance'].

In the example Github repo in the next section, I noticed that I only did xcom_push and xcom_pull for Tasks that ran sequentially.

Github Repo for learning

I created this repo for learning Airflow and trying out the above features:

https://github.com/aaronlelevier/apache_airflow_examples

Notes from YouTube video “A Practical Intro to Airflow”

These are my notes from Matt Davis’ talk at PyData SF 2016.

This video was super helpful in filling in the gaps in conceptually understanding Airflow in combination with the Airflow documentation.

Airflow Concepts

metadata db

sqlite

mysql

postgres

webserver

flask

scheduler

nothing can run without this

crawls the configured dir’s to find DAGs

celery

tasks are sent from the scheduler to run on Celery workers

would use rabbitmq or redis for Celery Queue

Airflow objects

Tasks

Executors (workers)

Code

Useful DAG Arguments

start_date - will say when to start, if in the past, Airflow will backfill the tasks to that date based on the schedule_interval

can put the start_date for the future, if we don’t want it to run yet

schedule_interval - cron config

max interval to run should be at fractions of hour, not per minute, because Airflow kicks off tasks every 30 seconds

default_args - arguments supplied to the DAG, but get passed to the Operators

max_active_runs - can set globally or per Pipeline how may parallel pipelines to run at the same time

concurrency - max number of tasks run in a single Pipeline to run at a sinble time

can retry tasks

can put timeouts in Pipeline for when tasks start

Useful Task Arguments

retries -how many times to retry

pool - a worker pool - can limit slots here

queue - named Celery Queues - and assign certain Workers to a Queue based upon the type of Worker if biefy, lightweight, etc..

execution_timeout - set a max time for a Task to run

trigger_rule - default is “all upstream tasks are successful. Can change to things like

“all upstream tasks failed”
“all upstream tasks are done” - either success or fail

args for python

env vars

template vars

provide_context - supplies a dict of Task arguments, like “execution date”, from Airflow context

op_args - positional arguments for the Task

op_kwargs - kwarg arguments supplied to the Task

Executors

CeleryExecutor - for prod, puts exec request on a queue and sends it to a Celery Worker

SequentialExecutor - good for debugging, will stop scheduling to run Tasks

LocalExecutor - uses Python’s multiprocess module to run in a diff process, so Scheduler can continue scheduling things

used in Prod for some companies - if don’t want a distributed worker queue

MesosExecutor

Local debugging

To use pdb, use the SequentialExecutor, and run airflow test to hit pdb debugger

Local Pipeline testing

start_date - should be some date in the past

schedule_interval='@once'

can delete a DAG using the UI, in order to re-run this Local Pipeline for testing

Separate Business Logic

develop code first, then integrate with Airflow

Deploying new code

to get new Python code, we have to restart the process, so new code is imported. to do this with Airflow, we can do

airflow scheduler --num_runs - this will stop the scheduler after num_runs has occurred. You need a separate mechanism to restart the scheduler. Speaker uses a bash script

CELERYD_MAX_TASKS_PER_CHILD - will configure Celery workers to restart after the “max tasks”

speaker sets the above to 1 max, so each run, always gets new deployed code

Conclusion

So far, so good. By no means is this all Airflow can do, and is the above blog exhaustive. Thanks for reading!

Tensorflow Dev Summit 2019

2019-03-09T00:00:00+00:00

This is my blog post summary of the Tensorflow Dev Summit 2019. I applied to go to the conference, but wasn’t accepted. Google sent me an email saying that I could still attend via the live stream, so I took off to watch the live stream and learn.

I attended the last Pytorch Dev Con in Octoboer 2018. I’ve been pretty active in following machine learning and learning more about it and experimenting in my free time for about 1.5 years. My full time job is as a Python Developer working on internal tools.

Okay, to the conference.

The conference

The conference started out with an overview of the current Machine Learning landscape. Recent advances BERT for NLP, and Open AI’s GPT-2 model for NLP were discussed.

Models have been built like Aircognizer that predicts air quality based on computer vision. There’s a Google Play app for this.

There’s break throughs all the time in open datasets such as the open images dataset.

I’m going to include a library section below. So may libraries were released or now have new versions and features. The big release is Tensorflow 2.0.

Tensorflow 2.0

Tensorflow 2.0 was released.

Keras API is now the recommended way to express TF models.

Eager execution is the default.

Functions with dynamic control flows can now be decorated with @tf.function and be used in TF, just like PyTorch with @torch.jit.script

Saved models use the same format in development and prod, so the same exported model can be used in both places, and doesn’t have to be converted.

Tensorboard can be used in Google Colab. Here is a link to an example. I just ran the example and it uses ngrok. Way to ngrok, that’s awesome!

Orchestration

TF has chose to standardize on Orchestration for what everyone is already using instead of providing their own solution. Supported Orchestration frameworks are Apache Airflow and Kubeflow

Online courses

Both Udacity and Deeplearning.ai are going to have online courses

Hackathons

There’s a Devpost Hackathon with $150k in prizes. Wow!

Interface programming

tf.raw_ops exists for using the same ops that TF uses

There’s a set of base components for checkpoints, layers, etc… and these base components can be extended to write your own components.

tf_upgrade_v2 to upgrade TF 1.x code to TF 2.0

Tensorflow.js

Now supports Electron apps

There are off the shelf models for use

Tensorflow Swift

There’s going to be a new fast.ai tensorflow swift class. Jeremy Howard, I a little bit couldn’t believe it, if you are reading this! I’m excited to take a look at the class though. I love Jeremy Howards teaching style and the way he explains things. If I can get time and the class aligns with other goals, etc… I am pretty interested.

Tensorflow Special Interest Groups (SIGs)

There are special interest groups within Tensorflow for different parts of the community. There’s a SIG for Rust.

Tensorflow Extended (TFX)

Tensorflow Extended is an end-to-end ML solution. As of now, all components have been open sourced by Google. This and TF 2.0 were the #1 things that blew me away.

Here’s a couple of screen shots from the talks that overview what goes into a production ML workflow and the components involved.

I went through the Tensorflow Extended tutorial for example Chicago Taxi Cab model. Everything worked amazing. It uses Apache Airflow, all parts of TFX, gRPC to send requests to TF Serving.

Libraries

Tensorflow Hub - TensorFlow Hub is a library for reusable machine learning modules.

Tensorboard - TensorFlow’s Visualization Toolkit

has a ton new features, and is also now natively supported in Jupyter Notebook and Google Colab

Tensorflow Model Analysis - Model analysis tools for TensorFlow

Tensorflow Probability - Probabilistic reasoning and statistical analysis in TensorFlow

Tensorflow Agents - TF-Agents is a library for Reinforcement Learning in TensorFlow

Tensorflow Tensor2Tensor - Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.

Tensorflow Federated - TensorFlow Federated: Machine Learning on Decentralized Data

Tensorflow Privacy - Library for training machine learning models with privacy for training data

Tensorflow Extended - TensorFlow Extended (TFX) is an end-to-end platform for deploying production ML pipelines

Tensorflow Lite - TensorFlow Lite is a lightweight solution for mobile and embedded devices

Tensorflow.js - A WebGL accelerated, browser based JavaScript library for training and deploying ML models.

Talks Reference

All talks from Tensorflow Dev Summit 2019

Conclusion

I’m running out of time for the day. This isn’t everything, but it covers a lot. I hope you enjoyed readying and found some useful references. Thanks.

Personal recap of this week

2019-03-03T00:00:00+00:00

This is a personal blog of how this week went. Warning this blog isn’t meant as boasting. The week went really good. I’m not sure why. I need to find out. If I could have more weeks like this, well then wow! I could only hope. I’m recording this week for myself because it’s been pretty special.

Details

Each section is called out in order of how it happened. No order of importance.

Work

On Thursday, I gave a presentation to the architects at my work. This was amazing and particulary humbling. At this point, I am a contractor. I work hard and stay focused on my goals at my workplace, and am focused on where I want this to go. For me to give a presentation to the architects, not to get too nervous, and to be able to answer most all their questions was particulary special. I appreciate this opportunity.

Python coding

I wrote proxy logic for calling any method in a foreing class in Python and being able to hook into that method and make calls before or after the method is called. This could be equated to a vistor decorator. I am proud of this because I had no idea of how to do it, but I believed that it was possible, and it was. My particular use case was to properly bust the cache. But, doing logging, or any other use case, where a before or after boto2.client('ec2') for example out of one of the may remote client visitor calls, well this pattern can be used a lot, needless to say.

Personal

Read 1 chapter of Accelerated C++

Got my C++ environment with libtortch working and contributed to a torch.jit feature branch

Most intense mountain bike ride yet… yes, I’m still on a Trek DS2

First code review with a senior site reliablity engineer, SRE. It was great feedback. I need to fix up my PR now :)

Passed a code challenge.

I went to a barber shop and he didn’t trim my hair as I asked and I asked him multiple times to correct it until he got it right. I’m proud of this because I stuck up for myself to get the right haircut that I wanted.

Family

Went to mount Palomar and played in the snow with my family.

My neighbors opened up to me about being able to buy their oldest son eye glasses for seeing in color because he’s color blind. They said God bless, because these glasses are expensive, and they don’t come from a lot. I look up to them for doing this for thier son.

Conclusion

God bless. Thank you. This isn’t something that I talk about a lot. I am just so thankful for my current situation, myself and my family for being in such good health. I am so thankful for all that I have. It’s obviously something that you can’t take for granted. Thank you.

Convert a PyTorch model to C++ - using maskedrcnn-benchmark and torch.jit - Part 1

2019-02-16T00:00:00+00:00

The next step in my project for putting a machine learning model in production is to put an initial model in production! So…, I’m somewhere towards the first full iteration. Obviously, a production machine learning model will need to be accessed, a data pipeline, etc…, and I haven’t done that yet, but first let’s convert a PyTorch model into C++.

Goal

Convert the maskedrcnn-benchmark PyTorch Python model to a C++ model using torch.jit

Story

I’m using the maskedrcnn-benchmark Github library.

There’s an open PR for torch.jit support.

I pulled down the PR branch, and tried building to see if I can assist.

I first synced up with the master branch. (maybe I should have built and run the PR branch first, not sure….)

Ok, now I’m trying to install the additional dependencies of the PR branch. So far, it looks like it requires OpenCV and Pytorch in C++ as dependencies.

When I run demo/Mask_R-CNN_demo.ipynb because getting error:

error: could not create 'maskrcnn_benchmark/lib/custom_ops.cpython-36m-x86_64-linux-gnu.so': No such file or directory

So, I think I need to compile and correctly link custom_ops.cpp

When I run cmake CMakeLists.txt, like below, here’s the output.

command line:

(venv) aaron@ubuntu-desktop:~/github/maskedrcnn-benchmark/maskrcnn_benchmark/csrc/custom_ops$ cmake CMakeLists.txt 

With error:

CMake Error at CMakeLists.txt:11 (find_package):
  Could not find a package configuration file provided by "Torch" with any of
  the following names:

    TorchConfig.cmake
    torch-config.cmake

  Add the installation prefix of "Torch" to CMAKE_PREFIX_PATH or set
  "Torch_DIR" to a directory containing one of the above files.  If "Torch"
  provides a separate development package or SDK, be sure it has been
  installed.


-- Configuring incomplete, errors occurred!
See also "/home/aaron/github/maskedrcnn-benchmark/maskrcnn_benchmark/csrc/custom_ops/CMakeFiles/CMakeOutput.log".

Okay, I will try to install LibTorch C++ version

cd ~/Downloads
wget -bqc https://download.pytorch.org/libtorch/nightly/cu100/libtorch-shared-with-deps-latest.zip

I’m going to wait for this to download and then build. It’s a big downlaod, so did it in the background using wget -bqc

Part 1 done. To be continued.

Learning Erlang

2019-02-10T00:00:00+00:00

Let me tell you how Learing is. By this, I mean coding. The blog that you are about to read is written towards people who have mildly thought about coding. It’s not that hard, and you can get yourself into it fairly easy and it pays well. A lot of people might stop because of the learning curve, but you really just have to progress at you own rate. If you’re getting stuck, and this is an actual quote from Joe Armstrong,

If a process fails, what should it do? Well, it should try to do something simpler.

I love that quote. Learning is very much the same way. I’m taking some of my time to write this, but hopefully it benefits all the readers more than my small amount of time. If you enjoy the article, please let me know.

The Learning Process

This is how I learn. Some learn by listening, reading or doing, but all actually progress towards doing. I learn by doing all 3 at the same time.

I am learning Erlang. I know nothing about Erlang. I’m terrible at it. I am on page 67 of Joe Simpson’s great book “Programming Erlang - 2nd Edition”.

Okay, well let’s start at the max code that I can think of. Let’s code in Intellij and write a whole program. Okay, here’s how far I got. I was going to write a program about me and some other objects on a Sunday, and have them interact. I started a aaron.erl module, that sunday.erl module was supposed to spawn. Here’s the first code I wrote:

Then I progressed to the colsole, I wasn’t ready for a file. Doing c(module_name). each time to compile changes for Erlang files is tough.

So, I went to the console. Here’s some code I next wrote:

Then I next tried codewars. They didn’t have a lot of Erlang. I tried the Erlang 7kyu 7x7 Skyscrapers Problem, I could probably solve this in Python. Erlang, I didn’t stand a chance!

Conclusion

That’s what learning is.

It’s try, succeed or fail, try again.

If you enjoy something, then sometimes losing is the next best thing to winning. So, if you win or lose, you enjoy it and keep going, and eventually find out how to win because winning is better than losing!

If you start to not enjoy it. Then pivot your learning towards a direction that you like more, and keep learning.

Training the maskrcnn-benchmark on my own hardware

2019-02-08T00:00:00+00:00

My goal is to train a Machine Learning model on my own hardware and my own labeled dataset. In order to do this, I’ve been ramping up on the tools and software necessary to do this. This blog post is about Training the maskrcnn-benchmark on my own hardware

Initial detour

I’ll start where I left off from my last post, which was titled “Training an ML model on the COCO Dataset”, but I hadn’t fully trained a model, I had run a model over a couple of mini-batches of the COCO dataset.

After my last post, I decided to upgrade my Linux kernal headers unknowningly. Well, come to find out Ubuntu 18.04 with linux-headers 4.15.0-45-generic doesn’t currently work with Cuda 10.0 drivers. I tried fixing it a variety of ways, and wasn’t sure why it was working, but in the end, I downgraded back to 4.15.0-29-generic and it is now working.

If you run into a similar issue and want to downgrade, how I did it was, check the linux-headers here:

$ ls /lib/modules/
4.15.0-29-generic  4.15.0-45-generic

Then in grub, you can set the preference for choosing what linux-headers you want to use at start up like so:

# /etc/default/grub

#GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=-1
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""

Then run sudo update-grub and reboot for the config to take effect.

Let’s train the model

Now that Cuda drivers are working, I trained the model using this command

nohup python tools/train_net.py --config-file "configs/e2e_mask_rcnn_R_50_FPN_1x.yaml" SOLVER.IMS_PER_BATCH 2 SOLVER.BASE_LR 0.0025 SOLVER.MAX_ITER 720000 SOLVER.STEPS "(480000, 640000)" TEST.IMS_PER_BATCH 1 &

nohup will run the process in the background.

This is a large model. My hardware is a Nvidia 1080Ti with 11GB of GPU memory. The model is usually run on a cluster of 8 GPUs with a mini-batch of 16. On a single GPU, they recommend a mini-batch of 2. This takes up 6-10GB of memory depending on the image size.

The run time is also long. At 720,000 iterations above on the 1080Ti, it will run for 3.5 days.

After 2 days, I got excited and wanted to see how the model is doing. Below is my trained model, and the original model.

My model

Original model

Okay, continue training

I started the model again. It will load the previous checkpoint, and continue training.

I first change the iterations to 300,000 thinking that I should lower to about how many I had left, but the model must know the total expected iterations, because when I did this, the model thought that it was done, and ran the validation set.

I changed the command back to the original 720,000 iterations, and the model loaded it’s last checkpoint, started training again, and calculated 1.5 days left.

The Nvidia 1080Ti GPU

So far, the GPU has been running like a champ. I got it used off eBay, so you never know, but so far so good. It’s been at 72c degrees max. The process hasn’t failed, and ran continuously for the first 2days.

What’s next

I’m going to let the GPU finish training, run validations for the original vs. my trained model and compare. This comparison will be for spot checking individual images, but also with the validation set. The model returns these statistics for overall model performance. The below are after two days of training, so my plan is to evaluate after the full 3.5 days of training and also the original model to see if I can produce the same results.

OrderedDict([('bbox',
              OrderedDict([('AP', 0.2995932418350566),
                           ('AP50', 0.5048334612699799),
                           ('AP75', 0.3199970069827478),
                           ('APs', 0.17024760135017813),
                           ('APm', 0.33163478543717106),
                           ('APl', 0.38858065511848544)])),
             ('segm',
              OrderedDict([('AP', 0.28114866256991533),
                           ('AP50', 0.47389863964406576),
                           ('AP75', 0.2971631326535043),
                           ('APs', 0.12308321652841664),
                           ('APm', 0.30504965888965696),
                           ('APl', 0.41763089921672736)]))])

February 10, 2019

The model finished training after 3.5 days. Final output is prety close to the original.

model_final.pth

This is the final output final that gets loaded by PyTorch

PATH = "model_final.pth"
model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))

Final scores

OrderedDict([('bbox',
              OrderedDict([('AP', 0.3787438742878195),
                           ('AP50', 0.5955128018290543),
                           ('AP75', 0.41442391560475855),
                           ('APs', 0.21670904937263522),
                           ('APm', 0.41267339913138185),
                           ('APl', 0.49431726896178174)])),
             ('segm',
              OrderedDict([('AP', 0.34392779458751377),
                           ('AP50', 0.5608730402994763),
                           ('AP75', 0.3648390120652486),
                           ('APs', 0.15867316023179875),
                           ('APm', 0.36995797379803463),
                           ('APl', 0.5052136454262031)]))])

I’m going to collect data and train the model on my labeled data.

Converting Labelme annotations to COCO dataset annotations

2019-01-26T00:00:00+00:00

This is a short blog about how I converted Labelme annotations to COCO dataset annotations.

mlearning

I created the repo mlearning for storing Machine Learning utilities, helper code, etc…

The first main addition to this repo is the converter that I wrote. I takes labelme annotated images, and converts them to the COCO dataset 2014 annotations format.

Why?

I’m training a model on the COCO dataset, so I need a way to generate my own labeled data that can be used by this model. I tried out a few data labeling softwares, like RectLabel and LabelBox, but they were freemium’s and didn’t give me the output, or weren’t that great to use. labelme is open source. I plan to contribute back my COCO dataset converter once I use it more, and it becomes more stable.

Let’s see the converter in action.

Example Output

Here’s example output from the mlearning Github repo.

%matplotlib inline

import os

from matplotlib import pyplot as plt
import matplotlib.pylab as pylab

from mlearning import util
from mlearning.coco import Annotation
from mlearning.plotting import plot_bboxes_and_masks

pylab.rcParams['figure.figsize'] = 12, 12

# must set file paths for one's own data!!!
ann = Annotation(path=os.path.join(os.path.expanduser('~'), 'Desktop/license_plate_detection'))

# this is the function that displays a random example
plot_bboxes_and_masks(ann)

Future plans

I plan to label more images and then train them on the model. I am still doing a proof of concept that I can train on my own hardware, and that the labelme-to-COCO dataset converter that I wrote works.

Training an ML model on the COCO Dataset

2019-01-21T00:00:00+00:00

My current goal is to train an ML model on the COCO Dataset. Then be able to generate my own labeled training data to train on. So far, I have been using the maskrcnn-benchmark model by Facebook and training on COCO Dataset 2014.

Here my Jupyter Notebook to go with this blog.

Okay here’s an account of what steps I took.

Getting the data

The COCO dataset can be download here

I am only training on the 2014 dataset.

I’m working with this project:

https://github.com/facebookresearch/maskrcnn-benchmark#perform-training-on-coco-dataset

And, it must be linked to the correct directories in order to use it with the Github project:

https://github.com/facebookresearch/maskrcnn-benchmark#perform-training-on-coco-dataset

Training

Here are some training commands. These worked. Funny enough, the first command is for 720,000 iterations and it reported that it was going to take 3+ days to complete on my GTX 1080Ti. Also, I could only load 2 images at a time and it took up 10 out of 11GB of memory. This is a big model!

1-16-2019

python tools/train_net.py --config-file "configs/e2e_mask_rcnn_R_50_FPN_1x.yaml" SOLVER.IMS_PER_BATCH 2 SOLVER.BASE_LR 0.0025 SOLVER.MAX_ITER 720000 SOLVER.STEPS "(480000, 640000)" TEST.IMS_PER_BATCH 1

1-17-2019

python tools/train_net.py --config-file "configs/e2e_mask_rcnn_R_101_FPN_1x.yaml" SOLVER.IMS_PER_BATCH 2 SOLVER.BASE_LR 0.0025 SOLVER.MAX_ITER 10 SOLVER.STEPS "(480000, 640000)" TEST.IMS_PER_BATCH 1

Some Things that maskrcnn does really well

everything’s a hyper param

logging - for lots of feedback

single tqdm output for training

COCO Dataset format notes

Things that I learned about the COCO dataset that will be important in the future for training my own datasets with this format are:

Images

Image annotations have this format:

{'license': 3,
'file_name': 'COCO_val2014_000000391895.jpg',
'coco_url': 'http://images.cocodataset.org/val2014/COCO_val2014_000000391895.jpg',
'height': 360,
'width': 640,
'date_captured': '2013-11-14 11:18:45',
'flickr_url': 'http://farm9.staticflickr.com/8186/8119368305_4e622c8349_z.jpg',
'id': 391895}

Annotations

Annotations have this format:

{'segmentation': [[239.97,
24,
04,
49,
84,
41,
5,
79,
62,
46,
13,
17,
55,
71,
37,
41,
41,
51,
54,
95,
87,
34]],
'area': 2765.1486500000005,
'iscrowd': 0,
'image_id': 558840,
'bbox': [199.84, 200.46, 77.71, 70.88],
'category_id': 58,
'id': 156}

segmentation explained:

I was confused how the segmentation above was converted to a mask. The segmentation is a list of x,y points. In this format: [x1, y1, x2, y2, etc...] In this code block, the segmentation list of points is reshaped to [(x1,y1), (x2, y2), ...] and is then usable by matplotlib

for seg in ann['segmentation']:
    poly = np.array(seg).reshape((int(len(seg)/2), 2))

poly becomes an np.ndarray of shape (N, 2) where N is the number of segmentation points.

bbox explained:

bbox is of format [x1, y1, x2, y2]. The bounding box points start at the top left of the image as point (0,0). The x1,y1 offset is from the (0,0) starting point, where the y size goes down, since it’s starting from the top left. Then the x2,y2 values are offsets from the x1,y1 points.

Conclusion

I’ve now learned 2 datasets. Pascal and COCO. Now I know a little more why most projects doing image tasks support both.

What’s Next

Next I want to label my own data and train on it. The last section of the notebook is my attempt at this using RectLabel

I reviewed 3 different applications for labeling data:

Labelbox
RectLabel
Labelme

My criteria for evaluating is that it should be free and I should be able to run the program locally and label my own data as I please. User friendly is better obviously.

If I can’t find something, then maybe I have to create a simple app for labeling data. Definitely doable, but it’d be a detour.

Labelbox seems like it used to be open source, but they turned it into a SaaS, and I couldn’t get it to run.

RectLabel is $5 which isn’t bad, but it didn’t generate the segmentation data in the format that I need.

Labelme seems exactly what I am looking for. Open source. There isn’t a script for exporting to COCO Dataset 2014 format, so maybe this is an opportunity to contribute as well :)

So labeling my own data and training on it is the next step. Okay, until next time.

Random… Extra

Some random notes about things learned when doing this.

commands

count files in a dir - in order to check that the file count matches what was expected, or when the zip file didn’t fully download, or to check the image file count vs. expected.

ls -1 | wc -l

wget in background with no timeout, so I can start the job from my laptop, but process runs as a daemon on the DL machine

wget -bqc --timeout=0 url

fastjar

If a zip file didn’t fully download, fastjar can be used to unzip it.

Trying to unzip the file will give you:

unzip error “End-of-central-directory signature not found”

Use fastjar if the whole zip file didn’t download:

sudo apt-get install fastjar
jar xvf something.zip

StackExchange reference

nvidia-smi

equivalent to “tail nvidia-smi”

# keeps passed traces
nvidia-smi -l 1

# doesn't keep past traces
watch -n0.1 nvidia-smi

Deep Learning Machine Build

2019-01-12T00:00:00+00:00

This blog is about my experience building my first Deep Learning machine. There are plenty of great blogs and videos on how to do this. If you’re interested in how to do this only, skip to the References section, and I put links to all of the blogs / videos that I used.

Acknowledgements

This blog is dedicated to my wife. Without her support, none of this would be possible. My wife told me 6 months ago that I had put a lot of time into machine learning, and I should stick with it. She wouldn’t let change my direction!

Preface

One thing that is ringing true for me lately and with this whole project is:

People can do anything. You might not know how to do it yet but just go for it!

Nobody helped me do this as in writing any actual code or physically assembling things. One person gave me advice on Wifi drivers, thanks Evangelos!, but other than that I figured it out :)

My Story

Okay, so the story…

I got into PyTorch. I had tried Keras and Tensorflow as well, but for some reason PyTorch stuck and I really liked the API. It has pretty much the same API as numpy but with GPU support and for machine learning.

I went to the Pytorch Developer Conference October 2, 2018, and had just been excited since.

I had been using Paperspace for running models on a GPU that I rented hourly at $0.50 an hour plus $8 per month for persistent storage. This worked okay, but my monthly costs were going up as I was getting more into PyTorch, so it started to feel like it was the right time to invest in my own machine. I started at $15 a month, and the last month was $34. The astute will notice this is $360 a year. So, if I can make a $1800 DL machine build last 5 years, then I break even at my current usage, with only power as an additional cost, plus I can use the machine any amount on top of that.

I decided to build the machine myself instead of paying someone to assemble it. I followed a great YouTube video. It’s also in the References section.

Installing the USB Wifi Adapter was probably the hardest part. I did a separate blog on it.

Installing the CUDA drivers was pretty easy, following Nvidia’s very thorough documentation. There’s a link in the References.

Some things I learned

Assempling the DL machine scared me. In the YouTube video they say to be careful at every point or you coulld bend a pin or break something. I was super careful, and didn’t break anything, it just took longer.

Coming from a software background and doing unit testing, it scared me that the feedback loop for assembling the DL machine and to know if it was successful was super long. Once assembled though, small teaks and confirming that things were powered on worked. Things that I had to test and tweak were:

the hdmi input - if you have a GPU the hdmi has to be plugged in to the GPU hdmi port, not the motherboard hdmi port
what usb ports are available. The command lsusb will tell you this

The whole build was a learning experience. I didn’t know anything about setting up a Wifi adapter. I learned a lot and did a separate blog about it here

I learned to follow the directions that come with the Motherboard. I was Googling for what goes where and a diagram, but there was instructions in the box that I didn’t look at until I thought to check the box!!

Same for the power supply, there’s instructions in the box.

RAM is CPU specific. There’s a list of supported RAM sticks that work with the specific CPU. I first ordered RAM, and it said Intel on it, but my CPU is AMD. I didn’t install it, did some research, and had to return it and get the right RAM. Lucky for me, Newegg has a great return policy.

“You will need a monitor, keyboard, and maybe mouse”. I first thought, no I just plug my Macbook into the DL machine. This was maybe me not thinking it through. It might be possible, but for the simplist install, use these things.

An ethernet cable is needed to set up the USB Wifi Adapter. Most motherboards don’t come with Wifi. I had to purchase this separately later once I realized that I needed it.

Parts list

Here is my parts list for the build:

Case - Anidees AI-Crystal-AR Mid Tower
CPU - AMD Ryzen 7 2700X
GPU - GIGABYTE GeForce GTX 1080Ti
RAM - G.SKILL TridentZ RGB Series 16GB (2 x 8GB)
SSD - SAMSUNG 970 EVO M.2 2280 500GB NVME SSD
Motherboard - ASUS Prime X470-Pro
Power supply - EVGA SuperNOVA 1000W
USB Wifi Adapter - COMFAST Wireless WiFi Adapter
USB thumb drive - SanDisk 64GB USB

Miscellaneous

tools to assemple - iFixit Mako Driver Kit - 64 pieces
27 inch LG monitor
keyboard
mouse
ethernet cable

Build Decisions

There were a lot of decisions to make along. Here’s a list of some of them and why I chose what I did.

GPU

I chose to go with a used Nvidia GTX 1080Ti because these go for $500 on eBay, compared to $800 for a RTX 2080 or $1k+ for an RTX 2080Ti. I was using a Nvidia p4000 with 8GB memory on Paperspace, so when comparing specs, the GTX 1080Ti performs better than the p4000 and has 11GB memory, so this would be an upgrade.

CPU

I was comparing the Intel 8700k or the Ryzen 2700x. Ryzen has more cores and more cache memory. Both are 3.7 GHz clockspeed. Here is a comparison.

RAM

Paperspace did give me 30GB RAM. I went with 16GB RAM. Not sure if I need up to 30GB. I never hit this limit on Paperspace. The motherboard has 4 RAM slots, so this could be upgraded.

Power supply

The suggested power supply is minimum 650w with this setup. I chose a 1000w power supply to future proof myself in case I wanted to get another GPU. The case has room for another, and this is Jeremy Howard of https://www.fast.ai/’s recommendation. One for running training jobs and a second for developing. I can only hope to get to that level someday.

Images

DL Machine

All the boxes

Jupyter Notebook

I’m putting a section for this so I remember!

First, port forwarding must be setup for the 8888 port for the Wifi Home network to port forward to the DL machine.

Launch Jupyter on the server

jupyter notebook --no-browser --port=8888

On the client setup port forwarding

ssh -N -f -L localhost:8888:localhost:8888 -i ~/.ssh/my.pem  username@hostname

Go to localhost:8888 in the browser and Jupyter should be loaded.

Useful commands

To see which ssh processes are running, this will show ssh port forwarding

ps aux | grep ssh

Kill a process by PID

kill -9 <pid>

List all Process PIDs accessing a Port

lsof -ti:<port>

Kill all processes accessing a port

lsof -ti:<port> | xargs kill -9

wget in background

wget -bqc url

Wifi Interface setup and commands

Make sure that the driver is downloaded. I used this Github repo for my driver rtl8812AU

Follow the README.md installation instructions:

# cd /usr/src/rtl8812au
# sudo make clean
# sudo make
# sudo make install
# sudo modprobe -a rtl8812au

Then check ifconfig -a to see if interface is present.

iwconfig - will show wifi interfaces

If the interface is present, from here run one of the following to register an ip address to the interface:

sudo ip link set wlp3s0 down
sudo ip link set wlp3s0 up

Or restart machine:

sudo shutdown -r

Check if it’s working

ping -c3 192.168.0.1

If you’re using port forwarding, the IP address may have changed, so check the new IP address and adjust if needed.

References

Nvidia official CUDA Toolkit Documentation:

https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#ubuntu-installation

Nvidia CUDA Downloads Page:

https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=deblocal

Setup DL Machine Tutorials:

Gaming PC Build YouTube video

https://linuxconfig.org/how-to-install-the-nvidia-drivers-on-ubuntu-18-04-bionic-beaver-linux

Medium - Build your own deep learning machine

The $1700 great Deep Learning box: Assembly, setup and benchmarks

Install USB Wifi dongle on Ubuntu 18.04

2019-01-05T00:00:00+00:00

To my benefit or downfall, whichever way you look at it, I bought the Comfast CF-915AC USB Wifi Adapter from Amazon. This proved to be super difficult to install. Me being a noob in Wifi networking in Linux. After 3 part time days of fiddling with it, it is now work. Here’s my steps and how I got it working.

Firstly, the USB Wifi Adapter came with a CD Rom to install the driver. I don’t have a CD Rom drive, so I hit Google, and started looking for what driver to use and how to install.

Under the COMFAST Wiki it’s listed as using the RTL8811AU Realtek chipset.

I tried looking for RTL8811AU drivers via Google, since the product only came with a disk for installation and couldn’t find any drivers that matched. The Comfast official driver download links returned 404 page not found errors in Chinese!

I then found this Ubuntu formus blog and it said that the RTL8812AU driver could be used.

I then followed this StackExchange answer to get the USB driver RTL8812AU correctly installed:

https://askubuntu.com/a/1045927/324695

Using ifconfig, the interface was present, but it didn’t have an IP address.

I am using netplan and my /etc/netplan/01-netcfg.yml currently looks like this:

network:
  version: 2
  renderer: networkd
  ethernets:
    enp6s0:
      dhcp4: true
      nameservers:
        addresses: [8.8.8.8,8.8.4.4]
  wifis:
    wlx40a5ef40734c:
      dhcp4: true
      nameservers:
        addresses: [192.168.0.1, 8.8.8.8]
      access-points:
        "my-wifi-router-name":
          password: "my-password"

This is becasue I have WPA security enabled, so the netplan yaml wasn’t work.

I then followed this StackExchange answer:

https://askubuntu.com/a/16588/324695

Using the wpasupplicant package, I generated a wpa.conf file as suggested, and using these commands could connect to the Wifi Router

# install package
sudo apt install wpa_supplicant

# generates a file
wpa_passphrase SSID PASSWORD > wpa.conf

# command uses the file to connect, and runs as a daemon
sudo wpa_supplicant -i wlx40a5ef40734c -c wpa.conf -D wext -B

Everything is now working. After a reboot, the Wifi still connects. I am smiling!

Steps only

Here are the commands / steps only extracted from the above blogs.

Install driver

First remove any old drivers in /usr/src

Then, install new driver:

sudo apt purge rtl8812au-dkms
sudo apt install git
git clone https://github.com/abperiasamy/rtl8812AU_8821AU_linux.git
cd rtl8812AU_8821AU_linux
sudo make -f Makefile.dkms install
sudo modprobe rtl8812au

Can view that driver is covered:

modinfo rtl8812au | grep A811

Run ifconfig -a and the Wifi interface should be up.

Configure netplan

Configure netplan yaml file, /etc/netplan/01-netcfg.yml as shown above

Run sudo netplan apply to apply changes.

wpasupplicant

Install this package for WPA security support

sudo apt install wpasupplicant

Reboot and check

Reboot computer and check

reboot

# should show the wifi interface now with an IP address if working
ifconfig -a

Logging in remotely

There are 3 steps to do this

1. Assign a static IP address to your home network

This can be done from logging into your Wifi router. The URL to login will be on the bottom of your Wifi router.

I have a Netgear router, so I

2. The DL machine should be DHCP registered to an IP address

For me, this was already done in the previous step using netplan

3. Add port forwarding

This is configurable when logged in to the Router config. For me, this Netgear guide helped me setup port forwarding.

4. Test

I tested by connecting to my phone Wifi hotspot with my laptop then tested that I could ssh to the DL machine

ssh -i ~/.ssh/my.pem username@hostname

Some interesting points

ifconfig

ifconfig will show internet interfaces that are up

ifconfig -a will show all internet interfaces whether they are up or down

ifconfig was showing the Wifi interface as up, but with no IP. This meant that the USB Wifi adapter driver (driver) installation worked and it was registered as an interface in netplan but couldn’t connect to the network. It was the WPA security blocking it at this point.

lshw

lshw -C network will show hardware. -C network to only show “network” class of hardware.

After successfully installing the driver, I could see the wifi network as DISABLED. Then running netplan apply this show the network as enabled.

Commands used

# list usb ports and connected devices
lsusb

# configure network interfaces
ifconfig

# configure wifi network interfaces
iwconfig

# get detailed wifi info
iwlist <interface> s

# dynamic kernal support module
dkms

# ip show an manipulate routing, network devices, interfaces
ip link set <interface> up/down

# list hardware
lshw -C network


netplan apply

netplan --debug generate

# check if there's internet
ping -c3 www.ubuntu.com

Commands

2019-01-01T00:00:00+00:00

I am consolidating my commands that I have spread accross blog posts to just this one blog post. I’m going to maintain and update this post, so I’m not searching accross posts for how to do something!

Jupyter

Jupyter Notebook commands

Remote notebook setup

Here is how to launch Jupyter on a remote server and then view and use Jupyter as if you were running it locally.

On remote server, run this. (Need to be in the directory of the project with all dependencies installed obviously)

jupyter notebook --no-browser --port=8888

Locally. First make sure if the port has been previously bound to, that it’s unbound.

# unbind port
lsof -ti:8888 | xargs kill -9

# connect
ssh -N -f -L localhost:8888:localhost:8888 -i ~/.ssh/my.pem  username@hostname

Go to localhost:8888 in the browser and Jupyter should be loaded.

lsof

List all Process PIDs accessing a Port

lsof -ti:<port>

Kill all processes accessing a port

lsof -ti:<port> | xargs kill -9

kill

Kill a process by PID

kill -9 <pid>

Kill process by name

pkill -f <name-of-process>

wget

wget in background

wget -bqc url

ps

To see which ssh processes are running, this will show ssh port forwarding

ps aux | grep ssh

ls

count files in a dir - in order to check that the file count matches what was expected, or when the zip file didn’t fully download, or to check the image file count vs. expected.

ls -1 | wc -l

fastjar

If a zip file didn’t fully download, fastjar can be used to unzip it.

Trying to unzip the file will give you:

unzip error “End-of-central-directory signature not found”

Use fastjar if the whole zip file didn’t download:

sudo apt-get install fastjar
jar xvf something.zip

StackExchange reference

nvidia-smi

equivalent to “tail nvidia-smi”

# keeps passed traces
nvidia-smi -l 1

# doesn't keep past traces
watch -n0.1 nvidia-smi

# list usb ports and connected devices
lsusb

# configure network interfaces
ifconfig

# configure wifi network interfaces
iwconfig

# get detailed wifi info
iwlist <interface> s

# dynamic kernal support module
dkms

# ip show an manipulate routing, network devices, interfaces
ip link set <interface> up/down

# list hardware
lshw -C network

netplan apply

netplan --debug generate

# check if there's internet
ping -c3 www.ubuntu.com

nvidia-smi

# keeps passed traces
nvidia-smi -l 1

# doesn't keep past traces
watch -n0.1 nvidia-smi

My init file is not running - init.d and chkconfig

2018-12-21T00:00:00+00:00

So, the problem to solve is that an init.d script isn’t running and I’m not sure why. When this problem first occurred I hit Google, and was still lost. After Googling and asking around, here is how to solve it and some helpful commands.

Initial Start

At the start, the contents of my script, let’s call it my-init, looked like this:

#!/bin/bash
# chkconfig:  - 85 15
# start(), stop(), etc...

I had attempted to add it to the init.d startup scripts by calling:

sudo chkconfig --add /etc/init.d/my-init

The first problem was that there were no run levels. This line:

# chkconfig: - 85 15

Has a - where the run levels should go. I changed my runlevels to: # chkconfig: 345 85 15 and then ran these commands to re-add my script:

# remove the old file
sudo chkconfig --del /etc/init.d/my-init

# add file back with new changes
sudo chkconfig --add /etc/init.d/my-init

This site has a good description of runlevels:

https://www.tldp.org/HOWTO/HighQuality-Apps-HOWTO/boot.html

Okay, now runlevels are fixed, but it isn’t running. If that’s fixed, then where to look next?

Look in `/etc/`

If my-init was added correctly, then the file should be in the /etc/init.d/ directory. Run ls /etc/init.d/ and the file should be listed.

Each runlevel has a directory under /etc/ named /etc/rc<run-level>.d/, so /etc/rc3.d/ for all init scripts that have a runlevel of 3.

If the script has been correctly added to runlevel 3 for example, run this command:

sudo ls -l /etc/rc3.d/

And it should show a symlinked file, like:

/etc/rc3.d/S85my-init -> /etc/init.d/my-init

The 85 is the order number that the script will be called at system boot. init scripts are called in ascending order from 1 - 99

Okay, so the script is correctly linked, but it’s still not getting called, where next?

ps and strace

We can look at all running processes and the command used to envoke them using:

ps -ef

The output is something like:

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 02:15 ?        00:00:01 /sbin/init
root      3177  1915  0 02:16 ?        00:00:00 /bin/sh /etc/rc3.d/S82something start

At this point, we should be looking for a process that is running that was invoked by the system start. The system is stuck on an init script with a lower level then ours. In this case 82

A couple of things can be done now. strace, which may be present or can be installed from the package manager, can be used to inspect the current running process and what it’s doing, by running:

strace -f -y -p <PID>

This will show the current output. If there’s nothing useful, then try invoking the failing startup script with verbose output using -x and see what is happening when it first get’s invoked. Command:

sudo bash -x /etc/rc3.d/S82something start

This should show something useful output.

At this point, the issue was solved and the script was looking for some content it couldn’t find. After that failed, it was polling, trying to find it again, and repeat, so it was never exiting.

Conclusion

For me, the problem to be solved and how these commands and concepts go together helped me remember and be able to debug when it happened relatively soon again on a different script and issue. I hope that this helps you as well. Thanks for reading.

-Aaron

Android Meetup - A/B Test Experimentation/Feature flagging

2018-10-20T00:00:00+00:00

This blog post is about the Android Meetup that I went to where a Sr. Android Dev on the Uber Eats team discussed A/B Test Experimentation and Feature flagging

A/B Test Experimentation, What is it?

It’s testing two similar groups with different features to see which features are more popular / better

why is this valuable

apps features are then driven by what customers want

because there if is a bug, the feature can be disabled vs. waiting for a new version of the app to be deployed and the User has to have on auto-update

auto-update - a lot of Users disable this and do manual app updating

can hide new feature behind a flag when it crashes

can flag based on an Android version

How are features enabled / disabled

toggle feature flags on the server

feature flag libraries

Morpheus - Uber

Ik - Facebook

their are open source libraries to do this

Deployment statistics

Android deploy

takes 30 minutes to role out a deploy

iOS deploy

takes 1 week

switches

geofences - like at the airport

devices

target by group

per app flavor

merging code with flags

great because code gets merged that isn’t running yet until the the flag is enabled

FirebaseRemoteConfig

config on the server to toggle feature flags

FirebaseRemoteConfig is loaded on App boot, and then static calls to get data from the cache

if / else pattern with feature flags

eliminate after sometime

Factory pattern to abstract if/else but if/else checks are just moved there

when to launch experiment

don’t experiment on new Users

wait for a User to be logged in

can’t be the first time the app is started

maybe the RemoteConfig callback comes later, and other experiments are being ran, so done on the 2nd bootup or later

commits

message says

how to test
- this says turn on this flag
- restart app
- do this thing
how to revert
- this says to toggle the flag the other way

Scaling

For servers - it’s for traffic

For mobile - it’s for developers

number of developers
as they leave
processes

merging code process at Uber

Lynting checks are run before committing

Espresso tests

Unit tests

No QA

Testing

Abstract Activites

Dagger for Mocks

Try to not use Android as much as possible in the logic - better for testing

big device lab

is international for testing
can remotely connect to a device lab

Uber

Don’t use Gradle - they use Buck (created by Facebook)

Use a module system - there are 1000 modules

40m APK size

obfuscated
minified, etc…

Unobfuscated Debug version - 100m

test code

Redex - for launching an app quickly

Obfuscation - use’s Google Proguard

Anthing for $$ - is internally built

b/c it can’t be down

Lots of 3rd party libs

Open source as much as possible

Buck - is open source

OkHttp - contribute to this

Architecture

MVC - is used, so Activity class is much smaller

things to learn

webview

in app webview

mockups

no testing via Invision mock ups

always testing on Users

Firebase

Google Play services is needed

A/B testing exists out of the box in Firebase

just has to be enabled
Users will get collected into the A/B groups via the Firebase infrastructure

Feature Flags

for new features

variants of current features

will get notified if a feature is a security vulnerability

Locally changing of feature flags

can do this Locally, not on the main FirebaseRemoteConfig

toggling flags

can do this programmatically after so many crashes

things to flag on

city
phone type
Android version

Build flavors

Only have 2 versions:

debug
prod

Libraries

Charles - shows http calls

A/B Test

has to be statistically significant

crashes

usage groups should be statistically significant

groups should have similar profiles in order to be comparable. i.e same phone or Android version

PhD’s evaluate if crashes are statistically significant and stats on feature usage to see what features should be used

Optimizely - library for measuring stats of usage

sample size - depends on the feature

Java version

version is 8 in the demo

other notes

take advantage of new features

Editor Config

show param names for methods (currently I have this turned off)

Locations

wifi trusted locations exists in Android O

code

Dagger - to initialize dependency graph

retrofit

RxJava

PyTorch Developer Conference 2018 - PTDC - part 1

2018-10-13T00:00:00+00:00

This is my cliff notes style blog about my experience at PyTorch Developer Conference that happened on October 2, 2018. This is part 1 because as I was going through my notes, there was so much content and things to read and explain more, that it couldn’t easily fit into one blog.

Talks

The talks section is in loose chronological order of the speaker schedule

Here are the links to all videos from the conference:

Keynote Jerome Presenti

Recent breakthroughs in AI are things such as DensePose. This library has been recently open sourced by FaceBook.

Today FB is announcing PyTorch 1.0

Current flow to get PyTorch into production is:

Pytorch -> ONNX -> Caffe2

PyTorch 1.0 is about making this a more seemless process with the torch.jit module.

We need testing and tooling for ML, like software development 20 years ago. A seamless process to get PyTorch into production should exist, so torch.jit was created.

Hardware breakthroughs like the volta have accelerated ML research.

Operator fusion now speeds up training times.

Deep Dive on PyTorch 1.0

The goal of PyTorch 1.0 is to make putting PyTorch models into production as seamless as possible. For this, the PyTorch team is also considering:

Research and flexibility is central to PyTorch.
Production constraints are:
- hardware efficiency
- scalability
- cross platform

Currently ONNX is the way to put PyTorch models into production.

With PyTorch 1.0 comes the introduction of the torch.jit module. This module allows the developer to write code once, and with the use of torch.jit.trace decorators and other helpers from the module, the Python PyTorch model will be output to run independently from Python, for instance in a C++ program, and be production ready.

PyTorch model development workflow that’s possible as of 1.0:

experiment with models in Python
extract the model using a torch.jit.script
optimize and deploy

Torch scripts do operator fusion for algebraic simplification.

`torch.jit` module

Functions are decorated with torch.jit.script, while subclasses of torch.jit.ScriptModule use a torch.jit.script_method decorator. Also:

records control flow
uses a subset of Python
more info here: https://pytorch.org/docs/master/jit.html#torch.jit.ScriptModule

torch.jit.trace does the following:

runs code straight through
records what happens
can run later without Python present
more info here: https://pytorch.org/docs/master/jit.html#torch.jit.trace

Models can then be saved to disk.

Tutorial for loading a PyTorch module to C++

PyTorch library hierarchy described as:

Aten
 | 
Autograd
 |      \
 |    Python script
C++ extensions

Custom C++ exetensions for PyTorch can be written

Uses PyBind
setup.py can load the custom extension

Deep learning C++ bare metal for ML

All links available at pytorch.org/cppdocs

Distributed training uses C10D

performance driven C++ backend with Python frontend
uses futures with async_op=True
distributed_c10d.py
communication overlap
smart gradient sharing

torch.distributed is deprecated in favor of C10D

Fairseq

Fairseq provides a sequence modeling toolkit

Seq2Seq library that has been instrumental at FB

Won the Global Machine Translation 2018 competition

Trains machine translations (MT) in both directions

PyTorch Translate

https://github.com/pytorch/translate

predict content language
predict view language

Uses this flow

Alten -> decode
            \
            C++ Beam search
           /
Encoder -

FBGEMM for quantization

PyTorch Text has DataLoaders and abstractions for NLP

AI at Facebook

2.2B Users

300+ trillion predictions made per day

FB AI Infrastructure runs many experiments per day

Things that FB needs to achieve it:

AI as a service
Data platform
AI foundation

Current library relationship

PyTorch - research
 |
ONNX - transferring
 |
Caffe2 - production

Things that help scaling ML learning

increased batch sizes
increased learning rate
overlap with training and balance

More notes about the current ML environment

Models are needed for a highly fragmented hardware world. Mobile is highly fragmented and faces size constraints. Python isn’t always present, so production models should run without needing Python.

Quantization has been proven to speed up training without hurting model accuracy.

LibTorch C++ frontend for PyTorch is now available. Or, for download here https://download.pytorch.org/libtorch/nightly/cpu/libtorch-macos-latest.zip

People I met

The community was so friendly and welcoming. Coming from a software development background and not data science, with my knowledge of data science from online classes I was able to communicate, learn and understand everyone at the conference.

Some people I met were:

Conor - ML Intern
Tom - CTO at Paperspace
Francisco - ML Intern at fast.ai
Fedro - Data scientist working with Yann LeCun
James - core PyTorch Dev

I’ll be working on part 2 of this blog in the same style as this one. It will be a continuation of material presented in the conference with lots of links. More great PyTorch to come :)

Python Unit Testing with MagicMock

2018-08-26T00:00:00+00:00

This blog post demostrates how to mock in Python given different scenarios using the mock and pretend libraries.

This blog post is example driven.

What is mocking

Mocking is the use of simulated objects, functions, return values, or mock errors for software testing. It is useful for creating specific application states to run automated tests against.

Mocking can be useful for simulating an API response, where the API response can be made a static return value, so the tests are deterministic vs. an ever changing live API call that takes longer to request, the API could be subject to rate limiting, or maybe it’s a live system that can’t be live tested at will, such as a payment system.

Let’s get started.

Initial Setup

Assume these libraries are imported

# python2
from mock import MagicMock, patch
# python3
from unittest.mock import MagicMock, patch

from pretend import stub

# common example of a library to mock against
import requests

Some basics of the `mock` library

@patch can be used as a decorator for unittest methods. It can target a production test method and make it’s return value a MagicMock instance.

@patch is the same thing as using the MagicMock class. It sets the mocked method as a MagicMock instance for the duration of the unit test method, then set’s back the method to reference it’s original definition when the unit test method completes.

A MagicMock instance can:

capture the arguments that the method is called with
count how many times it’s called
return values that we specify
return the same or different values each time the mocked method is called
be made to raise errors

Mock at the call site

This is the first example, so a lot is going on here.

The prod method must be targeted based on the call site of where the method being mocked is called, not where the method is defined.

The below example shows that even though the method lives in bar.py, when we use @patch to mock the method, it’s targeted using the Python path to foo.Biz.baz. This is the Python path to the call site.

When using @patch, the decorated unit test method will now require an additional argument. This is mock_biz in the example. mock_biz is a MagicMock instance.

# bar.py
class Bar(object):

    def biz(self):
        pass

# foo.py
from bar import Bar

def foo():
    Bar().biz()

# test.py
import unittest
from mock import patch, MagicMock
from foo import foo

class MyTest(unittest.TestCase):

    @patch("foo.Bar.biz") # not -> @patch("bar.Bar.biz")
    def test_foo(self, mock_biz):
        self.assertFalse(mock_biz.called)

        foo()

        self.assertTrue(mock_biz.called)
        self.assertEqual(mock_biz.call_count, 1)
        self.assertIsInstance(mock_biz, MagicMock)

Mock two things

When mocking two methods, the MagicMock arguments to the method should be passed in in the order that the decorators wrap the unit test method. So, in the below example, the @patch for bar.requests.put first wraps the unit test method, so it takes the place of the first argument, then the next is bar.requests.get, so it’s the 2nd argument.

# bar.py
import requests

class Bar(object):

    def sync(self, id, query_first):
        if query_first:
            requests.get('/remote/api/{id}'.format(id=id))

        requests.put('/remote/other/api/{id}'.format(id=id),
                                                     data=current_data())

# test.py
import unittest
from mock import patch
from bar import Bar

class MyTest(unittest.TestCase):

    @patch("bar.requests.get")
    @patch("bar.requests.put")
    def test_foo(self, mock_put, mock_get):
        Bar.sync(id=42, query_first=False)

        self.assertFalse(mock_get.called)
        self.assertTrue(mock_put.called)

Mock as an Argument Captor

Here is an example of the MagicMock instance capturing different arguments. It can capture positional and keyword arguments. In the below example, url and method are positional arguments to Bar.biz, and are stored in mock_biz.call_args[0]

data and headers are keyword arguments, and are stored in mock_biz.call_args[1]

# bar.py
class Bar(object):
    def biz(self, url, method, data, headers):
        pass

# foo.py
from bar import Bar

def foo(url, method='GET', data=None, headers=None):
    Bar().biz(url, method, data=data, headers=headers)

# test.py
class MyTest(unittest.TestCase):

    @patch("foo.Bar.biz")
    def test_foo(self, mock_biz):
        url = '/api/users/{id}'.format(id=1)
        data = {'phone_number': '+17025551000'}
        method = 'PUT'
        headers = {"Authorization": "JWT <your_token>"}

        foo(url, method, data=data, headers=headers)

        self.assertFalse(mock_biz.called)
        self.assertEqual(mock_biz.call_count, 1)
        self.assertEqual(mock_biz.call_args[0][0], url)
        self.assertEqual(mock_biz.call_args[0][1], method)
        self.assertEqual(mock_biz.call_args[1]['data'], data)
        self.assertEqual(mock_biz.call_args[1]['headers'], headers)

Mock a return value

The MagicMock instance return_value attribute’s value can be set to mock a return value.

This is a common use case for how to mock an API response.

# bar.py
class Bar(object):

    def biz(self):
        return 1

# foo.py
from bar import Bar

def foo():
    return Bar().biz()

# test.py
class MyTest(unittest.TestCase):

    @patch("foo.Bar.biz")
    def test_foo(self, mock_biz):
        expected_value = 2
        mock_biz.return_value = expected_value
        
        ret = foo()
        
        self.assertEqual(ret, expected_value)

Mock multiple return values

If the mocked method is called multiple times, and MagicMock.return_value has been set, the same value will be returned each time the mocked method is called.

In order to return different values each time, the value of MagicMock.side_effect can be set as in the below example.

MagicMock.call_args_list can be used to check what the different arguments each time the mocked method is called.

# bar.py
class Bar(object):
    def biz(self, i):
        return expensive_computation(i)
    
    def expensive_computation(self, i):
        pass

# foo.py
def foo():
    bar = Bar()
    for i in range(2):
        value = bar.biz(i)
        process_expensive_value(value)
    
# test.py
class MyTest(unittest.TestCase):
    @patch("bar.Bar.expensive_computation")
    @patch("foo.process_expensive_value")
    def test_foo(self, mock_process_exp_val, mock_exp_comp):
        value1 = 1
        value2 = 2
        mock_exp_comp.side_effect = [value1, value2]
        
        foo()
        
        self.assertTrue(mock_exp_comp.called)
        self.assertEqual(mock_exp_comp.call_count, 2)
        self.assertEqual(mock_process_exp_val.call_args_list[0][0][0], value1)
        self.assertEqual(mock_process_exp_val.call_args_list[1][0][0], value2)

Mock an Exception

MagicMock.side_effect is more commonly used to mock throwing an Exception.

This is very useful because maybe the application hit a particular error state that is seen in the logs that needs to be guarded against. Instead of having to get the application back in that exact state to cause the error, the method that threw the error can be mocked to do throw the error, and then code can be written and tested to react accordingly.

Here is an example:

# bar.py
class Bar(object):
    def biz(self):
        if some_condition():
            raise CustomException()

class CustomException(Exception):
    pass
            
# foo.py
from bar import Bar

def foo():
    Bar().biz()

# test.py
class MyTest(unittest.TestCase):

    @patch("foo.Bar.biz")
    def test_foo(self, mock_biz):
        mock_biz.side_effect = CustomException()
        
        with self.assertRaises(CustomException):
            foo()

Using mock objects

Here is an example of creating a mock object using the stub library in order to create the exact response structure that you want to simulate and react to.

# bar.py
import requests

class Bar(object):
    def biz(self):
        return requests.get('/api/users/')

# foo.py
from bar import Bar

def foo():
    response = Bar().biz()
    if response.status_code == 200:
        data = json.loads(response.content.decode('utf8'))
        process_users_data(data)

def process_users_data(data):
    pass

# test.py
from pretend import stub

class MyTest(unittest.TestCase):
    @patch("bar.requests.get")
    @patch("foo.process_users_data")
    def test_foo(self, mock_process_users_data, mock_get):
        fake_reponse = stub(
            status_code=200,
            content=json.dumps({'users': 'data'}).encode('utf8'))
        mock_get.return_value = fake_reponse

        foo()

        self.assertTrue(mock_process_users_data.called)
        self.assertEqual(
            mock_process_users_data.call_args[0][0],
            json.loads(fake_reponse.content.decode('utf8')))

Other info about mocking

Parent methods can be mocked, so if your code calls super(), the @patch should target the parent class’s method.

Constructors can be mocked. A return_value = None must be set because a Python constructor can’t have a return value.

Conclusion

Mocking has been super useful for me in my professional software developer career. Code gets more and more complicated as applications scale, so being able to isolate inputs and outputs to a function within a system is priceless for unit testing.

Integration testing, the degree of mocking, and overall test strategy has to be thought of a lot. It can’t just be “mock all the things” and then you have 100% test coverage, and an application that is bullet proof. Maybe the mocks are wrong, there’s too many, etc.. Maybe things that don’t need to be tested are tested. Over testing is a thing.

If you have any feedback, drop me a line or comment below.

Thanks

Using Git Aliases and How to Display the Current Git Branch Name in a Terminal Prompt

2018-06-22T00:00:00+00:00

Here are two Git workflow hacks that help me be more productive.

Git Aliases

Git aliases help me to be more productive. With how often we as developers type git commands, using aliases allows for less typing and less typos. Here are my current git aliases.

I like to keep the above git aliases in a separate file ~/.gitshortcuts from my ~/.bash_profile. Then I add source ~/.gitshortcuts to my ~/.bash_profile. This helps keep things neat.

Display the Current Git Branch Name in a Terminal Prompt

I found myself always typing git status to find out what branch I was on before running the next git command. Always having the current git branch name displayed removes the need to always check this. Here’s my terminal prompt.

Use backup branches (Bonus)

I worked with a senior developer that once told me:

Behind the scenes I’m always managing a backup branch

This really stuck with me. His convention was to create a backup branch with the suffix -BAK. This strategy is really helpful. If you’re unsure if a git rebase will work, and all conflicts will be resolved correctly or some other series of git actions, having a backup branch allows you to easily restore from the last time that you were sure about your changes without having to use git reflog and so on.

Conclusion

That’s it for now. I’m sure there’s a lot more git hacks that I should be using. Drop me a line and let me know!

Thank you for reading.

Intro to Graphs

2018-06-05T00:00:00+00:00

The goal of this blog post is to distill down my learning of Graph Theory. The following learning took about 8 hours of my time, so if I can give it to someone in a single blog post with links for further reading, then I hope that I’m adding value to the reader.

I initially did this research before getting into Neo4j, the most widely used graph database. I like to use motivation of doing something new to also learn new things in that area, so it becomes a richer learning experience.

At the time of this writing, the AWS Neptune graph database has recently been made publicly available, so this blog post is potentially more relevant.

What is a Graph?

A Graph is a finite set of connected nodes. These nodes are connected by edges.

In the below graph. Each number is a node. Each line connecting them is an edge.

Nodes

A node is a connected object in the graph. They can also called a vertex (singular) or vertices (plural).

Nodes should have a unique identifier, such as a name or id. They can also store any other amout of arbitrary information. In a social graph, for example, a node would be a person. In a computer network, different machines would be nodes, their IP address or host name could be a unique identifier of the node.

Edges

An edge is the connection between two nodes.

An edge can also be called an arc or a line.

Edges can be directed or undirected (more on this later).

Edges can store information as well. In the case of a social graph, they may store information like how long two people have been connected. In a computer network graph, they could store connection protocols.

Incidents

Incident vertices

Incident vertices are vertices that an edge connects. In the above graph, edge e’s incident vertices are {U,V}

Incident edge

An incident edge is the edge that connects a pair of vertices. Above, {U,V}’s incident edge is e

Wikipedia Reference: https://proofwiki.org/wiki/Definition:Incident_(Graph_Theory)/Undirected_Graph)

Stack Overflow reference: https://stackoverflow.com/a/16886038

Types of Graphs

There are two main types of graphs. Undirected and Directed. This section discusses these in more detail.

Undirected Graph

Undirected graphs are connected graphs, but their is no direction to those connections. There isn’t a direction though between the nodes.

An example of this is LinkedIn. People have connections, or edges in graph theory terminology, but there is not direction to the relationships.

Undirected graphs can be represented as a list of unordered pairs that represent all connections making up the graph.

Unidirectional Graphs

There are 2 types of directed graphs. The first is a unidirectional graph. This type of directed graph has relationships that go in a single direction.

An example of this would be an organizational hierarchy. An employee has a boss, or maybe more than one if he or she is so lucky, but you can’t be someone’s boss and that person also be your boss.

Directed graphs can be represented as a list of ordered pairs.

Here, edges can be called directed edges, arrows, or directed (arcs, lines).

This type of graph is also known as an oriented graph.

Bidirectional Graphs

A bidirectional graph is a directed graph, where edges can be directed in both directions.

An example of this is Twitter. In Twitter you can follow someone, and they can follow you. In this case the relationship could be in both directions, but not necessarily.

Graph Matricies

Different types of matrices can be used to represent a single graph’s information. These matricies can be used to compute different information and for performant lookups.

It is importatant to note that a single matrix may or may not be an isomophism of the graph. This means that all information about the graph may not be represented in a single matrix. Diagonal matrices, for example, tell the number of incoming edges per node, but not which nodes are connected.

Adjacency Matrix

An adjacency matrix is a square matrix that tells which nodes are connected by displaying a 1 if connected, or 0 if not connected.

In an undirected graph, as the above pictures, this matrix is symmetric.

For directed graphs, the adjacency matrix will have a 1 denoting the direction of the incoming edge, otherwise 0, so the matrix is most likely not symmetric.

Coding challenge example

I came accross an adjacency matrix in a coding challenge as part of a job interview. The challenge was to take a 3x3 matrix of unique random values from 1-9. And, then be able to accept a sequence of values, and say how many steps it takes to traverse back and forth on this random 3x3 grid where positions to traverse is a random sequence as well. An adjacent position is 1 step, otherwise it’s 2 steps.

Here’s a concrete example of this problem:

# Rando 3x3 grid
[[2, 5, 9],
 [1, 8, 3],
 [4, 6, 7]]
 
How may steps does it take to traverse [2, 5, 8, 3, 4]?
 
Can you take any random grid and also random sequence and comput this?

This can be solved by loading the random 3x3 data into an adjacency matrix. The adjacent positions in the grid are connections, or 1’s, otherwise they are 0’s. An adjacency matrix can be used as a lookup to solve how many steps it takes.

And adjacency list could also be used.

Degree Matrix

A degree matrix is a diagonal matrix that says the degree of the vertices. The degree of a vertex is the number of it’s incidence edges. The vertices are plotted on the rows and columns.

In the above matrix, vertex 1 in the top left most position has a degree of 4 incidence edges, edge 2 has degree of 3, and so on.

Incidence Matrix

Inidence Matrix of above graph

The columns are the edges, and the rows are the nodes. Edge e1 in column 1 is the incidence edge of vertices {1,2}, e2 is the incidence edge for vertices {1,3}, and so on.

[[1, 1, 1, 0],
 [1, 0, 0, 0],
 [0, 1, 0, 1],
 [0, 0, 1, 1]]

A incidence matrix is a matrix that says which edges connect which nodes by plotting the incidence edges as 1’s, otherwise 0’s.

Incidence matrices are square if the number of vertices equals the number of nodes, otherwise they’re non-square.

Directed Incidence Matrix

For unidirectional graphs these are represented with a -1 for outgoing edges, 1 for incoming edges, otherwise 0.

For a bidirectional graph they follow the same rules as a unidirectional incidence matrix, except for incoming and outgoing endges between nodes, the data is represented by a 2

Adjacency List

Adjacency list of the above graph

The rows represent a node and the columns are the connected nodes values.

[[2, 3, 4],
 [1],
 [1, 4],
 [1, 3]]

An adjacency list is a list of nodes’ adjancent nodes. The first element, [2, 3, 4], is node 1’s adjacent nodes, the second element, [1], is node 2’s adjacent nodes, and so on.

Continued Reading

Graph Data Structure And Algorithms

comprehensive list of graph algorithms

Neo4j Graph Algorithms

Neo4j at the time of this writing has 300+ algorithms for use with it’s graph database

Directed Acyclic Graphs

These are one particularly interesting naturally occuring type of unidirectional graph that is used in project management, and there are software frameworks that implement them as well.

Example Graph Project

This is from a demo project that I did. It’s a Bidirectional Graph of my Github followers using Neo4j, D3.js, and Python Multithreading. Here’s the link if anyone’s interested:

https://github.com/aaronlelevier/neo4j-github-followers

Conclusion

Graphs are very interesting. They appear in so many places like social networks, machine learning, decision trees, and a lot more. This is really just a quick introduction. There are many sub-types of graphs within the above three main categories of graphs mentioned.

This research was originally part of my pre-research in order to learn more about graphs before trying out Neo4j, the largest graph database platform. I first saw Neo4j as recommended experience to have on a job posting, and I didn’t know what it was, so I started to look into it. This research was in my spiral notebook, so now it’s public for other’s if they’re intersted.

Thank you for reading. Happy coding!

virtualenv Cheatsheet

2018-05-26T00:00:00+00:00

A virtualenv is one of the first things a Python programmer learns about. This blog post describes what it is, how to set it up, and examples of some concrete use cases.

What is a `virtualenv`?

A virtualenv is a way of having separate Python environments. It allows you to have different environments for Python 2 and Python 3 and the 3rd party package dependencies to go along with that environment.

Quickstart

Run the following commands to create and activate your first virtualenv

virtualenv for Python3

# installs PIP globally
curl https://bootstrap.pypa.io/get-pip.py | python

# creates a virtualenv
python3 -m venv venv

# activates the virtualenv
source venv/bin/activate

virtualenv for Python2

# installs PIP globally
curl https://bootstrap.pypa.io/get-pip.py | python

# installs virtualenv globally
pip install virtualenv

# creates a virtualenv
virtualenv -p python2.7 venv

# activates the virtualenv
source venv/bin/activate

Verify it’s working

You should see the command line now starting with (venv). This means that the virtualenv is active.

Run which python and it should point to a Python executable inside the virtualenv at venv/bin/python

Quickstart Explanation

First get pip. pip, which stands for “Pip Installs Packages”, is the Python ecosystem’s package installer. If it is not already installed. This can be done using the linked instructions, or with this command.

curl https://bootstrap.pypa.io/get-pip.py | python

If you already have pip or just want to check, run this command to check and output all installed 3rd party packages:

pip freeze

If you currently have other 3rd party packages, and want to clean up the gloal dependencies, then run this command to remove all 3rd party packages:

pip freeze | xargs pip uninstall -y

This is my prefered configuration. To only have virtualenv as the single global 3rd party package. Other packages are local to the virtualenv.

Now, install virtualenv globally.

pip install virtualenv

Check what version of Python is the default version.

python -V

On my machine, the previous command returned Python 2.7.14. So, when I create a virtualenv this will be the default Python version, unless another version is specified.

Create a new virtualenv

virtualenv venv

Why is it named venv?

By convention, the name venv is used as the name of the virtualenv, but any name can be used.

Now, activate the virtualenv

source venv/bin/activate

venv, the name of the virtualenv should show at the start of the command line prompt.

# before
~/Documents/github/jira (master) $

# after
(venv) ~/Documents/github/jira (master) $

If you run pip freeze at this point, there should be no output because this is a new Python environment. The global virtualenv that was just installed should not appear in the output.

At this point, any 3rd party Python package can be installed, just like virtualenv was installed globally, and as long as the virtualenv is active, it will be private to that virtualenv.

To deactivate the virtualenv, run:

deactivate

CheatSheet

Here are some use cases and common commands.

Create a `virtualenv` in Python3

The venv standard library package is available as of Python 3.3 and can be used to create a virtualenv

python3 -m venv venv

`virtualenv` with a specific version of Python

This will create a virtualenv in Python 3.5

virtualenv -p /usr/local/bin/python3.5 venv

Access the `virtualenv` without it being active

By using the path to the Python executable inside of the venv, the virtualenv and all of it’s 3rd party packages will be loaded:

~/Documents/github/jira (master) $ python
Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 12:01:12)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
~/Documents/github/jira (master) $ venv/bin/python
Python 3.6.2 (v3.6.2:5fd33b5926, Jul 16 2017, 20:11:06)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

`virtualenv` with cron

crontab -e
* * * * * /path/to/venv/bin/python /path/to/code.py

Include global Python packages when creating a `virtualenv`

virtualenv --system-site-packages venv

Delete a `virtualenv`

rm -rf venv

Store copy of Python package versions

This is done by convention in a file called requirements.txt

pip freeze > requirements.txt

Install all packages from `requirements.txt`

This command would be used when running another projects code, to install all dependencies of the project.

pip install -r requirements.txt

Conclusion

virtualenv has been very useful. Now days, with Docker, and containers, they’re not always necessary, but if you’re developing locally and have multiple Python projects, they’re a must. You won’t get package conflicts, and can maintain different Python version and environments.

Any questions?

Thank you for reading. If you have any questions or anything that I can add, please let me know. Thanks.

Multithreading in Python

2018-04-05T00:00:00+00:00

This blog post is about Processes, Threads, and the GIL in Python. Because of the way that the Python GIL operates, it may be different than one initially expects, so this blog post is an attempt to discuss this in more detail.

Some background

A Process is a instance of a computer program in execution. It has it’s own address space, data stack, memory, and auxiliary data to keep track of it’s execution.

Threads exist within a process. They can run in parallel to the main thread of a process and share the same data space.

The GIL, Global Interpreter Lock, in Python says that when in effect only one thread can run at a time per Python process.

How the GIL behaves

If the GIL is in effect, it is a Lock, and only one thread can run at a time.

With CPU tasks, the GIL will be in effect. If multiple CPU tasks are being run on the same Python process, the GIL will block the additional tasks if they are non yielding until one completes and the GIL is released.

Yielding means non-blocking, so the use of Threads or the yield statement in Python for example are non-blocking if the task itself doesn’t block.

The GIL is released when doing I/O. I/O invokes built-in operating system C code, so the GIL is released and then reacquired when the I/O completes. This means that threads can be used in Python to improve I/O program performance.

The GIL is also often released in extension code. Here is more detail.

Separate Python processes

Each Python process has it’s own GIL managing access to it’s main thread of execution. Python processes can run in parallel and there are primitives for sharing data between processes.

Some Python modules for multiple threads or processes

The threading module is a higher-level interface built on top of the Python’s thread module. This is shown in the example below.

The concurrent.futures module has the ThreadPoolExecutor and ProcessPoolExecutor class. With these classes, jobs are submitted to a worker pool of a given size and then executed.

The subprocess module allows for the spawning of new processes. This is commonly used in testing for running independent test code in parallel.

The Python queue module adds this functionality. An example of using a Queue to share data between threads is shown below.

Example of improved Python performance with multithreaded I/O

Here is an example of a single vs. multithreaded I/O program using the Github API to demonstrate the difference in performance.

Example output

$ python github_followers.py --username aaronlelevier
single_threaded time(sec): 4.647056104964577
multi_threaded time(sec): 1.0500583100365475
both single_threaded and multi_threaded count: 127

Summary

If you are doing Python I/O, whether it be HTTP data syncing or other I/O tasks, multithreading can be used to improve program performance.

Extension code may also be used for running Python code in parallel on a single process.

More info

Core Python Applications Programming - Multithreading by Wesley Chun

Python Concurrency From the Ground Up: LIVE! by David Beazley

Analyzing the Pandas Series Apply Method

2018-03-04T00:00:00+00:00

I saw the Pandas.apply method and started thinking about this pattern and if it’d be useful to implement on other objects. Here’s a brief blog about the pattern.

The general pattern outside of framework or language specifics is:

Apply an anonymous function to a value or an iterable

Pandas Example

In the case of Pandas.apply it let’s you apply a function to each item in a iterable, which, in the case of Pandas, is each item of a single Series in a DataFrame. Series implements the apply method.

import pandas as pd

df = pd.DataFrame({'number': [1, 2, 3, 4]})

series = df['number']

print(type(series), '\n')

is_even = series.apply(lambda n: n % 2 == 0)

print(is_even)

### OUTPUT ###
# <class 'pandas.core.series.Series'>

# 0    False
# 1     True
# 2    False
# 3     True
# Name: number, dtype: bool

Python object instance example

Can this pattern then be used with Python base class instances inheriting from object? Yes, here’s what that would look like.

class Person(object):
    def __init__(self, name, age):
        self.name = name
        self.age = age
        
    def apply(self, func):
        return func(self)
    
person = Person('Bob', 60)

person.apply(lambda p: "{p.name} is {p.age}".format(p=p))

### OUTPUT ###
# 'Bob is 60'

Defining the apply method allows the lambda to be immediately returned, instead of having to assign it to a reference, then invoke it.

f = lambda p: "{p.name} is {p.age}".format(p=p)
f(person)

### OUTPUT ###
# 'Bob is 60'

Python iterable example

Okay, so let’s use the apply pattern with a Python iterable and see what it looks like.

This code snippet subclasses the Python list standard type, which may or may not be kosher. The point is that the type being subclassed just has to implement the iterable interface. This could be anything, a Django QuerySet for example.

class PersonList(list):
    def apply(self, func):
        return [func(x) for x in self.__iter__()]

person_list = PersonList()

person = Person('Bob', 60)
person2 = Person('Jerry', 55)

person_list.append(person)
person_list.append(person2)

person_list.apply(lambda p: "{p.name} is {p.age}".format(p=p))

### OUTPUT ###
# ['Bob is 60', 'Jerry is 55']

Let’s do the same using Python’s map builtin function.

[x for x in map(lambda p: "{p.name} is {p.age}".format(p=p),
                [person, person2])]

### OUTPUT ###
# ['Bob is 60', 'Jerry is 55']

Python dict example for iter and items

Python iterable example #2 - dict

class EmployeeDict(dict):
    def apply(self, func):
        return [func(x) for x in self.__iter__()]
    
    def apply_items(self, func):
        return [func(k,v) for k,v in self.items()]

employee_dict = EmployeeDict({'manager': person, 'clerk': person2})

employee_dict.apply(lambda x: x)

### OUTPUT ###
# ['manager', 'clerk']

Use apply pattern with dict items

employee_dict.apply_items(lambda k,v: "The {}'s name is {}".format(k, v.name))

### OUTPUT ###
# ["The manager's name is Bob", "The clerk's name is Jerry"]

Summary and Note on `Pandas.apply`

The Pandas.apply method does a lot more than apply a lambda function. Here is the full method signature and link to the source.

It makes sense to do a lot more, because if your class has to defined an extra method apply when you could just be calling the builtin Python map function, then it should be worth it.

Reusable Test Database Pattern in Django!

2018-02-17T00:00:00+00:00

This is a short blog post about the resusabled database pattern that I use for working on Django projects.

Use Case

You have a large Django project where running migrations takes in excess of 30 seconds, etc… Some undesireably long about of time. This is expensive time wise, if you are working on one test, and this is the overhead to re-run the single test each time.

Inspiration

When chatting with my coworker, @approaching236, he recommended looking into how to do this in Django. He said that there’s a 3rd party package in Ruby on Rails that does this and Django should be able to do the same thing. It’s been a huge win as far as speeding up how fast we iterate when testing, needless to say!

Solution

Use The Django DATABASES configuration for a TEST MIRROR database.

This configuration lets you run tests against a database without having to create a test database, run all migrations, and drop the test database each time. Instead, it runs the tests on an existing database, and rolls back each transaction after each unit test method, or test class, so as not to mutate database state for unit testing, as you would expect.

To use this configuration, update the Django project like so:

# myproject/settings/test.py

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'test_db',
        'USER': 'username',
        'PASSWORD': 'password',
        'HOST': 'localhost',
        'PORT': '5432',
        'TEST': {
            'MIRROR': 'default',
        }
    }
}

To use this config, you will want to build an an empty database with the schema migrated one time. After that, migrations no longer have to be run, and unit tests can be re-run without the database migrations overhead.

Helper Bash Functions

Here are some helper bash functions that I’ve found useful with the reusable database pattern. Once in a while there is leaked database state, so I usually make an initial backup after creating the initial empty database, in case that occurs, then a restore of the backup is quick.

# drop, create, migrate schema of a given database
migratedb () {
    export DJANGO_SETTINGS_MODULE=myproject.settings.test
    dropdb test_db
    createdb test_db
    ./manage.py migrate
}

# create a backup of a given database
backupdb () {
    if [ $# -eq 0 ]
    then
        echo "db_name not specified"
    else
        db_name=$1
        db_name_copy=${db_name}_copy
        psql -c "DROP DATABASE $db_name_copy;"
        psql -c "CREATE DATABASE $db_name_copy WITH TEMPLATE $db_name;"
    fi
}

# restore database from a given backup
restoredb () {
    if [ $# -eq 0 ]
    then
        echo "db_name not specified"
    else
        db_name=$1
        db_name_copy=${db_name}_copy
        psql -c "DROP DATABASE $db_name;"
        psql -c "CREATE DATABASE $db_name WITH TEMPLATE $db_name_copy;"
    fi
}