Rob Durst

OpenClaw - Hello Bob! - A First Take.

2026-02-16T00:00:00-08:00

“This is a piece of garbage… are you sure you want it” - IT at a startup firesale as I try to pay $50 for one of the weathered Mac Minis.

Well, more than 8 years later, it turns out not only does ubuntu run well on it, it’s actually got sufficient specs to run an OpenClaw assistant.

Hardware Specifics: Mac Mini 2012 (A1347) - Intel Core i5-3210M, 4GB RAM, 98GB disk

Operating System: Ubuntu 22.04 LTS

So the goal here was to:

see what all the hype is about
see if maybe this is the one thing I’ve always needed to actually adopt Trello…

My setup.

Primary model: anthropic/claude-opus-4-6 (edit recently downgraded)
Fallback model: openai/gpt-5.2 (edit recently downgraded)
Web search: Brave Search API
Task management: Trello
Messaging: Telegram bot (locked to a single allowed user ID)
Orchestration: Docker Compose, running as a single gateway container

Some Lessons

Probably 1k+ blog posts about “how OpenClaw is a game changer”, “OpenClaw just replaced __”, etc. This is not a hype post, just a random list of things I learned setting this up this weekend.

Lesson #1: You Need Cost Controls on Day One

Within the first 24 hours I burned roughly $20 in Anthropic API credits - I told Opus 4.6 to “do research” on some topics. Opus is powerful but expensive and when you give it open-ended research tasks, it happily spawns tool calls, reads files, searches the web, and generates long responses. I keep heavy tasks to other interfaces: ChatGPT, Claude UI, or Claude Code (I have the max subscription). Also attempted to codify this within the TOOLS.md file: “Be lean with Anthropic API. Minimize tool calls, batch operations, avoid reading huge files unnecessarily. Spawn sub-agents for heavy research instead of doing it inline.”

Basically:

API Keys + Tokens != Claude Code Max

IDK why, but this initially caught me off guard.

Lesson #2: Rate Limits and Fallback are Not Plug-and-Play

Model fallback didn’t work out of the box. Had some hacking to get this to work. I may have just misconfigured things. Always a possibility.

According to Claude, this is a three layer system.

Model config in openclaw.json (which model + fallbacks)
Auth profile cooldowns (exponential backoff per provider)
Gateway-level rate limiting (10 attempts/minute, 5-minute lockout)

Actually still failing me today as I tried to set up a research and aggregate type morning brief related to my technical interests and it failed to run.

Still much to do to figure this out. The two dimensions of (a) rate limits and (b) token budgets was an unexpected hurdle.

Lesson #3: Hardening

There’s been a lot of security scrutiny around OpenClaw recently: exposed instances leaking API keys, malicious skills on ClawHub, etc. Beyond the basics (fresh/wiped machine, limit access to credentials, don’t expose to the public internet), I set my Claude Code agents on hardening the Docker setup: cap_drop: ALL, read_only: true, no-new-privileges, non-root user, gateway bound to 127.0.0.1 only.

Not a security expert, but yet-another-reminder to take this stuff seriously is never a bad thing.

Lesson #4: Cron!

Still a ways to go here, but I set up, among others, a couple useful columns in my Trello setup: Today and Tomorrow. Every morning my OpenClaw assistant will merge these and then send me an update on what to do today. Not 100% sure yet this message is useful, but automated board management is something that feels very appealing and worth exploring more fully.

Will follow up here after more experimentation, especially if this is something I continue to tune and it works its way into being a critical process for keeping myself organized.

Stay tuned!

My First Gleam Community Contribution ⭐️

2025-11-23T00:00:00-08:00

By no means am I a newcomer to the world of open source (first contribution was January 4, 2018):

I have contributed to numerous repos across the Stellar ecosphere including stellar-core
I implemented the prompt io method for pyret lang - (PR)
some friends and I launched an ice cream store on the Bitcoin Lightning Network - first of its kind (link)
I even contributed a couple PRs to wasm-bindgen many years ago

However, this first is extra special because it’s for the Gleam programming language community, a language I’m super excited about and will even be speaking about in a few months.

All contributors to the Gleam website this month... including me!

Here is my PR - “[Typo/Bug] Fix Link Reference in Hello Gleam Post”.

Funny enough, my first foray into Stellar core was also a small documentation fix 😁.

Excited for the future and to continue to dive in deeper and deeper to the Gleam ecosystem, not just as a user but also as a contributor.

My Experience at SRECon EMEA ‘25

2025-11-11T00:00:00-08:00

My talk is live (YouTube link)! My slides can also be found here and I blogged about my experience here.

Enjoy!

Gleam Gathering 2025: Talk Proposal Accepted 🎉

2025-11-06T00:00:00-08:00

Super excited to announce that I was selected as a speaker for the first ever Gleam programming language conference! The conference will take place at the Origin Workspace in Bristol on February 21, 2026. Check out the conference website.

Without giving too much away, I’ll be discussing my experience leveraging Gleam to develop the caffeine language compiler for internal reliability tooling at Spring Health.

Ever since joining the Rust community at their third annual conference in Portland, Oregon in 2018, I’ve been seeking that early programming language community experience and the opportunities that come with it. While my formal PL background is limited to brief research, independent studies, podcast binging, and hobby language hacking, I look forward to the day when I can find a way to make working on a compiler with a bunch of awesome, fun, like-minded people my day job.

Contextualizing Reliability

2025-07-07T00:00:00-07:00

Spend a few minutes in the SRE community and you will surely hear: “reliability is the most feature.” From the Google SRE Workbook:

The argument is sound and, as stated, “people don’t normally disagree.” However, what does this look like in practice?

This morning I spent some time contextualizing this argument with numbers.

Setting the Scene

Consider a typical web service. This service has users and we’ll make the following assumptions:

we aim to serve 100% of users
users are aware of and have access to the site
while users have their own reliability issues (i.e. network outages, slow connections, etc.), we’ll ignore those for now

Basic Definitions

Intent: the user has the intention, or desire, to use the site.

Able: the user, when navigating to the site, has the ability to accomplish their goals and extract value from it. Here able will function as a synonym for reliable.

The Interplay of Intent and Ability

As it stands, we have the ideal, 100%, and then reality, the combination of those with intent and the ability to use the site: 100% of those aware and with access can use the site, X% of those have the intent to use the site, and Y% of those are able to use the site.

Mathematically this looks like this:

ACCESS: 100%
INTENT:   X% = ACCESS - THOSE_WITHOUT_INTENT
ABLE:     Y% = X% - THOSE_WITHOUT_ACCESS

Visually:

Which is based on the following truth table:

Considering the extreme cases, lack of intent or lack of ability, we get:

No ability: if 0% of those with intent have the ability, 0% will access. Thus, even if 100% of users have the intent (i.e. design, marketing, engineering, etc. have done their job exceptionally well), no one can access the site.

No intent: if 0% of users have the intent of using the site, 0% will access. Thus, even if the site is completely reliable, if no one wants to access it, no one will.

In reality we have some combination of the two.

The Don’t Leave Any Money on the Table Principle

At the end of the day, we live in a capitalist society. Most web services provide some value to the user in exchange for time and/or money (which then, in turn, pays your salary). So, business leaders desire to convert as much of their addressable market to customers. In business lingo, it’s something like this:

Image source.

However, converting 100% of users with access to 100% with access and intent is a challenging (and maybe impossible?) endeavor. It would involve turning the following triangle into a square? 🤔

Image source.

As a site reliability engineer, designing an effective user funnel is outside my wheelhouse.

However, I do believe the following.

If we’ve invested the time and money to achieve a decent conversion rate, we are, up to a certain point, wasting resources if we leave any users with intent unable to access our site.

No site is perfectly reliable. In fact you don’t want it to be (each extra nine of reliability is exponentially expensive). However, there is a threshold of level of service that your users expect, which, if not met, results in missed business. Defining that threshold (service level objective) based on a decent proxy of the user experience (service level indicator) is not easy (heck… there is a whole book about this stuff). However, as hard as it is, service reliability engineering is a science. There are principles, best practices, patterns, and existing precedents (across industry domains, medicine, bridges, airplanes) we can follow.

Conclusion

Based on the above, any discrepancy between the level of user intent and the level of user ability is, in my opinion, money on the table. So, circling back to our original point of investigation, reliability is the most important feature: without it, the rest of your organization’s work loses its effectiveness and momentum and in the worst case, is completely wasted.

SRECon Emea 2025 Accepted!

2025-07-01T00:00:00-07:00

🎊 Accepted 🎊

Obligatory email screenshot.

A Spoiler 👀

Talk Short Description

Scaling a site-reliability culture from the ground up at a hyper-growth, resource-constrained startup is a uniquely challenging endeavor: plenty of playbooks exist, yet every startup’s socio-technical reality is its own puzzle. And while “reliability is the most important feature” rings true, tight deadlines and shifting priorities often sideline proactive reliability initiatives. Thus, since these cycles are a precious commodity, ensuring their success is paramount.

By retracing our SLO adoption journey, highlighting failures, missteps, and near wins en route to our eventual breakthrough, we uncover an effective litmus test for gauging readiness, or recognizing when a team isn’t quite there yet.

Today, this readiness framework guides how we assess the timing of reliability investments at Spring Health. It also serves as a practical tool for teams in fast-growing engineering orgs still early in their reliability journey, especially those navigating similar constraints.

En Route to Becoming an Active Member of the Global SRE Community

In preparation for becoming a more active participant in the SRE community, working on my personal brand.

Socials (short form):

Blogging (medium form):

Futhermore, while I have done some blogging here in the past:

I will be working to publish more, especially on the things I learn and build at Spring Health and from my side independent research work at Zen SRE Studio.

Publications (long form):

I setup my Orcid which will be used for my talk for the conference. Also linked my contribution (not authorship) from some undergrad work in 2020 (Nessie: Automatically Testing JavaScript APIs with Asynchronous Callbacks).

SRECon Emea 2025 Submission

2025-05-21T00:00:00-07:00

I Am Now An SRE

After starting the team and helping lead the development of Spring Health’s in-house provider availability system (a story that deserves its own post), I officially transitioned into an SRE role in mid-2024. My mission: to make Observability Driven Development the default. It’s an ambitious culture shift and one that leans more on the socio than the technical. That makes it a challenge, but also a rich learning opportunity (all problems are people problems at the end of the day, right???).

As part of this move, I’ve been leading our internal effort to adopt SLOs. This is actually our fourth attempt. Since I already used my learning budget on Local First Conf, I needed a different path to get to SRECon Emea 2025. So I’m going the speaker route and leveraging this story of our journey to SLO Readiness as my way to get there.

Today I officially submitted my talk proposal! While I very well might not be accepted, it’s the first step in the journey towards becoming a speaker and really a participant in not just SRE at Spring, but the entire SRE community. Excited!

While Hope is Not a Strategy may be the SLO saying, hopefully I follow up soon with good news… wish me luck!

Will post the details of the submission here later following the review period.

Digicus Whitepaper

2024-10-10T00:00:00-07:00

Scratch for Smart Contracts

Rob Durst
me[at]robdurst[dot]com

With it’s launch in 2015, Ethereum ¹ brought smart contracts to the forefront of the blockchain ecosystem. These smart contracts were based on the (original?) definition laid out by Nick Szabo in 1996 ²: “[a] smart contract is a set of promises, specified in digital form, including protocols within which the parties perform on these promises.”

As implemented today, smart contracts are typically turing complete and rely on a VM coupled with the underlying blockchain nodes. Like any typical VM, a “smart contract handling VM” handles some set of instructions, typically byte code, charging fees for execution, storage, etc. Contracts may be written in a “blockchain native” language like Solidity ³ or a pre-existing language that is compiled to a format the VM supports (i.e. Rust compiled to WASM which may run on Stellar’s Soroban ⁴). While these smart contract development ecosystems have matured greatly over the past few years, writing a non-trivial, secure, correct smart contract is not easy (Google smart contract hack and you’ll see a plethora of examples). Today, various tools exist to solve this problem. Grouped generally, there are a few approaches:

Allow folks to use the programming language they are most familiar with (Hyperledger Fabric ⁵)
Give folks the best developer experience possible
Provide a set of re-usable, well vetted, well tested, and well understood boiler-plate, templated smart contracts (OpenZepplin ⁶)

We propose a fourth approach: making the smart contract language itself more approachable. Fundamentally, the barriers to entry are overwhelmingly numerous. For example, in order to be a competent smart contract developer on the Stellar’s Soroban, you need to be familiar with how blockchains work, have a strong foundation in smart contracts, and be functional in the Rust programming language. Expecting non-technical participation and a painless onboarding experience for newcomer developers is not realistic. We believe a blockchain agnostic, visual, block-based programming environment will facilitate a more accessible onboarding experience for all. Specifically, we’ve built Digicus, an intuitive, simple, Lego-like IDE, backed by Digit, a compiler plugin framework for seamless transpilation between existing smart contract languages (i.e. Solidity) and frameworks (i.e Stellar’s Soroban Rust SDK) and our intermediate representation (DTR: Digicus Textual Representation). Digicus is the blockchain ecosystem’s Scratch ⁷, a smart contract 101 platform that will help onboard newcomers into our world, aiding their journey to “smart contract 201” and beyond.

Table of Contents
Existing Work and Inspiration
Solution
- An Overview
- The Digicus IDE
- A Compiler Plugin Framework
- Digicus Textual Representation
- Supported Instructions
- Instruction Execution Flow
- Valid Types
Conclusion
- Latest Status
Grant Acknowledgements
References

Existing Work and Inspiration

The idea of a platform like Digicus is not a completely novel idea.

A 2021 paper titled SmartBuilder: A Block-based Visual Programming Framework for Smart Contract Development states ⁸:

In this paper, we introduce SmartBuilder, a block-based visual programming framework for building smart contracts using extended Google Blockly libraries. It allows Hyperledger Fabric smart contract (also known as Chaincode) development learners or non-expert users to build smart contracts using visual blocks without writing a single code

This solution claims to have developed a commercial application of their proposed framework for the Aergo blockchain ⁹.

A second paper, Scilla: a Smart Contract Intermediate-Level LAnguage focuses on the idea of a smart contract intermediate language ¹⁰:

This paper outlines key design principles of Scilla—an intermediate-level language for verified smart contracts. Scilla provides a clean separation between the communication aspect of smart contracts on a blockchain, allowing for the rich interaction patterns, and a programming component, which enjoys principled semantics and is amenable to formal verification. Scilla is not meant to be a high-level programming language, and we are going to use it as a translation target for high-level languages, such as Solidity, for performing program analysis and verification, before further compilation to an executable bytecode. We describe the automata-based model of Scilla, present its programming component and show how contract definitions in terms of automata streamline the process of mechanised verification of their safety and temporal properties

This paper focuses on verification, not ease of development (however, a very interesting idea Digicus might be able to adopt later).

Furthermore, the idea of a compiler framework is heavily influenced by LLVM ¹¹.

LLVM defines a common, low-level code representation in Static Single Assignment (SSA) form, with several novel features: a simple, language-independent type-system that exposes the primitives commonly used to implement high-level language features; an instruction for typed address arithmetic; and a simple mechanism that can be used to implement the exception handling features of high-level languages (and setjmp/longjmp in C) uniformly and efficiently. The LLVM compiler framework and code representation together provide a combination of key capabilities that are important for practical, lifelong analysis and transformation of programs.

The beauty of LLVM as a general target for high level languages allows language developers to get (a) a bunch of optimizations and (b) a whole host of backends for free. Its importance to the compiler ecosystem cannot be understated.

Digicus is a novel combination of these ideas, designed practically with the intention of developing a production application.

Solution

An Overview

Our solution requires three distinct pieces of technology:

an IDE (a visual programming language)
a compiler plugin framework
a well-defined intermediate representation language

The IDE is where a user will spend most of their time viewing and modifying smart contracts. The IDE is integrated with Digit, our compiler plugin framework which is responsible for transpiling between source language(s), DTR, and target language(s). Within Digit resides an implementation of dtr_core, a Ruby gem that adheres to the DTR specification. While dtr_core is effectively the reference implementation, based on our appreciation of Rob Pike ¹² and his acknowledgement of the importance of the Go spec ¹³, please consider the specification of DTR to be the source of truth and dtr_core to be just an implementation.

Visual Programming Language

The smart contract developer tooling community has yet to find PMF for a tool to aid novice developers in the creation of secure and performant smart contracts. While we don’t naively believe this is due to a lack of effort, we instead attribute it to implementations focused on the wrong level of abstraction.

Today, there are more-or-less three levels of abstraction:

high level programming languages (i.e. Solidity)
visual programming languages that syntactically mirror source code
well-vetted template libraries (i.e. OpenZepplin)

Thus far, Digicus (and many of the earlier VPL attempts in this space) have been focused on (2). While not yet fully formed, designed, or even really described in this specification document, we believe there is a 2.5. In such a world, we will design an abstraction over high level source code which eliminates known security violations and/or performance degradations, guiding the novice developer towards a happy path solution (we call this a “shift left” solution” as we’re shifting the addressing of security and performance closer to the beginning of the SDLC).

[August 2024] We don’t yet know what this looks like. We’re engaging in conversations across our network in pursuit of a solution… stay tuned!

Architecture:

Our compiler plugin framework is leveraged by Digit via the Bean Stock Compiler Plugin (pod) Hub ¹⁴. Bean Stock sources pods via git submodules, where each submodule has a bean_stock_manifest.yaml file describing the pod. Here is an example:

name: soroban_rust_frontend
type: backend
version: 0.16.1
description: Frontend for Soroban Rust SDK. Compiles Rust to DTR.
command: make run FILEPATH
author: rob | me@robdurst.com
source: https://github.com/spaced-out-thoughts-dev-foundation/soroban_rust_frontend

The Digit server’s Dockerfile can then fetch and compile/setup each submodule upon docker build based on the manifest. Wthi this in place, Digit will be able to handle various transpilations at runtime as requested by Digicus.

The Digicus IDE

Below is an initial mockup of the MVP user interface.

When developing with Digicus, users will be able to name the contract, drag and drop block components into the interactive function creator, and then they will be able to perform minimal, chain agnostic testing of the contract. The UI/UX for this will be based heavily off of the well-tested, mature Scratch programming platform. Furthermore, a plethora of features will be supported to make this a first class smart contract development interface, including but not limited to:

common vulnerability detection
autocomplete (eventually some sort of Copilot-esque flavor as well ¹⁵)
realtime syntax/semantic error recognition with sensible warnings and recommendations for improvement
chain specific integration testing

The IDE will support two workflows: (1) upload and modify and (2) net new creation.

Workflow #1: Upload and Modify: users will be able to upload a smart contract from one of the supported smart contract programming languages/frameworks. Once uploaded, users can modify as they wish.

Workflow #2: Net New Creation: users will be able to create a generic smart contract from scratch. However, they will not always need to start from scratch; a library of common templates will be available for bootstrapping.

While a full IDE is the goal of this work, we anticipate the visualization feature to be immediately useful with the possibility to embed within existing smart contract analysis/explorer/etc. software today.

A Compiler Plugin Framework

Digit, Digicus’s compiler plugin framework, enables the ambitious software developer to add support for their blockchain smart contract programming language (or framework) of choice. To do so requires the development of a bidirectional transpiler, from source to DTR and DTR to source. We aid this development by way of example plugins and a central repository for hosting these such that they are discoverable and auto-magically integratable with the IDE.

A plugin is an executable binary with two methods:

to_dtr(source_language: String) --> String
from_dtr(dtr: String) --> String

The Spaced Out Thoughts Development Foundation aims to initially support:

Stellar’s Soroban Rust SDK

We’re biased, we love Stellar and given the novelty of their solution to global cross border payments, a strong technical foundation, and the very recent launch of Soroban, focusing here was a no brainer.

However, given the flexibility of the compiler plugin designed, one day, Digit will be able to support the following:

Digit enables the seamless translation of contracts not only from source to DTR, but from source A to Target B. There are numerous ecosystem examples of targeted transpilation libraries ¹⁶ and even blockchain interoperability¹⁷. Digicus aims to go one step further, the generalization of smart contracts as whole. We believe this is layer 3 at its finest!

Digicus Textual Representation

DTR is defined by ASCII text files with the .dtr ending. Each file contains the definition for a single contract and consists of five sections:

Contract: where the name of the contract is specified
State: where we define the type and initial data for each variable
UDTS: where we define user defined types
Interface: where we define each externally visible method (name, input, output, body)
Helpers: where we define each internal method (name, input, output, body)
Non-translatables: some source languages are more expressive than the target language. Thus, in an effort to allow easier transpilation back, we’ve included an optional section to store relevant metadata unique to the source.

If a section is omitted, it will be assumed to be non-existent.

Overall structure:

[Contract]: CONTRACT_NAME

[State]:
  * STATE_DEFINITION
  ...
  * STATE_DEFINITION
:[State]

[UDTs]:
  * UDT_DEFINITION
  ...
  * UDT_DEFINITION
:[UDTs]

[Interface]:
  * FUNCTION_DEFINITION
  ...
  * FUNCTION_DEFINITION
:[Interface]

[Helpers]:
  * FUNCTION_DEFINITION
  ...
  * FUNCTION_DEFINITION
:[Helpers]

[NonTranslatable]:
   TEXT_IN_ANY_FORMAT
:[NonTranslatable]

STATE_DEFINITION:

  [STATE_NAME]:
    * Type: TYPE_NAME
    * Initial Value: VALUE

Where TYPE_NAME can be any valid type. (TBD) do we want a 1-to-1 mapping of Rust types to .dtr types? Yes for MVP, but maybe as we implement this it will be (a) easier to generalize and/or (b) clear that new users need not care about this and so we can make smart decisions for them. Eventually we may rethink this when we expand to other frontend targets.

UDT_DEFINITION:

All user defined types are objects. An object is a named entity with zero or more typed fields.

[Foo]:
  * Bar: Integer
  * Baz: String

Enums are fairly typical. Consider how we might represent the two flavors of Rust enums as objects.

Typical enum:

// Rust
enum TrafficLight {
    Red,
    Green,
    Yellow,
}

...

// Digicus
// UDT
[TrafficLight]
  * Kind: Integer

// Function
[TrafficLightValue]
  * Input
  {
    traffic_light: TrafficLight
  }
  * Output: Boolean
  * Instructions
    $
      { instruction: conditional_jump, input: (greater_than, traffic_light.Kind, 3, 1), scope: 1 }
      { instruction: return, input: (False), scope: 1 }
      { instruction: return, input: (True), scope: 1 }
    $
...

Enum variant:

// Rust
enum Shape {
    Circle(f64),
    Rectangle(f64, f64),
}

...

// Digicus
[Shape]
  * Circle: Tuple<Float>
  * Rectangle: Tuple<Float, Float>

FUNCTION_DEFINITION:

  [FUNCTION_NAME]:
    * Input:
      {
        INPUT_NAME: TYPE_NAME,
        ...
        INPUT_NAME: TYPE_NAME
      }
    * Output: TYPE_NAME
    * Instructions:
      $
        { id: UUID, instruction: INSTRUCTION_NAME, input: (VALUE_NAME: VALUE,..., VALUE_NAME: VALUE), assign: ASSIGN_NAME, scope: SCOPE_LEVEL },
        ...
        { id: UUID, instruction: INSTRUCTION_NAME, input: (VALUE_NAME: VALUE,..., VALUE_NAME: VALUE), assign: ASSIGN_NAME, scope: SCOPE_LEVEL }
      $

Note:

the input section is optional
the output section is optional
output may be at most one value
an instruction need not have an input nor an output
an ASSIGN_NAME is a local variable which may be referenced by following instructions
INSTRUCTION_NAME is the name of a subset of supported rust expressions see supported instructions
SCOPE_LEVEL is an unsigned integer detailing which level of accessibility an instruction is executed within. It is required

Supported Instructions

Defining a set of common instructions across all blockchains is challenging. Thus, it is likely this list will be in flux;until otherwise stated, this list may be incomplete.

Operation Name	Inputs	Assign	Category	Description
assign	1	Required	Basic	given some input value, assign to `ASSIGN_NAME`
evaluate	>1	Optional	Basic	given a method name and 0 or more inputs, execute method. At this time, evaluate is a fairly loose catch-all for not explicitly defined operations
print	1	None	Basic	given some value, print it to standard out
exit_with_message	1	None	Terminating	immediately end execution, returning message
return	1	None	Terminating	return from function with input value
and	2	Required	Logical	assign to `ASSIGN_NAME` result of “and-ing” two values
or	2	Required	Logical	assign to `ASSIGN_NAME` result of “or-ing” two values
goto	1+	None	Control Flow	conditional if two inputs. In this case, first input is the condition to evaluate. If that is true, or there is only one input, move in code to the first input (an instruction id)
jump	1+	None	Control Flow	conditional if two inputs. In this case, first input is the condition to evaluate. If that is true, or there is only one input, jump to scope level
end_of_iteration_check	1	Required	Control Flow	check on input to see if at end of iteration. Return result to `ASSIGN_NAME`
field	2	Optional	Object	access a field on an object and assign result to `ASSIGN_NAME`
instantiate_object	1+	Optional	Object	initialize an object by first passing in the type of object and the passing in each initial values for its fields. Supported types here include: `Dictionary`, `List`, `Range`, `Tuple`, and `UDT`. For UDTs, the second input is the name of the UDT.
add	2	Required	Binary	assign to `ASSIGN_NAME` result of adding two value
subtract	2	Required	Binary	assign to `ASSIGN_NAME` result of subtracting two value
multiply	2	Required	Binary	assign to `ASSIGN_NAME` result of multiplying two value
divide	2	Required	Binary	assign to `ASSIGN_NAME` result of dividing two value
try_assign	2	Required	Basic	assign to `ASSIGN_NAME` the result of the attempted assign of input index 0 to input index 1
increment	1	None	Unary	An operation to increment the input (however that may be implemented)
unary	2	Optional	Unary	Basic unary operations like `!` and `-`
break	0	None	Control Flow	Breaks out of the current execution flow.

Instruction Execution Flow

Instruction flow is determined by two main factors:

the top to bottom position of the instruction within the collection
the scope of the instruction

Much like traditional computing, instructions here, are executed in the order defined, from top to bottom. However, the instruction pointer ¹⁸ is scope-aware; thus, if the current scope is 7, but the instruction pointed at is a scope of anything besides 7, the instruction pointer will transition to the next instruction in the collection. Scopes help us isolate functional blocks of logic and as we move through the execution of a program, we simulate a stack frame based on these scopes.

To illustrate, consider the representation of an if-else block in DTR:

// ====== scope 0 ======
// CONDITONAL_RESULT_1 = x > 10
{ instruction: evaluate, input: (greater_than, x, 10), assign: CONDITONAL_RESULT_1, scope: 0 }
// if CONDITONAL_RESULT_1, scope => 2
{ instruction: jump, input: (CONDITONAL_RESULT_1, 2), scope: 0 }
// else, scope => 3
{ instruction: jump, input: (3), scope: 0 }

// ====== scope 2 ======
// print("we are in scope 2")
{ instruction: print, input: ("we are in scope 2"), scope: 2 }
// scope => 0
{ instruction: jump, input(0), scope: 2}

// ====== scope 3 ======
// print("we are in scope 3")
{ instruction: print, input: ("we are in scope 3"), scope: 3 }
// scope => 0
{ instruction: jump, input(0), scope: 3}

// ====== scope 0 ======
{ instruction: print, input: ("we are back in scope 0"), scope: 2 }

We start by evaluating x > 10. If it is true, we navigate to scope 2, print “we are in scope 2”, then navigate back to scope 0 where we print “we are in scope 0”. If x is less than or equal to 10, then we navigate to scope 3, print “we are in scope 3”, finally navigating back to scope 0 where we print “we are in scope 0”.

Equivalent ruby code would look like this:

if x > 10
  print "we are in scope 2"
else
  print "we are in scope 3"
end

print "we are in scope 0"

Going “backwards” to repeat ourselves

There are times where it might be useful to repeat some logic. In most programming languages, you can accomplish this with a loop:

println!("Before loop")

for i in 0..42 {
  println!("Iteration: {}", i);
}

println!("After loop");

With only the basic scope-based sequential execution we’ve laid out thus far, the only way to replicate the above looping logic is by loop unrolling ¹⁹. However, that is not practical. Thus, we’ve introduced the concept of an instruction id and a goto instruction type. The goto instruction (as outline in the table of the preceding section) accepts a single input, the instruction id to navigate to. Thus, we can replicate the above code with the following collection of DTR instructions.

{ id: 0, instruction: "print", input: ("Before loop"), scope: 0 }
{ id: 1, instruction: assign, input: (0), assign: i, scope: 0 }
{ id: 2, instruction: evaluate, input: (less than, i, 42), assign: LOOP_CHECK, scope: 0 }
{ id: 3, instruction: jump, input: (LOOP_CHECK, 1), scope: 0 }

{ id: 4, instruction: "print", input: ("Iteration: {}", i), scope: 1 }
{ id: 5, instruction: "increment", input: (i), scope: 1}
{ id: 6, instruction: "goto", input: (3), scope: 1}

{ id: 7, instruction: "print", input: ("After loop", scope: 0) }

Here we assign 0 to i. Then we check if i is less than 42. If so, we navigate to scope 1. In scope 1, we print our message, increment i, then navigate back to the loop check. This code will continue executing until we fail the loop check. Once we fail, we don’t jump to scope 1 and so instructions with the ids of 4, 5, and 6 are skipped. Thus, we print “After loop” and the program terminates.

At this point, we’ve outlined the basics of instruction execution flow for DTR.

Valid Types

Basic Types

A.K.A. primitive types.

Digicus supports the following:

Address
BigInteger
Boolean
Float
Integer
String

Container Types

Types that contain types.

Digicus supports the following:

List: homogenous, single unary (ex. List<Integer>)
Dictionary: homogenous in dimension, binary arity (ex. Dictionary<String, Integer>)
Range: an iterator over some values (ex. Range<Integer>)
Tuple: n-ary arity (ex. Tuple<Integer, Integer, String>)

Conclusion

We believe Digicus will revolutionize the way budding smart contract newcomers onboard into this powerful and wonderful ecosystem. Furthermore, as with scratch we expect folks to eventually graduate from Digicus native smart contracts to leveraging the core programming language of their blockchain of choice. At this point, Digicus will become a sidecar IDE; opened in a window adjacent to their text editor, smart contract developers will be able to leverage the incredible tooling of Digicus to enhance their current development flow.

Presented visually, we believe the evolution of Digicus will look like this:

As time goes on, we will work with developers to solicit feedback and conduct surveys in order to ensure we have the greatest possible positive impact on this community. These survey results will be posted on our website.

Until then, please feel free to follow our coding journey here and experiment with the IDE here.

References

TypeScript's keyof and Structural Typing

2023-07-17T22:00:00-07:00

As I explore the possible solution space for designing a type for the Pyret programming language with Ben Greenman, I’ve spent some time grappling with the concept of structural type compatibility. Specifically, I am interested in TypeScript’s type system and the keyof operator. This blog post focuses on summarizing what I’ve learned thus far (DISCLAIMER: this post might contain inaccuracies as I continue to refine my understanding).

If these concepts interest you, stay tuned. I plan to continue exploring these ideas and ultimately aim to answer questions like: can we leverage this concept within Pyret?

Type compatibility

We’ll use the term type compatible throughout this post, so let’s take a moment to formally define it. As best summed up by Nominative and Structural Typing, type systems must be able to answer two type compatibility questions:

equivalence: given two type expressions T1 and T2, are T1 and T2 equivalent? In other words, are all objects of type T2 valid objects of type T1, and vice versa? Commonly written as T1 == T2
subtype: given two type expressions T1 and T2, is T1 a subtype of T2? In other words, are all objects of type T1 also objects of type T2? Commonly written as T1 <= T2

Generally, there are two main approaches to answering these questions: structural typing and nominal typing.

Structurally typed: type compatibility is determined by the structure of the types

Nominally typed: type compatibility is determined by the name of the types

Consider the following example.

foo {
    bar: () => string
}

baz {
    bar: () => string
}

In a nominally typed language, foo != bar, however in a structurally typed language, foo == bar.

What about Duck Typing? Duck typing is dynamic type compatibility at runtime; both structural and nominative typing are static. Furthermore, typically at runtime we just care if an object has a specific property more so than if an object is really some type (i.e. don’t care as much about the type’s name or entire type structure).

Which is better? Of course, it depends: one of the main pros of nominal typing is differentiation between two types of the same structure, i.e. MeasurementInFeet and MeasurementInMeters. Structural typing is advantageous for its flexibility.

While we’re comparing structural and nominal type systems, note that hybrid systems (I don’t believe this is a technical term) exist.

At a base level, C++ is nominatively typed–two classes with different names are not considered equivalent; and subtyping must be explicitly declared (via inheritance). However, the template system uses structural typing–any type can be an argument to any template; and the compiler won’t complain unless a particular instantiation of a template is incompatible with its declaration. – Nominal and Structural Typing

Here is a nice table summarizing type compatibility for a large set of programming languages: Comparison of programming languages by type system - Wikipedia.

Structural Typing

I have much more experience with nominally typed systems: C#, Python, Java, Rust (albeit a bit fuzzy), Haskell (which is mostly nominal), etc. Thus, I will focus here on structurally typed languages.

TypeScript

TypeScript is a structurally typed programming language that transpiles to Javascript allowing for engineers to gradually introduce types to an existing, dynamic, duck typed, Javascript codebase and/or leverage the expansive JavaScript ecosystem with an additional layer of safety.

Unless you’re already familiar with TypeScript’s type system, I’d recommend opening up a TS Playground online editor in another screen as you read along.

In a structurally typed language, since we care about structure, not about name, an analogous operator for .class or .type is keyof (hand wavy, hand wavy), where keyof returns a concatenated collection of object properties.

type Point = { x: number; y: number };

// "x" | "y"
type P = keyof Point;

TypeScript does have the typeof operator, however it is limited (see more on limitations here).

type Point = { x: number; y: number };

// typeof pt = { x: number; y: number }
const pt: Point = { x: 10, y: 11 }

keyof Examples

So in what ways is keyof useful? It may not be immediately obvious… it certainly wasn’t for me!

(1) accessing an object in a “safe” way via generic constraints

interface Food {
    isTasty: boolean;
}

let pizza = { isTasty: true }

function isMyAThing(food: Food, attribute: keyof Food): boolean {
    return food[attribute];
}

isMyAThing(pizza, "isTasty")
isMyAThing(pizza, "isHealthy") // will fail

Here isMyAThing does two interesting things: (a) indexes properties of a type (b) constrains input of attribute to just being valid properties of Food.

(2) map over properties of an existing type

type Boolify<T> = {
    // read "for each key in T translate to boolean
    [K in keyof T]: boolean
} 

interface SomethingUseful {
    name: string;
    age: number;
}

type SomethingLessUseful = Boolify<SomethingUseful>;

let notHelpful: SomethingLessUseful = { name: true, age: false };

Here Boolify maps all properties of the generic T to booleans,

(3) work in conjunction with utility types (i.e. Record, Pick, etc.) which manipulate the type (since its structural typing, that means manipulating the underlying properties)

Consider Record: which constructs an object type whose property keys are Keys and whose property values are Type. This utility can be used to map the properties of a type to another type. More formally: Record<Keys, Type> (Record Type - TypeScript Handbook)

type Qualities = {
    isTasty: boolean;
    isHealthy: boolean;
    calories: number;
}

interface Info {
    value: string
}

type Food = Record<keyof Qualities, Info>;

const asparagus: Food = {
    isTasty: { value: "sometimes" },
    isHealthy: { value: "yes!" },
    calories: { value: "idk" }
}

(4) create new types

type Food = {
    isTasty: boolean;
}

type Decoration = {
    isPretty: boolean;
}

// intersection type: combination of all properties
type DecorativeFood = Food & Decoration;

const weddingCake: DecorativeFood = {
    isTasty: true,
    isPretty: true,
}

// union type: one type _or_ the other
type DecorationOrFood = Food | Decoration;

const bacon: DecorationOrFood = {
    isTasty: true
};
const painting: DecorationOrFood =  {
    isPretty: true
}

// union type and keyof
type DecorationOrFoodDescription = keyof Food | keyof Decoration;

const bacon: DecorationOrFoodDescription = 'isTasty'
const painting: DecorationOrFoodDescription =  'isPretty'

Conclusion

With a basic understanding of keyof and structural typing, the next question is can we implement a similar keyof operator for Pyret?

Pyret’s core is written in JavaScript. Thus, we should be able to, more-or-less, translate keyof from TypeScript to JavaScript and implement it as a built-in method for Pyret. However this assumes that core types are implemented as objects in a fairly consistent way… if not, implementing keyof might be a heavier lift than expected.

We’ll explore this in a following post.

References

Napkin Math at Work

2022-08-12T10:00:00-07:00

I am about four months into my journey at Spring Health and it’s been a great learning experience thus far! Besides learning the ins-and-outs of Rails, a coworker and I have set up a load testing environment and utilized this to bring forth a few notable perf wins. Those stories are worthy of the official Spring Health Engineering Blog… stay tuned!

However, here’s a sneak peak.

System Awareness

Ice hockey is a rather complex sport. Towards the end of my >15 year “career” playing ice hockey, I became a penalty kill specialist; I specialized in ensuring my team did not give up a goal when playing down a man (or two).

keep your head on a swivel

Knowing where everyone was in relation to the player with the puck, or keeping your head on a swivel, while deducing the opposing team’s system (Brad on defense, the middle of the umbrella powerplay system, always turns up the shot while Pete, the top goal scorer, keeps trying to sneak back door) required a great sense of situational awareness.

At risk of sounding cheesy, I’ve found killing penalties fairly analogous to developing a performance engineering mindset. Both require proper observability practices (head on a swivel… or Datadog). Both require an understanding of what’s typical (Brad never shoots… customers traffic load follows the business day). Both require a mental model of the system (three man high umbrella… frontend react, backend a combination of queues, databases, servers, etc.). And finally, both require some basic stats (on the penalty kill we will always be outnumbered… 5 million requests taking 100ms across 10 containers, etc., etc., etc.).

Some Basic Stats

Yes, basic with a capital B. Basic stats and napkin math go a long way. Unconvinced? Consider the following…

Let’s say a company expects to process 100 million records next week. Since this is a fairly expensive job, they do it in batched tasks using separate resources from their production environment. You observe that, as is, the processing takes so long that it will still be in progress as the next batch of records come the following week.

Let’s add some more context.

million records
task/container at a time
records/task
containers (4 cores each)
400ms per processing per record

Thus, we can quickly guestimate how long processing currently takes.

1 million tasks = 100 million records / 100 batch size
250,000 concurrently executing tasks = 1 million / 4 containers
40 sec = 400ms * 100 batch size
10,000,000 sec = 40 sec/task * 250,000 tasks

~16.5 weeks = 10,000,000 sec / (60 * 60 * 24 * 7)

Putting on your thinking cap, you dig in to the data your APM presents and discover the following:

each separate record spends 175ms of processing time encrypting unused data
during processing, we batch 100 records at a time into a task, where each container can only execute one task at a time and each task is not that computationally expensive

Is it worth the time to implement the possible changes we discovered above?

Since we don’t need to encrypt the useless data, let’s cut that (and on a closer look, we actually encrypt two useless fields, so let’s cut both of those!!). The time to process a single record is now only 50ms! Furthermore, since each task is independent and computationally cheap, let’s reconfigure our cloud cpu resources to 16 containers, 1 core each.

With this, let’s redo our math.

1 million tasks = 100 million / 100 batch size
62,500 concurrently executing tasks = 1 million / 16 containers
50 sec = 50ms * 100 batch size
312,000 sec = 5 sec/task * 62,500 tasks
~0.5 weeks = 312,500 sec / (60 * 60 * 24 * 7)

Not only do these minimal changes sound trivial to implement, it’s clearly worth our time. With just a bit of research and kindergarten level mathematics, we’ve made a broken process much cheaper, efficient, and bottomline… feasible.

Conclusion

Napkin math is an art - it requires knowing what’s important and what you can gloss over. Used effectively, it’s a great tool for determining potential performance wins, ballparking future server traffic load, and quickly checking someone else’s numbers. For me, it’s be helpful in justifying how I spend my free cycles at Spring; it’s a very quick way to determine the biggest bang for your buck!

RSS Feed Up and Running! How I did it.

2022-04-15T10:00:00-07:00

Finally set up RSS feed so my followers can actually subscribe!

Turns out it’s quite easy to set up an RSS feed with Jekyll. It’s spelled out fairly well here, although it mostly focuses on getting set up with R Blogger. I will be paraphrasizing and focusing on a general RSS feed setup.

[First], set up your Jekyll blog. Add the gem jekyll-feed.

Gemfile

*** abbreviated ***

gem 'jekyll-feed'

*** abbreviated ***

_config.yml

*** abbreviated ***

plugins:
  - jekyll-feed

*** abbreviated ***

[Second] run bundle install to install locally. My local ruby env. is kinda messed up so I skipped this step. Test in prod right?!?

[Third] push to GitHub. Once it is pushed check that all is well by navigating to http://YOUR_PAGE/feed.xml

At first, if you do not have an RSS reader it may say something along the lines of No RSS reader installed.

[Fourth] install an RSS reader if you don’t have one.

As an example, let’s say you downloaded the RSS Feed Reader Chrome extension and navigate to my blog’s RSS feed, you should see something like this:

[Fifth] users should be able to subscribe to email alerts. While you could use the above Chrome extension, I am using FeedRabbit. Not 100% sure how well it works because I believe it emails notifications daily (and I just set this up today), but I imagine it gets the job done.

Springing into a New Adventure

2022-04-09T10:00:00-07:00

This announcement comes a bit pre-mature. While those close to me, my most immediate network, and my direct coworkers know, no one else does yet; I read online it’s common courtesy to wait until 2 weeks into your new gig to tell the world. Whatever. So. For anyone who follows my blog close enough to discover this post (good on you, there’s no RSS feed, so you must have wondered by here on your own accord), you get this announcement in advance (and with a bit of an explanation too).

EDIT (4/16/22) I do now have an RSS feed!

In true Linkedin fashion…

I am pleased to announce I have accepted a fulltime position as software engineer at Spring Health.

Immediate Reactions

Ok, Rob, so why?

DocuSign is a great company (see here) among other awards/top ten lists they have been on.
My managers have been wonderful
The work is very interesting
The company (except for the “outlook” which Wall Street throws a temper tantrum…) has been performing incredible!
Paper signatures WILL lose out in the long term

If anyone is considering DocuSign, it’s an absolutely fantastic company. Would recommend to anyone (and gladly help connect you with some of the platform folks doing technically interesting things).

So please Rob, fill me in.

My Career

I’ve talked to ALOT of people in the past few months. To those of you I’ve talked to, thanks for the help! Special shoutout to my girlfriend and parents… thanks for understanding my back-and-forth-back-and-forth.

Ultimately, I made a sort of thesis on what I was looking for. It largely stemmed from the following quote:

“It is well known the drunken sailor who staggers to the left or right with n independent random steps will, on the average, end up about square root of n steps from the origin. But if there is a pretty girl in one direction, then his steps will tend to go in that direction and he will go a distance proportional to n. In a lifetime of many, many independent choices, small and large, a career with a vision will get you a distance proportional to n, while no vision will get you only the distance . In a sense, the main difference between those who go far and those who do not is some people have a vision and the others do not and therefore can only react to the current events as they happen.” ― Richard Hamming, The Art of Doing Science and Engineering: Learning to Learn

Thus, it was time to take control of my career. But in order to take control of my career, I needed a vision.

Vision: become the world’s best performance engineer for hyper-growth startups.

From this vision I can derive some sub goals:

[technical] understand the nuts of bolts of all kinds of complex systems (live in rabbit holes) AND how they fit together
[technical] hack together some open source tools to accelerate performance engineering field as a whole
[technical] give a talk about some of the super cool stuff I did at work at a conference
[financial] buy a house in 5 years (perf. eng. are in high demand and typically do well in this regard)

But, this all comes from the primary vision. So how do I get there?

According to a conversation I had with the creator of the flamescope tool, there are three main avenues to improvement in this field (maybe we can abstract this to any field?):

experience
mentorship
reading

This person whom I spoke with was a huge believer in (1). He pointed out his colleague, creator of flamegraphs was very much a (3). Many of my coworkers at DocuSign pointed to (2) being incredibly important.

So, Spring Health. I believe this offers the opportunity to get all 3:

[experience] a startup with growing pains offers the opportunity to OWN systems which have performance issues
[mentorship] will be able to work with a previous manager who’s a “wizard” in perf things (as said by previous DocuSign coworkers)
[reading] I already have WAY too many books on my shelf to leaf through, (2) for me has provided a few reccomendations. Time to buckle down. Will probably dedicate a future blog post to however I figure out the best notetaking/knowledge retention strategy.

But Rob, Why Spring Health

Ok, ok, ok. So maybe there would be other places that could also offer 1-3. Why Spring Health?

Here is a random laundry list of reasons why:

one of the tech leads believes in blogging
I believe in the mission; I have dealt with mental health issues personally
financial compensation is on par with what lifestyle I’d like to live
company size is between Stellar Development Foundation (~50) and DocuSign (~6500) offering a new experience
at a smaller eng. org., I can, as a mid-level eng., own things and thus grow my career faster
culture of peer programming (my new manager has worked remotely for a decade)
eng. org. recently turned over, chance to make a big impact as culture develops (into hopefully a great technically curious & learning community like I loved in SF)
unlimited PTO to coordinate vacations and time off with my girlfriend
doing my work on a MacBook (did someone say 16 inch !!!!)
I won’t be penalized for working in Miami (and furthermore won’t be penalized for moving with my girlfriend when she starts fellowship in a little over a year)

Other Criteria

A couple previous coworkers at DocuSign gave me further advice, other criteria I could use to consider if this was a good fit. I also will work in a couple role models.

First Manager:

I am blanking a bit this AM, but his framework went something like this:

For: who is leading the company? What is the company doing?
What: what are you working on?
With: who do you work directly with?

Based on the number of these you can answer positively, you can determine how good the opportunity is:

1 –> good
2 –> great
3 –> dream

Here is how I’d answer:

TBH I don’t know a ton about the founders, but I do believe in the product and the mission, so let’s give this a 0.5
I will be able to own some perf things, so definitely a 1.
I get to work with an old manager, and a few others that seem like awesome coworkers based on my interview conversations, thus this is also a 1.

So, we get somewhere between a 2 and 3, passing this test.

Mentor: is this an outsized opportunity?

According to (yet-another-coworker), startups offer the greatest opportunity to own the largest portion of a codebase. Thus, if I can work hard, produce, ship, and gain the trust of my peers, there’s enough work to do such that opportunities here nearly limitless (while of less scope than DocuSign for the time being).

Thus, we also pass this test.

Richard Hamming: yes, another story from Hamming… sorry, not sorry.

Over on the other side of the dining hall was a chemistry table. I had worked with one of the fellows, Dave McCall; furthermore he was courting our secretary at the time. I went over and said, ‘‘Do you mind if I join you?’’ They can’t say no, so I started eating with them for a while. And I started asking, ‘‘What are the important problems of your field?’’ And after a week or so, ‘‘What important problems are you working on?’’ And after some more time I came in one day and said, ‘‘If what you are doing is not important, and if you don’t think it is going to lead to something important, why are you at Bell Labs working on it?’’ I wasn’t welcomed after that; I had to find somebody else to eat with! That was in the spring. In the fall, Dave McCall stopped me in the hall and said, ‘‘Hamming, that remark of yours got underneath my skin. I thought about it all summer, i.e. what were the important problems in my field. I haven’t changed my research,’’ he says, ‘‘but I think it was well worthwhile.’’ And I said, ‘‘Thank you Dave,’’ and went on. I noticed a couple of months later he was made the head of the department. I noticed the other day he was a Member of the National Academy of Engineering. I noticed he has succeeded. I have never heard the names of any of the other fellows at that table mentioned in science and scientific circles. They were unable to ask themselves, ‘‘What are the important problems in my field?’’ - Richard Hamming, You and Your Research

So, am I doing important work at Spring Health?

While this question is certainly asked more in a “researchy” sense, it is a fundamental question that gives this job (and maybe even life) meaning. I do believe by helping make the Spring Health platform more performant and usable, I will be able to aid more people in receiving the mental health care they require.

Furthermore, I believe by improving my own engineering skills such that I can hop into ANY hyper-growth startup in the future and I can do some very important work in my lifetime; of course this will only happen after I see the Spring Health opportunity all the way through… I hope to be at Spring for a long while, 4+ years such that I can actually do some very interesting work… yet-another-docusign-coworker stressed nothing impressive or interesting happens until years 2+ or 3+ at a company.

Steve Jobs: ok, one last one.

For the past 33 years, I have looked in the mirror every morning and asked myself: ‘If today were the last day of my life, would I want to do what I am about to do today.

While this is certainly a “day-to-day” question I must ask myself, the answer today is a definite yes. This is the best move for me, my career, and most importantly my life, the life I want to live.

Conclusion

So that is probably more of an explanation than I owe anyone. Maybe this is too personal to be shared online? Who knows?

This is a personal blog after all (with almost no SEO). Hopefully if someone else is in a career rut or trying to find a course for their career/life, this post can provide some guidance or at least motivation.

From the Archives: Independent Study 2019 Summary

2022-01-18T04:19:21-08:00

Was digging through old repos on GitHub and came across my write-up for my Fall 2019 Independent Study at Colby College. This is the work which will result in me being mentioned in Nessie: Automatically Testing JavaScript APIs with Asynchronous Callbacks which is being presented at the ICSE 2022 conference.

Independent Study Fall 2019

In a thematic continuation of my Spring 2019 indepedent study, where I designed and implemented the Sailfish programming language, this Fall I undertook a research project, exploring the extension of a feedback-directed random test generation framework, called LambdaTester, such that it could work with asynchronous code. My personal goal for this research was two fold:

explore an interest in a research-based career path
dive into JavaScript internals… stop saying “JavaScript is whack” and actually understand why it is the way it is

To complete this research, I collaborated with Professor Frank Tip of the Northeastern University Programming Language Lab and was advised by Professor Bruce Maxwell of Colby College.

What is LambdaTester?

LambdaTester is a test generation framework that employs four different strategies for generating callbacks to higher order methods in JavaScript. It is comprised of a two part algorithm. The first stage is callback discovery. Callback discovery looks to “learn” which methods expect callbacks as input and at which position(s). It does this via a 100 trial and error algorithm, which attempts different combinations of random values and a simple callback that logs a message. If this method is executed and logs the message, the callback’s position is recorded. The second stage is the actual test generation phase. This takes a few different inputs, such as the result from the first stage, an output directory, and whether or not to utilize a feedback-directed approach. The crux of this stage is the callback generation phase. There are four types of callbacks generated:

empty: a baseline test
quick test: using a haskell influenced library, generate random values to return
callback mining: follows the basic style/structure of callbacks from a corpus of code
feedback-directed: the most interesting and effective method, uses previously generated and derived values from outer scopes and previous tests

This framework was well received at OOPSLA 2018 and achieved impressive results, demonstrating incompatibility between versions of many well used, open-source, NPM packages. To learn more, checkout the paper linked above.

AsyncLambdaTester Basics

The extension I explored focused on asynchronous callbacks. What is the difference between an asynchronous callback is when the callback gets executed – either right away, or at some unspecified time in the future. Asynchronous code introduces complexity into code, breaking any sort of assumptions that can be placed on the order of execution of given statements. LambdaTester did not seek to introduce or generate asynchronous callbacks. Thus, an entire interesting area of possible errors was ignored. With asynchonous callback generation, ordering bugs and a whole new domain of faulty assumptions can really be tested. In order to implement this extension, I needed to implement two things:

classification of expected callbacks as either asynchronous or synchronous
a strategy for nesting method callbacks and reusing previously calculated/derived values

Unfortunately, due to extenuating circumstances (mostly job interview and prep) I was only able to finish the first step – I came up with an algorithm and partially implemented 2, but do not have enough concrete to present (or want to expose the secret sauce for in-progress research).

A Failed Attempt at Improving Callback Discovery Prediction

Before I even started the process of implementing an async/sync classification algorithm, I decided to step back and see if I could improve the current callback discovery algorithm. The problem with the current algorithm is that it was not exhaustive, thus for methods that expected something like: number, number, number, callback, the callback was not being discovered. Thus I underwent an interesting, albeit unsuccessful delve down the type inference rabbit hole. You can read all about it here: Adventures in PL: Inferring Function Signatures in JavaScript.

Classifying Callback parameters

The most clever and novel part of my work this Fall was the design and implementation of a method for classifying callbacks as either asynchronous or synchronous. The intuition came for this during a deep dive into how the Node.js event loop works. Here is a good video for the less familiar: https://www.youtube.com/watch?v=8aGhZQkoFbQ

As an example, consider the following code:

const foo = () => console.log("A");
setTimeout(foo, 0);
console.log("B");

A totally sensible guess into what the output may be is “A” then “B”… however, that is not the case. But how can that be??? The timeout is set to 0ms… Well, it turns out that the timeout fires and then throws its callback method onto the event loop. Then, the rest of the code in the call stack executes, so “B” is logged. Once the call stack is empty, the asynchronous method, foo, is called, and “A” is logged. Crazy right? Well, not so much once you get the hang of the event loop.

We can use this to our advantage. Consider a setTimeout method that instead accepted a method that would be executed synchronously. Then we would get “A” and then “B” as expected. Thus, if we execute methods as follows:

const foo = () => console.log("A");
someMethod(foo);
console.log("B");

We can classify the callback as asynchronous or synchronous depending on the ordering of the logged messages. It turns out this worked wonderfully, as tested against modules within the Node.js standard libraries.

One Caveat

However there were some cases where my method above failed… some really interesting edge cases. It turns out sometimes callbacks may be classified as both asynchronous and synchronous. So how can a method be synchronous and asynchronous? When working with various methods in the zlib module, AsyncLambdaTester classified methods as both asynchronous and synchronous:

Discovery phase...  
[{  
	name: 'deflate',  
	positions: [ '2: sync', '2: async', '3: sync', '3: async' ]  
}]

Let’s dive into some source code!

// Convenience methods.  
// compress/decompress a string or buffer in one step.  
deflate: createConvenienceMethod(Deflate, false)

Hmmm… so it turns out deflate is a sort of convenience method around a more general Deflate function that works on a Deflate object. Let’s jump to the convenience creator:

function  createConvenienceMethod(ctor, sync) {
	// skipping to the interesting code...  
	return  function  asyncBufferWrapper(buffer, opts, callback) {  
		if (typeof opts === 'function') { callback = opts; opts = {}; }  
			return zlibBuffer(new ctor(opts), buffer, callback);  
		};  
	}  
}

Here we can see that an asynchronous method is created. If options are given, then the callback is the third method, otherwise it is the second. Let’s jump to the zlibBuffer method and see what’s going on there:

function  zlibBuffer(engine, buffer, callback) {  
	// skipping to the interesting code...  
	engine.cb = callback;  
	engine.on('data', zlibBufferOnData);  
	engine.on('error', zlibBufferOnError);  
	engine.on('end', zlibBufferOnEnd);  
	engine.end(buffer);  
}

Ah! So here we see that the callback is invoked in three different scenarios by an engine object (the zlib engine, a project dependency). Thus, the callback is not immediately pushed to the event loop to be invoked asynchronously, but actually may be called earlier. Let’s experiment! Consider the following test code:

const zlib = require('zlib');  

const f = (i) => {  
	return  () => console.log(`cb: #${i}`);  
}  

zlib.deflate("", f(2));  
zlib.deflate(1, f(1));  
console.log("A");

When we execute the above, we get:

cb: #1  
A  
cb: #2

Ah ha! We have found the source of our confusion. Based on the realization of the zlib engine above, we see that if we give deflate the wrong type for the first parameter, the callback executes immediately – it is not pushed onto the event loop to be executed asynchronously. However, if deflate gets the correct type, even if the value is nonsensical/incorrect, it will execute asynchronously as expected.

Some Implementation Details - To Typescript We Go

An unanticipated sub goal that I accomplished during this study was gaining some familiarity with TypeScript. A rather lengthy and largely unnecessary, yet learning filled part of this study was cleaning up the LambdaTester codebase and writing all my extensions in TypeScript. This required, learning TypeScript, using linters to enforce code style, and restructuring the code into OOD-esque classes (much easier done in TypeScript than Node.js). Ultimately this turned out to be a wonderful use of my time as I handed off the project to another student at NEU. Instead of a jumble of attempts to generate asynchronous tests, I left a well commented, clean, organized codebase. As they say: “Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live.”

Conclusion

Ultimately my independent study was a success, although maybe not quite in the way I originally hoped. For one, I certainly learned much more about how Node.js works internally. However, I did not finish my implementation of AsyncLambdaTester and also decided not to follow the research career path. That being said, I owe Professor Tip and Professor Maxwell for allowing me to pursue this project! Even though it did not quite work out as hoped, I am glad I will be able to look back and say “at least I tried it” and not have doubts about my journey directly into industry!

Microsoft's Coyote vs. Async Server: Repro'ing Race Condition & Generating Test Corpus to Prove Fix

2022-01-12T04:19:21-08:00

A few months ago at DocuSign, I was part of an effort to make some server code fully asynchronous. We’d gone down the async rabbit hole and during load testing, witnessed throughput limited by an async deadlock induced bottleneck. This was the final step to full asynchrony for a dependent service (SPOA, my primary focus at the time). While a coworker wrote the first 98% of the PR to make the server fully async, I was tasked with bringing the changes over the line.

In doing so, I ran into some challenges, the most devious being an interesting race condition in the server’s graceful close behavior due to Linq querying it’s state to wait on a collection of Tasks while the data structure itself is still writable.

Some Psuedocode

State:
	- ConcurrentDictionary<Connection_Handler_ID, Task>

Connection_Acceptance:
	1. Accept the connection
	2. Handle connection asynchronously and add Task to ConcurrentDictionary

Connection_Acceptance_Loop (THREAD 1):
	1. while (true) { Connection_Acceptance }

Close (THREAD 2):
	1. Stop TCP Listener
	2. Get state from ConcurrentDictionary
	3. Wait for Tasks from ConcurrentDictionary to complete

The Problem

Due to Close and Connection_Acceptance_Loop being executed on separate threads, it’s possible to end up with the following execution sequence:

Call Close
Accept a new connection
Stop TCP Listener
Get state from ConcurrentDictionary
Handle Connection Async and add Task to ConcurrentDictionary
Wait for Tasks from ConcurrentDictionary to complete

Bug: here we end up not waiting for the task handled in (5) to complete since we grabbed a snap shot of the state from the ConcurrentDictionary in (4)

Conceptually its not too hard to come up with a solution. However, getting it right took a couple tries at finding the right synchronization primitive and was somewhat complicated by our desire to keep the main code path lockless.

While I could try to simulate all the possible states in my head or on paper, I came across a really cool Microsoft research project called Coyoyte. Coyote does exactly this, simulates all (or at least many) of the possible code execution sequences, saving the executed sequences in order to be able to repro interesting ones on, say, failure.

Thus, instead of just proving to myself and code reviewers that my solution made sense, I could intentionally create the failure scenario then run my solution against it to make sure it worked.

Setup - Hello World

While this tool sounds great in theory, I had no idea how to make it work. So, I started with a minimally useful hello world example to make sure everything is functioning.

// Program.cs
public class Public
{
  public static void Main(string[] args) { }

  [Microsoft.Coyote.SystematicTesting.Test]
  public static void Foo()
  {
    NUnit.Framework.Assert.AreEqual(1, 1);
  }
}

// Project.csproj
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
	<OutputType>exe</OutputType>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Coyote" Version="1.2.8" />
    <PackageReference Include="Microsoft.Coyote.Test" Version="1.2.8" />
    <PackageReference Include="NUnit" Version="3.13.2" />
  </ItemGroup>

</Project>

Testing with Coyote:

dotnet publish -r win-x64 -c Release -p:PublishSingleFile=true --self-contained true
coyote rewrite .\bin\Release\netcoreapp3.1\win-x64\some_project.dll
coyote test .\bin\Release\netcoreapp3.1\win-x64\some_project.dll -m Foo -i 100

By packaging as a single file we get the .dll with all dependencies included/linked. -m here is method and -i here is iterations.

Setup - Async Server

Unfortunately there are some limitations we must work around to use Coyote:

forced to mock TCP connect
cannot use SemaphoreSlim (part of testing code in Core/Core)
unable to Sleep.Thread (hacky but would have been helpful in mocking)

So, I boiled down the behavior to a minimum example that hopefully captures most of the interesting logic at play.

Coyote to the Rescue!

With an understanding of Coyote and a mocked async server, I sought to repro the ConcurrentDictionary LINQ query race. To do so, I created the following test method:

[Microsoft.Coyote.SystematicTesting.Test]
public static async System.Threading.Tasks.Task Foo()
{
    var expected = 0;

    MockServer server = new MockServer(() => Interlocked.Increment(ref expected));

    server.Open();

    server.Connect();
    server.Connect();
    server.Connect();

    var countTask = server.CloseAsync();

    server.Connect();
    server.Connect();
    server.Connect();

    var count = await countTask;

    Assert.IsTrue(expected <= count);
}

Here I connect 3 times before calling close and 3 times after. Since I don’t delete connections from the dictionary (if a connection is finished and we await on it, it’s just some extra CPU cycles), I’d expect the number of handled connections to be less than the number of connections we seek to await on.

intentionally using faulty logic, I get the following:

. Testing .\bin\Release\netcoreapp3.1\win-x64\test_test_shit.dll
... Method Foo
... Started the testing task scheduler (process:33436).
... Created '1' testing task (process:33436).
... Task 0 is using 'random' strategy (seed:1792470991).
... Telemetry is enabled, see http://aka.ms/coyote-telemetry.
..... Iteration #1
..... Iteration #2
..... Iteration #3
..... Iteration #4
..... Iteration #5
..... Iteration #6
..... Iteration #7
..... Iteration #8
..... Iteration #9
..... Iteration #10
..... Iteration #20
..... Iteration #30
..... Iteration #40
... Task 0 found a bug.
... Emitting task 0 traces:
..... Writing .\bin\Release\netcoreapp3.1\win-x64\Output\test_test_shit.dll\CoyoteOutput\test_test_shit_0_12.txt
..... Writing .\bin\Release\netcoreapp3.1\win-x64\Output\test_test_shit.dll\CoyoteOutput\test_test_shit_0_12.schedule
... Elapsed 2.3007777 sec.
... Testing statistics:
..... Found 1 bug.
... Scheduling statistics:
..... Explored 45 schedules: 45 fair and 0 unfair.
..... Found 2.22% buggy schedules.
..... Number of scheduling points in fair terminating schedules: 6 (min), 20 (avg), 66 (max).
... Elapsed 3.9781934 sec.

Investigating the cause of the bug I see:

<TelemetryLog> Telemetry is enabled, see http://aka.ms/coyote-telemetry.
<TestLog> Running test 'Public.Foo'.
<ErrorLog> Unhandled exception. NUnit.Framework.AssertionException:   Expected: True
  But was:  False

   at NUnit.Framework.Assert.ReportFailure(String message)
   at NUnit.Framework.Assert.ReportFailure(ConstraintResult result, String message, Object[] args)
   at NUnit.Framework.Assert.That[TActual](TActual actual, IResolveConstraint expression, String message, Object[] args)
   at NUnit.Framework.Assert.IsTrue(Boolean condition)
   at Public.Foo() in C:\Users\Robert.Durst\Desktop\test_test_shit\UnitTest1.cs:line 148
<StackTrace>    at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__274_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
   at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread)
   at System.Threading.Tasks.Task.ExecuteFromThreadPool(Thread threadPoolThread)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()


<StrategyLog> Found bug using 'random' strategy.
<StrategyLog> Testing statistics:
<StrategyLog> Found 1 bug.
<StrategyLog> Scheduling statistics:
<StrategyLog> Explored 45 schedules: 45 fair and 0 unfair.
<StrategyLog> Found 2.22% buggy schedules.
<StrategyLog> Number of scheduling points in fair terminating schedules: 6 (min), 20 (avg), 66 (max).

This means that we handled more requests than we were waiting for!

With the scenario I was interested in repro’d, I now had a corpus of tests to prove my solution against.

Conclusion

While it was somewhat trivial to map out all the interesting execution sequences for this code, it was very helpful to test my hypothesis and confirm the solution worked against more rigorous testing. Coyote is actually a fairly easy to use tool and I didn’t even use it very rigorously or test out its feature set to the fullest (can open failing tests in the Visual Studio Debugger). One take away for sure is that concurrency code is hard to get correct - it took more hours than I’d like to admit to get a reasonable mock working. If I come across some non-trivial concurrency code with few dependencies that I’d have to mock, I won’t second guess taking Coyote out for a second spin.

My experience: Why defensive coding matters

2021-08-11T05:19:21-07:00

See original post here.

This blog post is part of an ongoing series, From the Inside, which highlights the interesting everyday work of performance and infrastructure platform engineers at DocuSign. Every month we sit down with an engineer and ask him/her/they to tell us about an interesting experience: a bug fix, feature shipped, or lesson learned. This week we sat down with software engineer Robert Durst of the Engineering Systems Crew (ESC) team to discuss an interesting performance investigation his team undertook.

Background

DocuSign currently uses a hardware load balancing provider and is in the process of exploring a software alternative, HAProxy. As part of this exploration, we introduced a Stream Processing Offloading Agent (SPOA) to the HAProxy stack in order to reach parity with its hardware counterpart. This is essential during an initial mixed-mode topology; as part of the gradual transition we have been running both load balancers simultaneously, sharing traffic between the two.

SPOA communicates with HAProxy’s Stream Processing Offloading Engine (SPOE) via the Stream Processing Offloading Protocol (SPOP) over TCP, and a single SPOA server sits sidecar to each HAProxy instance. SPOA intercepts and processes every incoming request, executing logic and dictating actions HAProxy should take, such as blocking a request due to a denylisted source IP address. Instead of opting for an existing open-source implementation, we built our own C# SPOA server that best fits our performance requirements.

The problem

We ship SPOA as a Docker container on a Linux host. After noticing a spike in CPU usage on a machine running SPOA, we observed the following output from docker-stats:

CONTAINER ID   NAME              CPU %     PIDS
7d44334e4f7f   spoaserver.8000   200.24%    23
Having 200%+ CPU usage is certainly unexpected and a significant blocker before SPOA can be deployed to production.

Collecting traces: perfcollect

DocuSign is still early in its development of running C# services within Linux Docker containers, so we didn’t have an established process to investigate the problem.

Our initial approach involved installing dotnet-trace on the Docker container, waiting for the problem to reproduce, then analyzing the trace. Unfortunately, we not only observed no obvious offender, we also found an inconsistency between dotnet-trace and docker-stats over the CPU utilization of the container. Since dotnet-trace describes just the dotnet process(es) and docker-stats describes the entire container, something else within the container must have been consuming CPU time.

On Linux, Docker containers are simply isolated processes, forked from the main process and thus observable from the root operating system. So, what we needed was a tool which captured a trace across all processes and a way to tie Docker container processes and subprocesses together. Enter perfcollect, a Bash script developed and maintained by Microsoft that:

“leverages Linux Tracing Tookit-Next Generation (LTTng) to collect events written from the runtime or any EventSource, as well as perf to collect CPU samples of the target process.”

Visualizing traces: flame graphs

Early on in my time at DocuSign, my manager, Aaron Tyler, introduced me to the blog of Brenden Gregg. Soon thereafter, I bought Gregg’s Systems Performance book and became increasingly familiar with flame graphs. As per Gregg’s website:

“Flame graphs are a visualization of profiled software, allowing the most frequent code-paths to be identified quickly and accurately… [t]he x-axis shows the stack profile population … the y-axis shows stack depth, counting from zero at the bottom. Each rectangle represents a stack frame. The wider a frame is, the more often it was present in the stacks. The top edge shows what is on-CPU, and beneath it is its ancestry. The colors are usually not significant, picked randomly to differentiate frames.”

It turns out Gregg’s FlameGraph generation scripts work perfectly with the output generated from executing the perfcollect script. Thus, we now had a process for collecting and displaying traces:

Run perfcollect from root OS       --> perfcollect.zip
Unzip (perfcollect.zip)            --> out.perf
Stackcollapse (out.perf)           --> out.folded
Generate flame graph (out.folded)  --> flamegraph

Note: we run a slightly modified perfcollect script that gathers the PID of Docker processes to generate the .map files required for Docker process symbol resolution when generating flame graphs on a separate machine.

Analyzing traces: an unexpected culprit

With the tooling in place, it was time to find the root cause. Here is the flame graph for the offending SPOA machine, zoomed in on the dotnet process within the Docker container:

Reviewing the flame graph, it became clear that we spend a TON of time in DecodeString. This was certainly unexpected! Parsing through the source code, Aaron discovered that within the frame parsing code, we continuously read bytes until we read as many as explicitly defined by the incoming frame.

From the SPOP spec:

String := <LENGTH><STRING_VALUE>
Thus, without any sort of defensive coding, we entered an infinite loop triggered by data that adhered to the SPOP protocol, yet violated our assumption of sane input: specifying a string length longer than the string data itself.

With a probable root cause identified, it was time to identify the source of the buggy frames. We looked at our internal monitoring GUI, Kazmon, for help.

Every time we received a 128 frame type (UNEXPECTED FRAME, i.e. garbage) we saw another CPU get tied up. Furthermore, this happened at almost the exact same time every week.

To further debug and test our infinite loop theory, I did the following:

1. logged IP of incoming messages, and
2. added an assertion that the expected string length should be <= remaining frame buffer length, allowing an early exit in case of receipt of the nonsense data described above.

Not only did we see the assertion throw, meaning we hit this code path, we no longer saw the CPU spikes:

And we caught the offending sender red handed, identified by its IP address.

It turns out our vulnerability scanner was sending garbage to SPOA and we’d neglected to harden our implementation against legal, yet impractical message frames.

Takeaways

It turns out dotnet-trace may have been the right tool for this investigation, the buggy code was within the dotnet process. However the ability to generate a flame graph from perfcollect’s output proved invaluable. From this investigation, I developed a rudimentary understanding of Docker container internals and also established a process for future performance investigations. Furthermore, I came away having learned the following lessons:

Defensive coding is essential to protect against unexpected data, especially data that still adheres to the protocol.
Flame graphs are an extremely helpful tool during performance investigations, offering an interactive, single-page visual that can be used to identify anomalies quickly.

If you are interested in this type of work or distributed systems performance in general, check out the open role listings within my organization on the DocuSign careers page. We’re hiring!!

Two Flavors of Pipes

2020-06-27T05:19:21-07:00

On June 1, I started my journey at DocuSign. While starting remote has been a bit weird, I have had an incredible experience so far, shipped some code, learned some things, and even made a fool of myself a couple times. I am extremely grateful that I am able to find employment in these crazy times, especially meaningful and interesting employment. There is lots of C# at DocuSign, so I have been migrating away from C++ & Linux focused development into new, uncharted waters. A side effect of this is moving from Unix –> Powershell for some of the automated scripting. Just before I started at DocuSign, I gained a newfound appreciation for Unix (and Unix pipes) after reading Brian Kernighan’s latest book Unix: A History and Memoir. Never used pipes? Well, today is your lucky day! This post will show you how to use pipes in Unix and then how a similar effect can be created in powershell.

Pipes

Ah yes, so what is a pipe? This is a good place to start. Pipes, or more formally a pipeline, is basically a mechanism for chaining multiple processes together by feeding the output of one into the input of the next. A slightly more computer-sciency way of saying this is “a set of processes chained together by their standard streams, so that the output text of each process (stdout) is passed directly as input (stdin) to the next one” - (Wikipedia). So yes, a pipeline is probably almost exactly what you thought it was.

Douglas McIlroy

This concept that was first brought into the world of computing by Douglas McIlroy at Bell Labs (and subsequently implemented into Unix by the mythical Ken Thompson. Here is a fantastic quote about the invention of the pipe: “His ideas were implemented in 1973 when (“in one feverish night”, wrote McIlroy) Ken Thompson added the pipe() system call and pipes to the shell and several utilities in Version 3 Unix. “The next day”, McIlroy continued, “saw an unforgettable orgy of one-liners as everybody joined in the excitement of plumbing.”” (Wikipedia).

Pipes in Unix

Ok, time for an example. For anyone running a version of Unix, MacOS, Ubuntu, any Linux derivative, etc. this one is for you (feel free to open up terminal and follow along - I ran each line of code as I wrote this so the examples should work). I will be creating an example very similar to Brian Kernighan’s favorite example - we will be determining the misspelled words in a text file.

For starters, we have a textfile (text.txt):

I like to write commands using pipes and pipes are what I like to wriite

First, we want to read from the file. We can do this uisng cat:

> cat text.txt
I like to write commands using pipes and pipes are what I like to write

Then we want to lowercase all the words since our dictionary only has lowercase letters. To do this on the text from our text file, we use the | to pipe the output from our first command to the input of our second command:

> cat text.txt | sed -e 's/.*/\L&/g'
i like to write commands using pipes and pipes are what i like to write

The sed command takes a regular expression, matches every occurance of this expression, and replaces these matches with the second expression. In this case our find expression is .* which says to match on everything and our replace expression is L& which turns all ascii characters to lowercase.

Now, we want to put every word on its own line. Using the sed command again, we match against whitespace \s and we replace with a newline \n:

> cat text.txt | sed -e 's/.*/\L&/g' | sed -e 's/\s/\n/g'
i
like
to
write
commands
using
pipes
and
pipes
are
what
i
like
to
write

The next step is to alphabetize our list and then remove duplicates. Removing duplicates allows us to perform fewer checks against our dictionary in the next step:

> cat text.txt | sed -e 's/.*/\L&/g' | sed -e 's/\s/\n/g' | sort | uniq
and
are
commands
i
like
pipes
to
using
what
wriite
write

Finally, it is time to check for misspelled words. To do this, we want to check to see if each word is in the dictionary, if it is not, it must be misspelled. Assuming the computer has a textfile called dictionary, we will use the grep command to do just that. The grep command is a unix command for searching for lines of text that match a certain pattern (regex again!!). By using this with some flags, v which returns the inversion, f a path to a file to match against, and x matching only against whole lines, we get the desired effect:

> cat text.txt | sed -e 's/.*/\L&/g' | sed -e 's/\s/\n/g' | sort | uniq | grep -xvf dictionary
wriite

Yep, so next time you’re in a Zoom party with friends on a Friday night, share your screen and impress them with this.

Pipes in Powershell

Ok, so you run Windows but still want to impress your friends. How does the above translate? Are there pipes in the powershell? Questions, questions, questions.

It turns out there are pipes, and have been for quite a while. However, they come with one small caveat… when piping in the powershell, output is formatted as objects, not just unformatted streams of text. This actually turns out to be quite useful! Let’s dive in and see how we can recreate the above.

To start with, we could use cat, but that seems like cheating… let’s use a powershell cmdlet! Cmdlets are powershell methods. Many useful ones are included with Windows and you can of course write your own as well.

> Get-Content text.txt
I like to write commands using pipes and pipes are what I like to wriite

Ok, so far so good. Now let’s expose what I meant by passing objects instead of unstructured text. When we use the Get-Content command here, we get a string. We can access this string in the next pipe via the special $_ variable (variables in powershell are denoted by the $). Thus, we can actually combine the splitting of lines by whitespace and the lowercase conversion in one go (in a much friendlier, readable way… no regex!):

> Get-Content text.txt | % {$_.Split(" ").ToLower()}
i
like
to
write
commands
using
pipes
and
pipes
are
what
i
like
to
wriite

Now it is time to sort and remove duplicates. Windows ships some similar commands to what we used in Unix: Sort and Get-Unique:

>  Get-Content text.txt | % {$_.Split(" ").ToLower()} | Sort | Get-Unique
and
are
commands
i
like
pipes
to
using
what
wriite
write

Wow - this is almost a 1:1 translation so far and we even avoided regex! Ok, now it gets a little messy. Unlike grep above, Compare-Object does not show one-sided differences. As you may imagine, this is rather unfortunate when comparing against a large dictionary; it returns each difference with a SideIndicator, <= meaning only in left and => meaning only in right. In our case we want the words which are unique to the input so we filter =>:

>  Get-Content text.txt | % {$_.Split(" ").ToLower()} | Sort | Get-Unique | Compare-Object (Get-Content) dictionary) | Where-Object {$_.SideIndictor -eq "=>}
InputObject SideIndicator
----------- -------------
wriite       =>

Ah, so we have what we want, but now it is in this wonky table. IMO, this is an unfortunate side-effect of powershell’s object approach. However, it is not that hard to transform the output into the desired format:

> Get-Content text.txt | % {$_.Split(" ").ToLower()} | Sort | Get-Unique | Compare-Object (Get-Content) dictionary) | Where-Object {$_.SideIndictor -eq "=>} | Select-Object "InputObject" | Format-Table -HideTableHeaders
wriite

My apologies, I realize that this one-liner is now flowing into the abyss off the right side of the code box. That being said, I actually believe this one-liner is much more readable than the unix one, albeit that takes away some of the fantastical magic that comes with it when you try to impress your friends.

Conclusion

So, besides this being an interesting exercise, what’s the point? My point is that after two summers deep in the world of cryptocurrency in San Francisco, I was indoctrinated into a certain mindset. This mindset said things like:

closed source evil
Microsoft bad, Linux good
do you use Rust? Why don’t you use Rust?

This is of course a minor exaggeration. That being said, I am learning (or rather relearning if you talk to my 11th grade English professor) it is almost always best to take a nuanced approach - by considering the tradeoffs of a decision and both/all points of view, you end up with a much stronger, more comprehensive final outcome. In software this means using Linux when appropriate and Windows when it is not. In astrophotography, it means taking some wide-angle shots and some close-ups. In collegiate track and field 400m training, it means 1000m workouts and 50m speed work (yes 1k’s are no fun for short sprinters, but hey, I cut 4 sec off my 400 pr in one season, thanks Dave). So, since I’ll probably be living in both Windows and Unix systems for the next X years of my software engineering career, it’ll pay off to understand both.

Brush off the Bitwise

2020-05-19T05:19:21-07:00

A singular bit is the most atomic form of information. Yes, no. Is, isn’t. True, false. 1, 0; a singular bit may be interpret in many different ways depending on context. Consider now a collection of bits: 0110. Collections of bits are often interpreted as binary numbers. The numbers you and I use everyday are base 10, or decimals. Binary numbers are base 2, calculated right-to-left, where we increment an exponential factor of 2, starting with 0, multiplying this by the 1 or 0 bit and then summing up the results. I.e for 0110, we start on the right, calculating 2^0 * 0 and move to the left:

2^0 * 0 + 2^1 *1 + 2^2 *1 + 2^3 + 0 = 4

These bits may be manipulated by operations, known as bitwise operations.

Bitwise Operations

The following are six, fundamental bitwise operators. I’ll accompany the definitions with some python code which may be executed and experimented to better grasp the concepts. Python is nice because you can just open up an interpreter.

NOT: not is a unary operator, also known as the compliment, which flips a bit, i.e. 0 –> 1 and 1 –> 0.

>>> ~1
-2

Now this may seem odd, and did stump me for a second. However, it makes sense when you consider how 1 is represented here. In python, 1 is an integer, specifically a member of the Int class. This class has a useful bin method which returns the binary representation of the number.

>>> bin(1)
'0b1'

This makes sense. However, consider the following:

>>> bin(-2)
'-0b10'

Negative two is represented in two compliment form. It turns out that, in order to keep things symmetric, all numbers are interpreted in two’s compliment. Therefore we get the following:

>>> ~-2
1
>>> ~1
-2
>>> 0
-1
>>> ~-1
0

Which makes sense if you investigate the binary representations:

>>> bin(0)
'0b0'
>>>bin(-1)
'-0b1'

TLDR: not just flips the bits, however it may be unintuitive at first when used in your favorite higher level programming language depending on how the numerical value is represented. For a little more wtf fun, try ~False in python.

AND: given two bits, returns 1 only if both bits are 1. AND, along with the rest of the bitwise operators, are usually performed on bit patterns, or collections of bits.

>>> 1 & 1
1
>>> 1 & 0
0

OR: given two bits, returns 1 if either bit is 1.

>>> 1 | 1
1
>>> 0 | 1
1
>>> 0 | 0
0

XOR: given two bits, returns 1 only if 1 of the two bits is 1.

>>> 1 ^ 1
0
>>> 1 ^ 0
1
>>> 0 ^ 1
1
>>> 0 ^ 0
1

Shift Left: given a collection of bits, slides all the bits left, padding the back with zeros. Thus, for the number 3 (11) we get 6 (110). Left shift is the equivalent of x * 2^n for some x shifted n bits, i.e. for 3 shifted 1 to the left we get 3 * 2^1 = 6.

>>> 3 << 1
6
>>> 10 << 3
80

Shift Right: given a collection of bits, slides all the bits right, padding the front with zeros. Thus for the number 3 (11) we get 1 (1). right shift is the equivalent of x / 2^n for some x shifted n bits, i.e. for 3 shifted 1 to the right we get 3 / 2^1 = 1. Notice how we round down here, thus the formula is only an approximate.

>>> 3 >> 1
1
>>> 10 >> 3
1
>> 4 >> 1
2

Applications of Bitwise Operations

It turns out we can combine these operators in interesting and useful ways. In computers, we cannot access bits directly, instead we must access them in collections of 8, as a byte. Therefore, if we want to represent data as a boolean True or False value, we end up wasting 7 bits. However, with bitwise operators, we can pack eight booleans into a byte and reclaim this wasted space. Consider the following bit-packed boolean collection: 10101101. Let’s pretend the 4th from the left, i.e. the 3rd index in normal cs terms, indicates the current state of shelter-in-place for your hometown. How can we access this value?

The basic intuition here is we want to determine if the bit we are interested in is “on”. To do this we can use AND with the 4th from left bit as the only “on” bit as the comparison. Then we shift that value 4 places to the right so that it is either a 1 or a 0. See here:

>>> 173 & 16 >> 4
0

For a more complex example, let’s say Governor Newsom decides to only lift the shelter-in-place restrictions if his neighbors are also lifted. It turns out we can do the check and also flip California’s bit (4th from left) all in one go.

First check if the left neighbor is lifted:

>>> 173 & 32 >> 5
1 

Then check if the right neighbor is lifted:

>>> 173 & 8 >> 3
1

Now combine this and shift to the 4th index, representing whether California should lift the restrictions:

>>> ((173 & 32 >> 5) & (173 & 8 >> 3)) << 4
16

Now toggle California’s shelter-in-place to on (technically this would toggle it off if it was already on).

>>> 173 ^ (((173 & 32 >> 5) & (173 & 8 >> 3)) << 4)
189

It turns out querying and turning on/off certain bits is super useful. This is called masking.

Conway’s Game of Life

Conway’s Game of Life, a simulation of cellular automata, is a classic and also a project in Colby College’s Data Structures and Algorithms class. However, complex data structures are a bit overkill for this simple simulation. In fact, if we use bitwise operators, we can use a bit-packed array as our data structure. Using an unsigned long, or a 64-bit integer, I did just that.

You can check out the source code here.

References

A Semester of Coding Music

2020-05-08T05:19:21-07:00

A Semester of Coding Music

There’s a quote by Jim Rohn that goes “you are the average of the five people you spend the most time with.” Considering I spent the last two summers deep in Silicon Valley’s blockchain scene, surrounded by people immersed in cutting edge tech always planning for the next Burning Man, it was nearly inevitable that I enroll in a class called algorave (offered for the first time this year at Colby College). This course was Silicon Valley in a nutshell: experimental electronic music via pure, functional programming (Haskell). While I’ve never dabbled in creating music or designing visuals my final performance this afternoon ended up pretty ok.

Here is a link to my performance.

What is an Algorave?

An algorave, algorithm plus rave, is:

an event where people dance to music generated from algorithms, often using live coding techniques. Algoraves can include a range of styles, including a complex form of minimal techno, and the movement has been described as a meeting point of hacker philosophy, geek culture, and clubbing. - Wikipedia

There are many different platforms and languages allowing for a variety of techniques for music creation. In my class, we focused on using TidalCycles, a Haskell DSL that allows users (even the musically challenged) to create cool patterns via the SuperDirt synth.

The TLDR of TidalCycles

Below is a sampling of what you can do with TidalCycles, a.k.a. Tidal. In general, Tidal has four components:

samples
functions
effects
transitions

To play a simple bass drum sample, you would type: d1 $ s "bd"

This takes the sample bd and plays it once every cycle. I don’t formally know music, but if you do, you can setcps, set the cycles per second, to match what you may expect in terms of beats per second.

To reverse some samples every 2 cycles, we can use the every function:

d1 $ every 2 rev $ s "bd bd hh"

For this part of Tidal, Haskell works wonderfully, as you compose functions in a typical Haskell style. Consider the following which reverses, then slows, then increments the selection of the first sample every 2 cycles:

d1 $ every 2 ((|+ n "2") . slow 2 . rev) $ s "[bd bd hh]"

note: samples like bd or hh often have multiple varieties which can be specified by bd:2 or s "bd" # n "2".

To make things more interesting, we can apply effects to samples. The following will randomly pan the sample:

d1 $ s "bd" # pan rand

Finally, let’s assume you are playing a set and want to have a spiffy fade in of a nice percussion kick. With transitions you can do just that. The following fades in the reverbkick sample gradually over the course of 10 cycles onto synth input 1.

xFadeIn 1 10 $ s "reverbkick"

There is certainly a lot more you can do with Tidal… the above is just a small taste :) To learn more checkout the website and the docs.

My Takeaways

As a computer science major and coding enthusiast, I came into this class from a coding perspective: I viewed this class as coding music, not creating music via code. However, coding in this unconventional way taught me a few unexpected lessons.

Lesson 1: Code is Merely a Tool

During the 2017-2018 school year, I took a gap year where I spent the majority of my time in San Francisco. The first part of this year was spent at a coding bootcamp, where I learned fullstack JavaScript (MERN stack). From there I got the programming language bug and very quickly doubled the languages I could write basic projects in from three to six. However, at this time my focus was on learning languages for the sake of learning languages. It turns out this approach was largely not beneficial in the long run… even though I read an entire book on Rust and listened to the wonderful New Rustacean Podcast, if you held a gun to my head right now and said “write me a proper linked list in Rust” I would surely fail.

Skip ahead a couple years. I am now learning Haskell for the second (third?) time, but this time in the context of a DSL for live coding music. I am focused much more on composing functions to create cool sounds than all the fun and fancy lingo like monads, functors, etc. One weekend I decide to try to tackle a paper that has been confusing me for a while (the last two times I tried to learn Haskell I could not make it through). This time it was easy - it all clicked - it all made sense. The next weekend I even implemented a parser in Haskell based on the principles from this paper. It ended up being 78% smaller than the equivalent C++ parser I wrote last year and certainly more than 78% easier to reason about.

Lesson 2: Build For Yourself First

One of my personal epiphanies, realized during the course of this quarantine is I have succumbed to what I’m calling “toy project syndrome.” I am not sure if this was inspired by religiously following programming Reddit + HackerNews or if it was an emulation of the blockchain hype train. Either way, it goes something like this:

find a cool new topic in computer science
learn just enough to build a toy X
write a blog post about toy X
reap the gratification of telling the world about toy X
go back to 1, neglecting the more intellectually challenging and thought provoking journey of creating a real world X or contributing to an established open source X

While the intentions of this syndrome come from a genuine desire to maximize breadth of knowledge, at some point, there is a need to progress beyond the beginner stages of some topic.

Thus, for this class, when I wanted to dive into visual creation and needed a tool for easier, stateful handlers, I silently dived head first into the code, creating something that was useful first. Besides ensuring nothing out there like this already existed, I did not tell the world about my idea before having written any code; only once I had a product that I used myself did I share.

This solves two things:

creating something you actually use means it won’t just be thrown out at the end of the process
building isn’t done for the gratification from upvotes, but the joy of actually creating something that solves your own problems

So, while yes, this is me promoting what I built, and yes I did learn a heck of a lot more building my own programming language than this simple JavaScript API wrapper, my language is dead and this library is not.

HydraFriend

Ok, a little bit of self promotion. I can not resist. Hydra is a wonderful tool built by Olivia Jack that allows for live coding of visuals. Often times, people will pair up where one person codes music and the other visuals - it turns out it is a bit hard to do this solo. Seeing that it is possible to hook-up SuperCollider to Hydra and capture music events, I figured it would be helpful to have structures for maintaining state in order to create more complex visual patterns. To better understand why this is helpful, consider the following vanilla Hydra (taken from examples):

// state
freq = 10
numSides = 0

// "event handler"
msg.on('/play2', (args) => {
	var tidal = parseTidal(args)
	if(tidal.s === "sd"){
		freq = freq + 10
	} else if (tidal.s === "bd"){
		numSides = numSides + 1
	}
})

// visual
osc(() => freq, 0.0, 0.8)
	.kaleid(() => numSides)
	.out()

You can only imagine how complex this may get if we wanted to build an entire audio-reactive visual for more than two samples. What if we wanted the handler to also maintain more complicated state, like only triggering every 5th message, or cycling from 0 to 360? For this, we can define Handlers and Shape Generators. The following is an example from the HydraFriend wrapper I have started hacking together.

const { Handler, HydraFriend, Shape } = require("hydrafriend");

// define a new octagon shape
const octagon = new Shape()sides(8);

// handle bd by inverting every 5
const handle_bd = new Handler("bd", () => octagon.invert());
handle_bd.every(5);

// register all our handlers
const hf = new HydraFriend();
hf.register(handle_bd);

While HydraFriend is not yet complete, it is already much easier to create a solo visual + audio set when reasoning about and preparing visuals in this manner.

Lesson 3: Functional Programming is Intuitive

Over the past few years, I have been around numerous people in situations where they were first learning to code: from a coding bootcamp (JavaScript), to tutoring CS100 classes (Python), to this music class (Haskell). If I were to ask you, “hey, which of the following will probably lead to the most frustration?” you would almost certainly say Haskell. However, in my experience, that is certainly not the case. My musically inclined peers, whom had never coded before and were self-proclaimed not computer people, were able to easily create incredible music with Tidal, composing functions in ways that many graduating CS Majors would never even imagine. It was a truly incredible experience… one I continue to try to convey to the Colby CS department in hopes of the creation of a formal functional programming class.

Conclusion

Suffice it to say MU213 was an incredible class! I’d like to publicly thank my professor Ryan Maguire for introducing us to this cutting edge field. Whether you’re a student at a large university, a small liberal arts school, or a graduated, life long learner, I’d totally recommend an unconventional coding experience. You may be surprised at what you learn!

To Categories From First Principles

2020-04-03T05:19:21-07:00

Welcome! Somehow you have managed to find the first post in my Category Theory Series. Embarrassingly, my motivation for diving into Category Theory partially comes from a desire to understand the humor in this video. Yet from the most innocent of beginnings, here I am, here you are. What’s this nonsense all about?

Some People Smarter than Me Say Things About Category Theory

So why is this stuff interesting? Since I have no clout here, I am just YAPB (yet-another-programming-blog) I will phone a friend, or well, just look for Category Theory quotes from really smart people.

“Category theory is both an interesting object of philosophical study, and a potentially powerful formal tool for philosophical investigations of concepts such as space, system, and even truth.” - Stanford Encyclopedia of Philosophy

“Category theory can be helpful in understanding Haskell’s type system.” - Haskell Wiki

“In my opinion, category theory isn’t so much another country-on-the-map as it is a means of getting a bird’s-eye-view of the entire landscape. It’s what lifts our feet off the grass and provides us with a sweeping vista from the sky.” Tai-Danae Bradley PhD

“Category theory is becoming a central hub for all of pure mathematics…. We believe that it has the potential to be a major cohesive force in the world, building rigorous bridges between disparate worlds, both theoretical and practical.” - Seven Sketches in Compositionality: An Invitation to Applied Category Theory

To Categories From First Principles

Preface

STOP: do not read any further. Before beginning your journey here, you must first unthink and unknow any presumptions you may have about Category Theory, Sets, Math, the World, etc. Let’s begin with an absolutely clean slate.

Note: this explanation is meant to be incredibly simple and for ANYONE of ANY background. Thus, some of the mathematical concepts, especially my negligence of Set Theory, are not 100% correct and may lead readers askew if they dive deeper into Category Theory without this knowledge (edit after Reddit feedback).

Object

We will start with an object. An object is simply a thing, yep, literally anything. This thing has no meaning. Here are some things:

cat: 🐈
face: 😀
one: 1
pizza: 🍕

In your everyday life, these objects may have meaning. You may be able to also do things with them, like eat a 🍕 or feed a 🐈. Unfortunately, that is not true in this world, at this time. These are objects, nothing more - do you get the point yet?

Arrow

It may be desirable to transition, or travel, from one object to another. In this world, that is possible via one way roads, called an arrow. We don’t know anything about these one way roads. In real life, they could be bumpy, pristine, fast, slow, curvy, straight, etc. Here all we know is that the road goes from some object A to some object B. Consider the following arrows (➡️):

f: 🐈 ➡️ 🍕
g: 1 ➡️ 2
h: 😀 ➡️ 😀

Notice how the last arrow goes from face to face. Don’t be confused. If you are confused, please reread the preface. All we can assume about an arrow is that it goes from A to B. Also notice that these arrows are directed meaning 1 ➡️ 2 goes from 1 to 2, but not 2 to 1.

Set

Sometimes it’s pretty cool to have more than one of something (unless that thing is parking tickets). This collection of things exists in our world! We call it a set. Here are a few examples:

A set of objects: O = { 🍊, 🍋, 🍌 }
A set of arrows: A = { f, g, h }
The empty set: ∅ = {}

Note: sets can come with some very helpful properties a la Set Theory. While these are important and may be useful later, let’s ignore them for now. Instead, a set is simply a collection of things.

Graph

By now, you’re probably wondering, “what’s the point of any of this if I can’t do anything with it?” Well look no further. Consider a world called graph made up of a set O of objects and a set A of arrows between these objects (some may call these edges and vertices).

A graph G with: O = { 🍊, 🍋, 🍌 } and A = { f, g, h } may look like this:

Let’s introduce some operations:

domain (dom): given an arrow return the object to its left
codomain (cod): given an arrow return the object to its right

Example:

While addition of these trivial operations may seem pointless, we are now able to talk more about our world: “this object is the domain of that arrow”. Before, while you and I could see that two objects are related by an arrow, we had no way of formally describing which object was on which side of the directed arrow. These operations will be important as we continue to build up our world.

Axioms vs. Observations

In the next expansion of our world, Category, we will begin to introduce some axioms about the system. Having not left my house for over 30 hours, I have entered a sort of (phsychadelic free) philosophical existence. If you’re not an experienced Mathematician, and even if you are, you may wonder the following:

What is an observation?
What is an axiom?
What is the difference between an axiom and an observation?
Why do we need axioms?

While I will answer all these, in the process of doing so, you will understand better how mathematics, as a universe is built (and simultaneously understand my process for constructing Categories).

Scene: one day, I take you to an apple orchard and you notice that the first tree has 10 apples on it, the second tree has 37 apples on it and the third free has 104 apples on it.

Tree	Apples
1	10
2	37
3	104

This is an observation.

Observation: “a record or description so obtained “ - Merriam-Webster Dictionary

Since you took some basic math classes, you deduce a pattern:
prevApples + 7 + (20 * 3^(tree#-2)):

tree#	prevApples	formula	apples
1	n/a	n/a	10
2	10	10 + 7 + (20 * (3^0))	37
3	37	37 + 7 + (20 * (3^1)	104

Note: technically this is not mathematically sound, how can you explain the first tree? Ignore this for now… this is more or less the point of this example.

We call this a hypothesis, or conjecture.

Hypothesis: “an educated guess” - every grade school science teacher ever

However, you continue walking through the orchard and see more trees. It turns out your hypothesis did not hold:

Tree	Apples	Predicted
1	10	10
2	37	37
3	104	104
4	229	291
5	430	838

In fact, you were not even remotely close. It turns out, after some laborious calculations, the pattern is actually:
apples = 3 * (tree#)^3 + 2 * (tree#)^2 + 5

With some fancy logic skills, we can utilize a technique called induction to prove this formula works. Once we have proven a hypothesis, it is called a theorem. As stated in an article that really motivated this section:

“Proofs are what make mathematics different from all other sciences, because once we have proven something we are absolutely certain that it is and will always be true. It is not just a theory that fits our observations and may be replaced by a better theory in the future.” mathgion.org

From this, we observe that math tends to follow a pattern like this:

observe something
question whether that observation holds in other situations
postulate that it does and construct a hypothesis
gather some more data, a.k.a examples, and see if hypothesis holds
formulate a proof (via induction, contradiction, etc.)
verify logic holds: via math friends or a proof assistant like Lean, Coq, etc.
tell the world and publish results

From this process, we continuously define the world around us, developing a more concrete system for building, creating, and understanding.

Cool right? Well, I still have not gotten around to axioms yet. Let’s go back to the proof of induction from above. In this proof, I had to rely on some basic building blocks, or principles to base my argument on. i.e. if I was to prove to you 1 + 1 = 2, we would need to agree on a few things:

the definition of equals
how one adds things
what even is a number

While this may seem like a recursive process that never ends, at some point you hit rock bottom… “ok Rob, just believe me, this is true!!!”. This is where we discuss axioms.

Axiom: “a statement that is taken to be true, to serve as a premise or starting point for further reasoning and arguments” - Wikipedia

In each branch of math, logic, philosophy, etc. there are fundamental truths or observations that are more or less “self explanatory.” Since we are building up a mathematical system here, Category Theory, we need to introduce some of our own fundamental building blocks. In doing so, we can then construct an entire system, or world, where our observations (or here our operations) have more meaning because we can prove things about them, things that ALWAYS hold!

Commutative Diagrams

The significance of commutative diagrams is that, for any two objects A and B, every path (of arrows) between these objects, when composed, results in equal arrows. Thus, in the diagram above (for associativity), we have h ; (g ; f) = (h ; g) ; f = h ; g ; f. Let’s take a walk…

h ; (g ; f)
- Start: 🐈
- Take g ; f: 😀
- Take h: 🍌
- END: 🍌
(h ; g) ; f
- Start: 🐈
- Take f: 🍕
- Take h ; g: 🍌
- END: 🍌
h ; g ; f
- Start: 🐈
- Take f: 🍕
- Take g: 😀
- Take h: 🍌
- END: 🍌

From the above, it’s clear 1 = 2 = 3.

So what?

Well, we now have the ability to combine and discuss equality of arrows. With these tools, we can begin to reason about transitions between objects within our world in interesting ways. Without the addition of any more operations or axioms, we can draw parallels to the real world to make some sense of this:

Driving (get from one location to another):
[objects]: locations
[arrows:] roads

Surfing the internet (get from one webpage to another):
[objects:] webpages
[arrows:] links

Baking (combine ingredients to make tasty treats):
[objects:] ingredients
[arrows:] directions like mix, chop, put in oven, etc.

Coding (get from one type to another):
[objects:] types
[arrows:] functions

But actually though, who cares?

Ok, tough crowd. Let’s discuss a hypothetical situation using concepts from above in the world of driving. Let’s say you want to get to strabucks - you’re halfway through some ridiculous article on Category Theory and need an energy boost. Since you’re new in town, so you ask a stranger for directions. Here is how it goes:

You Hello, I just moved to Seattle, where is the closest Starbucks?

Stranger 1: Welcome! Well, if you take road A to road B, take a right onto H, you’ll come up to E. From there, you’ll see one on your left (path: E ; H ; B ; A).

Stranger 2: Hold on there mate, I know a better route. Instead of taking H, you can avoid the notorious pothole and instead take G to ID, which goes between G and F, and end up on F. From there you’ll go a few blocks before arriving (path: F ; ID ; G ; B ; A).

You: thanks for the input, however, if I know that G is the beginning of ID (domain) and F is the end of ID (codomain), can’t I just go straight from G to F?

Stranger 2: oh yeah, you’re a bright fellow. By the unit law you’re correct (new path: F ; G ; B ; A)!

Stranger 3: well how about just going from A to B to G to C to D, a scenic route great for newcomers, and then to E (path: E ; D ; C ; G ; B ; A)?

You: ok, ok, I need to map all this out. Here we go, these are my options correct? [all nod in approval]

Stranger 3: how about I simplify this map a bit. Since traffic in Seattle is the same everywhere and all roads are equally good (in this parallel world, not real Seattle), we get the following (note, as before ; means composition and describes transitions, applied from right-to-left):

Stranger 2: oh!! I see where you’re taking us [takes the paper]. From here we can construct the following commutative diagram:

Stranger 1: so by this commutative diagram and the associativity, this young lad can either take A to D to C ; B or he can take A to B ; D to C. These routes are in fact equal!

You: awesome, thanks y’all!

And from there you hop in your car and get on your way.

Conclusion

If you made it this far, congratulations! I did not anticipate the length of this first blog post. Size aside, I hope it is now clear how powerful and useful this stuff is; with four operations and two axioms AND ONLY these four operations and two axioms, we can already reason about transitions and relations between objects within very different worlds (or structures).

Comb Sort

2019-11-04T04:19:21-08:00

Since I have a few hours to kill on the plane this evening (and I won’t be falling asleep anytime soon due to the delicious Pumpkin Spice Latte I just consumed), I decided to write a post about Comb Sort. Context: on Wednesday I will be presenting this to my Analysis of Algorithms class… this algorithm did not come up during my interview prep, so if you are deep in the process, stressing about how you need to know yet-another-sort-algo don’t fret!

Yo, Bubble Sort Kinda Blows

While Bubble Sort maintains an admirable O(1) space complexity, it has an O(n^2) time complexity. Compared to any decent sorting algorithm, quicksort, mergesort, etc. this is quite inefficient. So, one may wonder, is it possible to speed it up? Where are the inefficencies? Where is the bottleneck? While I cannot promise you Bubble Sort will ever be as fast as a decent O(n log n) algorithm (and it won’t), there is room for optimization!

An Abstract Intuition

Let’s pretend you are a hunter and its your job to hunt as many turtles as possible. Now, turtles are slow creatures, so, we will assume they do not run away from you, We will also assume there are infinitely many turtles, just for the sake of argument.

Since you are a tough guy, your weapon of choice is a club. Thus, you must travel within a yard of each turtle in order to be in striking distance.

For many seasons, you are able to provide for your family. However, your family grows. By the time your family exceeds 100 people, you are having problems keeping up – your strategy for traveling within striking distance of each turtle seriously inhibits the efficiency of your hunting. You begin to wonder, how can you improve upon your hunting? Would it be possible to reduce the distance you travel between turtles?

It turns out the answer to this question is yes! A neighboring family has discovered a hunting algorithm that works considerably more efficiently: in order to reduce the distance traveled, they use a grappling hook to bring the turtles closer.

This preprocessing allows them to perform less steps-per-turtle, allowing them to feed families 100x larger than yours with ease.

Turtle and Rabbits

So how can we apply this to Bubble Sort? (With my best Gavin Belson impression) “Consider the tortoise…”.

In Bubble Sort, there are two, let’s call them, “outlier values”, turtles and rabbits. Assuming an ascending order sort, we have the following definitions:

Turtle: a small value at the end of the list

Rabbit: a large value at the beginning of the list

While Rabbits are cheap to move, Turtles are incredibly expensive to move. Consider the following list:

[X, X, X, X, 1]

If every X > 1, 1 must be swapped (or inverted) 4 times. Now consider a boat load more X’s and several more turtles… I think you see where I am going.

Key Intuition: a bottleneck of bubble sort is the inversion count related to turtles. What if we could reduce the inversion count by increasing the swap gap from 1 to some much larger n (where n < length of the array).

Comb Sort

This is the idea behind Comb Sort! In Comb Sort land, it’s open season and you’re going turtle hunting with a long range sniper. Comb sort works by establishing a gap, and swapping values across that gap. Starting with a gap of length of array - 1 and decrementing by the magic 1.3 until we get to 1, we essentially preprocess the unsorted array, reducing the work of Bubble Sort at the end.

In code it looks like this:

# the magical 1.3
GAP_FACTOR = 1.3

def swap(arr, a, b):
    arr[a], arr[b] = arr[b], arr[a]

def getNextGap(gap):
    gap *= GAP_FACTOR
    if gap < 1:
        return 1
    return int(gap)

def combsort(nums):
    n = len(nums)
    gap = n
    swapped = True

    while gap != 1 or swapped == True:
        gap = getNextGap(gap)
        swapped = False
        for i in range(0, n - gap):
            if nums[i] > nums[i + gap]:
                swap(nums, i, i + gap)
                swapped = True

    return nums

How Effective is this Preprocessing?

Curious to see how this performed, I created a list of 10,000 randomly shuffled numbers and clocked (using cProfile) Bubble Sort and Comb Sort. The numbers line up with the intuition:

Bubble Sort:

~25,000,000 swaps
~8.79 seconds

Comb Sort:

~65,000 swaps
~0.05 seconds

While this data may be seriously skewed by some arbitrarilly bad distribution of the random shuffle in my one sample point, this at least goes to show that preprocessing the unsorted list before performing Bubble Sort is an effective strategy for optimization.

Try it yourself.

References

Adventures in PL: Inferring Function Signatures in JavaScript

2019-10-21T05:19:21-07:00

Last Friday morning, I sat in my small single in Pierce dormitrory staring at the TypeScript compiler plugin I had just hacked together (with some help of course). Given a small code snippet of TypeScript:

import { map } from 'lodash';
map()

my plugin returned a list of type suggestions for the given function, much like what happens when you hover your mouse over a function in VsCode’s Intellisense. It turns out these type suggestions themselves were actually not necesarilly types… as this depended on the type annotation header being used. For example, for the above code snippet, I get something along the lines of this:

{
  name: 'collection',
  documentation: [ { text: 'The collection to iterate over.', kind: 'text' } ],
  displayParts: [...],
  isOptional: false
}
{
  name: 'iteratee',
  documentation: [ { text: 'The function invoked per iteration.', kind: 'text' } ],
  displayParts: [...],
  isOptional: false
}

Which really did not help a whole lot. So, as I checked the clock and saw it was now a staggering 5AM, I came to wonder…

Inferring Callback Argument Positions in JavaScript

In investigating the extension of a tool for generating feedback-directed, random tests for JavaScript, I spent some time experimenting with the tool’s callback position inference algorithm. For the less informed, this is non-trivial in JavaScript since it is a dynamically typed language. That means there is no type signature, type annotation, or type anything to clue a person or a static analyzer into what types are expected.

It turns out LambdaTester’s algorithm did generally well for most common methods, but failed for methods that:

had complex type requirement pre-condition checks, ex: foo(Number, Number, Number, Number, Callback)
had optional arguments, ex: readFile(Number, [optional], Callback)

This makes sense given the algorithm’s basic design: For 100 iterations, randomly permute the set of argument input types (and values) with a callback as the final argument from position 1…5.

So, as a curious and naive undergrad, I set out to see if I could do better.

Putting the Brute in Brute Force

With a couple months of algorithm practice as part of interview preperation, I knew the first move was to check out a brute force solution. This is rather obvious and was quick to implement. Basically, for the first five argument positions (a hard coded assumption made by the paper) I generated all possible combinations of types with a callback at the end. Satisfyingly, this exhaustive approach was correct, finding every single callback position. However, as with any of these brute force algorithms, it did not scale well. For this particular input size, my code tried over 1500 combinations, a fairly non-trivial 15x more than LambdaTester’s algorithm. Some quick math:

type = number | null | object | array | string | boolean

POS_1: cb (1 = 1)
POS_2: type, cb (6 * 1 = 6)
POS_3: type, type, cb (6 * 6 * 1 = 36)
POS_4: type, type, type, cb (6 * 6 * 6 * 1 = 216)
POS_5: type, type, type, type, cb (6 * 6 * 6 * 6 * 1 = 1296)

Total: 1555

Thus, we more generally have a (6^(n-1)) + (6^(n-2))… or roughly an exponential 6^n complexity, which is really bad, especially given the initial algorithm is constant time, or 100 iterations no matter the size of n arguments we want to discover.

How can we optimize this?

Using my problem solving strategies from The Algoithm Design Manual, I searched for ways to reduce repeated work or prune the search space. Ultimately turns out there is no good way to do so since:

every test is unique and so no work is redundant
any test can be the right answer (cannot infer a direction based on a previous outcome)
related to 2, there is no way to determine the partial correctness (output is binary)

Hmmm… time to phone the audience.

***

Type Inference in the Wild

Avoiding the more intensly type theoretical papers, I ended up focusing my search on three, well-used, open source projects:

TypeScript: superset of JavaScript that transcompiles to JavaScript, supporting futuristic ECMAScript features and some static typing
Flow: lightweight, static type checker for JavaScript
Google’s Closure Compiler: JavaScript to JavaScript optimizing compiler

While all three more-or-less have some degree of type inference, the Closure Compiler is less focused on it, and Flow seems to be on the decline:

Recently a bunch of open-source projects originally created at Facebook published plans to be rewritten in TypeScript. At Facebook we strongly value the independence of individual teams in creating their roadmaps, and in doing the best they can for the products they build. The projects that have decided to switch to TypeScript have external contributors whose lives will be much easier with this switch, and we respect these decisions.

and furthermore, was less interesting because it seemed to be veering on the side of required annotations (something I naively believed to be due to a deficiency in innovation). So, I jumped headfirst into TypeScript’s source code with the thinking “if it can tell me what types are and are not correct, then it must have some idea around what types are correct and expected…”

AND WE ARE BACK AT THE BEGINNING OF THIS POST…

Conclusion - Gradual Typing and this Thing Called Heuristics

This evening, before diving back into my partial solution from above, I decided to do a bit of searching into this concept I had seen around the internet called Gradual Typing. Settling upon an article by Jeremy Siek, I learned that gradual typing is a type system where static and dynamic types coexist, often via type annotations…

WAIT A MOMENT… isn’t this kind of like TypeScript!

Going back to Google, I did another search and it appears that TypeScript is a soft sort of gradually typed language (not going to pretend I 100% know what that means… just learned about gradual typing three hours ago). Ah, this makes sense. Thus, it is expected that you cannot fully infer every single method type signature!!!

Ok, so it seems this TypeScript, type annotation crutch is doomed to fail; furthermore not every JavaScript library has a type header file. Now what? Well, coincidentally, I got to the seventh chapter in the Algorithm Design Manual, a chapter discussing heuristics. The basic idea here is that we can use some stats and probability to get a mostly correct answer, sacrificing correctness for, typically, a significant improvement in efficiency. However, this comes with a catch. We must be able to define a cost function to weigh our progress towards the optimal solution. As discussed above, since ever input is unique and only correct or incorrect, no such cost function exists.

Well, it turns out my work here is done. I may not have much to show for my adventure here, but I have thoroghly convinced myself that the current algorithm is very likely the best.

Dynamic Programming

2019-10-12T05:19:21-07:00

So everyone who ever lived and learned algorithms, at some point in their journey, excitedly typed out a “how dynamic programming works” blog post. Ladies and Gentleman, this is not it. Instead, this is simply just some fun with python and proving to myself that this dynamic programming stuff is cool and useful. If you are looking for a how to on dynamic programming, I would totally recommend The Algorithm Design Manual!

I am working through this book in preparation for job interviews and really love the style and content of this book – as I still work through parts of the bible (CLRS), I find this book a sort of “abridged” version more or less perfect for someone interested in learning more than you can ever learn grinding away on LeetCode, but still not too off in theory land.

Fibonacci and Dynamic Programming

This is the dynamic programming example everyone uses… it has almost been beaten to death imo. However, it honestlt took me a few times to really “get it”, to understand what is really going on here and how it relates to the concept of dynamic programming as a whole. To solidfy my understanding of how fibonacci can be sped up with dynamic programming, I coded up four solutions, from worst to best.

Solution 1: Recursion

You’re typical solution that overflows the stack and runs slow.

def fib(n):
    if n < 2:
        return n

return fib(n-1) + fib(n-2)

Solution 2: Recursion and Memoization

The optimized solution that still overflows the stack.

def fib(n):
    if n in seen:
        return seen[n]

    seen[n-1], seen[n-2] = fib(n-1), fib(n-2)
    return seen[n-1] + seen[n-2]

Solution 3: Iteration and Linear Space Memoization

The optimized, stack friendly solution most people are happy with.

def fib(n):
    dp = [0] * (n+1)
    dp[1] = 1

    for i in range(2,n+1):
        dp[i] = dp[i-1] + dp[i-2]

    return dp[n]

Solution 4: Iteration and Constant Space Memoization

The best, A+ solution.

def bestFib(n):
    l, ll = 0, 1

    for i in range(2, n+1):
        l, ll = ll, ll + l

    return ll

So how do these all compare (using an input of 40)?

Solution 1: 300,000,000+ function calls Solution 2: 81 function calls Solution 3: 4 function calls Solution 4: 4 function calls

Don’t believe me?!? See below!

And what is the damage done by using 3 instead of 4?

Jumping into some profiling fun, I installed the guppy library. In doing so, I was able to see the size of the heaps of each respective algorithm after it terminated. It turns out, on sufficiently large data sets, it is incredibly substantial (obvious I know). For example, I saw a 100x difference in memory usage, which resulted in a 10x slow down from 3 to 4 with a large n value. Pretty crazy (yet totally expected…)!

Spring 2019 Independent Study: The Sailfish Programming Language

2019-05-12T05:19:21-07:00

I spent the last three and a half months reading through Compilers: Principles, Techniques, and Tools and implementing my own programming languages, Sailfish. As part of the final submission for my project to complete this independent study, I wrote a manual for the language, describing how it works, and some of the design processes along the way.

Link to the Manual here.

[Medium] A Restrospective On My Time As Organizer of SF Cryptocurrency Developers Meetup

2018-08-09T05:19:21-07:00

A Retrospective on my time as Organizer of the SF Cryptocurrency Devs Meetup

[Medium] An Open Source SDK and Serde Magic: My First Two Months as a Member of the Rust Community

2018-05-09T05:19:21-07:00

A Retrospective on my time as Organizer of the SF Cryptocurrency Devs Meetup

[Medium] I Just Wrote a Stellar Smart Contract Pt. 2: Let’s Dig a Little Deeper

2018-04-07T05:19:21-07:00

I Just Wrote a Stellar Smart Contract Pt. 2: Let’s Dig a Little Deeper

[Medium] How I Beat CryptoKitties

2018-04-02T05:19:21-07:00

How I Beat CryptoKitties

[Medium] I Just Wrote a Stellar Smart Contract

2018-02-21T04:19:21-08:00

I Just Wrote a Stellar Smart Contract

[Medium] So You Want to Buy Ice Cream on the Bitcoin Testnet: Block and Jerry’s

2018-02-04T04:19:21-08:00

So You Want to Buy Ice Cream on the Bitcoin Testnet: Block and Jerry’s

My First Open Source Contribution!

2018-01-04T04:19:21-08:00

I am Now Officially an Open Source Contributor

One of the coolest things about the blockchain/crypto craze in my opinion is the wealth of amazing open source projects. A goal of mine for a while has been to contribute to one of these projects, a sort of rite of passage in the crypto dev community. Well, I have finally made my first contribution!

The project I chose to contribute to is StellarTerm, a Stellar Trading Client. I chose to address the need for a minimal, yet informational account history table. Here is what I came up with:

You can even hover over the assets to get a cool little popup (I did not design the popup):

And you can filter by effect (action) type:

Here is proof of my contribution:

Quite exciting stuff! If you’re also an open source contributor to a cool crypto project, please feel free to comment below.

Cheers and Steem on!

Horizon School of Technology Blog Series

2017-12-01T04:19:21-08:00

During my time at the Horizon’s School of Technology Coding Bootcamp I wrote a series of blog posts:

Bitcoin Edge Dev ++: My Experience

2017-11-03T05:19:21-07:00

Bitcoin Edge Dev++: An Incredible Experience and How I Got Here

I had an absolutely incredible last two days learning the high level technical overview of Bitcoin. As I’ve had more and more people lately ask me about how to begin in this space, I wanted to begin this post with a quick recap on how I got to where I am.

To organize this post I am going to split it up into 5 parts:

My Journey into the World of Blockchain
Bitcoin Edge Dev ++ An Overview
Day 1
Day 2
Now What?

[PART 1] My Journey into the World of Blockchain

My journey into the blockchain world started about 11 months ago midway through December of 2016; thus I am in the weird “new” by some people’s standards and “old” by other people’s standards position. A local tech consulting company, Collaborative Consulting (now CGI), put on an internship competition, a case competition challenging students to come up with the best combination of blockchain and college transcripts. Interested in winning the competition and securing an internship position, I opted for an independent study during my January term to focus on learning about blockchain technology. What started as a desire to win quickly transformed into a desire to learn. I found myself engrossed in white papers and side learning tangents, obsessed with understanding every detail about this exciting technology.

Since then, my blockchain obsession has taken me many incredible places:

Presented about blockchain technology at CGI’s inaugural East Maine Tech Night
Published a Cryptocurrency Ticker App to the Pebble App Store
Attended the MIT Bitcoin Expo
Became a daily Steemit user and even a witness for a week
Played around with Ethereum smart contracts
Found a few local cryptocurrency enthusiasts and helped start a local meetup
Attended MANY cryptocurreny/blockchain meetups in SF
Attended Silicon Valley Tech Week
Attended Tech Crunch Disrupt and got to ask Vitalik Buterin a question
Helped Co-host a Steemit meetup with @ned as speaker (120+ people came!)

And now, I had the privilege of attending the inaugural Bitcoin Edge Dev++ 2 Day Technical Bootcamp. It is finally time to get serious about developing!

[PART 2] Bitcoin Edge Dev ++ An Overview

As per the Bitcoin Edge website, here is the mission of the program:

Bitcoin Edge Dev++ Tutorial is meant to focus on scaling the development capacity of the ecosystem via education of developers in the field of cryptocurrency and helping the industry streamline the process of developer training. The primary focus of this tutorial will be a basic first-principles introduction to cryptocurrency as well as cryptocurrency-specific engineering methodologies, security practices, and standard operating procedures.

Consider a sports analogy for this situation:

The NFL has run out of talent. To solve this problem, they setup a two day bootcamp taught by top NFL coaches and players to educate and train the next level of talent and get them excited about playing pro football.

Pretty damn cool right?

Of course excited and hoping to maximize the situation, I prepped for most of the month leading up. My prep included:

the first six chapters of http://www.learncpp.com/ in an effort to get a little bit of C++ knowledge under my belt
Mastering Bitcoin, an incredible book for getting a high level technical understanding of Bitcoin (most of what I learned this weekend was actually covered wholly, or partially, in this book)
Bitcoin Dev email list, Jimmy Song’s newsletter, Reddit, and Twitter to stay up to date

Walking in the doors on Thursday morning, I was well prepared.

[PART 3] Day 1

Both days of the two day event were held on Stanford’s campus. Let me just say, WOW!!! Stanford’s campus is absolutely beautiful. The palm trees, the adobe architecture, the modern buildings… just an incredible place. I showed up a few minutes early and got a good seat, front and center. We even received some swag (as I had secretly been hoping for).

Here is the schedule for the day:

While I certainly learned too much to explain here in detail, here is a brief synopsis of what I learned. We started with the most basic elements of any crypto system… cryptography. Jimmy Song’s system went from elliptic curves to finite fields to a combination of the two, to ECDSA to asymmetric cryptography to digital signatures, to P2PKH addresses and signatures to P2SH addresses and signatures. While it may seem like a lot, it is in fact a very nice, logical progression. Luckily I had just taken a cryptography class last semester so, I was one of the least confused people in the room.

John Newbery, core developer and active contributor was up next. He explained the basics behind blocks, the blockchain, transactions, and the mempool. While at times it got technical and down to the byte (literally) it was fascinating to see how all these moving parts worked together to create the Bitcoin ecosystem.

With day 1 done, I had a pretty good overview of how the entire Bitcoin system worked; Day 1 was layer 1, focused on the fundamentals of the Bitcoin protocol. Day 2 was layer 2, things built on top of layer 1, ex) Lightning Network.

[PART 4] Day 2

I found Day 2 to be slightly more interesting as it focused less on the fundamentals and more on applications (and even some code).

Here is the schedule for the day:

We began with the last of the fundamentals: HD wallets, mining, and common attacks. Most of this was knowledge I had coming into the day and was covered in Mastering Bitcoin.

However, the second part of the day was incredibly exciting!

RPC: RPC stands for remote procedure call and allows for people to connect and interact with their Bitcoin node remotely. I had always been wondering how this worked because in theory, if I wanted to create a Bitcoin application, it would require me to connect to a node in order to access the Bitcoin network. This session presented us with a basic application and explained the basics of interacting with and setting up full nodes.

Lightning Network: WOW the Lightning Network is awesome! For those of you who are unfamiliar, at its most basic level, the Lightning Network is a second layer protocol for Bitcoin. It works by rooting an initial multisig transaction on the blockchain and then setting up a payment channel between two people off-chain. Via a multi-hop architecture, you can then create a network effect between these channels and connect any two people, allowing for fast, private, cheap transactions that don’t bloat the Bitcoin blockchain. Cool right? Following Tadge’s explanation of the LN, we got play around with a live demo! I received fake bitcoins on the lightning network from the creator of it!!!!! What a great time!

Crosschain Swaps: Crosschain swaps allow you to privately and securing transact assets between two different blockchains. This presentation was followed by a simple demo. While exciting, the premise is sort of cryptocurrency laundering (at least from some people’s perspectives). Very cool for some people, not so much for the government :)

[PART 5] Now What?

I am by no means a Bitcoin expert, and am still years of experience away from being technically skilled enough to be a core developer. However, I am a pretty skilled full stack developer with a high level technical understanding of Bitcoin. So ultimately, I believe the best way for me to contribute to the ecosystem is to use the skills I have and the knowledge I gained and try to develop layer 2 applications. Here are some thoughts I have on where to go next and things I want to experiment with:

Develop a front end facing Bitcoin application using my React/web dev skills
Build a simple RPC application
Develop a simple JS app to mimic Bitcoin’s fundamentals
Mess around with the Blockstream Bitcoin Satellite

Now that I am pretty knowledgable in the Bitcoin basics, I am more than happy to help anyone who may be struggling and needs a little bit of clarification! Like I said, I am not an expert by any means, but I undoubtedly know more this week than I did last week… and I believe that if I continue knowing a little bit more each week, at some point I’ll be pretty good at this stuff.

Hope everyone has a wonderful Fall weekend! Special shoutout to all my fellow Steemians at SteemFest! Cheers and Steem on!

Bitcoin Core Devs Meetup: Forking Meetup Panel Notes

2017-06-03T05:19:21-07:00

Overview

Today, @sahil.hingorani and I attended the San Fransisco Bitcoin Meetup’s Forking Panel discussion in the heart of San Fransisco. The meetup consisted of a strong panel of speakers who discussed the current state of Bitcoin, including the recent forking proposals, Segwit, and Bitcoin Cash. As someone really interested in this space, I took lengthy notes on my computer during the panel discussion. This post is essentially a concise version of the notes I took. Enjoy!

On the Panel

Here are the panelists from left to right.

Eric Martindale: left Blockstream to devote more time to building Fabric, an experimental peer-to-peer information market.

Elizabeth Stark: Cofounder at Lightning Labs and fellow at Coin Center.

Christopher “JJ” Jeffrey: CTO of Purse.io and creator of BCoin, an advanced fullnode implementation of bitcoin.

Laolu “roasbeef” Osuntokun: Cofounder at Lightning Labs and a btcd contributor.

Some Background Information - Relevant Terms

FYI 1: Hard-fork vs. Soft-fork

Currently, there are two acronyms floating around that confuse a lot of people: UAHF and UASF.

UASF stands for User Activated Soft Fork. UAHF stands for User Activated Hard Fork. I will get to what these two solutions seek to do, but before I go there, what exactly are HF’s and SF’s?

Hard Fork: a hard fork is a splitting of the blockchain, resulting in two coins. It essentially creates two coins, two blockchains, and two networks that originated from the same blockchain. An example is Ethereum and Ether Classic. Both started as Ethereum and both can trace their origination to the same blockchain, but now the two are completely different entities.

The UAHF that people are talking about will create Bitcoin Cash and is proposed by Bitmain. This is their plan for opposition against UASF and BIP148.

Soft Fork: a soft fork is a backward compatible upgrade that make minor changes to the network. An example of a soft fork is the P2SH protocol that allowed for new Bitcoin transaction types. The difference between a soft fork and a hard fork is that a soft fork only needs majority rule and is compatible with nodes that have not participated. Hard forks are not compatible with non-participating nodes and thus cause disruption and confusion within the system.

BIP148 is a USAF.

FYI 2: Segwit

Segwit stands for segregated witness. This allows for certain parts of transaction data to be left out of transaction blocks, and thus more transactions can fit in a single block.

FYI 3: BIP91

Allows for Segwit2x (Segwit + 2mb increased block size) to be activated if 80% of the miners signal acceptance, versus the previously required 95%.

FYI 4: BIP148

A UASF that seeks to enforce all Segwit ready nodes to run Segwit. Segwit was accepted into the system by miners in a MASF (miner activated soft fork), aka BIP91.

FYI 5: Lightning Network

The lightning network is the second layer of the Bitcoin protocol. What does this mean? Before yesterday I had no idea.

Basically, the lightning network is a separate entity that works on top of the blockchain. What that means is that an application can do some really cool things that the blockchain may not be able to handle while functioning on the lightning network. Then, later the lightning network can finalize these actions on the blockchain.

An example is reversible transactions. On the blockchain, this is not possible. But, with a second layer protocol like the lightning network, reversible Bitcoin transactions may become a reality.

FYI 6: Bitcoin Cash

Some people (even some who originally backed Segwit2X) are unhappy with the direction of Bitcoin and are creating their own currency. While this carries the Bitcoin name, it is an altcoin like Ethereum, Litecoin, etc. Bitcoin Cash is a fork of Bitcoin with no Segwit and 8mb block sizes instead of 2mb. If you act before August 1, ViaBTC will freeze your BTC in exchange for Bitcoin Cash. Many people from the meetup believe this is a scam.

The Panel Discussion

Question 1: What is the Problem?

a) Bitcoin’s network is reaching its limit and cannot keep up with the number of transactions on the system. Bitcoin is a scarce resource, meaning that with more demand and a limited supply, prices will rise. Unfortunately, with a fixed transaction size, fee’s must increase more and more as people essentially bid to get their transaction included on the blockchain. Thus, micro (small) payments become obsolete, going against the original intentions of Bitcoin visionary, Satoshi Nakamoto.

Solutions to this issue are to make transactions smaller and more efficient (Segwit) and to make the block sizes bigger (Bitcoin Cash).

b) Issue with hard forks. Bitcoin Cash proponents seek a hard fork. Nearly all the panelists agreed that a hard fork is not a desirable solution. This would create confusion between the two competing blockchains and create unhealthy vulnerabilities that would ruin the reputation of Bitcoin as a whole. One of the panelists even went so far as to say a contentious hard fork may seriously damage and even destroy Bitcoin.

c) Who governs Bitcoin? During the discussion of why hard forks are not desirable, the question of governance came up. The golden question here is who governs Bitcoin, the miners or the coders? The obvious argument seemed to be the miners, since they maintain the system. However, these miners are incentivized to protect their investments and so their interest should be aligned with those of the ecosystem. Thus, if they are making moves that instill distrust in the average Bitcoin user, they would likely alter their actions to regain the trust of the average user in fear of damaging the value of their bitcoins.

Question 2: Why didn’t Segwit2X go smoothly?

For those on the panel who support Segwit, Segwit2X signaling means it is time to celebrate. Sure, they had to make some concessions and increase the block size to 2mb, but the Segwit supporters argued that passing Segwit far outweighed the negative effects that may come with increased block size. However, there are still those that are very opposed to Segwit. These people are supporters of UAHF and are proponents of Bitcoin Cash. So, even though it seemed like a half-way point was met, there are still miners and large organizations making plans to split Bitcoin because they are unsatisfied with portions of the agreement.

Question 3: What is your vision of Bitcoin? Is Bitcoin a currency or a stored value?

Here is the unabridged version of my notes for this question :)

Laolu (playing devils advocate in support of Bitcoin Cash): it is a currency, not a stored value (DEFEAT ENTIRE PURPOSE if not used as a currency) NEED more people on the system, MORE supply, less fees, LET MARKET DECIDE BLOCK SIZE.

JJ: into currency and value store BIG ADVOCATE OF LIGHTNING, lightning debate between idealism and pragmatism, SEGWIT2X just good because we got segwit and progress is the MOST IMPORTANT and put before the privacy issues.

Elizabeth: believe it is both currency and stored value, lightning allow for smaller transactions, lightning allows for local consensus and then the bigger network can confirm the rest, NOT EITHER OR SCENARIO.

Martindale: blockchain bad for payments, credit cards way faster, struggle to understand urgency to grow network because SATOSHI vision is not met, need to do this first because use case is not met, bitcoin is a trust anchor and a reference point for all, multiple implementations good for the long run, bitcoin protocol detailed and complex, many edge cases exist.

Question 4: What is happening in November?

Second half of Segwit kicks in and people think 3rd coin maybe as mining pools may go 50/50. Basically, even though there was lots of optimism coming from the panel, we are by no means out of the woods yet.

Question 5: How do we come together as a community and what can we do better?

We need to NOT duplicate work and effectively communicate better as a community
Make lightning more user and developer friendly (basically everyone there was a HUGE lightning supporter)
Party, pop some bottles, and celebrate Segwit
Scaling and developing beyond the blockchain, aka layer one (and not just lightning)

Random Tangent 1: Segwit and Scripting?

Segwit allows for more script versioning meaning more complex codes can be put on the Bitcoin protocol. Some of these include:

Complex contracts (vault, covenant)
Time locks (smart contracts)
Witness programs (GO STEEM!!!!)

Segwit also allows for new contracting languages and the implementation of sidechains. Sidechains are huge because with sidechains and script versioning you can do any change you want without a hard fork.

At this point in the discussion, the panel went into a description of the lightning network. Here it got a little technical. The lightning network is not Turning complete, meaning it basically cannot loop. The people on the panel viewed this as a good thing. Basically, the computational challenge with turning complete systems (the halting problem) is you CANNOT REASON the expense of a program because you cannot predict the outcome. Thus you have problems such as those seen on Ethereum (Ethereum supports Turning complete languages).

Random Tangent 2: Why Ethereum sucks…

Inevitably, the time came to Shit on Ethereum. Basically, the panel concluded that Ethereum is way worse with its scaling issues. Also Ethereym has very few full nodes validating the network and at the end of 2018, it will be at 1TB!

Post Discussion Questions

There were many, but these are the best ones.

As developer, how can I help ecosystem?

Help get better documentation out there
Lead outreach events
Recruit more developers
Once Bitcoin evolves into more layers, other can get involved without dealing with blockchain
Help develop LAPPS - lightning network apps
Find an open source mentor on a project you are building; seek individual who can guide you to find a place to contribute

Why is Bitcoin the best?

According to the panel, Bitcoin is the best because:

It has the best developers
It is the most scalable
It has established the desired network effect
It is about to really explode with all this extra utility
It has the best memes

Conclusion

As someone biased towards DPOS (Delegated Proof of Stake - think Steem, Peerplays, EOS, Bitshares) systems, I am a little weary of Bitcoin. However, the optimistic vibe radiating from the panel and throughout the room of 150 people was contagious. It is hard not to be excited about the future of Bitcoin and the future of cryptocurrency as a whole. I still believe there is a second mover advantage in this space, and the coin that is able to solve Bitcoin’s problems AND find a killer application for this technology will ultimately be the most successful. No matter how optimistic people seem to be, I am still not convinced that a platform that has grown so large can really make the changes it needs without disrupting a majority of its users.

Either way, this was an incredible experience, and I am itching to see how it will all play out. Even though I am not currently invested in Bitcoin, I am interested in the tech debates and curious to see how this will effect the many, many altcoins… and now that ICO tokens are illegal securities, EVERYTHING just got more interesting.

Anyway, I hope this helps some people! I will begin attending these meetups more regularly this Fall since I will be living in San Fransisco.

Cheers and Steem on!

Trezor Unpackaging and Review

2017-05-21T05:19:21-07:00

Finally Received My White Trezor in the Mail Today!

The Trezor

This is my first hardware wallet, so I was unsure what to expect. Here is a picture of the package front:

And of course a picture of the package back:

The package was surprisnghly hard to open. While there were seals, taking the seals off did not make the package any easier to open. However, once I opened the package, I found:

The Trezor
The USB cable to connect the Trezor to a computer
A keyring for easy holding of the Trezor
Two recovery seed key papers (to be filled out once the Trezor was set up)

So, I booted up my computer and headed over to Trezor’s website. Once I plugged in my wallet, the firmware began to install (it does not come with firmware, forcing the user to install the most up to date version). Then, I began the actual setup process:

Create a pin. The pin creation process was actually very cool. You create a pin using the standard nine digit, 3x3 square pin pad. However, the pin numbers do not show up on the computer screen – you use the pin numbers from the Trezor’s filled in pin pad and hit the matching space on the pin pad on the computer. This way, the Trezor may create a random pin pad each time, but you keep the same pin passphrase. Thus, if a hacker were to get your pin combination from the computer, they would not be able to get into your Trezor without seeing the corresponding numbers from your one-time pin pad. Having just taken cryptography, I think this is REALLY cool. It is essentially an implementation of the one-time pad. The one-time pad is a cryptographic encryption technique where two people use the same one time encryption key to send a message. This one time encryption key changes every time. Trezor has essentially created a pseudo-one-time pad with their pin pad passphrase.
Write down recover seed. Once you have set the pin passphrase, you then will see 24 words come up in a very specific order. The Trezor goes through the words twice. This is how you can recover your Trezor if you lose the pin passphrase.
Send some money to your new wallet! Since I had liquidated 40 STEEM I sent $35 worth to my new wallet!

After exploring the Trezor software some, I also noticed they support a few other coins as well.

And here is my Trezor with some of the swag it came with.

First Impression

My first impressions were very positive. The packaging was simple, yet had everything I thought should be included (although for the less intuitive techy, there were no instructions so if your first intuition was not to go to Trezor.io, or plug in the Trezor you may be confused on how to begin). The setup was very fast and easy. The Trezor itself is simple, yet complete – as advertised on the website. From setup to money transfer, it took maybe 10 minutes. So far, highly recommend!

Note: Not endorsed by Trezor, simply a happy customer.

Cheers!

Blind Digitial Signature Voting Scheme (Non-technical Explanation)

2017-04-13T05:19:21-07:00

The Non-Technical Guide for the Average Reader and Less Tech-Savvy Cryptocurrency Enthusiast

The Problem:

Yesterday I wrote an article about my implementation of a blind digital signature voting scheme… however I got too technical. I used some scary math terms like “pick a unit of a field modulo p.” Yuck!

The Solution:

This article is in response to that article, which may be found here. The GOAL of this article is to inform the more casual reader about blind digital signatures. I will split this article into three parts:

Digital Signatures
The Zero Knowledge Proof
Blind Signatures and Voting

Digital Signatures

Nearly everyday people are asked to sign documents.

This signature is a way of verifying the authenticity of a document; it gives a person a receipt that verifies a transaction was valid or a document authorized/official.

In the 21st century, we deal with digital documents, digital receipts, digital messages, and digital transactions. How can we verify/authenticate these things in the same way we do for physical things? The answer is digital signatures. A digital signature is essentially a mathematical scheme for verifying/authenticating data. A digital signature addresses three things:

Authenticates the identity of the sender
    a) Every sender will produce a unique mathematical signature
    b) Thus, a signature may be tied to a user's identity so it is possible to prove I sent data to you
Forbids the sender from denying having sent the data
    a) As a consequence of the first bullet point, it is possible to prove someone sent data
    b) Therefore, I cannot deny that I sent data to you
Verifies the data was not tampered with
    a) Signatures rely on (and are mathematically generated by) the actual data
    b) Therefore if the data is tampered with, the signature will not verify

This system is based on public key cryptography. In public key cryptography, people use public keys to encrypt data and send it to the user of the public key. The user will then use their private key to decrypt the data. While everyone knows the user’s public key, they don’t know the user’s private key. Thus, only the user can decrypt messages sent to himself/herself.

Digital signatures are sort of a reverse of this system. Users sign documents using their private key and these documents are “tied” to the user’s public key; to verify a document, the verifier uses the public key of the signer, the signature, and the document.

Here is a high level diagram of the digital signature process:

The Zero Knowledge Proof

Before we jump into blind signatures, which are simply an application of digital signatures, let’s explore the idea of a zero knowledge proof. A zero knowledge proof is: a way for person A to prove to person B they know something without revealing the information they know. Here is the most classic example:

We have Alice in a purple shirt and Bob in a green shirt. Alice wants to prove to Bob that she knows the password to a secret door in a cave. Alice proposes that instead of giving the password to Bob for him to try out himself, she prove to him that she knows the password. Bob agrees and they both go to the cave. Alice proposes that if she can always appear out of the cave from the path Bob calls out, she must know the password to the door in the cave. So, she and Bob devise a plan. Alice will venture randomly down either path A or path B. Once Alice is down that path, Bob will stand at the entrance and then call out a path. If Alice knows the password to the secret door, she will be able to appear from either path no matter which path she starts down.

Alice randomly selects either path A or path B.
Once Alice has entered the cave, Bob enters the cave. He calls out either A or B. In this case he called out A and Alice started down path B.
Alice, returns down the path Bob has called out. They repeat many times. If Alice knows the password, she will always be able to return down the path Bob calls out; if Bob calls the path Alice starts from, Alice must simply turn around and if Bob calls out the path Alice does not start down, Alice will simply complete the loop by opening the secret door.

In a paper by David Chaum, he outlined an application of digital signatures called blind signatures. Basically, in certain cases, people may need an authoritative figure to authenticate or verify some data, however the person may not want that authoritative figure to know the data.

The best example is a secret election. It goes like this:

Problem: A trustee wants to hold a secret election. The electors cannot meet and will have to send in their votes. However, they value privacy and want their votes to be verifiable.

Solution: Blind Signature Voting Scheme.

An elector casts their ballot, puts the ballot in a carbon envelope, and puts that envelope in a plain envelope with a return address.
The trustee receives the envelope, opens the envelope, verifies the person is eligible to vote, signs the carbon envelope, puts that envelope in a plain envelope with a return address and sends that envelope back to the elector.
The elector receives the envelope, opens the envelope, opens the carbon envelope, verifies the signature on the ballot, puts the ballot in a plain, unmarked envelope, and sends the envelope to the trustee.
The trustee receives the envelope, opens it, and has a verifiable ballot from an anonymous source. Part 4 image lost :(

This is the voting scheme I implemented and discussed in my last post. I hope this less technical approach makes more sense!

Conclusion

Hope this article is easier to follow @pitterpatter @steemitqa @spartanza.

If you have any questions please ask below!

To answer @spartanza:

Does this use, or can it use a digital asset residing on a Blockchain as an identifier or the representation for the digital signature?

For Bitcoin, identification is done via wallet address. Each wallet address is a series of characters that is generated from a user’s public key. Attached to this public key is a private key. Thus, the user can digitally sign transactions from their wallet with their private key. In most attacks/hacks on cryptocurrencies, people trust their private keys to third parties and these third parties are hacked. Thus, the hacker gets the private keys of wallets and can use these keys to send verifiable transactions from the hacked wallets to themselves.

Can it work with a Blockchain?

Yep! It works with Bitcoin, other cryptocurrencies, and MANY other applications (one being voting).

Bibliography:

Zero Knowledge Proof Example: http://pages.cs.wisc.edu/~mkowalcz/628.pdf

Zero Knowledge Proof Images:

CC BY 2.5, https://commons.wikimedia.org/w/index.php?curid=313643
CC BY 2.5, https://commons.wikimedia.org/w/index.php?curid=313645
CC BY 2.5, https://commons.wikimedia.org/w/index.php?curid=313648

Blind Digitial Signature Voting Scheme

2017-04-12T05:19:21-07:00

You may be wonder, “What the hell is a blind digital signature voting scheme?” Well, let’s break it down into pieces. This is essentially RSA + digital signatures + blind signatures + voting. We will look at this step by step!

RSA

RSA is a public key cryptosystem. Essentially, Alice comunicates with Bob by taking Bob’s public key and encrypting a message with that key. Then Alice sends the encrypted message to Bob. Bob receives the ciphertext (encrypted message) and uses his private key to decrypt the message. Then Bob can read the message. This is secure because only Bob can read the message from Alice because only he has the private key to decrypt the message. EX) Think of your mailbox. Anyone can put mail in your mailbox, but only you have the key to open it. From a technical standpoint, this works as so:

Key Generation:
    Pick two large primes. Lets call those two primes p and q.
    Calculate:
        a) n = (p)(q)
        b) phi(n) = (q-1)(p-1)
        c) e = (relative prime of the units of phi(n)) * Higher Level Math Term *
        d) d = modular inverse of e * Higher Level Math Term *
    public key = (n,e) and private key = (n,d)
Encryption
    Receive Bob's public key (n,e)
    Choose m, message
    Encrypt message --> m^e = C
    Send C to Bob
Decryption
    Receive message C
    Remember private key (n,d)
    Decrypt message --> C^d = m

Digital Signatures

Digital signatures work sort of like real world signatures. Whenever you sign a receipt, you are verifying that you agree with the purchase and you are using your credit card; essentially you are “confirming” your identity. Confirming is in quotes because this system is archaic and not reliable. Digital signatures are essentially a mathematical signature that cannot be replicated (at least not easily).

As with RSA, digital signatures use public and private keys. Say I want to send a bitcoin to Bob, well, Bob would want some way of verifying that I in fact agreed to that transaction (everyone likes free money, but Bob doesn’t want me claiming later I was cheated). Well, I can digitally sign the transaction. Essentially, in the Bitcoin world, people are identified by pseudo-anonymous addresses that are random characters. For each address, the owner of the wallet owns a private key. Through math, the only one who may create messages or send transactions from a wallet is the one who possesses the private key associated with that address.

Blind Signatures and Voting Scheme (with RSA)

Consider now a system where someone needs something verified but doesn’t want the signing party to know the data they are signing off on; the best example is at a voting center. While a voter needs an election official to confirm their voting eligibility and authorize their vote, they don’t necessarily want the official to know who they voted for. Thus we have blind signatures. Here is a technical implementation of a blind signature scheme (using RSA), as outlined by David Chaum in this paper:

Signing Authority Creates Public and Private Information:
    a) Generates random p and q
    b) Computes n=pq and phi(n)=(p-1)(n-1)
    c) Picks e such that e is a relative prime in the field of units modulo phi(n) * Higher Level Math Term *
    d) Computes d, where d is the inverse of e modulo phi(n) * Higher Level Math Term *
    e) Publishes to PUBLIC: (n,e)
Voter Prepares Ballot for Signing by Signing Authority:
    a) Generates random r such that 1<r<p
    b) Chooses favorite candidate, option, etc. on ballot
    c) Creates message: m = candidate + r
    d) Generates f such that f is a relative prime in the field of units modulo n * Higher Level Math Term *
    e) Computes g, where g is the inverse of f modulo phi(n) * Higher Level Math Term *
    f) Computes blinded message (disguises his message): m' = m*f^e mod n (where n and e are public knowledge)
    g) Sends m' to signing authority
Signing Authority Authorizes Ballot
    a) Signing authority receives m'
    b) Signing authority verifies voter is eligible to vote
    c) If voter is eligible, signing authority signs ballot: s' = (m')^d (where d is the secret exponent of the signing authority)
    d) Sends s' back to voter
Voter Unwraps Blinding of Ballot
    a) Receives s'
    b) Computes s = (s')(g)
    c) Sends the signature s in to the ballot receiving location
Ballot Received, Verified, and Counted
    a) Compute:
        1) ballot = s^e
        2) signature = s
        3) verification (true or false) --> Use ballot, signature, (public e, aka public key of the signing authority)
    4) vote = ballot - random-extra (predetermined length of random-extra makes easy to subtract)

Conclusion

So I really hope all that made sense! If you have any questions please ask below! If there are any small mistakes, SORRY, it has been a long day.

If you want to try this out for yourself, I uploaded the code to my GitHub: https://github.com/robertDurst/BlindVoting

To run the code:

Download
Download necessary modules (I think you just need websocket)
Open two terminals
Terminal 1: python web_app.py --> The result window of the polling
Terminal 2: python electionPoll.py --> The polling window

Again, comments, questions, love, accepted below!

Cheers!

Rob Durst

OpenClaw - Hello Bob! - A First Take.

Some Lessons

Lesson #1: You Need Cost Controls on Day One

Lesson #2: Rate Limits and Fallback are Not Plug-and-Play

Lesson #3: Hardening

Lesson #4: Cron!

My First Gleam Community Contribution ⭐️

My Experience at SRECon EMEA ‘25

Gleam Gathering 2025: Talk Proposal Accepted 🎉

Contextualizing Reliability

Setting the Scene

Basic Definitions

The Interplay of Intent and Ability

The Don’t Leave Any Money on the Table Principle

Conclusion

SRECon Emea 2025 Accepted!

🎊 Accepted 🎊

A Spoiler 👀

Talk Short Description

En Route to Becoming an Active Member of the Global SRE Community

SRECon Emea 2025 Submission

I Am Now An SRE

Digicus Whitepaper

Scratch for Smart Contracts

Table of Contents

Existing Work and Inspiration

Solution

An Overview

The Digicus IDE

A Compiler Plugin Framework

Digicus Textual Representation

Supported Instructions

Instruction Execution Flow

Valid Types

Conclusion

References

TypeScript's keyof and Structural Typing

Type compatibility

Structural Typing

TypeScript

keyof Examples

Conclusion

References

Napkin Math at Work

System Awareness

Some Basic Stats

Conclusion

RSS Feed Up and Running! How I did it.

Springing into a New Adventure

Immediate Reactions

My Career

But Rob, Why Spring Health

Other Criteria

Conclusion

From the Archives: Independent Study 2019 Summary

Independent Study Fall 2019

What is LambdaTester?

AsyncLambdaTester Basics

A Failed Attempt at Improving Callback Discovery Prediction

Classifying Callback parameters

One Caveat

Some Implementation Details - To Typescript We Go

Conclusion

Microsoft's Coyote vs. Async Server: Repro'ing Race Condition & Generating Test Corpus to Prove Fix

Some Psuedocode

The Problem

Setup - Hello World

Setup - Async Server

Coyote to the Rescue!

Conclusion

My experience: Why defensive coding matters

Background

The problem

Collecting traces: perfcollect

Visualizing traces: flame graphs

Analyzing traces: an unexpected culprit

Takeaways

Two Flavors of Pipes

Pipes