I think many aspiring early to mid-career engineers find themselves stuck here. If you’re struggling with this, I hope this article can provide some guidance. What I needed at the time were tools, like habits, that would allow me to disconnect from work when 5pm rolled around. Similarly, I wanted to prevent work, and work thoughts, from bleeding into my personal time. This an ongoing journey for me, but I have since developed habits that have helped me make a serious dent here.

Before we start, an obligatory note on therapy: if you’re struggling with your work-life balance, a therapist can help you gain clarity on your situation - why are you feeling this way, and what behaviors, or externalities, contribute to your malaise. From a place of clarity, you can often derive your own solutions. If you can afford it, I seriously recommend it.

Obligatory disclaimer aside, this is what worked for me:

1. Get work out of your personal devices

When starting a new job, the idea of installing your work’s messaging app in your personal phone can feel like a great convenience. There’s some truth to that. As knowledge workers, a lot of our work involves communication, so with just your chat client in your phone you can have a pretty productive afternoon. Better yet, you can do this while being away from your desk, waiting for a doctor’s appointment, or lining up for the checkout at the store; always online, always reachable. For people in my age range, being constantly online for friends and peers is familiar and a little cool, so it takes some scar tissue to realize it’s a different beast when it comes to work.

When I first uninstalled my work messaging client off my personal phone, during Christmas no less, I immediately sensed a weight off of my shoulders. I took some time after uninstalling it for me to stop mindlessly unlocking my phone in search for the app, waiting for the dopamine hit of seeing a new message, a thread update. Since then, I have sworn off having work accessible from my personal devices. Ultimately, keeping work constrained to company hardware creates a handy physical boundary that you can fallback on to build better work-life habits.

2. Manage the intensity of your day

The final hour of the work day can be pretty productive: there are often fewer meetings then, so it can feel like a good time to start a demanding task, like an intense debugging session, or an incident investigation. However, finding yourself in the middle of intense focus when 5pm rolls around is hardly conducive to committing your work and logging off on time. If, like me, you enjoy obsessing over puzzles (read: bugs) and can’t stop mulling over them until you’ve made a breakthrough in your understanding (that you’ll immediately need to validate), you need to plan so you can be in the right mindset when it’s time to disconnect.

For that reason, I avoiding starting intense tasks near end of day and also try to start the last hour of my day by making a quick mental plan of how I can finish or checkpoint all the work I’m multitasking at that point in the day. That “final hour agenda” keeps me honest to how much I can really get done before 5pm and lets me bring things to a satisfying conclusion on time.¹

Consider the intensity of work across your day as well. In the past, I’ve had days where I’m fully “plugged in” for hours, alt-tabbing left and right, multi-tasking on every screen and making progress on many projects at once. I still do this sometimes (it’s fun and it can feel great in the moment) but we’re not built for long stretches of multi-tasking; it’s also good to leave some energy in the tank for the unexpected. Plus, if you’re going at 100% for most of the day, the last hour of the day might not suffice to help you wind down.

3. Be disciplined in your note-taking

People who are really into note-taking software like Obsidian and Notion often refer to the process of curating a good personal wiki as “building a second brain”. The analogy resonates because taking good notes has allowed my brain to rest easy knowing that I can rebuild context on the things I was working on by looking back at my notes. For the longest time, I was inadvertently always musing about work because it helped me remember the little details. Taking notes has helped me stop relying on this habit.

My note-taking approach is simple, but it works for me:

• I maintain a To-Do list that reflects the active work that I’m either driving, or simply participating in. I have a general rule of trying not to have more than 8 items in the list, which helps me gauge how much capacity I have for handling the unexpected. If the list ever exceeds more than 8 items, it’s a signal something needs to be deferred onto the backlog, and removed from the list.
• For each project, either active in my To-Do list or “in the backlog”, I maintain a project-based set of notes. These are just for me, and include observations, findings, reminders, handy commands, etc. These notes get messy while capturing information throughout the day, but a quick periodic clean-up suffices to keep them trustworthy.

4. Have routines outside of work hours

It’s easy to continue working after hours when you have nothing else planned in the evening. It’s much harder if you’ve made competing plans, have an appointment, or paid for class that requires you to be elsewhere; those things put constraints on your time that force you to eventually disconnect. If you’re struggling with working too much, it helps to have these “competing plans” not as one-offs, but as routines to provide a constant reminder that it’s time to go.

For me, that’s been a combination of evening gym classes, routine events with friends like weekly meetups and games, and other things of the sort. The key is that these needs to be routine (reliable, expected, at the same time, same place) so that you can start to plan around it. Obviously, making plans like these are a healthy thing to do in general, it’s not just to help you manage your work-life balance; that’s the more reason to do it. Still, I think the default for most people is to schedule these things as one-offs, so I recommend instead turning these into routines.

Parting thoughts

Everyone who struggles with overwork struggles for different reasons, but for me the key is to manage my mindset - I need to be in the right mindset to allow me to disconnect come 5pm, satisfied with the work I did throughout the day, and with something different to look forward to after hours. Similarly, I benefit from experiencing friction in trying to work after hours, which sends me a signal to reconsider before engaging. My advice ultimately serves to nurture those goals. If these systems don’t work for you, I recommend working on gaining clarity on your situation, what leads to your overwork, and how do internal and external factors lead to it. Please take care of yourselves.

All the best.

TLDR; we just made Durable Functions for Python way faster. Try it out by installing azure-functions-durable 1.1.0 on PyPI.

The Durable Functions (DF) programming model aims to make it easy for developers to write stateful workflows that take full advantage of the performance and scaling characteristics of serverless. As such, and since the release of the Durable Functions for Python experience, we’ve been working hard to optimize the performance of DF Python apps to accelerate the parallel processing of tasks, and the execution of concurrent orchestrations. We’re excited to share that we’ve made dramatic improvements to the SDK’s runtime performance, and we can’t wait to get you to try it out!

How to try it out:

Install the package

To try out the new experience, install azure-functions-durable 1.1.0 (or a later version) from PyPI.

This is a non-breaking change release, so your existing applications can benefit from these improvements without any code changes after you upgrade to the latest SDK.

The performance improvements

The exact performance improvements you will see depend on your workload, your orchestration’s structure and, since we’re dealing with Python, it will especially vary depending on your Python language worker’s settings.

That said, we can consider for example just a simple fan-out-fan-in orchestration over 5k activities.

## orchestrator function in “Orchestration/__init__.py” 
def orchestrator_function(context: df.DurableOrchestrationContext): 
    activities = [] 
    for i in range(5000): 
        activities.append(context.call_activity('Hello', str(i))) 
    yield context.task_all(activities) 
    return "Done" 

## activity function in “Hello/__init__.py” 
def main(name: str) -> str:  
    return f"Hello {name}!" 

In this case, we’ll leave the default Python language worker settings intact, which we generally recommend be fine-tuned. We’ll also leverage the latest durable-extension release. Finally, we’ll run this benchmark on top of the Azure Functions Consumption Plan for Linux. We compare the duration of each orchestration in the graph below.

We see that the SDK in version v1.0.3 takes about 246 minutes to complete, whereas version v1.1.0 takes merely 13 minutes! It’s a dramatic speed-up of about 18x!

Again, your mileage might vary, but we expect that generally all major kinds of workloads will benefit, in runtime performance, from this release.

The technical story

There’s a bit of a technical story to these changes and, while we don’t have the time and space to get into the nitty-gritty details here, we do want to share some high-level intuitions about what’s changed.

If you’re a Durable Functions user, it hopefully comes as no surprise that DF leverages an event-sourcing-based replay mechanism for fault tolerance. Your orchestration code is replayed multiple times to determine, among other things, what scheduled work is yet to be completed, and what work needs to be scheduled for execution. Given that computationally expensive tasks should be offloaded to activity functions, most of the time spent by an orchestration function will be spent doing this replay-based bookkeeping. This makes the performance of our replay behavior one of the main factors in overall orchestration performance.

During replay, the SDK needs to reconstruct an orchestration’s state at multiple points in the execution. To do this, it needs to reason over an array of “History” events that record things like: “timer scheduled”, “timer fired”, “activity with name X and input Y was scheduled”, “activity with name X and input Y failed”, etc. Using these events, Tasks are determined to have reached a terminal state, or not.

In our old replay algorithm, a single Task could require an exhaustive search over the History to determine if it had finished executing. This meant that we were incurring, in the worst-case scenario, on a runtime complexity of \(O(H*T)\), where H is the size of the History and T is the number of Tasks.

Given that the History size is mainly controlled by the number of Tasks, we can simplify this expression to be \(O(T^2)\), meaning that the time spent on replay increased quadratically in the number of Tasks. This cost isn’t really noticeable in small orchestrations, but it makes a huge difference if your orchestrator makes a lot of Activity calls.

So how can we optimize this procedure? Our approach was to aim at rebuilding an orchestration’s state by processing the History array only once, which we knew we could do by making use of additional metadata in our History events. Our changes can be summarized in two steps.

First, we implemented a lightweight Task state-machine in Python, to facilitate modeling the full lifecycle of DF Tasks. Each Task state machine also monitors the lifecycle of its sub- Tasks, which are needed to implement our WhenAny and WhenAll abstractions. Using these, we can easily propagating task-resolution information across the DF orchestration.

Second, our History array contains metadata that roughly maps to lifecycle updates of Tasks from within the Durable Extension. Now that we have a Task-like state machine in Python, we make use of this metadata to update our Python tasks as we iterate through the History events, allowing us to more efficiently reason about Task states.

Putting it all together, these changes allow us to make a linear pass over the History events, updating our Task state machines in Python at every relevant event, and using that to more efficiently perform our replay-based bookkeeping and get back to scheduling “real” work. And there you have it!

What’s coming

The technical challenges highlighted here are present in other DF implementations as well, particularly in our JavaScript and TypeScript SDKs. Porting these improvements to those SDKs is high in our priority list.

There’s also much work being done behind the scenes to continue improving our out-of-process DF experiences, so be on the lookout for updates like this in the future.

Keep in touch!

This is a big change so if you encounter any issues, please report them to our GitHub repository here.

Keep in touch with me on Twitter via @davidjustodavid

Other members of Durable Functions team can be found on Twitter via @cgillum, @comcmaho, and @AzureFunctions.

I’m super excited to announce that Durable Functions for Python is now generally available! Folks, this has been in the works for some time now, and a lot of effort has gone into it. The rest of the Durable Functions team and I are really thankful to everyone that contributed, and continues to contribute, as well as the community feedback on GitHub. All in all, we look forward to seeing the Durable programming model available for Python devs in the serverless space.

To get started, find the quickstart tutorial here! You can also check out our PyPI package and our GitHub repo. Finally, we encourage you to bookmark and review our Python Azure Functions performance tips here.

If you aren’t very familiar with Durable Functions but are a Python developer interested in serverless, let me get you up to speed! Alternatively, if you’re a programming languages nerd, like me, and are interested in programming models and abstractions for distributed computing and the web more generally, I think you’ll find this interesting too.

The serverless programming trade-off

If you’ve been in the serverless space for any amount of time, then you’re probably familiar with the programming constraints that this space generally imposes on a programmer. In general, serverless functions must be short-lived and stateless.

From the perspective of a serverless vendor, this makes sense. After all, a serverless app is an instance of a distributed program, so it needs to be resilient to all sorts of partial failures which include unbounded latency (some remote worker is taking too long respond), network partitions / connectivity problems, and many more. And so, if your functions are short-lived and stateless, a serverless vendor trying to recover from this kind of failure can just retry your function again and again until it succeeds.

You, as the programmer, take on these programming constraints in exchange for the many benefits of going serverless: elastic scaling, paying-per-invocation, and focusing on your business logic instead of on server management and scaling specifics.

Stateless woes and short-lived blues

In general, these programming constraints aren’t necessarily a big deal. After all, the serverless space is well-known as good fit for generating “glue code” that facilitates the interaction between other larger services; this is often well suited for those coding constraints.

But what happens when you do need state and long-running services? Here’s where we would have to get creative and perhaps take on a burden of a different kind. For example, to bring back state, you might take a dependency on some external storage, like a queue or a DB. Things get hairy quickly as your apps grows in complexity: you might need to add more storage sources, figure out a way to retry operations, a strategy for communicating error states, and will encounter many other difficult ways to shove-in state back to your stateless program. This is all possible to overcome, and folks have done it, but it’s not pretty. Can we do better?

Breaking serverless barriers with Durable Functions

Durable Functions is the result of identifying many application requirements that were previously difficult to satisfy in a serverless setting and developing a programming model that made them easy to express.

Let’s take a look at an example, in Python, of course.

Let’s say you’re trying to compose a series of serverless functions into a sequence, turning the output of the previous function into the input to the next. An example of this could be a serverless data analysis pipeline of 3 steps: get some data, process it, and render the summarized results. To make things more interesting, you also want to catch any exceptions in that pipeline and recover from them by calling some clean-up procedure.

So, how do we specify this with Durable Functions? See below.

def orchestrate_pipeline(context: df.DurableOrchestrationContext): 
    try: 
        dataset = yield context.call_activity('DownloadData') 
        outputs = yield context.call_activity('Process', dataset) 
        summary = yield context.call_activity('Summarize', outputs) 
        return summary 
    except: 
        yield context.call_activity('CleanUp') 
        Return "Something went wrong" 

Let’s put the calling conventions and syntax minutia to the side for a moment. The snippet above is simply calling serverless functions in a sequence and composing them by storing their results in variables and passing those variables as the input to the next function in the sequence. Additionally, it wraps that sequence in a try-catch statement and, in the presence of some exception, it calls our clean-up procedure.

The snippet above is deceitfully simple. Remember that this is a serverless program, so each of these functions could be executing on a remote machine, and so normally we’d be required to communicate state across functions via some intermediate storage like a queue or DB. But here, it’s just simple variable-passing and try-catches. If you squint your eyes, it feels like the first naïve implementation you’d come up with if this was running on a single machine. In fact, that’s the point.

Durable Functions is giving you a framework that abstracts away the minutia of handling state in serverless and doing that for you. It integrates deeply with your programming language of choice such that syntax and idioms like if-statements, try-catches, variable-passing all work and serve as familiar tools of specifying a stateful serverless workflow.

Diving deeper into Durable Functions

We’re only scratching the surface here. The Durable Functions programming model introduces a wealth of tools, patterns, concepts, and abstractions to facilitate the development serverless apps. They are documented at length here so I encourage you to follow-up there if you want to learn more.

Once you start getting the hang of things, you can dive into more advanced concepts like Entities, which allow you to explicitly read and update shared pieces of state in serverless. These are really convenient if you need to implement a circuit breaker pattern for your application. You can learn more about them here and here.

Taking a step back from all these, let’s look at the bigger picture: programming a distributed application. The question of what are the right abstractions when programming for distribution is a question as old as our ability to ping a remote computer. That question has only grown more pressing as more of our applications have gone to the cloud and our business logic has gone serverless. Within this landscape, Durable Functions provides an answer to the challenge of implementing serverless workflows with ease; and that’s why I’m so excited to share this.

So, why Python?

Durable Functions is currently available in .NET, JavaScript, and most recently in Python as well. Considering the explosion of Data Science-centric use-cases in the Python community, I think serverless is well-suited for powering the kinds of large-scale data gathering and transformation pipelines that are so common in this domain.

Additionally, Python is a very productive high-level language which I think aligns well with the “batteries included” promise of serverless; I think many Python devs will feel right at home with a framework like Durable Functions that abstracts over the challenges of mainstream serverless coding constraints.

Finally, like many others, Python is a language that’s very close to my heart. Like for many other younger programmers, Python is the language I learned to program with and, while in recent years I’ve become quite fond of more obscure PLs (you know who you are), Python is very much still the language that comes to my mind when I think about code. So that’s all to say that I’ve poured my heart into this work and it’s quite meaningful to me to finally have it ready. Even now with the release announcement, we still have many more ideas and projects to help this Python library mature, become more performant, more expressive, and all that good stuff. We hope you enjoy it!

Learn more and keep in touch!

Keep in touch with me on Twitter via @davidjustodavid and give @cgillum a follow for more Durable Functions goodness

We’re also quite active on the project’s GitHub issue board so you can always reach out there! Finally, thank you for reading. If you have any questions and/or want to nerd-out about this, DM me!