It doesn’t take psychological studies to know The Trump Effect is a thing. (Though there are many.) Just look around. Friendships are fracturing. Hate crimes are on the rise.

Some prefer to believe The Trump Effect is not a thing. Disagreeing with the existence of The Trump Effect doesn’t mean it doesn’t exist. People who are afraid of diversity and agree (on some level) with the hateful rhetoric that is popping up everywhere deny its impact. “Correlation does not equal causation,” they might say. Or they might simply say “nuh-UNHH!”

I worry about those people, but not as much as I worry about the folks who are aware of The Trump Effect and deny its impact on their own lives. They point to what’s going on around them and on the news as evidence of how hateful rhetoric infects society in a multitude of ways, but may believe their own households are impervious its effects.

I know people who are in relationships where both partners agree on politics yet are suffering from an increase in stress and friction in those relationships. It doesn’t take a behaviorist to explain this: the news is freaking people out. People who are stressed may sleep less, or be more irritable, or be distracted by whatever is stressing them. As a result, a lot of people are bringing their anxieties home where they rub off on their relationships.

Maybe they don’t want to talk. Maybe they can’t talk about anything other than politics. Maybe their fight or flight response is triggered by current events, which makes them fearful of the vulnerable moments that make up a healthy relationship.

(Note: I’ve been listening to Brené Brown’s “The Power of Vulnerability” this past week, and it has influenced my thoughts on all this.)

For example, opening up about anxieties is a vulnerable moment. Initiating romance is a vulnerable moment. Opening up to vulnerable input from a partner requires vulnerability to receive. Basically, anything situation where a person puts themselves “out there” is at odds with a state of fight or flight.

When people stop putting themselves out there in their relationships, they disengage. Few relationships can tolerate recurring or long-term disengagement. For some, the fight or flight responses began as Hillary and Bernie were knocking each other down while Trump was gaining momentum. For others, it began when Trump continued winning primaries. Somewhere in early 2016, the impact of politics on relationships really began in earnest, and it hasn’t let up since.

This is not a partisan problem. Democrats may feel they’re the only ones suffering from The Trump Effect, but it really goes both ways. Conservatives feel under siege for their beliefs, and the stress of feeling like they’re under attack from liberals triggers the same prolonged fight or flight response that liberals feel every time Trump lies.

I’m trying to set aside my political beliefs enough to recognize that today’s political climate is messing everyone up. If people are too stressed, wrung out, defensive or any other emotional state to allow vulnerability with their loved ones, the whole problem gets worse. With no emotionally safe place to recharge, we all disengage more from the warm parts inside ourselves. Our optimism is suppressed, and we’re less able to imagine positive outcomes across the board.

On this Election Day Eve (November 5, 2018), I encourage you to take a look at your close personal relationships and how they have fared over the last two and a half years. If those relationships have suffered, that could have happened for any number of reasons. But to think that what’s going on in the world has nothing whatsoever to do with those struggles is to be dishonest with yourself.

The outcome of the 2018 midterms is going to exacerbate this emotional disengagement for a bunch of people. You may find yourself feeling better on Wednesday than you have recently, or you may feel much worse. Take stock of how things have been in your inner circle now so you can be more aware of how things are different Wednesday.

The Trump Effect is Affecting You was originally published in claylo on Medium, where people are continuing the conversation by highlighting and responding to this story.

I’ve walked the streets of New York City alone after midnight without fear. I have jogged along the beach in Los Angeles under a full moon, by myself. I get in my car at all hours of the day and night, concerned only about not closing the seatbelt in the door.

I take these things for granted. Until recently, it never occurred to me that my lack of fear is the result of male privilege.

I’m a six-foot, two-inch tall white guy. In my adult life, I’ve ranged from fit to overweight, but the width of my shoulders and my upright, chest-out posture has remained constant. When I walk down the street, I look like someone not wise to mess with. The odds of anything happening to me physically that I do not expressly permit are barely measurable. I take these facts for granted also.

I’ve never contemplated how to carry my car keys so they could be used as a weapon if necessary. I’ve never searched my house for possible intruders after entering.

The only things I am afraid of are things I let myself be afraid of. I am in complete control of that aspect of fear; no one has ever been in my life who imposed a sense of fear upon me. Again: privilege.

Over the past few years, I have become increasingly aware that the world I experience is quite different from the world others live in. Sure, I’ve always known about the Haves, the Have-Somes, and the Have-Nots. It’s my lack of fear that I’ve come to realize is pretty rare. I understand now how fortunate I am to be relatively fearless.

Now I See

Despite my own experience, I have been aware of sexual assault. I read the news. I listen to friends talk about their experiences. I lost count of the number of seasons of Law & Order: Special Victims Unit I’ve watched. Still, for a long time, I considered what I read and heard and saw atypical.

I’m awake now. I see clearly that women have been assaulted, objectified, and oppressed my entire life, and for far longer than that.

I think it was Brock Turner’s story that jolted me awake. Then Bill O’Reilly, the Pussy Grabber in Chief, and Roger Ailes. I was kind of stunned as all that unfolded, blinking deer-in-the-headlights like until the Supreme Court nominee hearing last week. The contrast between Dr. Christine Blasey Ford’s entirely credible testimony and Brett Kavanaugh’s dodgy, belligerent, and teary rebuttal was finally enough to force me into action.

But what action?

What can I do to change all this? It’s tempting to say “not much” — especially since the oppression of women is all around us. It’s overwhelming to consider the scope, the pervasiveness of the messages that bombard us every day, all telling us in one way or another that it’s normal and acceptable that women are treated and considered the way they are. It’s everywhere. And if it’s overwhelming to me, a white guy who can think about these problems at my leisure, I can barely imagine what it’s like for women to get through a day with the deck stacked against them. When I think about it, I think “there’s no way I would put up with that shit!” … which is of course because dudes are allowed to get spitting mad. Women who try to speak out are more often than not ignored, doubted, shamed, or all of the above.

I have wrestled with this idea of what I can do … mostly because I was thinking about it as an action I can take that has a beginning and end, such as taking out the trash or doing the dishes. What I came to understand is there is no box to check. If I want to make a difference in the world for women, it must become my new approach to life. I can’t do something — I must be something.

What I Can Do

I can set a positive example for my two sons.
I can make choices about the type of media I expose my sons to, and steer them away from the Porky’s-like content I consumed in the 1980s.
I can correct and educate my boys when they (unwittingly or not) behave in ways that perpetuate the patriarchy.
I can intervene when I see women being abused verbally or physically.
I can vote for candidates and ballot issues that aim to combat the oppression of women. (Often that will mean voting for women.)
I can be mindful of my responses to things that may be affected by a lifetime of societal training to consider women “less than.”
I can keep fighting against my own social programming.
I can amplify women’s voices through social media and IRL conversations.
I can talk and I can write.
I can see.
I can listen.
I can believe.

What You Can Do

You can do the same stuff, dude. It’ll take practice. You’ll slip up and fall into old thought patterns and behaviors. This won’t be like flipping a switch. Be mindful of when you slip and re-double your efforts.

Fair warning: there may not be a fundamental change to this situation in your lifetime. I hope there will be, but it’s naive to believe we can change thousands of years of bias in a hurry. Our combined actions won’t stamp out the oppression or attempted oppression of women any more than racism has been eradicated.

What I can promise you is this: if you don’t adjust your behavior (and I’m talking to all you guys), nothing will change. And it must change.

Calling All Dudes was originally published in claylo on Medium, where people are continuing the conversation by highlighting and responding to this story.

I Hear You’re Feeling Down

The gloom of winter began almost imperceptibly in September, as it always does. Minutes of light shaved off the day, barely noticed until the shavings became ten-minute hunks of light hitting the floor and sputtering out.

Your mood followed a similar pattern. The bustle of back-to-school mixed with the increasingly impossible daily political news. While you were trying to make sense of what was happening, his campaign signs multiplied in places they didn’t seem to belong, such as lawns in your neighborhood. The walls felt as if they were closing in.

Then a recording of him bragging about sexually assulting women surfaced. Hope sparked and began to spread. Maybe the worse outcome of the worst election cycle in history could still be averted!

Then Comey’s October Surprise.
Then election night.
Then despair.

Can You Show Me Where It Hurts?

How could this have happened? Don’t people understand? Why do so many people vote against their own best interests?

The impact of the election outcome hit you as a series of slow-motion body slams. The programs you and your loved ones cherish were suddenly in jeopardy. His cabinet picks were a continuous stream of cruel jokes—a climate change denier to lead the Environmental Protection Agency, a public education opponent as the Secretary of Education, an anti-vaxxer as Secretary of Health and Human Services. What was bad to begin with became worse than you’d imagined. Despair began to feel normal.

After the inauguration, the insults and affronts to everything you held dear about America came in a rush. So many outrageous events on a daily basis it took all your energy to keep up. You supported the marches, the protests, the defiance … the fight became nearly all-consuming. Coupled with the regular (and sometimes substantial) stresses of daily life, your relationship with your significant other took a beating. Friends fell by the wayside. Being a good friend was impossible with everything going on. You didn’t have the energy to explain, and felt you shouldn’t have to anyway. Can’t everyone see the world is coming apart?

There Is No Pain, You Are Receding

His relentless waves of embarrassing tweets.
His blatant support of white supremacists.
His racism oozing from every pore.
His ignorant ravings escalating global tensions.
His ceaseless lies.

Feeling battered, exhausted, defeated, you withdrew from most of it. Nearly a year of fighting and it all keeps getting worse.

How will you weather this storm that feels like it will never end? Do your job, take care of the kids, hunker down. Put your own needs on the sidelines. Board up the windows, sandbag the doors. We are all hurricane victims in the emotional sense.

You don’t even bother with the people worrying about you anymore. There’s too much to explain. Plus, why don’t they feel it like you do? How can they worry about you when so many people are being hurt every day? You can’t make them understand if they don’t get it already.

When I was a child
I caught a fleeting glimpse
Out of the corner of my eye
I turned to look but it was gone
I cannot put my finger on it now
The child is grown
The dream is gone
I have become comfortably numb

—Pink Floyd. “Comfortably Numb.” The Wall, Columbia, 1979

Hey.
WAKE UP.

Zoom in. Repair your relationships. Nurse your wounds. Let others help you. If you aren’t taking care of yourself, you can’t keep fighting. There’s a reason they tell you to put your own oxygen mask on before helping those around you.

Yes, it’s overwhelming. Yes, it’s relentless. Yes, you feel guilty thinking about your own pleasures with so much suffering going on.

But remember: without those pleasures, despair will consume everything. Take a chance on some hope. Stick your neck out a little bit. There are still good things out there, maybe more than you think. Having some good times won’t extinguish your fire for resistence. Good times are the fuel that fire needs.

Take a breath. Feel good. Draw strength from the people who love you. Remember what you’re fighting for. (And remember even Muhammad Ali needed help getting his gloves on.)

Then keep fighting.

Special thanks to Roger Waters and David Gilmour for allowing me to hijack portions of Pink Floyd’s Comfortably Numb lyrics for this post.

Like this post? Tap the clap, baby!

You Have Become Comfortably Numb was originally published in claylo on Medium, where people are continuing the conversation by highlighting and responding to this story.

Hands-on Learning was originally published in claylo on Medium, where people are continuing the conversation by highlighting and responding to this story.

I think I understand the sort of aw-shucks tone you’re going for here and in a few other spots in your apology. (Specifically, the “yeah, I kinda needed that” line near the end.) I don’t know what you think it conveys, but there’s some unquestionably creepy shit here, man.

To me, this one sentence undermines all that follows. Either you don’t really believe any of it (or enough of it) was creepy, or you’d like the reader to shrug and laugh it off with you.

I offer this critique to urge you to be careful: the aw shucks style that’s useful and accessible to midwestern boys like us is doing you a disservice here. If you’re going to own this stuff, then fucking own it.

“Probably”? was originally published in claylo on Medium, where people are continuing the conversation by highlighting and responding to this story.

“I don’t think you need a transfusion tonight, but you need to see someone tomorrow.”

So that’s an alarming thing to hear. Especially when the “you” referred to is you.

Let’s rewind.

Eight years ago, I noticed some bright red blood in the can when I’d finished my business in the business office.

That can’t be good, I thought. I oughta ask someone about that.

A few months later, the third such event occurred, and came with an intense desire to go to sleep. That prompted me to make a gastroenterologist appointment.

A week and a few Massive Invasions of Personal Space (MIPS) later, the doc said I had A Thing. A Thing that was bleeding.

“Uh, that sucks,” I said. “What do we do about it?”

“Well,” said the doc. “You could have A Thingectomy. That’s one option. But when you look at your situation” — he gestured to the entire half of the room where I was sitting — “you can expect to have these kinds of problems.”

“My situation?”

Doc nodded and continued. “You’re overweight. You lead a sedentary lifestyle. You eat a limited range of things, all of which are bad for you. You avoid roughage like you’re allergic to it.”

“I am allergic to grass, actually,” I said. Doc ignored me.

“As long as you keep making these choices, you’re going to have problems. A Thingectomy can cause all sorts of other problems. Usually people are fine afterwards, but every once in awhile a patient winds up with a whole new set of problems that make the original Thing they started with seem really great.”

“Plus,” Doc said, “when you consider that your choices are choices, I generally don’t recommend A Thingectomy. Just make better choices.”

Right. I went on my merry way, pondering my life choices and silently agreeing that A Thingectomy sounded like a very poor life choice. I mused on what concessions I’d be willing to make right up until I got home, where my two toddlers instantly hijacked my train of thought.

Blink twice and it’s 2013. The periodic bloody event that occurred every few weeks progressed into multiple times a week. Each time it happened, I gazed into the crimson water, searching for clues.

Look at your situation, Doc said in my memory.

Oh yeah. Maybe I should do something about that. I imagined dramatically reducing my carbohydrate intake, as I’d done ten years prior to lose weight. I remembered the grueling carb withdrawal symptoms. As I left the business office, the thoughts faded slowly into the background, leaving a residue of anxiety — NO CARBS!! BUT, THINGECTOMY!! — smeared all over everything.

Over the next two years, the red drip became more frequent. With a big sigh, I decided one day that since I was in my early forties, after all, I’d better start getting used to using various “adult absorption” products. Depressing as hell, but I was too tired to think about action plans. I especially had no time for A Thingectomy, and suddenly a nap sounded good.

Looking back, it’s noteworthy that the time period when CHRONIC BLOOD LOSS (as New Doc would write on my chart in the future) became something I was intimately familiar with was the same time I slid back into depression. I wrote about that party here. It’s funny. There are a lot of overlapping symptoms between chronic depression and anemia.

Time passed. I actively avoided thinking about my daily crimson tide, because ABSORBENT PADS and OMFG. I don’t think I was in denial (he said, the quintessential denial denyer) because I knew allllll about My Situation.

My energy ebbed. I had a harder time focusing. I bumped up my ADD medication but it didn’t help much. So much in life was demanding my attention and energy it was no wonder I felt lousy much of the time. I was just beat. Right?

But My Situation, I kept thinking. I was gaining weight. I was still not exercising or making improved choices about food, excercise, sleep — the things that contributed to My Situation. So of course I get tired easily. I’ve been told that happens to fat dudes, and I’m firmly in that camp these days, so yeah. I’m this tired because I’m that out of shape.

My world collapsed in many ways during 2016, as I’ve mentioned. Through it all, I experienced a lack of ability to work very well, extreme fatigue that came on suddenly, and more. I noticed these changes, was confused by them, and ultimately decided it must be due to Life’s Crazy Ways and my ongoing recovery from my holiday bout of pneumonia. Right?

In parallel to all this, my 5 Hour Energy consumption went from every so often to three times a day. And no Mom, I didn’t connect the dots. Yes ma’am, I should have.

Friends told me I was spacing out in conversations, or suggested I wasn’t even listening. My already sketchy memory seemed to be getting worse.

Through it all, I thought Ok, I’m going to have to do something about this every time I visited the business office. But I sure as hell don’t have time for A Thingectomy in the foreseeable future. I staggered on.

Yeah, actually staggered. I sometimes found myself so exhausted I’d stumble, bouncing off door frames on my way to bed.

I decided that something was definitely amiss when I sat down in the business office and heard a sound like urination. Now, typically I would not find this worth mentioning, as I was in the place where such sounds are common. It then occurred to me I was not, in fact, urinating. When given the opportunity, the blood just flowed in a steady stream. Not like the REDRUM scene in The Shining or anything, but steady.

Wondering if I’d just hit the threshold for “heavy flow,” I made an appointment for a physical. Due for one anyway, I figured.

During the two weeks until that appointment rolled around, the blood faucet stopped flowing like it had been. I had major head rushes once or twice a day. Sometimes my thighs felt tingly like they’d fallen asleep while I stood in the kitchen making school lunches for the boys. My pulse sometimes shot up to 130bpm (according to my trusty Apple Watch) when I did super-strenuous things like Get Up From Chair. My hearing started getting funky, like intermittently muffled.

I think there may just be something wrong with my blood circulation, I thought.

It was a head-scratcher in the physical. Pulse was good. Blood pressure was good. Heart sounds like it should. Hm, we mused.

“Maybe it’s related to the sinus infection I had last week?” I offered, flexing my I’ve-Seen-All-the-Seasons-of-House-M.D. muscles. New Doc wasn’t impressed.

“Maybe. Let’s order some blood work and make sure there isn’t something else going on,” New Doc said.

Which brings us back to the beginning of this story: a call from New Doc, asking if maybe I’ve had any gastrointestinal bleeding that I forgot to mention during the exam. Oh, yeah.

I was given an “emergency referral” to a gastroenterologist, whom I went to see the following afternoon. I asked GDoc if there was something in the body chemistry that allowed some people (y’know, ALL THE WOMEN) to bleed regularly without having these kinds of problems.

“Oh, a lot of women are anemic. Many of them are and don’t know it. It’s under-diagnosed,” GDoc said.

Okay then. GDoc explained that the sensations I’d been experiencing are what happens as your red blood cells and hemoglobin become depleted. Given the rate of blood loss, and that I was not consuming enough critical ingredients needed for my body to make more blood, GDoc figured my body has just shut down blood production. Probably a month or so ago, right around when things really started feeling worse.

Which brings us to the current moment. I’m on water only today, as I get set for my double ended MIPS tomorrow. I’m on 325mg of ferrous gluconate once a day. (GDoc promises I’ll feel much better when my body starts making blood again.) I’ll learn about My Thing tomorrow after I sober up from what they’ll give me to make me calm enough to allow the MIPS to proceed without any of the hospital staff being injured by resistance on my part.

As I prep for tomorrow’s adventure, I encourage you to read Dave Barry’s take on his experiences with MIPS.

I’m still sort of amazed I didn’t connect the dots any sooner. If you’re feeling a little wooshy, you might want to call the doc sooner than I did. 😉

When the Well Runs Dry was originally published in claylo on Medium, where people are continuing the conversation by highlighting and responding to this story.

Just putting this here while it’s still legal to post such a photo here in the United States.

Putin In LEGO was originally published in claylo on Medium, where people are continuing the conversation by highlighting and responding to this story.

Howdy. It’s been awhile. Maybe you’ve been catching my snark on Facebook, Quora, or perhaps Twitter. Or, like a lot of people, maybe you thought I’d limped back into the Batcave and succumbed to my wounds.

That didn’t happen (🙌 Alfred!), but it has been a busy couple years since I wrote anything longer than a tweet.

After a few years without writing, I started wondering if I had anything else to say.

(I’ll pause while the people who know me stop laughing.)

The fact is, the most powerful, poignant work I’ve written lately fit on the back of my thumb.

“LESS.”

I need less stuff. I need less distraction. I need to do less talking (and more listening).

So … given that I sincerely believe that I need less of almost everything, where does writing fit in? After all, less is more, right?

I don’t know yet. Hang around, and we’ll figure it out together.

On Writing Again was originally published in claylo on Medium, where people are continuing the conversation by highlighting and responding to this story.

When we started using Amazon EC2 at Mashery in the fall of 2006, it had been in private beta for about three days. Meaning they’d announced it was available, but you had to get an invite to use it.

We managed to score an invite, and I immediately began building Mashery’s entire infrastructure around EC2. There was some angst over this decision internally and from our investors, but in retrospect it was one of the most important (and best!) decisions we made as a young company.

One of the things I remember vividly about the month in which I got my feet wet with EC2 was my take-away from reading the sparse documentation. It didn’t come right out and say it, but the vibe between the lines was:

WE MAY RIP THE RUG OUT FROM UNDER YOUR FEET AT ANY MOMENT.

I had this sensation that ANY instance, at ANY time, would just blink out of existence without any warning (other than the ominous undertone of the documentation).

So we built for fault tolerance from the beginning.

In the years since, I’ve seen services flail due to some EC2 hiccup or another. The buzz around Infrastructre as a Service has shifted from 2006 “IF YOU USE IT YOU ARE CRAZY” tones to “YOU ARE CRAZY IF YOU DON’T USE IT”, and with that shift has come this pleasant sensation that cloud infrastructure never goes down.

Snap out of it.

ANYTHING TECHNICAL has the potential to just up and crap the bed at ANY TIME. A service built without anticipating failure deserves the downtime it experiences.

I’ve heard all the ways of saying that planning for failover is premature optimization, but let’s face it: if the service isn’t built to fail well from the beginning, it’s unlikely ever going to get around to adding graceful fault tolerance.

What most companies do when faced with Technical Failure Awakening is build a “we’re down! Sorry!” page that lives on some other server, and points DNS to that page when their primary infrastructure fails. That’s not a solution, folks, it’s just a slightly better bedwetting than an completely unresponsive server. These guys have gone from failing their customers 100% to failing them 99.9%.

There are plenty of ways to get around this — it takes planning, and prioritization from the top.

Here’s the catch: failing well is hard. In the current “OMG Launch Immediately and Iterate!” and “Customer Service is the Only Defensible Strategy” culture of internet companies, no one has paused to recognize that these are mutually near-exclusive. Planning to fail gracefully in ways that don’t negatively impact customers is HARD WORK. It takes TIME, and simulated failures. (See: Chaos Monkey)

This is by no means a complete list of failure avoidance resources, but here are a couple of things to start with:

(No, I don’t work for DynDNS — I just think they have a great product offering at a reasonable price. Similar services can be had from UltraDNS or Akamai if you’re interested in paying 1999 prices.)

Amazon RDS doesn’t currently support auto-failover replication between regions, which is the only thing that would have saved people from this week’s outage. Those that are particularly concerned about having serious uptime can’t be fully bound to what their cloud service offers them. Cloud services are super-awesome, but it may make sense to use them in conjunction with other techniques that aren’t (yet) packaged up for mass consumption. Replication is often one of those hard problems that require custom configurations that off-the-shelf product features won’t cover. Remember: that reality doesn’t minimize the importance of replication to fault tolerance.

Go forth and stay dry under the clouds.

Photo credits: iStockphoto, CaptainD

Originally published at claylo.com on April 21, 2011.

Where There Are Clouds, It Sometimes Rains was originally published in claylo on Medium, where people are continuing the conversation by highlighting and responding to this story.

Gene Kranz was 37 years old when Apollo 13, the space mission he served on as lead Flight Director, launched. A few days after takeoff, the Apollo 13 Service Module exploded, jeopardizing the lives of the three astronauts on board. While he never used the phrase “Failure Is Not An Option” (other than in the film version of the story), it was the attitude of mission control.

In light of the AWS EC2 crisis, which has left some companies with their services crippled or completely disabled for over 24 hours, it’s a good time to reflect on whether or not failure is an option to you. I know that most of us aren’t writing software controlling the launch codes, but we are writing software that businesses, and therefore people’s livelihoods, are dependent upon.

Evan Cooke, Twilio’s CTO, wrote a great post yesterday about how they managed to avoid any downtime. Pay attention to his update at the end of the post regarding how they’ve resisted EBS adoption.

By the way…
At Mashery, we did extensive I/O testing on EBS versus RAID0 striped ephemeral stores, and there’s no comparison. RAID0 blows EBS away on performance. Sure, you don’t get the magical features off-the-shelf that Amazon provides for EBS, but then again … how magical are those features feeling today?

So ask yourself: is failure an option?

The Choice is Yours

(Link)

When you’re designing a system, you make tradeoffs. Choices. You must choose among variables like:

• Development speed/Time-to-market
• Development cost
• Ongoing costs
• Quality
• Performance
• Resiliency
• Durability

… just to name a few. Most people are familiar with the Project Triangle: “Good, Fast and Cheap: Choose any two” … but a simple triangle doesn’t fit today’s projects.

Most of the above items are self-explanatory, but I specifically separated resiliency and durability. A project/site/service may be resilient, which to me means it can go down and bounce back quickly. A durable site, on the other hand, won’t go down. There’s also a variance between the two on whether or not a site can rebound without data loss. A resilient service may bounce back, but suffer data loss … or, perhaps a resilient site doesn’t have any data to worry about. There aren’t many services that have value without some data, but many services provide value with minimal data dependencies.

Heed The Heptmogrifier

As a handy reminder, I give you … The Heptmogrifier. Adjust as you like, but remember that your head will explode if you try to set all sliders to the max.

(I know “performant” isn’t really a word, but it’s used often enough that I’ll deal with those complaints.)

This is what you’re really working with when you deal with “internet-scale” applications.

The traditional good/fast/cheap tradeoffs will dictate the project’s code quality. But those three factors are influenced heavily by the choices you make about the other four.

Sustainable: How much is this thing going to cost to keep running, given the choices made?

Durable: How susceptible is the service to failure? More importantly, how likely is data loss as a result of failure?

Resilient: How quickly can the service recover from a failure? It’s extremely difficult for a service to have zero downtime. So, even if you’re building something durable, factor in what it takes to failover. Example: a DNS TTL of 60 seconds, with an every-30-second health check and a two-strikes policy, might take as much as 120 seconds to detect a failure and switch traffic elsewhere.

Performant: How do the choices you make in other areas affect performance overall? Or, perhaps some aspects of the system will need performance expectation adjustments based on the choices you make around durability?

Not all of these choices are as painful as they sound. A comment on my previous post suggested that latency was too high across regions to consider. This simply isn’t true for most applications. A half-second lag for master-slave replication across the United States is nothing when you consider that you’ll be able to cut over to that other region in 2 minutes, promote that slave to a master, and be back up and running before most people finish reading the TechCrunch article about how the sky has fallen.

Talk to Your Investors/Stakeholders

They probably don’t care about the technical details, but they will certainly care about their investment going down the tubes because the service they invested in goes toes up due to incomplete failure planning. It’s not a popular conversation, and “how will I scale?” is often how questions about uptime and availability are framed.

It’s time to think about it differently. I agree that scaling to millions of users is a nice problem to have — a “high class problem,” as a friend is fond of saying. However, surviving an outage like what’s happening with Amazon EC2 East is an entirely different story. Your service may find it difficult to get to 10,000 users if it faceplants every time an upstream provider has a hiccup. (Investors, please weigh in below — feel free to call bullshit if I’m off the mark.) Make sure you and your other stakeholders are on the same page.

Take control of your destiny.

Reject failure as an option.

In doing so, you’ll not only live to fight another day, you’ll be around to take customers your competitors alienated by choosing … poorly.

Image credits: NASA, Indiana Jones and The Last Crusade

This post relates to this one, published a day prior:

Where There Are Clouds, It Sometimes Rains

Originally published at claylo.tumblr.com on April 22, 2011.

Failure Is Not An Option was originally published in claylo on Medium, where people are continuing the conversation by highlighting and responding to this story.