If you’ve read The Economist, donated to Wikipedia, or contributed to The World Monuments Fund, you’ve interacted with systems that Diana Montalion helped to architect. She was also a speaker at Codecamp_Festival last year, where she talked about the skills we need now in architecture. 

If you’ve missed Diana’s talk last year, have no fear! She will also be joining us in 2023. If you’re still on the fence, have a look at her take on From software to systems: skills we need now.

The skills we’re going to cover are tech agnostic, it’s not about Kafka, or Kubernetes, or Terraform. It’s the kind of skills that you need regardless of what the programming language is. 

I called this talk From software to systems, but those things mean a lot of different things. How do we define software and how do we define a system? 

I started my career as a software engineer, as a back-end developer or a full stack developer, if you remember when full stack was actually possible to be. I mostly worked on big websites with a lot of traffic and back then, we used a single piece of software and wrote a lot of code. We extended and extended and to make it performant, we added a lot of caching. The software itself created the boundary of space-time, so we created a digital space. There were lots of discussions about aggregating information versus encouraging traffic and encouraging traffic always won. At the time, you wanted everyone to come over to your house.

And time was requested, there would be a request for a piece of information and the software would provide it. Every one of these pieces of software was part of an emerging information system. The internet is THE information system. It’s a giant knowledge graph and then it’s expanded to be beyond even what we would consider the boundaries of the internet to be. Soon, my clients were quickly outgrowing pretty much everything we knew to do.

One example is The Economist. Way back when, my teammates and I built a web property so that what was in The Economist magazine, could now be on The Economist website or The Economist Digital. And within six years or so, one print article had about 40+  destinations and not only that. They were making films and podcasts, they were having different types of media and there was no infrastructure to support this. The first thing I did when I came back again to do systems architecture for them, was that I tried to draw a model of the current system and I discovered two things. One, that nobody actually knew what was going on, because it was all stuck together, siloed, and two, my model had a box in the middle with the word Suzanne in it. And that’s because Suzanne literally cut and pasted things. That’s how things got distributed.

More recently, I have been working with the Wikimedia Foundation. This to me, is an iconic example. Wikipedia is built and designed around a single piece of Wiki Software. The original concept is that the Albert Einstein info is in a web page, it’s an HTML webpage. But now, we think of that content as ubiquitous. We want to ask our smart devices questions and get the information. To make that transition, there are a lot of unique challenges, but one of them is that the community also works on the software too. So the boundaries even between information and software are very blurred.

So, fundamentally the last eight years of my career have been spent restructuring space-time. A space is no longer defined by a piece of software and time is no longer defined by a page request. We’ve been building asynchronous software, microservices, where we don’t transform the legacy software, we create abstraction, we start to figure out ways to iteratively move into the infrastructure that we need. We’ve been building event-based interactions, cloud-native platforms that talk to other cloud-native platforms and hypermedia data structures, all these things.

My focus shifted to the relationships between software. It is less about how I do this in a piece of software and more about what are the relationships between software and the patterns that we need, in order to make those relationships healthy and emergent.

So I became an enterprise software architect, in part because I love it, I really love the pattern stuff, the pattern is what gets me up in the morning. But also, because it was a survival choice. We weren’t going to be able to do what we needed to do, unless we thought differently. I’ve been part of some complex transformations over time, and what I can tell you, is that almost every initiative hits the exact same iceberg. That iceberg is almost never the lack of Kubernetes experts, that’s not what sinks the ship. What does sink the ship is that we don’t think in systems.

The term, software crisis, came out 50 years ago and we still haven’t resolved it. As complexity increases, our thinking and communication structures do not scale with it, and so, we end up trying to do something different inside of our old mental models. And more than that, we don’t think together in systems. Organizations are a communication structure, they’re a thinking structure that dictates how we think and communicate and so, if the organizational structure tends not to change, that means that the structure of our thinking and communication doesn’t change, which leads to us not being able to build something different.

What do I mean by thinking in systems? To explain that, I’m going to talk about linear thinking, which is what we don’t want. Most of us don’t think of linear thinking as linear thinking. When we say the word thinking, this is what comes to mind, linear thinking. It’s what we’re taught in school and it also is the structure of education. We are so surrounded by linear thinking, that it is in the vast majority of cases the default. That is not a bad thing, because we need linear thinking to build, design, deploy and develop software. But different types of thinking go together to create what we need to do and we tend to lean very heavily toward linear thinking. 

Linear thinking is when we want things to be predictable, rational, repeatable, top-down, dualistic and concerned with control. The mechanisms are created in order to create control, but they don’t help us design systems, they don’t help us resolve challenges when there’s uncertainty and complexity. We also expect predictable, top-down, rational from our socio-technical systems, we expect people to behave this way and communicate this way.

Non-linear thinking means many things: systems thinking, parallel thinking, there’s so much overlap in how people are trying to describe it, that I’ve defaulted to the term non-linear thinking The thing that I most want if there’s only one idea that you’ll remember after this, is that non-linear thinking is a practice and the better you get at it the more complexity you’re going to want to take on and then you’re going to have to practice more. So if you’re lucky, you actually never arrive at expertise because you’re always increasing your capability and your capacity to solve problems without over-reliance on linear approaches.

Non-linear thinking is making sound decisions, while happily immersed in complexity and uncertainty. Non-linear thinking is concerned with how relationships produce effect. When you have this and you have that, and you put them together to get a third thing, that neither of those two could have done on their own, it’s a real challenge in the transition from software to systems, where you have this and that and there’s not a lot of attention put on the and part. We’re not good at the relationship between the parts.

Systems are counter-intuitive, so when you’re solving a problem outside of the linear construct, what you’ll discover is that, what we think we know, is almost always incorrect, it’s not going to serve us. Often, when you go into an organization and do an analysis and you’re looking for the leverage point, the place in the system where no one is paying attention to symptoms, you actually can make a change, and unlock the capability or improve the health of the system. It’s hard to find that leverage point, and often when you find it, people already know about it, that’s the pain point. But they’re almost always doing the exact wrong thing, pushing it in the exact wrong direction and that’s because of counter-intuitiveness. Even when you do find the leverage point, almost nobody will believe you. In my experience, it can often take a year of talking about ways that we might transform before I start getting emails back that have my suggestions in them.

It’s a process that requires a lot of internal integrity, which I didn’t need when my focus was exclusively on software. The feedback loop in software development is so much shorter. 

The structure of communication in an organization is the system that they’ll design. Show me a code base, and I can tell you a lot about the team that wrote it, and how they are working. You can see it in the outcome. And this makes sense because what we think and what we think together and talk about, is precisely what we’ll build. It doesn’t come from anywhere else. Everything that’s in code is just thoughts and discussions. So if we want to build something different, we would have to change the structure of our thinking and communication.

The challenge in doing that is that we have two blockers. And they’re pretty big blockers. The first is that we are spectacularly terrible at non-linear thinking. We are so tangled in our cognitive biases and logical fallacies and we really have to practice to be untangled. The second blocker is that we don’t know that we’re spectacularly terrible. As a matter of fact, the worse we are at non-linear thinking, the more we’re sure we’re excellent. And here’s the paradox: in order to know that you’re not doing well in non-linear thinking, you have to actually be good at non-linear thinking, to see it. 

So, in the fullness of systems, we are open to and embrace skills that we call soft skills. I’m going to talk about six skills, but these are really doorways. Pick any doorway that interests you or any combination of doorways like it’s a buffet. They really all go together, not really inseparable, so it doesn’t matter where you start.

1. Craft conceptual integrity – this is actually more of an intention than a skill. It’s kind of a North Star and it’s a good mantra to have. Fred Brooks says it’s the most important consideration in systems design. He also doesn’t do a great job of defining it and it’s really hard to define because it’s like art. You sort of know it when you experience it, or you know it when it’s missing. The way I like to describe it though is creating harmony from a cacophony of many good but independent and uncoordinated ideas. I have three suggestions of practices that can help raise your sensitivity to whether or not conceptual integrity is something that’s part of what you’re trying to do. The first is to think cross-functionally. What’s the impact, the relationships and other perspectives? What is the why of what you’re doing and try thinking outside the boundary of the engineer experience? The second is learning about nonTech systems. There are some examples of books about actual building architecture that help you think about components and spaces, but not in terms of microservices, in terms of literal space. It helps create a language for talking about system patterns. The third is to study relevant-to-you patterns. Every time we go and learn about the patterns that are specific to us is a real benefit to our being able to discern when we’re engaging in systems talk.
2. Cultivate self-awareness –  you can’t improve your thinking and experience if you’re not aware of your thinking and experience. The ability to work with our own minds is an essential tool for what we do. In order to create conceptual integrity in the world, we have to create conceptual integrity in ourselves. It can be very confusing to be interacting with everybody’s thoughts and opinions all the time and still hear your own in the cacophony of many good but uncoordinated ideas. What we think is what we’ll communicate and then what we communicate is what we build. Some examples of skills that come in handy in this area are journaling and meditation, and I know it sounds weird, but for me, they really help me to stay aware of what I’m thinking and about all the things that I’m experiencing. The other thing is that these practices help you hear your intuition and sometimes you can just have a sense of something that is wrong but the cognitive understanding doesn’t come until later.
3. Respond rather than react – people usually panic. They overreact from their own perspective. If there is no communication between people, and no information shared between them, the whole system falls apart, whereas, if there was a systems view of what is happening, we could overcome this. There’s a lot of blame on systems. Remember counter-intuitiveness? What we’re blaming is almost always wrong.  When you’re reacting, when you’re in a meeting and someone says something really stupid that really gets on your nerves, the worst thing you can do is react. Because you will almost always push it in the wrong direction. Rather than reacting to confusion around you or what’s happening in the system or how to focus on solving a problem, maybe you can go for a walk or go for a run. The pattern where you can’t think your way out of reacting results in just adding more on top of your reaction. And then, suddenly you have this major drama. Dropping out of that pattern with experience is the first step to not reacting and enabling your ability to respond. The second skill I would recommend is saying “yes, and”. This is an improv practice, so when improvisational comedians go on stage, they will always use the  “yes, and” exercise. If I say no, then the whole scene falls apart. It just does not work. I’m not agreeing, I’m acknowledging. And if we acknowledged what other people said before we tell them they were wrong, we would find solutions faster. It can be a lot harder than it sounds, but the results will be something else. And the last skill here is to try something. Doubt is the experience of uncertainty and doubt only changes when you have a different experience, so try something. When you’re really stuck, it often means that you actually need some experience in order to continue having the conversation. Do different things at once and eventually, it will help you to discern where you want to go.
4. Build systematic reasoning – respond rather than react. Responding is work, argumentation. We are an opinion-sharing culture. You post something in a forum and then everybody gives you their opinion and somehow you’re supposed to make some integrated sense out of that. I don’t find opinions very productive, but also, opinions are not how we’re contributing value. It’s great to have them, but that’s actually not knowledge work. The real, deeper work comes from using the ability to respond, the ability to arrive at the best possible conclusion, under the circumstances, when conditions are uncertain. Using this as a method of inquiry, build your response. So every time you leave a meeting reacting, the next step is establishing how to build a response to what it is that I’m hearing. Conditions and systems are always uncertain. You put a relationship between three things together, and you put them in the world and then you get something you did not design. That might be a really good thing, innovation. It might be hacking Facebook as a part of an election cycle. Systems will surprise you, that is pretty much a given. You can build systemic reasoning in a very straightforward way. Whenever you make a claim, give three to five reasons that convinced you. Show me what your reasons are and then strengthen them, to get cohesion. If you don’t have cohesion like you think you do,  and you built it into the system, what would happen?
5. Make Art(ifacts) – write it down, model things together anytime that you’re stuck on something. Keep the why connected to the what. Make sure everybody is sharing that same understanding. Create layers in your artefact so that people that need different information can get that. Your architecture is a knowledge graph, so make it like an interconnected knowledge graph. Explore frameworks that describe the system. How do you communicate your system, how do you organize artefacts so that it communicates the system as opposed to how you develop in your local environment?
6. Synthesize –  synthesize knowledge, experience and good judgment, into decisions based on valid reasons. Not your own knowledge and experience exclusively, because systems are complicated. Synthesize expertise, synthesize experience and use sound judgment in order to come to decisions. 

Watch Diana’s talk here.