Where Does Complexity Go?

Complexity. So much of engineering and design revolves around taming complexity. So much of product management is understanding your users’ tolerance for complexity.

It’s tempting to think that we’re drowning in complexity nowadays as a result of how interconnected and powerful software has become, but even the early luminaries of computer science recognized complexity is at the heart of building software.

Of course, this observation is not limited to software or even technology. Other fields don’t always use the same language, but you can see the same idea repeated whenever smart people reflect on their experience solving problems.

Physicists are smart. Let’s let them weigh in.

As nice as those are, I think the philosophers are going to win this one.

The simplest explanation is usually the right one.

Occam’s Razor (paraphrased)

Welp, I’m convinced. Simplicity good. Complexity bad. Case closed.

Tesler’s Law

Unfortunately, there’s another idea to contend with, one inspired by physics: Tesler’s Law or The Law of Conservation of Complexity. It’s similar to the idea we’ve covered before of accidental vs essential complexity.

Every application has an inherent amount of complexity that cannot be removed or hidden. Instead, it must be dealt with, either in product development or in user interaction.

Like so many great ideas in computer science, it came out of Xerox PARC. According to Wikipedia:

While working for Xerox PARC in the mid-1980s, Larry Tesler realized that the way users interact with applications was just as important as the application itself. The book Designing for Interaction by Dan Saffer includes an interview with Larry Tesler that describes the law of conservation of complexity.

So we should control complexity, remove when possible, and look for the simplest truth that will work.

But what do we do with the complexity that we cannot avoid? Where does it go? Who shoulders the burden?

Unfortunately, too often in the history of software, the answer has been the end user. We software creators didn’t have the ability or budget to craft a good user experience; so the users had to use their intelligence and time to compensate.

It’s kind of amazing to see Tesler talking about this in the 80’s, because when I started my career in the early 2000’s, user experience still felt like an afterthought for most systems, certainly for enterprise systems. The software shipped, and when users were inevitably confused, they were lovingly encouraged to RTFM.

As thoughtful user experience and better design have started to permeate the software industry over the last decade, this attitude is slowly changing.

Yet, if there is an inherent complexity that we cannot remove and cannot hide, how do we design an experience that feels simple to our users?

Conceptual Compression

The answer is of course that it depends, as it does with most hard engineering and design questions. It depends on the characteristics of the userbase, the flows of the system, the variability of task contexts, and probably a dozen other factors.

But there have to be some universal approaches, general strategies. A few years ago, I heard DHH give a keynote at RailsConf, where he introduced the idea of Conceptual Compression, which is still the best general answer I’ve heard to this question.

Building stuff with computers means building on top of abstractions. CPUs, 1s and 0s, assembler, C compilers, database drivers, memory management, and a million other concepts are required to make our applications work. But as we progress as an industry, fewer people need to know all of them, and thank heavens for that.

It takes hard work to improve the conceptual compression algorithms that alleviate application programmers from having to worry about the underpinnings most of the time, but it’s incredibly rewarding work. The more effectively we compress the concepts of yesterday, the lower the barriers to entry become. And we need low barriers if we are to get more people started making applications.

DHH

He’s talking about conceptual compression from the perspective of a developer, but when I look at the excellent user experiences created by our design team, this idea of compression applies equally well.

Compressing Complexity

When you have a piece of essential complexity, you can’t hide it entirely. It will inevitably leak through, often creating processes that are mostly correct most of the time but never exactly what the user wants, with surprising behaviors that feel nondeterministic.

If you try to remove the essential complexity altogether, you will force users to work around your system. A couple years down the road, you’ll learn that the teams have started using spreadsheets to actually do their work and just recording the results in your system, after the real work is done. Your elegant system that was going to be a platform for innovation is now merely a source of busywork.

Compressing the complexity tries to walk a middle path. Reduce the complexity presented to the user, while making the full uncompressed view discoverable for those who want or need to wade through the full murky quagmire of complexity.

The goal is to let users glimpse the essential complexity, to understand there’s a world of choices and ramifications they can choose to explore, but also to guide them through a simple experience that still works if they choose to ignore most of the complexity.

How To Compress

This is easier said than done though. It requires a deep understanding of the problem space and a deep understanding of your users.

To facilitate this, at Simple Thread we begin projects with a discovery process focused on interviewing stakeholders and users, building out personas and archetypes, user needs and flows, etc. If you work on an internal cross-functional team or have a product marketing team, you may be able to rely on less formal approaches. But if you’re going to present a simple experience of a complex system, there’s no getting around your designers and engineers having a deep understanding of the problem domain and how your users navigate it.

Without sufficient understanding, compressing complexity can feel similar to hiding and have the same negative outcomes. The key difference is that hiding tries to pretend the complexity doesn’t exist, while compressing acknowledges the complexity and tries to handle most of it for the user.

The other thing to keep in mind is that a simpler experience for the user often requires a more complex implementation behind the scenes. It’s a tradeoff, like so much in design and engineering.

It’s tough to distill the idea of compressing complexity into a few easy tips, but I can share what I’ve seen working here at Simple Thread:

• Do the essential work of getting a deep understanding of the users and the problem space.
• Use that understanding to identify the key decision points to drive a simple experience that gives most users what they need in most cases.
• Acknowledge that it’s rarely possible to have a simple solution give all users what they need in all cases.
• Instead build in easily accessible ways to perform more complex actions and fiddle with more detailed options.
• Look for that graceful balance between simple for the user, and still pragmatic to build.
• Beware that an unreasonable amount of engineering or design behind the scenes can be a sign that you are attempting to oversimplify complexity that is essential to the user. In programmer terms, you might have picked the wrong abstraction.

If you have other mental frameworks for thinking about complexity or pragmatic tips, please share in a comment. And if you ever want to chat about controlling complexity in your software products, I’m always happy to talk about this evergreen challenge.