The Law of Unintended Consequences

Bet you never thought about it, but Energy-efficient traffic lights can’t melt snow. Talk about the Law of Unintended Consequences. You want to save money by buying more energy-efficient traffic lights, but then you end up spending an untold amount of money on traffic light cleaners, or potentially, the EMS/Police who have to deal with the aftermath of stop lights not working after heavy snows. Then there are the health insurance costs, department of transportation costs to fix the barriers that might get damaged, car insurance costs, etc… The consequences are complex and far reaching. But once you see it, it just seems so obvious! Of course LEDs aren’t going to melt snow, they don’t produce very much heat at all!

Unfortunately, while using incandescent bulbs, this wasn’t a problem. So it wasn’t a problem that they knew needed to be addressed. Trust me, if incandescent bulbs didn’t melt the snow previously, then you can be guaranteed it would have been at the top of their list to find a solution to this problem. But they picked a solution based on one factor that brought some other problems along with it.

Does this sound familiar? If you are a software developer, it should. You probably have to deal with some of the most complex systems that have ever been devised by humans. Thousands of inputs, outputs, moving parts, pieces of business logic, and probably a team of about 5 people to do it all! Right? Of course, companies often don’t have the resources to bring in dozens of developers to write a really great system, they just need a “good enough” system to allow them to function.

So What is a Developer To Do?

But when you’re pushed up against constant deadlines, resource issues, feature requests, etc… what do you do? Well, you try to do as little work as possible, right? You try to find some way to solve parts of your problems so that you don’t have to write code in order to do it. Some of these problems, such as data storage, are so commonplace (and complicated) that we don’t even consider building them anymore.

Other problems aren’t so generic and commonplace, but often we can leverage third party products, tools, or libraries in order to get us part of the way there. Great! We can just use tool X or library Y and now we just need to plug it into our software at point Z and everything is great, right? Not really, there is a big part of the picture that is missing here.

People who have a problem and pick a tool based on features are doing the same thing that the people who picked the LED stoplights probably did. They said, does this have all of the features that we need to replace the incandescent bulbs? Bright enough? Check. Lower energy? Check. Lasts a long time? Check. Sounds good! And that is the same thing that people will do with third party tools. They look at their set of problems, look at the set of features that the product offers, but they don’t look at everything else that the software, library, or tool brings along with it.

Consider a blog post that I read this morning by Jimmy Nilsson, titled “The Big Picture of Software Development”. In this blog post Jimmy takes a very introspective look at what we are, how we do what we do, and why we do it (go read it, it is a GREAT post). I’m in a particularly contemplative mindset currently, and this blog post really started me thinking. Particularly this part:

I find that quite a lot of people strongly believe that products will help a lot. After 20+ years in this business, I believe that products are more often the problem than the solution so care must be taken.

In the same way there is the comment about companies believing they need expert help from the vendor of the product they are having trouble with. To me, that’s a smell regarding that product.

“We can’t solve problems by using the same kind of thinking we used when we created them.”
–Albert Einstein

Care must absolutely be taken. If you are picking a tool that solves 5 problems, you need to make sure that you aren’t going to get 20 more problems. One of the biggest problems I see are tools that aren’t isolated. They are either pervasive in their reach, or require heavy integration at every level of your application. For example, a database should be a fairly isolated piece of your application. It has a single purpose and you interact with it in a very particular way. You can isolate the code to deal with the database into certain key areas, and then you have very little problems. But have you ever worked in an application where they threw database access code all over the place? It became a nightmare, didn’t it?

Unfortunately, many tools make it hard or impossible for you to isolate them from your code. And once they are in there, they are rarely going to come out. And that is a commitment. A commitment that you are going to have to deal with whether it is good or bad. Often, by the time you realize it was bad, it is far too late to reverse course. You have entangled your application in a web of dependencies, and there is no going back.

How Can I Change This?

But how do you deal with this? We obviously can’t expect to build every little piece of our application that we need. That would be ludicrous in this day and age. But there are a few key things we can do to mitigate these issues.

1. You an lean toward externalizing dependencies. Instead of pulling something into your application, can you just make a call out to it? Can you use a hosted service that you don’t need to worry about? There can be other issues here, but this should be a strong consideration.
2. Pick tools that perform targeted tasks, instead of tools that do a million different things. The more a tool does, and the more you use it for, the more your application will become shackled. This leads me to my next point…
3. Consider how much you can isolate the tool. Can you keep the library or tool in one small part of your application? If so, you are going to be in a much better place. Especially if the relationship starts to go south. This might take some research and forethought, but you can usually get a good idea of just how pervasive a tool is going to be.
4. Consider flexibility. If it would take twice as long to build a tool than it would to integrate with a third party tool, but you are concerned that the purchased tool might not meet all of your needs, then consider building it. If you integrate with the third party tool first, then figure out that it doesn’t meet your needs, you’ll likely have to spend time getting the tool out, then spend more time developer the replacement.
5. If you can’t externalize your dependencies, then lean toward libraries. Instead of buying a software package or tool, look for a library. They are generally smaller, more flexible, and easy to abstract away from your software. Libraries are usually written with the mindset of fitting them into your application, whereas with software packages and tools you will often find yourself shoving your application into them.
6. Look for open source. If you can find a library, tool, etc… (whether it be free or commercial) that fits your needs, make sure it is open source. You might laugh and say, well, I’ll never go into the code of a third party product and fix or change anything! Well, that may or may not be the case, but when you are using a popular open source tool, there are thousands of developers who are in the code. They have the same problems that you have, and they are solving them.

Good Dependencies

Just to avoid a million different revisions of “but what about the .NET framework?”, “what about Rails?”, “what about Django?”, “what about <insert required dependency here>?”. You have to take dependencies on something. Like I said, we can’t be expected to write every line in our applications these days. We are expected to be much more productive than that. Picking a good framework and libraries which will help us build our applications more effectively is key to success. Knowing when to leverage third party services, tools, and applications is more of an art than a science. You’ll start to get a better feel for it after you have been bitten a few times.

Final Thoughts

So, am I just saying to not heavily integrate with other systems? No. What I am saying is that if you need to heavily integrate with a third party library you need to have a list of Pros and Cons. There are Cons. Most people don’t look for them, but they are there. You need to think about them, and you need to be objective. I love NHibernate, but I also know that many .NET developers aren’t familiar with it and there is a learning curve. So that would be something that I would begrudgingly list in the Cons column.

I honestly believe that most of the pain that we feel in software development today is self inflicted. But until we start looking at our development practices and really ask ourselves why we are feeling this pain, things will never get any better. Like the Albert Einstein quote from Jimmy’s article said, “We can’t solve problems by using the same kind of thinking we used when we created them.”