Reality Driven Development – Why do good programmers produce bad software?

How come in a world of powerful languages, IDEs with intellisense, O/R mappers, unit tests, numerous libraries, copious amounts of third party tools, continuous integration, tons of code samples, thousands of developer forums, and more technical books than you can shake a stick at, is it so freakin hard to produce good software?

There is a blog post by Oliver Steele that has gotten some recent attention (despite the fact that it was written in 2004!) and it is called "The IDE Divide". In this post Oliver Steele talks about a divide in the programming world between what he calls the "language mavens" and the "tool mavens". He points out that people are often either big proponents of a particular high level language, or group of high level languages (an example he uses is Haskell), or they put tools above languages and focus more on how much more productive tools can make them. While I don't have any fundamental problems with his post (it does actually make several good points and draws several important conclusions), I think it does underscore a particular mindset that often occurs in software development where we see tools and languages not problems and solutions. If something goes wrong, then we need a tool to fix it! If this project failed, well if we had been using language X or tool Y then it would have been easier! We often want to put the implementation above the problem.

I think we do this for several reasons. (keep in mind that in this list the word "customer" is overloaded, a customer is anyone who is consuming your software)

1. It's the fun part! For most of us we started off hacking away at software in our younger years before we ever knew what specs or customers were. Because of this, and because the fact that most of us just want to write code, we often enjoy the architecture/design/development part of the software more than the requirements gathering. Go figure. This can be a good thing, if we didn't enjoy the construction part then all of our jobs would be hell, but at the same time, you need to figure out what you are going to build before you build it. As Alan Stevens put it in a recent tweet "Knowing what to build is the hardest part. Everyone focuses on the how to build it part because that's an easier discussion." And he is completely correct, discussing construction is more natural for us because that is what we like to do and that is what most of us were trained to do. How many blog posts have you seen on requirements gathering?
2. We think we already know what needs to be written. By the time that most programmers hear about a new piece of software that needs to be written they are already designing it in their head. Once we start thinking about it we start building up layer upon layer of preconceived notions about what we are going to be building. I can't tell you the number of times that a project has been put in front of me that needs a quote and I see the customers initial request and start thinking about what we would have to do in order to build such a system. Then when you start asking the customer questions about what needs to be built you realize that the system is nothing at all like you would have envisioned. There is either a barrier of language, lexicon, or experience between you and the customer.
3. A lot of methodologies at first glance support this behavior. A lot of people will take the parts of agile development that they want and throw the rest out. They will latch onto the parts about not having a big spec up front and writing tests first and think that this means that they should just jump right in and let the application evolve. Well, that is a great idea as long as you want crappy software. Even in most agile methodologies there is a strong emphasis on gathering requirements and defining things like user stories up front so that you *know* what you are building before you build it. You may not know exactly *how* you are going to build it, but you have a general idea of *what* you are building.
4. This is what a lot of us are used to. There are *a lot* of development shops out there that still writing software as if the developers are operating in a vacuum. A team of people will hammer out a detailed spec (assuming you have a spec) and then they will "throw it over the wall" and the development team will disappear to their cubes for 3 months and then out pops a grotesque conglomeration of specification and assumption. Then they get a long "you screwed this up list" and go back and start the same process to get the software to do what the customers actually wanted, the only problem is that they are using the same system that screwed it up in the first place. Look ma, no collaboration!
5. We aren't agile. We think that specs and requirements are something that you only get before you build software. How many times have you completed a task only to show a customer your proud accomplishment and you receive a list of things that are completely wrong with your design? You may have heard the term "impedance mismatch" when talking about ORM software, well this is impedance mismatch for requirements. Your customers requirements go in one end and then there needs to be a translation layer into your requirements. This translation is neither clean nor is it deterministic. Another point, you need to have customer feedback. At at a previous job I held I was given a project and the specs weren't very clear at all. I asked my boss to allow me to go talk with the people that were using the software and he told me that I didn't want to do that because it would just start down a road of constant changes and requests! There are two problems here, first you need to set customer expectations. They should not have the idea that every little tweak or suggestion they have is going to get implemented, but writing an application for someone without feedback is a lot like throwing javelin with a blindfold. Secondly, to assume that changes are bad is how you end up with bad software. Software needs to evolve in order to meet business requirements.
6. Your specifications are not very specific. You look at your specs, you see a lot of holes, and so you want to just fill them. Having a lot of gaps to a certain extent is a good thing! Your customer often doesn't know what they want any more than you do. The only way to get it right is to evolve both of your opinions in the process by delivering early and often. This way your customer can start to get a clearer picture of how they want the system to work while you start to see the evolution of the customer's decisions which will make it easier for you to predict their needs. To paraphrase Steve McConnell if your customers specifications spelled out the software 100% you'd be better off just executing the spec. Keep in mind also that good specs tell you what needs to happen (and may give a little guidance on how), bad specs tell you in detail how to do it.

You might be thinking right now "So, you are saying that I should put the spec first? We just don't have time to hammer out a big spec before (or during) our software development!" Well, I'm not telling you to make a big spec document. I think that big spec documents are something that hurts software development, not helps it. I'm not even necessarily saying that you should follow a formal agile methodology, what I am saying is that you *need* to take the time to figure out what you are going to write. You need to figure out what problem you are solving or what task you are trying to accomplish. You need to talk with your customer and figure out what they need from the system and what pain they are having that needs to be alleviated. Whether you put it on note cards, a white board, or in a word doc, you need to figure out what you are going to do.

So, why do good programmers who use all the tools at their disposal write bad software? Because we aren't writing the correct program and we aren't solving the right problem. We aren't going to our customer and getting feedback and we aren't keeping up with changing requirements. Most often this is because we don't know what our requirements are. Generally speaking, our customers are not experts in application design, and usually they don't understand how to give you requirements. The only process they have for giving you requirements can be to look at an application and tell you what is or is not correct. By acknowledging that the software you are writing is only partially correct, you will mentally prepare yourself for the inevitable changes that will come down the pipeline.

So, just remember that you can use BDD, TDD, DDD, MVC, ORM, (enough?) all you want, but if you are not writing what your customer wants and needs then you are just wasting time. So the next time someone tells you not to do something because it may result in change requests just tell them "we aren't writing software because we have a bunch of hard drive space that needs to be filled, we are writing software to solve problems and unless we are solving the right problems we might as well just stop writing code right now."

…stepping down from soapbox.