Code Simplicity

FOSSCoach 2008

2008-07-01T21:14:53Z

If you’re going to be in Portland, Oregon or attending OSCON and you want to hear me talk, I’ve proposed a session for FOSSCoach called Code Simplicity: Software Design In Open Source Projects.

Registration for FOSSCoach is free, but is limited to 100 people, so sign up if you want to come.

-Max

Comments: 0

The Never-Shipping Product

2008-05-31T07:42:25Z

When you work as a professional programmer, you almost always know somebody (or are somebody) who’s going through one of the most common development horror stories in the book:

“We started working on this project five years ago, and the technology we were using/making was modern then, but it’s obsolete now. Things keep getting more and more complex with this the obsolete technology, so it keeps getting less and less likely that we’ll ever finish the project. But if we re-write, we could be here for another five years!”

Another popular one is: “We can’t develop fast enough to keep up with modern user needs.” Or, “While we were developing, Google wrote a product better than ours much faster than us.”

When I hear things like that, the first thing I ask myself is “How did that happen? Why did it take so long for them to finish their product that they ran into that problem?”

The answer lies in complexity. The more complex a task is, the harder it is to complete it. So you start out with something simple that can be completed in one month. Then you add complexity, and the task will take three months. Then you take each piece of that and make it more complex, and the task will take nine months.

Complexity builds on complexity–it’s not just a linear thing. It’s not like, “We have ten features, and so adding one more will only add 10% more time.” No, one new feature will have to be coordinated with all ten of your existing features. So if just the feature itself takes 10 hours of coding time, there will be another hour of coding time for that feature interacting with each other feature. The more features there are, the higher the cost gets of adding a feature.

Some projects start out with such a complex set of requirements that they never get a first version out. If you’re in this situation, you should just trim features. Don’t shoot for the moon in your first release–get out something that works and make it work better over time.

There are other ways to add complexity than just adding features, too. The most common other ways are:

Expanding the purpose of the software. Generally, just don’t ever do that. Your marketing droids might be drooling over the idea of “making a single piece of software that does your taxes and cooks dinner”, but you should be screaming as loud as you can whenever any suggestion like that comes near your desk. Stick to your purpose–your software just has to do what it does well, and you will be succeed, if that purpose is something people need.
Adding programmers. Yes, that’s right–adding more people to the team adds complexity, it does not make things simpler. Remember The Mythical Man Month? What he says in there is true because of the complexity equation I explained higher up in this article–if you have ten programmers, adding an eleventh means spending time to groove in that one programmer, plus the time to groove in the existing ten programmers to the new guy, plus the time spent by the new guy interacting with the existing ten programmers.
Change things: Any time you change something, you’re adding complexity. Whether it’s a requirement, a design, or just a piece of code, you’re introducing the possibility of bugs, the time required to implement the change, the time required to validate that the new change works with all the other pieces of the software, the time required to decide upon the change, and the time required to track the change, and the time required to test the change. Each change builds on the last in terms of all this complexity, so the more you change, the more and more time each new change is going to take. It’s still important to make certain changes, but you should be making informed decisions about it, not just changing everything on a whim.
Lock-In to bad technologies. This is where you make a bad decision about your backend or libraries and then are stuck with it for a long time because you’re so dependent on it. Obviously “bad technology” is very subjective, but sometimes there are obvious good choices and obvious bad ones. For example, if you need an embedded scripting language, Lua is generally considered to be a good choice, and “write our own” would probably be one of the worse choices, depending on the situation. It’s definitely relative–it’s not like there’s only one good choice and all the rest are bad, but some choices might make your life easier than others.
Poor design or no design. Basically, this just means “a failure to plan for change.” Things are going to change, and that requires design work, to maintain simplicity while the project grows. Failing to do this can introduce massive complexity very fast, because suddenly each new feature quadruples the complexity of the code instead of just adding a little bit to the complexity.
Re-inventing the wheel. For example, if you invent your own protocol when a perfectly good one exists, you’re going to be spending a lot of time working on the protocol, when you could just be working on your software. You should basically never have any huge invented-in-house dependency, like a webserver, a protocol, or a major library, unless that is your product.

The thing about all of these is that they’re insidious. Most of them only do long-term damage–something you won’t see for a year or more. So when somebody proposes them, often they sound harmless! And even when you start implementing them, maybe they seem fine. But as time goes on–and particularly as more and more of these stack up–the complexity becomes more apparent and grows and grows and grows, until you’re another victim of that ever-so-common horror story: “The Never-Shipping Product.” (Which is nowhere near as cool as The Neverending Story, believe me.)

-Max

Comments: 0

Specific Solutions

2008-04-22T14:39:14Z

So, I’m a huge Kyle XY fan, and I was entertaining myself this morning by watching the various “behind the scenes” clips that they have on the website. Of course, before each clip was an ad–the same ad every time–for The Sims. No matter how silly the ad may be, being forced to watch it over and over did eventually get me thinking–although not about buying The Sims:

Why has The Sims sold 100 million copies, while Second Life, an ostensibly much more flexible and powerful universe, only has about 2 million active users? They look pretty similar, and you might guess at first glance that they’d have somewhat similar audiences. But, although 2 million users is nothing to scoff at, 100 million absolutely trounces it. So why the big difference?

Well, of course, The Sims has EA Games behind them, who have a massive distribution channel and a lot of marketing power, but the Internet buzz and general promotion of Second Life is pretty good too, so although EA has the edge, that doesn’t explain a 50-to-1 difference in sales. There must be something actually different about the products themselves.

Well, at first glance, The Sims is very user-friendly and Second Life is (from what I’ve heard) hard to use. The Sims does a limited scope of things very well, and Second Life does an unlimited number of things through a difficult interface, with mediocre results.

But fundamentally, why is it that products like The Sims succeed so much more than things like Second Life? And why does The Sims have a better interface, why do people want to play The Sims more than they want to play Second Life? Well:

It’s easy to make a system that does something specific, and hard to make a system that does everything.

When something is easy to make, you can spend a lot more time focusing on the little details–the polish. When something is hard to make, you spend all your efforts just making it work, and there’s no time left to sand off the rough edges. Specific solutions allow you to handle a problem with a level of grace, efficiency, and quality that could never be achieved by a generic, do-it-all solution.

A great example is the Colossus computer, built in 1943 to break encrypted German radio messages in World War II. That was all the Colossus did–it was fundamentally incapable of doing any other task. However, it would have taken a 5 MHz general-purpose computer to break that code at the same rate as the Colossus–a level of speed that desktop computers didn’t reach until the 1970’s, 30 years later.

This is something that I have been trying to get across to programmers for quite some time–you don’t need to solve all the world’s problems with one piece of code, you only need to solve the problem you’re solving. Whether you’re designing a whole system or just a tiny piece, your code doesn’t need to do any more than is called for by the known requirements. Sure, keep it extendable for the future–that’s part of making a quality solution. But all solutions should be specific solutions to known problems.

The Sims satisfies a specific, known need–the desire of people to play out specific types of fantasy lives. It succeeds because it’s not infinitely flexible. Second Life is a nightmare of complexity because it tries to be “everything to everyone”–a situation in which you almost always end up being not enough to anyone.

When people have problems with complexities or confusions in their software, I encourage them to be The Sims. Be Notepad. Be the Google main page. Don’t try to anticipate every need in the world, just solve the ones you know exist. Everything tends to fall into place when you really know your requirements and just go to solve them.

-Max

Comments: 3

Complexity and the Wrong Solution

2008-04-12T21:11:59Z

Often, if something is getting very complex, that means that there is an error somewhere far below the level that things are getting complex on.

For example, it’s very difficult to make a car move if it has square wheels. You’re going to be spending lots and lots of time figuring out how to make the car work, when really it should just have round wheels.

Any time there’s an “unsolvable complexity” in your program, it’s because there’s something fundamentally wrong with it. If the problem is “unsolvable” at one level, maybe you should back up and look at what’s underlying the problem. Maybe you put square wheels on the car, and now you’re trying to figure out how to make it go fast.

Programmers actually do this quite often. For example, “I have this terribly messy code, now it’s really complex to add a new feature!” Well, your fundamental problem there is the that code is messy. Clean it up, make the already-existing code simple, and suddenly adding the new feature will be simple.

What Problem Are You Trying To Solve?

If somebody comes up to you and says something like, “How do I make this pony fly to the moon?”, the question you need to ask is, “What problem are you trying to solve?” You’ll find out that they really need to collect gray rocks. Why they thought they had to fly to the moon, and use a pony to do it, only they know. People do get confused like this.

So when things get complex, back up and you look at the problem that you’re trying to solve. Take a really big step back. You are allowed to question everything. Maybe you thought that adding 2 and 2 was the only way to get 4, and you didn’t think about adding 1 and 3 instead, or just skipping the addition entirely and just putting “4” there. The “problem” is “How do I get 4?” Any method of solving that problem is acceptable, so figure out what the best method would be, for the situation that you’re in.

Discard your assumptions. Really look at the problem that you’re trying to solve, and think about the simplest way to solve that problem. Not “How do I solve this problem using my current code?” Not “How did Professor Bob solve this problem in his program?” No, just how, in general, in a perfect world, should that problem be solved? From there, you might see how your code needs to be re-worked. Then you can re-work your code. Then you can solve the problem.

-Max

Comments: 16

Truncated Posts in RSS?

2008-04-10T23:24:06Z

Hey everybody. So, some of my posts are rather long, and so I truncate them with a (Read More…) link that takes you to the full post. That link also shows up in the RSS, as a (more…) link.

For the frontpage of codesimplicity.com, I think that’s pretty useful–it makes it a lot easier to browse the various articles. But I was wondering, for those who read this in an aggregator or via RSS–for the long posts, do you prefer getting only the short version (with the “more…” link) or would you rather just have the whole post?

Update: It looks like, from the comments, that full text in the feed is the clear winner! That’s actually my preference too, when reading sites, but I usually am not reading posts as long as the ones I write. I’ve installed and activated the Full Text Feed plugin for WordPress, so in the future the feeds will contain the full text even though the front page and email notifications will still be cut by the “more” link.

-Max

Comments: 11

Instant Gratification = Instant Failure

2008-04-10T08:10:41Z

The broadest problem that I see in the software industry is that companies are unwilling to engage in strategies that only show results in the long term. Or, more specifically, that organizations are unaware that there is any such thing as a long-term strategy.

In the US, it’s probably a symptom of a general cultural problem–if an American can’t see an instant result from something, they think it doesn’t work. This leads to fast food, french fries, and fat people. The healthy way to eat (protein and vegetables) has a delayed effect on the body (you don’t get the energy for over an hour), and the bad way to eat (endless carbohydrates without nutritional value) has an instant result–immediate energy.

Software is always a long-term process. I wrote the first version of VCI in about three weeks, and that was insanely fast. Any actual application (VCI’s just a library) takes months or years of person-hours, even if you keep it small. So you’d think that organizations would be far-sighted about their development strategies, right?

Unfortunately, it just doesn’t happen. Competitor X comes out with “Shiny New Feature” and The Company says “we must have Shiny New Feature RIGHT NOW!” That’s not a long-term winning strategy, that’s just short-sighted panic. If you have users, they’re not all going to get up and go away in the next five minutes just because somebody else has one feature that you don’t. You should be looking at trends of how many users you’re gaining or losing, not just responding mindlessly to the immediate environment.

So what’s a good long-term strategy? Well, refactoring your code so that you will still be able to add features in the future, that’s a good one. Or spending some extra time putting some polish on your features and UI so that when the product is released, users are actually happy with it. Not adding features that you don’t want to maintain, if they’re not important enough–that’s another one.

Remember that Mozilla did poorly for years, only to finally start gaining dominance in a market that Netscape had lost, because they had a long-term plan. Granted, Mozilla made some decisions early on that caused some things to take longer than they should have, but they still won out in the long term, despite failing in the short term.

Of course, it can be hard to convince people that your long-term plan is right, sometimes, because it takes so long to show results! When I started refactoring Bugzilla about four years ago, there was pretty constant resistance, particularly when I would review patches and say, “You need to wait for the new architecture before this can go in,” or “This needs to be fixed to not be spaghetti code.” But once the refactoring really got rolling (after about two and a half years), it suddenly became way easier to add new features and nearly all the developers became big supporters of refactoring.

I read so much “advice” on “how to run your software business” that just focuses on instant gratification–what you can get done right now. “Add features!” “Get millions of dollars instantly from VCs!” Unfortunately, the way the universe seems to work is that you can destroy something in an instant, but it takes time to create something. So in reality, the closer you get to “instant gratification”, the closer you get to destruction of your product, your business, and your future.

If you want a good plan, pick one that admits that creation takes time. It doesn’t have to take forever, but it’s never instant.

-Max

Comments: 11

If It Ain’t Broken…

2008-04-04T23:58:36Z

Okay, so remember our third law? (You can’t break things if you don’t change them.) Well, that has a very important related rule, that every engineer on Earth knows, but sometimes forgets:

Never “fix” anything unless it’s a problem, and you have evidence showing that the problem really exists.

Of course, most of us know this as “If it ain’t broken, don’t fix it.” However, these wise words are frequently ignored, because many developers don’t have a good understanding of what “broken” means. Often, developers just imagine that users have a problem with something, and start fixing it. Or they go off and develop features that don’t solve anybody’s problem. This is far, far more common than you might think.

So that’s why I say you should have evidence that there’s a problem. Without that evidence, you could just be fixing things that aren’t actually problems, and if you go around fixing things that aren’t broken, you’re going to break things. And not only could you be generating bugs, but you’re wasting your time and adding complexity to your program for no reason. You need evidence that there’s a problem, before you start coming up with a solution.

What do I mean by “evidence”? Well, five users report that when they push the red button, your program explodes. Okay, that’s good enough for me! Or you push the red button, and you notice that the program explodes.

However, just because a user reports something doesn’t mean it’s a problem. Sometimes a user just didn’t realize that your program had some feature already, and so asked you to implement something else silly. For example, you write a program that sorts a list of words alphabetically, and a user asks you to add a feature that sorts a list of letters alphabetically. Your program already does that. (Actually, it already does more than that. This is often the case, with this sort of confused request.) So in this case, the user thought there was a problem, but there wasn’t even really a problem, even though he could maybe present “evidence” that he couldn’t sort a list of letters. (He just didn’t realize that he should use the word-sorting feature.)

Note that if you get a lot of requests like the above, it probably means that users can’t easily figure out how to find what they need. There’s some complexity that needs to be reduced on your end.

Finally, sometimes a user will report that there’s a bug, but actually that’s the program behaving exactly as you intended it to. In this case, it’s a matter of “majority rules.” If a significant number of users think that the behavior is a bug, it’s a bug. If only a tiny minority (like one or two) think it’s a bug, it’s not a bug.

The most famous error in this area is what we call “premature optimization”. That is, some developers seem to like to “make things go fast”. But they do it before they know that it’s slow! This is like a charity sending food to rich people. “We just wanted to help people!” Right—illogical, isn’t it? They’re solving a problem that doesn’t exist.

The only parts of your program where you should be concerned about speed are the exact parts that you can show cause a real performance problem for the users. The rest of the code, the primary concern is simplicity, not “make things go fast”.

There are infinite ways of violating this rule, but the way to follow it is so simple: just get real evidence that the problem is valid before you fix it.

-Max

Comments: 0

The Fourth Law of Software Design: Complexity vs. Ease of Maintenance

2008-03-11T03:19:03Z

Okay, so if we never change our software, we can entirely avoid defects. But change is inevitable! Particularly if we’re going to add new features. And after all, one of our goals was to make software easy to maintain, and to maintain software, it has to be changed here and there. In other words, we will be making changes. So “don’t change anything” can’t be the ultimate defect-reduction technique.

Well, like I said in the my design philosophy it helps to keep your changes small. But if you want to avoid even more defects, and eliminate them even from your small changes, there’s another law that can help you. And it doesn’t just reduce defects–it keeps things maintainable, makes it easy to add new features, improves the overall understandability of your code, and knowing it helps you make better software, all around. This Fourth Law of Software Design is:

The maintainability of a system is inversely proportional to the complexity of its individual pieces.

Where “maintainability” means “ease of maintenance.” Extreme unmaintainability would be the total inability to maintain some part or the whole of a piece of software. Perfect maintainability is impossible, but it’s the goal you strive for–total change or infinite new code with no difficulty.

This law is largely empirical, meaning that I figured it out by observation, not by logic. However, it does have a logical basis:

The simpler something is, the easier it is to understand. For example, a beach ball is very simple–a single large round object that you throw around–and is something that anybody can understand.
The more complex something is, the harder it is to understand. For example, a jet plane is very complex, and takes extensive training to use and understand. Complexity is not the only factor that makes things hard to understand, but with enough complexity, anything can become hard to understand.
The less you understand something, the harder it is to fix or modify it.
Thus: The more complex something becomes, the harder it is to modify (maintain) it.

However, you’ll notice that I didn’t say anything about the complexity of the whole system, in the law. I only mentioned its individual pieces. Why did I do that?

Well, an average-sized computer program is so complex that no human being could comprehend it all at once in their mind. It’s only possible to comprehend pieces of it. So we actually always have some large, complex structure for our whole program. What then becomes important is that the pieces can be understood when we look at them, and that we understand how the pieces relate to each other. The easier it is to understand the pieces, the more likely it is that any given person will understand them. That’s particularly important when you’re handing your code off to other people, or when you go away from your code for a few months and then have to come back and “re-learn” what you did, by reading your own code.

Let’s make an analogy, to demonstrate the principle. Imagine that you’re building a 30-foot tall steel structure. There are two ways to make it–you could make it out of a bunch of small girders, or you could try to forge three huge pieces of steel and put them together. With the girders approach, it’s easy to make or buy the individual pieces. The three huge pieces, on the other hand, have to be carefully custom-made and worked on extensively. With the girders, if one breaks you just replace it with an identical spare part. With the “huge pieces” approach, when one breaks you have to evacuate the structure, remove 1/3 of it, create a whole new custom piece, and then add that back in without collapsing the whole structure. The girders are simple, the huge pieces are complex.

So why do people sometimes write software with the “huge pieces” approach instead of the girders approach? It’s because there’s a perceived savings of time when you’re first creating the software, with the “huge pieces” method. With a bunch of small pieces, there is a lot of time spent putting them together. You don’t see that with the huge pieces–there’s three of them, they snap together, and that’s it. But the part that’s missed here is that it took way more time to create the three huge pieces than it did to create the girders. When you’re making a huge, complex single piece, any tiny error means that the whole thing has to be fixed or re-worked. And per observation in the practical world of programming, you will spend far more time fixing and re-working those huge pieces than you will putting together the small girders. So even though the time spent creating the “huge pieces” might seem like “productive, important time” and the time spent putting together the girders might seem like “busywork” or “wasted time,” the “girders” approach is actually more efficient.

I could go on and on about this, but you can find out about it for yourself. If you don’t believe me, spend a few years working on a software project where all the parts are very complex. I don’t recommend that you do that, but if you need any proof of this law for yourself, that would be a good (if painful) way to get it. Of course, you could also just apply the law and see if your software keeps on being maintainable–that’s a much less painful demonstration.

So how do we use this law, in the practical world of programming? Well, generally I recommend that people make the individual components of their code as simple as possible. Ideally this would start way down at the assembly language level, but you don’t always get simplicity there. Nor do you always get a simple programming language. But with what you have, strive for simplicity. Make everything as simple as possible. Don’t be afraid to be stupid, dumb simple. There is no limit to how simple you can make something, because if you go too far, your “simplicity” will start becoming complex. (In other words, you’ll be overengineering.) So just be as simple as you can possibly be, and if you overdo it (which almost never happens), it’ll be pretty obvious.

-Max

Comments: 3

The Third Law of Software Design

2008-03-07T20:04:22Z

So now we know that there is more future time than present time and that software will change as time goes on.

Our next law is, once again, axiomatic, and needs no derivation:

It is impossible to introduce new defects in your software if you do not change anything about it.

This is important–and categorized as a law–because defects violate our purpose of helping people. If something is a defect, by definition it is not helpful to people, and we need to avoid it.

This is also sometimes stated more informally as “You can’t introduce new bugs if you don’t add or modify code.” I’m not sure that “code” entirely covers “anything about it,” so I didn’t state it that way.

Of course, the reverse would be:

It is possible to introduce defects into your software if you change something about it.

Which leads to:

The more changes you make, the more likely you are to introduce a defect.

The funny thing is that this seems to be in conflict with the second law, and in fact it is. It’s the balancing act between the second and third law that requires your intelligence as a software designer.

Combining all three laws, we get:

The best design is the one that allows for the most change in the environment with the least change in the software.

And that, pretty simply, sums up my design philosophy.

However, it’s important to limit that somewhat. Although that may be the best code design, that rule doesn’t necessarily lead to the best user-facing design. An equivalent law for users would be something like, “If you never use the program it won’t break,” but I’m not sure that’s so useful. This third law is about preventing bugs, not about making things work nicely. You still want things to work nicely and do what people want–I’m just telling you here how to avoid bugs.

Another thing to know here is that, given our first two laws, it’s an error to write a system that “does everything we could ever possibly need,” but not make it flexible enough to cope with future change. That might seem like a good way to “avoid future changes in the software”, but really you’re just bringing all that change into the present, introducing the same number of bugs, and then not allowing any room to grow. And no program will do everything you could ever possibly need–there will always be future requirements that you cannot predict. This is covered more in Designing Too Far Into The Future.

On the other hand, you can overengineer to the point where your design is so flexible that creating and maintaining it is extremely difficult. That would be the point where you reach a level of flexibility that is not necessary to the real future (thinking about this in relation to the First Law).

However, overengineering is a much less common error than designing too far into the future. When in doubt, expect change, and plan your code in ways that will make change as simple and small as possible.

-Max

Comments: 7

The Second Law of Software Design

2008-03-03T10:12:46Z

Now that we know that the future is important, our second law answers the question, “What’s going to happen in the future?” To any programmer who’s worked for any amount of time, this law will be obviously true once you see it, but it’s still good to derive and prove it.

This law is derived from things that we know about the physical universe. From physics we know:

Nothing stays still.

That is, there is no matter or energy anywhere that isn’t moving. Even the atoms in your desk are vibrating furiously, back and forth. It is actually impossible to make them stand still.

So, from that we can then assume:

Anything that exists in the physical universe will change.

Now, that might not be so obvious in some respects. After all, couldn’t you just have something vibrate back and forth forever in one space? Well, let’s ignore that that vibration is “change”, because it’s not really the kind of change I’m talking about. You have to look at it this way: as soon as a vibrating object strikes another vibrating object, one of the vibrations will change. With enough objects in the universe, change then becomes unavoidable. There are enough objects in this universe (particularly the world we live in every day) that change is thus extremely common and likely. Also, people change things, above and beyond these physical laws.

So, you can guarantee this:

The world around your program is going to change.

It could be that you wrote a program for four-wheeled cars and now everybody drives 18-wheel trucks. It could be that you wrote a program for 10 year olds and then 10-year-old education got so bad that they couldn’t use it anymore. Really, it could be anything–the only thing you can guarantee is that something is going to change.

This leads us to our second law of software design:

Your software is going to change.

It’s not going to change by itself, but unless you just give up and decide you don’t want people to use your software anymore, you as a programmer are going to have to accommodate the demands of a changing environment. And that changing environment includes you and your users, too. One day you could wake up and decide that you don’t like how “feature A” works and re-write that piece of the program. Or you might suddenly get new users who don’t like how “feature B” works and have to change that. Really, the possible causes of change are limitless–the only thing you can guarantee is that something will change.

That leads us nicely into the next statement we can derive here, looking at our second law in the context of our first law:

The longer your program exists, the more probable it is that any piece of it will have to change.

That is, as you go into the infinite future, you start tending toward a 100% possibility of every single piece of your program changing. In the next five minutes, no part of your program will probably change. In the next 10 days, a small piece of it might. In the next 20 years, I’m betting that a majority of it (if not all of it) will change.

This is our primary concern in designing for the future–that we allow for and expect change, not that we design ourselves rigid structures that will “stand forever” but suddenly become useless when the environment around them changes.

When you design for the future, you are designing for change.

-Max

Comments: 10

The Goals of Software Design

2008-03-06T23:54:42Z

Now that we know what software design is and the purpose of software, the next step is to define the goals of this science of software design.

From the purpose of software, we know that when we write software, we’re trying to help people. So, one of the goals of a science of software design should be:

To allow us to write software that is as helpful as possible.

Secondly, we usually want people to keep on being helped by our software. So our second goal is:

To allow our software to continue to be as helpful as possible.

Now, that’s a great goal, but any software system of any size is extremely complex, so allowing it to continue being helpful is quite a task. Maintaining it over time can be quite a bit of work. Even just creating the system in the first place can be nightmarish if you don’t have some guidelines to follow, or haven’t already had experience doing it. So that leads us to our third goal:

To design systems that can be created and maintained as easily as possible by their programmers, so that they can be–and continue to be–as helpful as possible.

When software is easy to create, a programmer can spend more time focusing on being helpful to the user, and less time focusing on the details of programming. Similarly, the easier it is to maintain a piece of software, the easier it is for a programmer to have that software continue to be helpful.

This third goal is probably the one traditionally thought of as the goal of software design, even if it’s never stated explicitly. However, it’s very important to also have the first and second goal to guide us.

The phrase “as easily as possible” in the third goal is very important. The idea is to make things easy to create and maintain, not to make them difficult or complex. That doesn’t mean that everything will be immediately easy–sometimes it takes time to learn a new technology or design something well. But in the long run, your choices should lead to easier maintenance and creation for your software.

Sometimes the first goal (being helpful) and the third goal (easy maintenance) are a little bit in conflict–sometimes making your software helpful can make it harder to maintain. However, I believe that these two goals have been much more in conflict, historically, than they need to be. It is absolutely possible to create a totally maintainable system that is yet extremely helpful to its users. And in fact, if you don’t make it maintainable, it’s going to be quite difficult to meet our second goal of continuing to be helpful.

All this, of course, leads us quite nicely back into our Primary Law, which is where we’d be going next if this were a book. The only thing I’d add to the Primary Law article is that under most circumstances, there’s a lot more potential help to be done in the future than there is in the present. (If there’s no potential help to be done in the future for your software, then it doesn’t have much of a life expectancy anyway, so we don’t worry too much about that situation.) In other words (and as I’ll be talking about more, later), by focusing on the second goal (continuing to be helpful), it is often possible to achieve the first (being helpful). That doesn’t mean first goal is unimportant–I’m just giving you something to think about.

-Max

Comments: 0

The Purpose of Software

2008-02-26T20:20:31Z

Whenever you engage in some activity, it’s a good idea to have some idea of what the purpose of that activity is. What is the end goal, and why are you doing it? For example, when I sleep, the goal is to be rested. When I talk, the purpose is to communicate and be understood.

Similarly, when we write software, we should have some idea of why we’re doing it, and what the end goal is.

Now, is there some way that you could sum up what the purpose of all software is? If there was such a statement possible, it would give orientation to our whole science of software design, because we’d know what we were going for.

Well, I think I’ve managed to derive a single purpose that would fit all software:

To help people.

For example, Bugzilla exists “to help people track bugs.” Firefox exists “to help people browse the web.” Remember, you could browse the web entirely by reading HTML yourself–but Firefox (or your web browser of choice) sure does help.

Even if you wrote a piece of software to help animals or plants, it would still be “to help people help animals or plants.”

Software is never there “to help inanimate matter.” Software does not exist “to help the computer.” It always exists to help people. Even when you’re writing libraries, you’re writing “to help programmers”, who are people. You are never writing “to help the computer.”

Now, what does “help” mean? Well, that is somewhat a subjective thing, and somewhat not. Look it up in the dictionary. There are many things you could help with–organizing a schedule, writing a book, planning a diet, anything. What you help with is up to you, but the purpose is always to help.

People who cannot conceive of helping another person will write bad software–their software won’t help people. In fact, I might theorize (as a guess based on some personal observations) that your potential ability to write good software is limited only by your ability to conceive of helping another.

The purpose of software is not “to make money” or “to show off how intelligent I am.” Anybody writing with those as their only purposes is violating the purpose of software and is quite likely to get into trouble. Granted, both of those are a way of “helping” yourself, but that’s a pretty limited scope of help and, I would argue, is likely to lead to lower-quality software than software genuinely designed to help individuals do what they need or want to do.

So anyhow, there you have it. When we are making decisions about software, our guiding principle can be how we can help. It’s possible to help more or less, to help fewer or more people or things. With that as a yardstick, it’s possible to understand our Laws Of Software Design and how we should be making decisions about software.

-Max

Comments: 6

What Is Software Design?

2008-02-25T23:37:02Z

On my last blog, one of the commenters very correctly pointed out that I hadn’t actually told you what I meant by “software design.” And, in fact, looking around the web a bit, I’m finding that what I mean by “software design” isn’t fully covered by most current definitions.

For the sake of this definition, let’s say that the process of making software is composed of three parts: administrative decision-making, technical decision-making, and actual coding. Of course, there’s also testing, releasing–there’s lots of parts to software in the real world. I’m just making an artificial division here to help define one part of the three I mentioned.

By “administrative decision-making” I mean the sorts of decisions that would be made primarily by managers in a software organization. These are scheduling, cost estimates, what programmer to assign to what task, etc. There are a lot of theories and study in this area–I think that it’s actually fairly well-covered (even if not completely made scientific yet) by lots of people. That is not the area I’m talking about when I saw “software design.”

The other side of those three parts is “coding”. That’s where you sit down and actually write the program, typing strange words onto a screen in the hope that the computer will do something. This process is partially covered by the study of computer science, which gives us mathematical ways of modeling code and information. It’s also covered by the manual of whatever language we’re writing in. Coding is what you do after you’ve already made decisions–it’s just the actual process of telling the computer what to do, or figuring out how to make the computer perform the actions you want.

The last piece, in the middle, is what I’m loosely calling “technical decision-making.” Often, the process of technical decision-making and coding happen so fast that they are mentally indistinguishable, particularly if you are an experienced programmer. You just “know” what to do and type it in. However, the decision-making and the coding are actually separate processes. Technical decisions would be things like: “Do we go with a functional programming language or a procedural programming language?”, “Should we have unit tests?”, “How should we style our code?”, “Should we optimize for speed?”, and even-less-high-level decisions, until you get down to the actual coding. Basically, anything that happens in your mind, on a piece of paper, on the whiteboard, etc. before you start programming, that’s software design as I mean it. Anything that involves the overall design of the system or the technical decisions you make while creating the system would fall under this category.

It could just as easily be called “software creation”, but I think that would get too confused with coding and computer science. Similarly, I started out calling it the subject of “software”, but that has the same problems. “Software architecture” is too limiting, as it’s often thought of in terms of classes and objects, and might not include things like unit tests, code style, or other technical decisions you have to make in the process of creating software.

I suspect that in time I will come up with an even-more-precise definition, but this should at least give you some idea of what I mean, for now.

What I think we primarily lack in the field of software design is a series of fundamental truths on which we can base our technical decisions. Experienced software developers “know” what “the right thing to do” is, but why is that the right thing? What makes that “right?” Many Perl developers, for example, would claim that Perl is “the right way”, in direct conflict with the way other languages work. There are definite warring camps in this field–how can we figure out which way we should go?

Well, that’s one of my goals with this science (or subject) of software design–to help us be able to figure out where we should go in any given situation.

So, in my view, any science of software design would have to consist of:

An explanation of the purpose of software.
An explanation of the goals of the science.
A series of fundamental truths on which to base decisions.

And this would allow us to achieve:

The ability to make decisions that achieve the stated purpose of software.
Some way of understanding what causes errors (decisions that do not achieve the purpose) in software design.
A method (or methods) of preventing future errors.
A method (or methods) of fixing errors that already exist.

In my view, a science is only as useful as it can be applied. Physics is very useful because we can use it to build things, fix things, design new things, etc. I would like the science of software design to be directly applicable to the practical process of writing software, and the kinds of decisions that real programmers have to make every day. I can’t promise that the study will be perfect, but I can promise you that it will be useful.

-Max

Comments: 3

The Primary Law of Software Design

2008-02-21T20:51:31Z

Ideally, any science should have, as its base, a series of unbreakable laws from which others are derived. What is a law? Well, in the field of science, it’s something that:

Is universally true, without exception.
Predicts phenomena that, when looked for, will be found to exist in the real world.

Some of the best laws are axiomatic, a big word meaning “obviously true.” For example, “Yesterday happened before today” is an axiomatic statement–the definition of the word “yesterday” makes that obviously true.

For the science of software design, we are lucky to have an axiomatic basic law which is senior to all others:

There is more future time than there is present time.

This is obviously true. “Now” is an infinitely small moment that quickly becomes another “now.” The future is infinite.

So, since the future is infinitely large and the present is infinitely small, we can derive another obvious statement:

The future is more important than the present.

Now, when we’re talking about an infinitely small present and an infinite future, what I’ve said there is pretty obvious. In the real world, though, we have to ask the question: how much future are we talking about? What’s more important, the next five minutes or the next ten years?

Well, in order to answer that question, we have to make one assumption: that you want your program to continue to exist and be used in the future. If you only want it to exist for the next five minutes, then those are the most important. If you want it to continue to be around for 10 years, then those 10 years are the most important.

So all together, this tells us how good our software design needs to be–it needs to be exactly as good as there is future time in which our software must exist.

If you’re writing a program that’s only going to be used once, you don’t have to worry about its design. If you’re writing a program that’s going to be used and modified by astronauts on a 100-year voyage to Alpha Centauri, you have to be really good.

So, this science of software design is a thing where you have choices–if you follow and understand all of its laws, you will have a program that survives very well into the future. If you follow none of them, your program won’t continue to exist very long, or at the very least, it will become more and more difficult to ensure its continued existence.

Now, keep in mind that sometimes in life, there are situations where what you do for the next five minutes determines whether or not you live or die. Similarly, in the real world of software, there can be situations where what your organization does right now determines whether you go bankrupt or not. These are totally valid decisions according to this law, because if your software falls entirely out of existence right now, then the next ten years don’t matter. There is no future existence for something that no longer exists. (Another axiomatic statement!)

However, such situations are generally the exception, not the rule. If you found yourself constantly in a situation where the next five minutes determined your life or death, wouldn’t you want to get out of that? Similarly, if you find yourself in an organization that always insists that the next five minutes are more important than the next ten years, perhaps you should consider leaving.

Now, don’t pass any of this off as unimportant just because it’s “obvious.” What matters isn’t the statement itself, but the importance placed upon the statement. This is the senior law of software design, from which all other aspects of good design flow without exception. If you know of any exceptions, I’d be happy to hear them so that I could refine the science using a higher primary law. But I’m not aware of any exceptions.

When thinking about this law, I usually apply it from the viewpoint of a programmer, not as a user. Sure, a program that is written now and can still be used in 20 years would be fantastically designed from a user’s perspective. But what I’m concerned about mostly is whether or not a software developer will be able to fix or modify this program 20 years from now. That’s an entirely realistic concern–there are programs that have been around for 20 years or more.

There are also other important things to think about in relation to this law–what you can and can’t know about the future. But that’s a subject for another blog.

-Max

Comments: 19

There Is No Science Of Software

2008-04-09T22:23:18Z

What we think of today as being “computers” started out in the minds of mathematicians as purely abstract devices–thoughts about how to solve math problems using machines instead of the mind.

These mathematicians are the people we would consider the modern founders of “computer science.” Computer Science is actually the mathematical study of information processing. It is not, as some people believe it to be, the study of computer programming. In fact, there is no science of computer programming. To understand how that could possibly be true, and what I mean, you have to know the history of programming.

The earliest computers were built under the supervision of computer scientists by highly skilled electronic engineers. They were run by highly-trained operators in tightly-controlled environments. They were all custom-built by the organizations that needed them (mostly governments, to aim missiles and crack codes), and there were only one or two copies of any given model.

Then, along comes UNIVAC and the whole notion of “commercial” computers. Now, there’s only so many advanced theoretical mathematicians in the world. If you start shipping out computers to everybody, you can’t ship a mathematician along with each one. So although some organizations, such as the United States Census Bureau, almost certainly had some highly-trained operators for their machinery, other organizations undoubtedly got their machine and said, “Okay, Bill from Accounting, this is yours! Read the manual and have at it!” And there went Bill, diving into this complex machine and doing his best to make it work.

Bill there is our first “working programmer.” He might have studied math in school, but he almost certainly didn’t study the sort of advanced theory needed to conceive and design the machine itself. But he can read the manual and understand it, and by trial and error, make the machine do what he wants.

Of course, the more commercial computers you ship, the more Bills you have and the fewer highly-trained operators you have. And if there’s one thing that Bill has, it’s job pressure. He has demands from management to “Get that task done now!” and “We don’t care how it’s done, just do it!” He figures out how to make the thing work according to its manual, and it works, even if it crashes every two hours.

Eventually Bill gets a whole team of programmers to work with him. He has to figure out how to design a system and split up the tasks between different people. Instead of being studied and mapped out like a standard science, the whole art of practical programming grows organically, more like college students teaching themselves to cook than like NASA engineers building a space shuttle.

So there’s this hodge-podge system for software development, and it’s all very complex and hard to manage, but everybody gets along somehow. Then along comes The Mythical Man Month, a book by a guy who actually looked at the process of software development and pointed out some things about it–most famously that adding more programmers to a project doesn’t necessarily make it faster. He didn’t come up with a whole science, but he did make some good observations about programming and managing software development.

Of course, then came a flurry of software development methods: the Rational Unified Process, the Capability Maturity Model, Agile Software Development, and others. None of these claim to be a science, just a way of managing the complexity of software development.

And that, basically, brings us up to where we are today: lots of “methods,” but no real science.

A science is composed of observations, experiments, and laws. Although we have hundreds of books of observations about practical programming, and “the world is our laboratory”, all of that has resulted in very few laws, which are the most important part of a science. That is, instead of coming up with methods to work around complexity, there ought to be some fundamental rules to follow that would lead to simplicity–ways to avoid complexity entirely and explain why what “works” in software development does work.

In reality, there are two missing sciences here. The first one is being worked on actively, and includes the various methods I mentioned above. That’s the science of managing software development. The fact that conflicting, equally-valid “opinions” seem to exist within the field indicates that the fundamental laws of software management have not been worked out. However, there is at least attention being given to the problem.

The other science, though, gets very little attention in the practical world of programming: the science of writing software. Very few people are taught that there is a science to writing software, in school. Instead, they are just shown “This is how it works in this programming language, now go write some software!”

The science I’m talking about is not Computer Science. That’s a mathematical study. I’m talking about a science for the “working programmer”–something that gives fundamental laws and rules to follow when writing a program in any language. One that’s as reliable as physics or chemistry in telling you how to create an application.

Some people might say that such a science is not possible, that software development is too variable to ever be described by simple, fundamental laws. Some people also once said that understanding the physical universe was impossible because “it is the creation of God and God is unknowable.” So unless you’re telling me computers are unknowable, I think that making such a science would be entirely possible.

Actually, I’ve started to collect some basic hypotheses for such a science, and I’m writing a book about it. I might post some of them here, but actually, all of what I’ve written in the blog so far actually comes from those hypotheses. That is, I haven’t really posted any of my hypothetical “laws” yet, but I have been posting data that I’ve derived from them.

Anyhow, I think that the primary source of complexity in software is actually the fact that this science is lacking. If programmers actually had a science for simplicity in software, there wouldn’t be nearly so much complexity, and we wouldn’t need crazy processes to manage that complexity.

Whenever there’s a problem with something, my instinct is to go philosophically above the level of the problem and look at it from there. If there’s some “unsolvable” effect happening to something, there must be some unknown cause on a higher level. So there you go–if the problem is complexity, then maybe it’s because there was no science of simplicity in the first place.

-Max

Comments: 10

Simplicity and Strictness

2008-02-13T21:19:27Z

As a general rule, the stricter your application is, the simpler it is to write.

For example, imagine a program that accepts only the numbers 1 and 2 as input and rejects everything else. Even a tiny variation in the input, like adding a space before or after “1″ would cause the program to throw an error. That would be very “strict” and extremely simple to write. All you’d have to do is check, “Did they enter exactly 1 or exactly 2? If not, throw an error.”

In most situations, though, such a program would be so strict as to be impractical. If the user doesn’t know the exact format you expect your input in, or if they accidentally hit the spacebar or some other key when entering a number, the program will frustrate the user by not “doing what they mean.”

That’s a case where there is a trade-off between simplicity (strictness) and usability. Not all cases of strictness have that trade-off, but many do. If I allow the user to type in 1, One, or " 1" as input, that allows for a lot more user mistakes and makes life easier for them, but also adds code and complexity to my program. Less-strict programs often take more code than strict ones, which is really directly where the complexity comes from.

(By the way, if you’re writing frameworks or languages for programmers, one of the best things you can do is make this type of “non-strictness” as simple as possible, to eliminate the trade-off between usability and complexity, and let them have the best of both worlds.)

Of course, on the other side of things, if I allowed the user to type in O1n1e1 and still have that be accepted as “1″, that would just add needless complexity to my code. We have to be more strict than that.

Strictness is mostly about what input you allow, like the examples above. I suppose in some applications (like, say, a SOAP library), you could have output strictness, too–output that always conforms to a particular, exact standard. But usually, it’s about what input you accept and what input causes an error.

Probably the best-known strictness disaster is HTML. It wasn’t designed to be very strict in the beginning, and as it grew over the years, processing it became a nightmare for the designers of web browsers. Of course, it was eventually standardized, but by that time most of the HTML out there was pretty horrific, and still is. And because it wasn’t strict from the beginning, now nobody can break backwards compatibility and make it strict.

Some people argue that HTML is commonly used because it’s not strict. That the non-strictness of its design makes it popular. That if web browsers had always just thrown an error instead of accepting invalid HTML, somehow people would not have used HTML.

That is a patently ridiculous argument. Imagine a restaurant where the waiter could never say, “Oh, we don’t have that.” So I ask for a “fresh chicken salad”, and I get a live chicken, because that’s “the closest they have.” I would get pretty frustrated with that restaurant. Similarly, if I tell the web browser to do something, and instead of throwing an error it tries to guess what I meant, I get frustrated with the web browser. It can be pretty hard to figure out why my page “doesn’t look right”, now.

So why didn’t the browser just tell me I’d done something wrong, and make life easy for me? Well, because HTML is so un-strict that it’s impossible for the web browser to know that I have done something wrong! It just goes ahead and drops a live chicken on my table without any lettuce.

Granted, I know that at this point that you can’t make HTML strict without “breaking the web.” My point is that we got into that situation because HTML wasn’t strict from the beginning. I’m not saying that it should suddenly become strict now, when it would be almost impossible. (Though there’s nothing wrong with slowly taking incremental steps in that direction.)

In general, I am strongly of the opinion that computers should never “guess” or “try to do their best” with input. That leads to a nightmare of complexity that can easily spiral out of control. The only good guessing is in things like Google’s spelling suggestions–where it gives you the option of doing something, but doesn’t just go ahead and do something for you based on that guess. This is an important part of what I mean by strictness–input is either right or wrong, it’s never a “maybe.” If one input could possibly have two meanings, then you should either present the user with a choice or throw an error.

I could go on about this all day–the world of computers is full of things that should have been strict from the beginning, and became ridiculously complex because they weren’t.

Now, some applications are forced to be non-strict. For example, anything that takes voice commands has to be pretty un-strict about how people talk, or it just won’t work at all. But those sorts of applications are the exception. Keyboards are very accurate input devices, mouses slightly less so but still pretty good. You can require input from those to be in a certain format, as long as you aren’t making life too difficult for the user.

Of course, it’s still important to strive for usability–after all, computers are here to help humans do things. But you don’t necessarily have to accept every input under the sun just to be usable. All that does is get you into a maze of complexity, and good luck finding your way out–they never strictly standardized on any way to write maps for the maze.

-Max

Comments: 13

When Is Backwards-Compatibility Not Worth It?

2008-02-12T02:35:36Z

This title might seem a bit like a contradiction to my last post! Well, you really shouldn’t break your API, if you can help it. But sometimes, maintaining backwards compatibility for any area of your application can lead to a point of diminishing returns. This applies to everything about a program, not just its API.

A great example of the backwards-compatibility problem is Perl. If you read the perl5-porters summaries with any regularity, or if you’re familiar with the history of the Perl internals in general, you’ll have some idea of what I mean.

Perl is full of support for strange syntaxes that really, nobody should be using anymore. For example, in Perl, you’re supposed to call methods on an object like $object->method(). But there’s also a syntax called the “indirect object syntax” where you can do method $object. Not method($object)–only the case without the parenthesis is the indirect object syntax.

Really, nobody should be using that syntax, and it’s not that hard to fix applications to call their methods the right way. And yet that syntax is maintained and supported in the Perl binary to keep backwards compatibility.

Perl is full of things like this that block forward progress because of historical problems.

Now obviously, this is a balancing act. When there are a huge number of people using something, and it would be really difficult for them to change, you pretty much have to maintain backwards compatibility. But if maintaining that backwards-compatibility is really stopping forward progress, you need to warn people that the “old cruft” is going away and ditch it. The alternative is infinite backwards-compatibility and no forward progress, which means total death for your product.

This also gives one good reason why you shouldn’t just add features willy-nilly to your program. One day you might have to support backwards-compatibility for that feature that you added “because it was easy to add even though it’s not very useful.” This is an important thing to think about when adding new features–are you going to have to support that feature forever, now that it’s in your system? The answer is: you probably are.

If you’ve never maintained a large system that’s used by lots of people, you might not have a good idea of: (a) How many people can be screwed if you break backwards-compatibility, and (b) How much you can screw yourself over by having to maintain backwards-compatibility. The ideal solution there is: just don’t add features if you don’t want to support them for many, many future versions to come. Sometimes it takes a lot of programming experience to make that sort of decision effectively, but you can also just look at the feature and think, “Is it really so useful that I want to spend at least 10 hours on it in the next three or four years?” That’s a good estimate of how much effort you’re going to put into backwards-compatibility, QA, and everything else for even the smallest feature.

Once you’ve got a feature, then maintaining backwards-compatibility is generally the thing to do. Modern Bugzilla can still upgrade from 2.8, released in 1999. But it can do that because we wrote the upgrader in such a way that old upgrader code doesn’t require any maintenance–that is, we get that backwards-compatibility for free. We only have to add new code as time goes on for new versions of Bugzilla, we almost never have to change old code. Free backwards-compatibility like that should always be maintained.

The only time you should seriously consider ditching backwards-compatibility is when keeping it is preventing you from adding obviously useful and important new features. But when that’s the case, you’ve really got to ditch it.

-Max

Comments: 4

Ways To Create Complexity: Break Your API

2008-02-07T08:13:12Z

An API is a sort of a promise–”You can always interact with our program this way, safely and exactly like we said.” When you release a new version of your product that doesn’t support the API from your old version, you’re breaking that promise.

Above and beyond any vague philosophical or moral considerations about this, the technical problem here is that this creates complexity.

Where once users of your API only had to call a simple function, now they have to do a version check against your application and call one of two different functions depending on the result. They might have to pass their parameters a totally different way now, doubling the complexity of their code if they keep both the old way and the new way around. If you changed a lot of functions, they might even have to re-work their whole application just to fit with the way your new API works!

If you break your API several times, their code will just get more and more and more complicated. Their only other choice is to break their compatibility with old versions of your product. That can make life extremely difficult for users and system administrators trying to keep everything in sync. You can imagine how quickly this could spiral out of control if every piece of software on your system suddenly broke its API for interacting with every other piece of software.

For you, maintaining an old API can be painful, and getting rid of it can make life so much simpler. But it’s not complexity for you that we’re talking about particularly here, it’s complexity for other programmers.

The best way to avoid this problem altogether is don’t release bad APIs. Or, even better (from the user’s perspective), create some system where you promise to always maintain the old APIs, but give access to more modern APIs in a different way. For example, you can always access old versions of the salesforce.com API merely by using a different URL to interact with the API. Every time you interact with the SalesForce API, you are in fact specifying exactly what version of the API you expect to be using. This approach is a lot easier in centralized applications (like salesforce.com) than in shipping applications (like Bugzilla), because shipping applications have to care about code size and other things. Maintaining old APIs is also very difficult if you only have a small team of developers, because that maintenance really takes a lot of time and attention.

In any case, releasing an unstable or poor API is going to either complicate your life (because you’ll then have to maintain backwards compatibility forever) or the life of your API users (because they’ll have to modify all of their applications to work with both the “good” and “bad” API versions).

If you choose to break your API and not provide backwards-compatibility, remember that some API users will never update their products to use your new API. Maybe they just don’t have the time or resources to update their code. Maybe they are using a tool that interacts with your product, but the maintainer of the tool no longer provides updates. In any case, if the cost of fixing their code is greater than the value of upgrading to new versions of your product, they could choose to remain with an old version of your product forever.

That can have a lot of unforseen consequences, too. First they keep around an old version of your product. Then they have to keep around old versions of certain system libraries so that your product keeps working. Then they can’t upgrade their OS because the old version of your product doesn’t work on the new OS. Then what do they do if some unpatched security flaw is exploited on their old OS, but they’re still tied to your old product and so can’t upgrade their OS? Or some security flaw in your old product is exploited? All of these situations are things that you have to take responsibility for when you choose to break your API.

And yet, having no API can lead to the same situation. People create crazy “hacks” to interact with your system, and then they can’t upgrade because their hacks don’t work on the new version. This is not as bad as breaking your API, because you never promised anything about the hacks. Nobody has the right to expect their hacks to keep working. But still, if management orders them to integrate with your product, those clever programmers will find any possible way to make it work, even if it sticks them with one version of your product forever.

So definitely make an API if you have the development resources to do it. But put a lot of careful thought into your API design before implementing it. Try actually using it yourself. Survey your users carefully and find out exactly how they want to use your API. Generally, do everything in your power to make the API as stable as possible before it’s ever released. It’s not a matter of spending years and years on it, it’s just a matter of taking some sensible steps to find out how the API should really work before it’s ever released.

And once it’s released, please, please, if you can help it, don’t break the API.

-Max

Comments: 5

What Is Overengineering?

2008-01-30T07:57:38Z

Software developers throw around this word, “overengineering,” quite a bit. “That code was overengineered.” “This is an overengineered solution.” Strangely enough, though, it’s hard to find an actual definition for the word online! People are always giving examples of overengineered code, but rarely do they say what the word actually means.

The dictionary just defines it as a combination of “over” (meaning “too much”) and “engineer” (meaning “design and build”). So per the dictionary, it would mean that you designed or built too much.

Wait, designed or built too much? What’s “too much”? And isn’t design a good thing?

Well, yeah, most projects could use more design. They suffer from underengineering. But once in a while, somebody really gets into it and just designs too much. Basically, this is like when somebody builds an orbital laser to destroy an anthill. An orbital laser is really cool, but it (a) costs too much (b) takes too much time and (c) is a maintenance nightmare. I mean, somebody’s going to have to go up there and fix it when it breaks.

The tricky part is–how do you know when you’re overengineering? What’s the line between good design and too much design?

Well, my criteria is this: When your design actually makes things more complex instead of simplifying things, you’re overengineering. An orbital laser would hugely complicate the life of somebody who just needed to destroy some anthills, whereas some simple ant poison would greatly simplify their life by removing their ant problem (if it worked).

This isn’t to say that all complexity is caused by overengineering. In fact, most complexity is caused by underengineering. If you have to choose, the safer side to be on is overengineering. But that’s kind of like saying it’s safer to be facing away from an atom bomb blast than toward it. It’s true (because it protects your eyes more), but really, either way, it’s going to suck pretty bad.

The best way to avoid overengineering is just don’t design too far into the future. Overengineering tends to happen like this: “Okay, I need some code to reverse a string. Well, might as well make a whole sytem for rearranging and modifying the letters in a string, since we might need that some day.” Essentially, somebody imagined a requirement that they had no idea whether or not was actually needed. They designed too far into the future, without actually knowing the future.

Now, if that developer really did know he’d need such a system in the future, it would be a mistake to design the code in such a way that the system couldn’t be added later. It doesn’t need to be there now, but you’d be underengineering if you made it impossible to add it later.

With overengineering, the big problem is that it makes it difficult for people to understand your code. There’s some piece built into the system that doesn’t really need to be there, and the person reading the code can’t figure out why it’s there, or even how the whole system works (since it’s now so complicated). It also has all the other flaws that designing too far into the future has, such as locking you into a particular design before you can actually be certain it’s the right one.

There are lots of common ways to overengineer. Probably the most common ways are: making something extensible that won’t ever need to be extended, and making something way more generic than it needs to be.

A good example of the first (making something extensible that doesn’t need to be) would be making a web server that could support an unlimited number of other protocols in addition to HTTP. That’s kind of silly, because if you’re a web server, then you’re sending HTTP. You’re not an “every possible protocol” server.

However, in that same situation, underengineering would be not allowing for any future extension of the HTTP standard. That’s something that does need to be extensible, because it really might change.

The issue here is “How likely it is that this thing’s going to change?” If you can be 99.999% certain that some part of your system is never going to change (for example, the letters available in the English language probably won’t be changing much–that’s a fairly good certainty) you don’t need to make that part of the system very extensible. (Even so, it’s still good to leave a tiny little room to expand, in the very rare chance that somebody adds something like, say, the Euro symbol to the language.)

There are just some things you have to assume won’t change (like, “We will never be serving any other protocol than HTTP”)–otherwise your system just gets too complex, trying to take into account every possible unknown future change, when there probably won’t even be any future differences. This is the exception, rather than the rule (you should assume most things will change), but you have to have a few stable, unchanging things to build your system around.

The second way (making something too generic) goes like this: Imagine that the Bugzilla Project suddenly went insane, and instead of saying that Bugzilla was a “bug tracking system”, we decided to make it into a “generic system for managing data in a database through forms.” It would become terribly complex, and it would also stop being a very good bug tracker, because it would be trying to be “everything to everyone” instead of just focusing on adding good bug-tracking features. That would definitely be overengineering–we’re just trying to track bugs, but suddenly we’re a generic form system? Yep, sounds like “orbital lasers” to me.

In addition to being too generic on the whole-program level, individual components of the program can also be too generic. A function that processes strings doesn’t also have to process integers and arrays, if you’re never going to be getting arrays and integers as input.

You don’t have to overengineer in a huge way, either, to mess up your system. Little by little, tiny bits of overengineering can stack up into one huge complex mass.

Good design is design that leads to simplicity in implementation and maintenance, and makes it easy to understand the code. Overengineered design is design that leads to difficulty in implementation, makes maintenance a nightmare, and turns otherwise simple code into a twisty maze of complexity. It’s not nearly as common as underengineering, but it’s still important to watch out for.

-Max

Comments: 1

How Simple Do You Have To Be?

2008-01-26T09:26:06Z

Sometimes, when you’re working on a project, there’s a question of, “How simple do we really have to be?” “How much do we have to simplify this thing?” “Is it simple enough?”

Well, of course, simplicity is relative. But even so, you can still be more or less simple. From the relative viewpoint of your user, your product can be hard to use, easy to use, or somewhere in between.

So, how simple do you have to be?

Honestly?

If you really want to succeed?

Stupid, Dumb Simple.

Apple knows this, one hundred percent. They make products that are so simple they’re practically idiotic. They aren’t the most technically advanced products (there are lots of phones that beat the pants off the iPhone in terms of features, for example), but they’re all usable by total morons. Big bright buttons, no clutter, straightforward interfaces, big text that explains everything–it’s like an elementary school textbook, but prettier and more expensive.

That’s the level of simplicity I’m talking about–the kind where you’d say, “Hey, my whole family could use this thing!”

But often, people really just don’t understand how stupid, dumb simple they have to be, to get to that level. For example:

When I’m at the mall, there are maps that tell me where everything is. On these maps, I want a huge red dot, with the words “YOU ARE HERE” in gigantic letters right there. Usually, though, I get a tiny red (or green, or yellow) dot (or triangle, or square) in the middle of the map that I really have to search for to see it. And then, off to the side there’s some text that explains, “The tiny green triangle means ‘You are here!’” Add this up to the general confusion of trying to find anything on these maps, and I could be spending five or six minutes just standing in front of this thing, trying to figure out how to get where I’m going.

To the guy that designed the map, this is all totally reasonable. He spent lots of time designing this map, so it was important enough to him that he would spend several minutes on it, learn all about it, figure it out, etc. But to me, the map is a very, very minor part of my existence. I just want it to be as simple as humanly possible, so that I can use it quickly and get on with my life!

Many programmers are particularly bad about this with their code. They assume that other programmers will spend a lot of time learning all about their code, because after all, it took a lot of time to write it! The code is that important to them, so isn’t it that important to everybody?

Now, the programmers I know are generally an intelligent bunch. There’s always a few exceptions, but mostly they’re very bright. But it’s still always a mistake to think, “Oh, other programmers will understand everything I’ve done here without any simplification or explanation of my code.” It’s not a matter of intelligence–it’s a matter of knowledge. That programmer who is new to your code doesn’t know anything about it. He has to learn! The easier you make it for him to learn, the faster he’s going to figure it out, and the easier it will be for him to use it.

There’s lots of ways to make your code easy to use–simple documentation, simple design, step-by-step tutorials, etc.

But, if your code isn’t stupid, dumb simple to use, people are going to have trouble with it. They’re going to use it wrongly, create bugs, and generally muck things up. And when all this happens, who are they going to come ask about it? Yes, you! You are going to be spending time answering all their questions…. (Mmmmmmm, sounds fun, doesn’t it?)

I know that none of us like being talked down to, or treated like we’re idiots. And sometimes that leads to creating things that are a little complicated, so that we feel like we aren’t “talking down to” to the user or the other programmer. We throw in some big words, make it a little less than simple, and people respect our intelligence but feel kind of stupid because they don’t get it. They might think we’re way smarter than they could ever be, and that is kind of flattering. But really, is that helping them?

On the other hand, when you make things stupidly simple, you’re allowing the user to understand your product. That makes them feel smart, lets them use the program, and doesn’t reflect badly on you at all. In fact, your users will probably admire you more if you make things simple than if you make them complex.

So when this question comes up of “How simple do we have to be,” you might as well ask yourself, “Do we want people to understand this and be happy, or do we want them to hate us and be frustrated?” And if you pick the former, then there’s only one level of simplicity that will assure your success:

Stupid, Dumb Simple.

-Max

Comments: 2

Complexity Is a Prison

2008-01-23T06:44:10Z

Sometimes, I think, people are worried that if they make their code too simple, then either:

Somehow they’re not demonstrating how intelligent they are, or how valuable they are, to their managers, or
The project will become so simple to work on that anybody can just steal their job!

It’s almost as though if they actually did their job right, then they’d lose it. Now, stated that way, that’s obviously a nonsensical viewpoint. But, if you’ve ever worried about it, here’s something to think about:

What if your code is so complex that you’ll never be able to leave your job?

What if you made something so complicated that nobody else could understand it? Well, then you personally would be tied to that project forever and ever. If you wanted to work on some other project at your organization, your managers would protest, “But who else will maintain this code?” Whoever was assigned after you to work on your code would constantly be walking into your new office, saying, “Hey, how does this part work?”

Maybe you have no conscience, and you’ll just leave the code to some hopeless replacement and ditch the company. However, I’m guessing that most people would feel tied to a project if they were sure that nobody else could ever take it over successfully. And really, even if you just take off and leave, somebody’s going to be calling you up and saying, “Um, hey, you know that one piece of code where…” You’ll get emails from “the new guy”: “Hey, I hear you wrote this code, and I have this problem…” If you can’t make somebody else understand your code and have them truly take it over, then you’re going to be stuck with a piece of that job forever.

In the Bugzilla Project, I am doing the best I can to work myself out of a job. I love working on Bugzilla, but I don’t want to be tied to it every moment of my life. I want to go on vacation sometimes. I want to write music! I want to be able to leave my computer for a month, and not have the whole world collapse. So I work to make Bugzilla simple enough and well-designed enough that somebody else can take over the parts I work on, some day. Maybe then I can go on and work on other things in Bugzilla, or some other programming project that I have, or maybe I could go make an album! Who knows! I just don’t want to be imprisoned by my own code.

If job security is so important to you that you’d tie yourself to a single job forever just to get it, then maybe you should re-evaluate your priorities in life! Otherwise, when you’re making decisions about your project, one thing to remember is this: complexity is a prison; simplicity is freedom.

-Max

Comments: 2