Software is, fundamentally, a solid object that is made of knowledge. It follows all the rules and laws of knowledge. It behaves exactly as knowledge behaves in just about any given situation, except that it’s in concrete form. For example, when software is complex it tends to be mis-used. When software is wrong (i.e., has a bug), it tends to cause harm or problems. When people don’t understand some code, they tend to alter it incorrectly. One could say these things of knowledge just as one could say them of software. Bad data causes people to misbehave; bad code causes computers to misbehave. I’m not saying that computers and people can be compared–I’m saying that software and knowledge can be.

One wishes to have knowledge in a sensible and logical form. Similarly, one should also desire to have software–particularly the code–in a sensible and logical form. Because code is knowledge, it should translate to knowledge in one’s mind almost immediately upon viewing it. If it doesn’t, then some part of it is too complex–perhaps the underlying programming language or systems, but more likely the structure of the code as created by its designer.

When we desire knowledge, there are numerous ways to acquire it. One could read about it, think about it, perform observations, do experiments, talk about it, etc. In general, we could divide these methods into acquiring the data for ourselves (via observation, experiment, thought, etc.) or getting data from somebody else (reading, talking, etc.).

There are some situations in which we must get data for ourselves, particularly when it applies to us in some unique way that we couldn’t rely on others to work out correctly. As an extreme example, walking on my own legs likely took tremendous amounts of personal experimentation when my body was much smaller. I probably had some assistance, but that knowledge had to be developed by me.

There are far more situations, however, in which we must rely on secondhand data. If one wants to do a good job at living, there’s a lot to know–one simply could not acquire so much information on their own. This is where the help of others comes in: the data they know, the lessons they’ve learned and can teach us.

It seems likely that these same principles describe when one should write code themselves or use existing code. You pretty much couldn’t write all the code yourself down to the hardware level and come up with some of the most useful software we have today. For sure, there are some things that only we are uniquely qualified to write–usually the specific logic of the product that we’re working on. But there are many more things that we must rely on existing code for, just like we must rely on existing secondhand knowledge to survive as individuals.

It’s also possible we could use this principle somewhat for deciding how to divide up work between developers. Would it be faster for somebody to create a piece of code out of their firsthand knowledge, or would it be faster for a group of people to look at the existing system (secondhand knowledge) and start to contribute their own parts (which will, in time, essentially become their firsthand knowledge)? The answer depends on the situation, obviously, and though the basic idea here may not be too novel (some programmers already know the system better than others and so they’re faster) the way we came to the idea is what matters. We first theorize that software is knowledge, and then suddenly we can see a clear logical line down to some existing principle that is already known to be generally true. Pretty handy, and indicates we could likely derive other, more useful information from this principle.

Of course, this is not, by itself, a science or a scientific system. It’s just an idea, one that seems to work well for deriving principles about development. I would say it is one of the broadest philosophical theories that I’ve been able to develop about software, in fact. It seems to cover all aspects and explain all behaviors. I could actually sit here and theorize about this idea all day, but I’m not here to ramble, just to give a brief summary and then see what you have to say about it.

-Max

Comments: 1

What if software development was a science–one with laws, rules, facts, and definitions that told you with certainty which directions to take and which directions to avoid? Not a dogmatic system which restricted you only to some particular methodology, but a series of principles that freed you to think for yourself and make the right decisions for your situation?

What if then, all of this was in a book, that book was only 90 pages long, and it was understandable by every single person working in the software industry, programmer or not? Would it make the world a different place? Find out for yourself:

Code Simplicity: The Science of Software Development.

I’ve spent the last several years developing, testing, and refining a series of scientific laws for software development. Some of what I’ve been doing, you’ve seen in this blog, but the book isn’t just a regurgitation of these articles here. It’s a complete, organized treatise on this new science–a series of principles that I hope will not just change your software, but also bring sanity, order, and happiness to your life as a software developer. Then, once your team reads it, it will bring understanding and insight to your group’s direction and discussions. And finally, when every software developer has read it, it will change the world of software development.

But it all starts here, with you. Help me change the world. All you have to do is read a book, and then if you think you got something out of it, tell other people about it, so maybe they will read it too.

• Direct from O’Reilly, available in print and for all e-readers, on sale for 50% off until April 4.
• In the Kindle Store
• And available in print at numerous other online bookstores, as well.

-Max

Comments: 15

• You have to add “hacks” to make things keep working.
• Other developers keep asking you how some part of the code works.
• Other developers keep mis-using your code, and causing bugs.
• Reading a line of code takes longer than an instant for an experienced developer.
• You feel scared to modify this part of the code.
• Management seriously considers hiring more than one developer to work on a single class or file.
• It’s hard to figure out how to add a feature.
• Developers often argue about how things should be implemented in this part of the code.
• People make utterly nonsensical changes to this part of the code very often, which you catch only during code review, or only after the change has been checked in.

That’s what I can come up with off the top of my head. What are some others?

-Max

Comments: 13

The only thing I care about is that your program helps me accomplish my purpose. That’s a truly remarkable feat, and if your program does it, you should be proud. There’s no need to make your program take up more of my attention just because you think it’s important.

Now of course, your program important to you! When you work on code for a long time, it’s easy to become attached to it. It was so hard to write. Your cleverness is unbounded, shadowing lesser mortals in the mountain of your intellect. You have overcome some of the greatest mental obstacles man has ever faced. Truly, you must shout this from the tops of every tower, through the streets of every city, and even unto the caves of the Earth.

But don’t. Because your users do not care. Your fellow developers might be interested, but your users are not.

When you’re truly clever, what will show up for users is that program is awesome. It’s so awesome, the user hardly notices it’s there. That is true brilliance.

The worst offenders against this ideal are programs that pop up a window every time my computer starts. I know your software is there. I installed it. You really don’t need to remind me. If my purpose is to start up my computer so I can use it, how is your pop up window helping me accomplish that? It’s not, so get rid of it.

There are smaller ways to cause problems, too, that all revolve around asking for too much time or attention from the user:

• “Users will definitely be okay with clicking through three screens of forms before they can use my product.”
• “I’m sure that users will want to learn all the icons I invented for this program, so taking away the text labels for those icons is fine!”
• “I’m sure it’s okay to stop the user from working by popping up these dialog boxes.”
• “Users will totally want to search through this huge page for a tiny little piece of text so they can click on it.”
• “Why should we make this simpler? That would be a lot of work, and it’s already pretty easy…for me.”

And so on.

The true humility required of a developer is the willingness to remove their identity from the user’s world. Stop telling the user the program is there. Don’t think that the user cares about your program, wants to spend time using its interface, or wants to learn about it. It’s not your program that they care about–it’s their purpose. Help them accomplish that perfectly, and you will have created the perfect program for them.

-Max

Comments: 9

1. Getting people interested in contributing
2. Removing the barriers to entering the project and contributing
3. Retaining contributors so that they keep contributing

If you can get people interested, then have them actually contribute, and then have them stick around, you have a community. Otherwise, you don’t.

If you are just starting a project or need to improve the community of an existing project, you should address these points in reverse order. If you get people interested in a project before you do the later two steps, then people won’t be able to enter and won’t stick around when they do enter. You won’t actually expand your community. So first, we want to be sure that we can retain both existing and new contributors. Once we’ve done that, then we want to remove the barriers to entry, so that interested people can actually start contributing. Only then do we start worrying about getting people interested.

So let’s talk about how you accomplish each step in reverse order:

Retaining Contributors

For the Bugzilla Project, this was our biggest challenge. Once somebody started contributing, what made them keep contributing? How did we keep people around?

Well, we had an interesting advantage in answering these questions, in that we are one of the older open-source projects in existence, having been around since late 1998. So we had a tremendous wealth of actual data to work with. We mined this data in two ways: First, we did a survey of all our past developers who had left the project, asking them why they had left. This was just a free-form survey, allowing people to answer any way they wanted. Then, we created a graph of the number of contributors over time, for the whole ten years of the project, and correlated the rise and fall of the graphs to various actions we took or didn’t take over time.

Once all this was done, I sent an email that out to the developers Bugzilla Project, describing the results of the research. You can read the whole email if you’d like, but I’ll summarize the findings here:

• Don’t freeze the trunk for long periods.

  The Bugzilla Project has a fairly-standard system of having stable branches that receive little change (for example, the “3.4″ branch where we commit bug fixes and do minor releases like 3.4.1, 3.4.2, etc.), and a main-line “trunk” repository where all new features go, and which eventually becomes our next major release.

  In the past, before a major release, we would “freeze” the trunk. This meant that no new features could be developed for several weeks or months until we felt that trunk was stable enough to call a “release candidate.” Then we would create a new stable branch from the trunk and re-open the main-line trunk for features. However, while trunk was frozen, there was no feature development happening anywhere in the Bugzilla Project.

  Graph analysis showed very clearly that every time we would freeze, the community would shrink drastically and it would take several months after we un-froze for the size of the community to recover. It happened uniformly, every single time we would freeze, over many years and many releases.

  Traditional wisdom in open-source is that people like to work on features and don’t like to fix bugs. I wouldn’t say that that’s exactly true, but I would say that if you only let people fix bugs, then most of them won’t stay around.

  We addressed this issue by never freezing the trunk. Instead, we branch immediately at the point that we normally would have “frozen” the trunk. The trunk always stays open for new feature development. Yes, this means that for a while, our attention becomes split between the trunk and the latest branch. We’re committing the same bug fixes to the branch and the trunk. We are also doing feature development on the trunk simultaneously with those bug fixes. However, we’ve found that not only does the community expand more rapidly this way, but we also actually get our releases out more quickly than we used to. So it’s a win-win situation.

• Turnover is inevitable.

  The survey found that the number one reason that contributors leave is that they no longer have time to contribute, or that they were contributing as part of their job and now they have changed jobs. Essentially, it is inevitable that most contributors eventually leave.

  So if it community members are definitely going to be leaving, the only way to consistently expand the community is to figure out how to retain new contributors. If you don’t get new members to stick around, then the community will continuously shrink as old contributors leave, no matter what else you do.

  So while retaining existing contributors is important–after all, you want people to stick around and contribute for as long as reasonably possible–what matters the most is retaining new contributors. How do you do that? Well, that’s a lot of what the rest of these points are about.

• Respond to contributions immediately.

  The Bugzilla Project has a system of code reviews that requires that all new contributions be reviewed by an experienced developer before they can become part of Bugzilla. There have been various complaints about the system over the years, but analyzing the survey data showed that people leave the project because getting a review takes too long, not because the reviews are too hard. In fact, the reviews can be as hard as you want as long as they happen almost instantly after somebody submits a contribution.

  People don’t (usually) mind having to revise a contribution. They even generally don’t mind revising it several times. But they do mind if they post a patch, don’t get a review for three months, and then they have to revise it, only to wait another three months to be told that they have to revise it again. It’s the delay that matters, not the level of quality control.

  There are other ways of responding rapidly to contributions, too. For example, immediately thanking somebody for posting a patch can go a long way toward retaining new contributors and “converting” them into long-term developers.

• Be extremely kind and visibly appreciative.

  For nearly every person who responded to our survey, the factors involved in not staying–beyond “my job changed” or “I didn’t have time”–were surprisingly personal. I know that we all work with computers, and perhaps we’d like to think that engineering should be a totally cold scientific profession where we all do our jobs correctly according to the requirements of the machine, and not worry about our emotional or personal involvements. However, nothing could be further from the truth–the personal interactions that people have with community members, the amount they feel appreciated, and the amount they feel assaulted, are actually the most important aspects of retaining community members.

  When people contribute on a volunteer basis, they aren’t getting paid in money, they are getting paid in admiration, appreciation, the sense of a job well done, and the knowledge that they are helping create a product that affects millions of people. When somebody has contributed a patch, you need to thank them. It doesn’t matter if the patch is total crap and has to be re-written entirely, you need to thank them. They have put some work into this, and if you don’t appreciate that, they will leave before they even start. After all, most people get little enough appreciation at their workplace–they stay there because they get paid in money! They don’t need to work for free with some other organization if it also doesn’t appreciate their work, or even worse, assaults every aspect of their contribution before even thanking them for it.

  Of course, you still need to correct people on the faults in their contributions. “Kindness” does not include putting bad code into your system. That isn’t kind to anybody, including the contributor whose skills probably need to improve, and who may go on believing that something they did in error was in fact correct. You have to still be careful reviewers and very good coders.

  What this does mean is that in addition to telling people what’s wrong with their contribution, it’s important to appreciate what’s right about their contribution, even if it’s simply the fact that they took the time to contribute. And you have to actually tell the contributor that you appreciate the contribution. The more frequently and genuinely that you do this, the more likely you are to retain the contributor.

• Encourage a total absence of personal negativity.

  One thing that drives people away from a project with lightning speed is when they get personally attacked for attempting to do something positive. A “personal attack” can be as little as an unpleasant joke about their code, instead of just a straightforward technical description of what is wrong. Saying something like, “What is wrong with you?” instead of actually leaving some helpful comment. Disguising personal criticism as “an attempt to help them code better” or “help them get along with others.” No matter how well-justified these actions may seem to be, they are all personal attacks that are extremely dangerous to your community.

  Now truthfully, coding and working on a collaborative project with people who have different viewpoints can get really frustrating sometimes, and I’ve been an offender in this area just as much as anybody has been. But we all have to learn that it’s not okay to insult other developers as people just because we’re personally frustrated with them.

  The solution isn’t just to say “everybody, now bottle up your frustrations until you explode,” though. There are lots of practical solutions. One of the best is to set up some specific system for handling problematic contributors. If there’s some contributor that Bob just can’t live with, there needs to be somebody in the community who Bob can go to to help work things out. We’ll call this go-to person the “community moderator.” So Bob tells the moderator about the problem, and maybe the moderator sees that other contributor really was being a terrible person or bad coder, and so this “community moderator” gently corrects that contributor. But it’s also possible that there was some communication problem between Bob and the other contributor that the moderator just needs to help resolve.

  This “moderator” system isn’t the only way to deal with the problem. You can resolve the problem in numerous ways–the most important thing is that you do resolve it. Without some channel or method for dealing with personal frustrations, individual contributors will take these frustrations out on each other. You will in fact foster an environment where it’s okay for one contributor to personally attack another contributor, because that’s the only avenue they have to resolve their problems, and nobody’s stopping them.

Basically, those last two points can be summed up as: be really, abnormally, really, really kind, and don’t be mean.

We’ve been applying all of these principles in the Bugzilla Project for the past several months, and we saw an increase in the number of retained contributors almost immediately after we started applying them. I’m finally starting to feel like the community is growing again, after shrinking almost continuously since 2005 due to violations of all of the above points.

Removing the Barriers

The next step is to remove the barriers to entry. What prevents people from getting started on the project?

Usually, the biggest barrier is a lack of documentation and direction. When people already want to contribute, their next step is figuring out how to contribute. They will go to your project’s website and look around. They will wonder, “Who do I talk to about this? How do I start contributing? What do you guys want me to work on?”

For the Bugzilla Project, we solved this problem in several ways:

1. A list of easy starting projects.

   Whenever we see a bug or feature request that looks like it would be easy for a newcomer to solve, we tag it as a “good intro bug” in our bug tracker. This gives us a list of good introductory projects that anybody can come and look at without having to ask us “where do I get started?”

2. Create and document communication channels.

   People will almost immediately want to talk to somebody else about the project. You should have email lists and also some method of instantaneous communication like an IRC channel. For example, we have an email list for Bugzilla developers and also an IRC channel where almost all our contributors hang out. In fact, we don’t just have a normal IRC channel–we also have a web page that people can use to chat in that IRC channel. That way, people don’t have to install an IRC client just to come talk to us. Setting up that web page enormously increased the number of new people coming into the channel and communicating with us. (And the increase was entirely positive–I can’t think of a single person who used the web gateway to cause us trouble.)

   Then once you have these channels, they need to be documented! People have to know how to get into them, they need to know that they exist. We have a wiki page that explains how to talk to us if you want to contribute. (Note that this is separate from our support page that describes how to get support for the project.)

   Also, as a final but perhaps obvious point, the existing community has to use the communication channels. If the main contributors do all their work in an office and just talk to the people next to then and you don’t use the mailing lists or IRC channels, then the community members aren’t going to want to use those communication systems either. After all, the new contributors aren’t there to talk to each other–they’re there to talk to you!

3. Excellent, complete, and simple documentation describing exactly how a contribution should be done.

   Fully document every step of your development process, and put that documentation onto a public web site. Don’t invent a new process, just document out what the existing actual process is. How do people get the code? How can they submit patches or other contributions to you? How do those contributions become an official part of the system?

   We have a very simple page that describes the basic steps of our whole process, and links to documents that describe each step in more detail. It also specifically encourages people to get into communication with us, so that we know that they are there and want to help.

4. Make all this documentation easy to find.

   This is a simple final step, but sometimes projects forget it! You can have all the wonderful developer documentation in the world, but if new contributors can’t find it super-easily, then you’re not actually removing any barriers to entry! We have a big “Contribute!” button on our website that describes all the different ways that people can contribute (not just code!) and links to more information about each of those.

We saw a definite upswing in the number and quality of contributions once we completed all these steps. Also, having everything documented and clearly stated on a public website meant that we no longer had to personally explain it all, every time, to every new contributor.

Direction and documentation aren’t the only things you can do though. Ask yourself, “What is stopping people from contributing?” and remove all the barriers there that you reasonably can.

Getting People Interested

How do you make people think, “Gee, I want to contribute to this project?” That’s the first step they have to take before they can become become contributors. Well, traditional wisdom states that people contribute to open-source projects because:

• They like helping.
• They enjoy being part of a community.
• They want to give back.
• They think that something is wrong and they need/want to fix it.

So you may want to make it apparent that help is needed, that an enjoyable community is there, that giving back is appropriate and appreciated, and that there are problems that need solving.

Now, to be fair, this is an area that I don’t have fully mapped out or figured out for the Bugzilla Project, yet. So I don’t have a lot of personal experience to draw on. But if we analyze other projects, we can see that some good ways of getting contributors are:

• Be a super-popular product.

  This may seem obvious, but it is indeed the primary way of getting new contributors. If a zillion people use your product, it’s statistically likely that many of them will want to contribute. The Linux Kernel and WordPress are good examples of this–they have millions of users, so there’s just bound to be a lot of contributors, provided that the “barriers to entry” and the “retaining contributors” aspects of the project have also been handled.

  One way to become a super-popular product–even if you’re just starting out–is to be heavily needed. The Linux Kernel was very much needed when it was first written, which is probably one of the reasons that it became popular as quickly as it did. It desperately needed to exist and didn’t exist yet.

• Be written in a popular programming language.

  Generally, people are more likely to contribute to a project if it’s written in a language that they already know. WordPress has a huge contributor community, and it’s in PHP. Say what you will about PHP, it is extremely popular. There’s a large number of people who already know the language, which increases the likelihood that some of them will start supplying patches for your code.

  This not the only reason you should choose a particular programming language, but it’s certainly a major motivator if you’re going to have an open-source project. I may think that Eiffel is a remarkable language, but if I wrote an open-source project in it, I would have a very hard time getting contributors.

Beyond those points, there are lots of clever ways of getting people interested in contributing to your projects, including speaking at conferences, publishing blogs, encouraging people on a one-to-one basis, and other methods that basically add up to “contact and encourage.”

I’d love to hear some of your ideas in this area, though. How do you get new people interested in contributing to your project? Has anything been particularly successful?

Summary

An open-source community is somewhat of a fluid thing–there are always going to be people coming and going for one reason or another. What’s important is that the rate of people entering and staying is greater than the rate of people leaving. All of these points help assure that, and hopefully they also make our communities productive and enjoyable places to be for everybody, even ourselves!

-Max

Comments: 39

Readability of code depends primarily on how space is occupied by letters and symbols.

What does that mean? Well, it means two things:

Code should have the proper amount of white space around it. Not too much, not too little.

There should be the proper amount of space within a line of code to separate out the different parts. Separate actions should generally be on separate lines. Indentation should be used appropriately to group blocks of code.

With this principle, it’s actually the absence of code that makes things readable. This is a general principle of life–for example, if there was no space at all between letters and words in a book, it would be hard to read. On the other hand, it’s easy to see the moon against the clear night, because there’s a lot of clear black space that isn’t the moon. Similarly, when your code has the right amount of space in it, you can tell where and what the code is easily.

For example, this code is hard to read:

x=1+2;y=3+4;z=x+y;print"hello world";print"z is"+z;if(z>y+x){print"error";}

Whereas with the proper spacing in, around, and between the lines, it becomes easy to read:

x = 1 + 2;
y = 3 + 4;
z = x + y;
print "hello world";
print "z is" + z;
if (z > y + x) {
    print "error";
}

There can also be too much or wrong space, however. This code is also hard to read:

    x            =          1+        2;
y = 3            +4;

  z = x    +      y;
print    "hello world"         ;
 print "z is " + z;
if (z  >     y+x)
 {        print "error" ;
        }

Code itself should take up space in proportion to how much meaning it has.

Basically, tiny symbols that mean a lot make code hard to read. Very long names that don’t mean much also make code hard to read. The amount of meaning and the space taken up should be closely related to each other.

For example, this code is unreadable because the names are too small:

q = s(j, f, m);
p(q);

The space those names take up is very little compared to how much meaning they have. However, with appropriately-sized names, it becomes more apparent what that block of code is doing:

quarterly_total = sum(january, february, march);
print(quarterly_total);

On the other hand, if the names are too long compared to how much meaning they represent, then the code becomes hard to read again:

quarterly_total_for_company_x_in_2011_as_of_today = add_all_of_these_together_and_return_the_result(january_total_amount, february_total_amount, march_total_amount);
send_to_screen_and_dont_wait_for_user_to_respond(quarterly_total_for_company_x_in_2011_as_of_today);

This principle applies just as well to entire blocks of code as it does to individual names. We could replace the entire block of code above with a single function call:

print_quarterly_total();

And that is even more readable than any of the previous examples. Even though the name we used–print_quarterly_total–is a bit longer than our other names for things, that’s okay because it represents more meaning than other pieces of code do. In fact, it’s even more readable than our block of code was, by itself. Why is that? Because the code block took up a lot of space for, effectively, very little meaning, and the function takes up a more reasonable amount of space for the same meaning.

If a block of code takes up a lot of space but doesn’t actually have much meaning, then it’s a good candidate for refactoring. For example, here’s a block of code that handles some user input:

x_pressed = false;
y_pressed = false;
if (input == "x") {
    print "You pressed x!";
    x_pressed = true;
}
else if (input == "y") {
    if (not y_pressed) {
        print "You pressed y for the first time!";
        y_pressed = true;
        if (x_pressed) {
            print "You pressed x and then y!";
        }
    }
}

If that were our whole program, that would probably be readable enough. However, if this is within a lot of other code, we could make it more readable like this:

x_pressed = false;
y_pressed = false;
if (input == "x") {
    handle_x(x_pressed);
}
else if (input == "y") {
    handle_y(x_pressed, y_pressed);
}

And we could make it even more readable by reducing it to this:

handle_input(input);

Reading “handle_input” in the middle of our code is much easier than trying to read that whole first block, above, because “handle_input” is taking up the right amount of space, and the block is taking up too much space. Note, however, if we’d done something like h(input) instead, that would be confusing and unreadable because “h” is too short to properly tell us what the code is doing. Also, handle_this_input_and_figure_out_if_it_is_x_or_y_and_then_do_the_right_thing(input) would not only be annoying for a programmer to type, but would also make for unreadable code.

Naming Things

It was once said by a famous programmer that naming things was one of the hardest problems in computer science. These principles of readability give us some good clues on how to name things, though. Basically, the name of a variable, function, etc. should be long enough to fully communicate what it is or does, without being so long that it becomes hard to read.

It’s also important to think about how the function or variable is going to be used. Once we start putting it into lines of code, will it make those lines of code too long for how much meaning they actually have? For example, if you have a function that is only called once, on one line all by itself, with no other code in that line, then it can have a fairly long name. However, a function that you’re going to use frequently in complex expressions should probably have a name that is short (though still long enough to fully communicate what it does).

-Max

Comments: 27

Well if you’ve ever experienced any of those things, your next thought might have been something like “**** this computer” or “**** the silly programmer who made it behave this way”. After all, aren’t programmers and hardware designers to blame for the crazy behavior of the system? Well, yes, to some extent they are. But after being intimately involved in software design for many years, I now have another reaction to poorly-implemented features. Instead of becoming angry with the programmer who implemented the system, I ask myself, “Who was the software designer who authorized this feature?” Who stood by silently and let this feature happen when they had the power to stop it?

Granted, sometimes there is no software designer at all, in which case you’re practically guaranteed to have a broken system. But when there is a software designer, they are ultimately responsible for how the system is put together. Now, quite a bit of this job involves designing the structure of features before they go into the system. But there’s also another part of the job of a software designer–preventing bad ideas from being implemented. In fact, if there’s any lesson I’ve learned from my years in the software industry, it’s this:

The most important word in a software designer’s vocabulary is “no”.

The problem is that if you give a group of humans total freedom to implement any random idea that comes into their mind, then nearly every time they will implement bad ideas. This isn’t a criticism of developers, it’s more of a fact of life. I have great faith in the intelligence and capability of individual developers. I admire developers’ struggles and achievements in software development. It’s just an unfortunate fact of existence that without some central guidance, people in a group tend to evolve complex systems that don’t help their users as well as they could.

An individual designer, however, is usually capable of creating a consistent and enjoyable experience for the users and developers both. But if that individual designer never steps up and say “no” when another developer starts to do something the wrong way, then the system will collapse on itself and become a chaotic mess of bad ideas. So it is very important to have a software designer who has the power to say “no”, and then it’s important for that designer to actually use that power whenever it is appropriate.

It is truly amazing how much you can improve your product just by saying “no” to any idea that really deserves a “no”.

Recognizing Bad Ideas

Before you can apply this principle, there is one thing that you have to know: how to recognize bad ideas. Thankfully, there are a lot of software design principles that help clue you in on what is a bad idea, and lead you to saying “no” when it’s truly needed. For example:

• If the implementation of the feature violates the laws of software design (for example, it’s too complex, it can’t be maintained, it won’t be easily changeable, etc.) then that implementation is a bad idea.
• If the feature doesn’t help the users, it’s a bad idea.
• If the proposal is obviously stupid, it’s a bad idea.
• If some change doesn’t fix a proven problem, it’s a bad idea.
• If you aren’t certain that it’s a good idea, it’s a bad idea.

Also, one tends to learn over time what is and isn’t a good idea, particularly if you use the above as guidelines and understand the laws of software design.

Having No Better Idea

Now, sometimes a designer can recognize a bad idea, but they still implement it because they can’t think of a better idea right now. This is a mistake. If you can think up only one solution to a problem but it is obviously stupid, then you still need to say no to it.

At first this may seem counter-intuitive–don’t problems need to be solved? Shouldn’t we solve this problem in any way we can?

Well, here’s the problem: if you implement a “bad idea”, your “solution” will rapidly become a worse disaster than the original problem ever was. When you implement something terrible, it “works”, but the users complain, the other programmers all sigh, the system is broken, and the popularity of your software starts to decrease. Eventually, the “solution” becomes such a problem that it requires other bad “solutions” to “fix” it. These “fixes” then become enormous problems in themselves. Continue down this path, and eventually you end up with a system that is bloated, confused, and difficult to maintain, just like many existing software systems today.

If you often find yourself in a situation where you feel forced to accept bad ideas, it’s likely that you’re actually near the end of this chain of events–that is, you’re actually building on a series of pre-existing bad ideas from the system’s past. In that case, the solution is not to keep “patching” over the bad ideas, but to instead find the most fundamental, underlying bad ideas of the system and redesign them to be good, over time.

Now ideally, when you reject a bad idea, you should provide an alternate, good idea in its place–that way you’re being constructive and moving the project forward, instead of being viewed as a roadblock on the path of development. But even if you can’t come up with a better idea right now, it’s still important to say no to bad ideas. A good idea will come eventually. Maybe it will take some study, or perhaps it will suddenly come to you while you’re standing in the shower one day. I have no idea where the idea will come from or what it will be. But don’t worry too much about it. Just trust that there is always some good way to solve every problem, and keep looking for it until you find it. Don’t give up and accept bad ideas.

Clarifications: Acceptance and Politeness

So it’s important to say “no”, but there are a few clarifications required on what I really mean, there. I’m not saying that every suggestion is wrong. In fact, developers are usually very bright people, and sometimes they really do nail it. Many developers make perfect suggestions and do excellent implementations. And even the worst solutions can have good parts, despite not being excellent as a whole. So many times, instead of actually saying “no”, what you’ll be saying is something more like, “Wow, there’s a part of this idea that is really good, but the rest of it is not so great. We should take the best parts of this idea and build them up into something awesome by doing more work on them.” You do have say no to the parts of an idea that are bad, though. Just because one part of the idea is good doesn’t mean that the whole idea is good. Take what’s intelligent about the idea, refine it, and build good ideas around it until the solution you’ve designed really is great.

Also, it is still critically important that you communicate well with the rest of your team–having the responsibility of saying “no” doesn’t give you the right to be rude or inconsiderate. If you continuously say “no” without any hint of kindness, you are going to fracture your team, cause upsets, and ultimately end up wasting hours of your time in argument with the people you’ve upset. So when you have to say “no”, it’s best to find a polite way to communicate it–a way that expresses appreciation for the input, positive suggestions of how to improve things, and lets the person down easily. I understand how frustrating it can be to have to slow down and explain things–and even more frustrating to repeat the explanation over and over to somebody who doesn’t get it the first time–but if that’s what it takes to have an effective development team while still saying “no” to bad features, then that’s what you have to do.

-Max

Comments: 3

However, before somebody can even start on the path of becoming a better software developer, one thing has to be true:

In order to become an excellent programmer, you must first want to become an excellent programmer. No amount of training will turn somebody who does not want to be excellent into an excellent programmer.

If you are a person who is passionate about software development–or even just somebody who likes being good at their job–it may be hard to understand the viewpoint of somebody who simply doesn’t want to get any better. To fully grasp it, it can be helpful to imagine yourself trying to learn about some area that you personally have no desire to be great in.

For example, although I admire athletes, enjoy playing soccer, and sometimes enjoy watching sports in general, I’ve never had a desire to be a great athlete. There’s no amount of reading or education that will ever turn me into a great athlete, because I simply don’t want to be one. I wouldn’t even read the books in the first place. If you forced me to take some classes or go to some seminar, it would leave my mind as soon as I took it in, because I would simply have no desire to know the data. Even if I was playing sports every day for a living, I’d think, “Ah well, I don’t have any passion for athletics, so this information simply isn’t important to me. Someday I will be doing some other job, or some day I will retire and not have to care, and until then I’m just going to do this because they pay me and it’s better than starving.”

As hard as this can be to imagine, that is what happens in the minds of many “bad” programmers when you tell them how or why they should write better code. If they don’t sincerely want to be the best programmers that they can be, it does not matter how much education you give them, how many times you correct them, or how many seminars they go to, they will not get better.

So what do you do? To be fair, I may not be the best person to ask–if I’m going to do something, I feel that I should do my best to excel in it. Perhaps the best thing you can do is encourage people to follow that concept. You could say to them something like: “If you’re going to be doing it anyway, why not do it well? Wouldn’t it at least be more enjoyable to be doing this if you were more skilled at it? What if some other people were impressed with your work, how would that feel? Would it be nice to go home at the end of the day and feel that you had done something well? Would your life be better than it is now, even if only a little? Would your life get worse?”

At the very worst, you could ask somebody to list off all of the consequences of “being a great programmer” until they felt some relief on the subject or started to see the idea differently. You could ask them something like, “What would happen if you were a great programmer?” and keep asking for more answers to that question until they felt better about it or started really seeing how good it could be to be excellent. You don’t have to respond to their answers with any positive or negative comments, just listen and politely acknowledge the things they’re saying. The idea is to give them the chance to really examine the possibility for themselves, and maybe come to some new and interesting conclusions, by themselves–not by you telling them what to think or disagreeing with their answers, but just by communicating to you what would really happen if they became great.

However you do it, the bottom line is that people must be interested in improving themselves before they can get better. How you bring them up to that level of interest doesn’t really matter, as long as they get there before you waste a lot of time giving them an education that they’re just going to throw away as soon as they hear it.

-Max

Comments: 5

1. It is more important to reduce the Effort of Maintenance than it is to reduce the Effort of Implementation.
2. The Effort of Maintenance is proportional to the complexity of the system.

And that is pretty much it. If all you knew about software design were those two principles and the purpose of software, you could evolve every other general principle of software development.

-Max

Comments: 9

• Potential Value of Implementation (V_i for short): How “valuable” could it be to implement this? For example, if we add something to the program that could directly prevent somebody from dying, that’s very valuable. If it simply might prevent a future typo in a single error message, that’s hardly valuable at all.

  The “value” can be for the end user or for other programmers. When we’re talking about design decisions that affect only code, and not the end user, flexibility is one of the major values–how important could this flexibility be?

  The potential value is separate from how likely it is that the situation will occur where you will need it. That’s the next issue.

• Probability of Value (P_v for short): What is the chance that this value will be, in fact, realized by an end user (for a feature or functional change) or by another developer (for some design decision)? If we’re adding in code flexibility to allow for potential contact with extraterrestrial apes, that’s not a very probable occurrence. If we’re adding in a feature that is immediately useful to every single user, that’s 100% probability. (The number of users that a feature will be useful to is also part of the Probability of Value.)
• Effort of Implementation (E_i for short): How hard will it be to implement this? This is a one-time cost–the immediate difficulty of performing the work required to create this thing the very first time. This would probably be measured in person-hours.
• Effort of Maintenance (E_m for short): How much effort will it require to maintain this in the future? (This includes any effort added to maintaining the entire program by implementing this.) Will this complicate maintenance for the whole system? This is an amount that increases over time. Similar to Effort of Implementation, this would be most likely measured in person-hours.

What we’re trying to determine is the Desirability Of Implementation (D for short). This answers the questions “Is this something we should do or not?” and “What should the priority of implementing this be?”

The simplest form of the equation is:

D = (P_v * V_i) / (E_i + E_m)

Or, in English:

The Desirability of Implementation is directly proportional to the Probability of Value and the Potential Value of Implementation, and inversely proportional to the total effort, consisting of the Effort of Implementation plus the Effort of Maintenance.

However, there is a critical factor missing from the simple form of this equation: time. What we actually want to know is the limit of this equation as time approaches infinity, and that gives us the true Desirability of Implementation. So let’s look at this from a logical standpoint:

The Effort of Implementation is a one-time cost, and never changes, so is mostly unaffected by time.

The Value of Implementation may increase or decrease over time, depending on the feature. It’s not predictable, and so we can assume for the sake of this equation that it is a static value that does not change with time (though if that’s not the case in your situation, keep this factor in mind as well). One could even consider that the Effort of Maintenance is actually “the effort required to maintain this exact level of Value,” so that the Value would indeed remain totally fixed over time.

The Probability of Value, being a probability, approaches 1 (100%) as time goes to infinity.

The Effort of Maintenance, being based on time, approaches infinity as time goes to infinity.

At first glance, that might sound as though design is hopeless, because maintenance becomes infinite effort–an amount that no Potential Value could surpass–and it seems like every possibility must be accounted for, because given infinite time the probability seems to indicate that every possibility will occur. Those are not true statements, though, because you have to think about the rate at which both of those items increase. If the fundamental effort of maintenance is very small, then even as time goes on, it will remain small. You could say that there is a “coefficient of maintenance” on any design decision or feature, and that that determines how rapidly maintenance effort will accumulate over time. As far as the Probability of Value goes, if it is a very tiny number, it may remain tiny until thousands or millions of years have passed–so if the Effort of Maintenance increases at a great rate, then it will easily outstrip the Probability of Value and the Desirability of Implementation will approach zero as time approaches infinity.

What this equation actually tells us is that the most important factors to balance, in terms of time, are probability of value vs. effort of maintenance. If the probability of value is high and the effort of maintenance is low, the desirability is then dependent only upon the Potential Value of Implementation vs. the Effort of Implementation–a decision that a product manager can easily make. If the probability of value is low and the effort of maintenance is high, the only justification for implementation would be a near-infinite Potential Value of Implementation.

This interestingly indicates why small improvements in programming languages and development frameworks result in such enormous changes in the resulting products–because tiny reductions in the Effort of Maintenance can make tremendous changes in the Desirability of Implementation. Features that otherwise would be thrown away by a product manager as impossible become part of the basic design plan. Polishing the UI becomes more desirable, because it requires less effort for both implementation and maintenance.

And finally, I think that this communicates most exactly and truthfully why simplicity is so important–because simplicity is what determines the “coefficient of maintenance” that I talked about above. The effort involved in maintaining simple code increases very slowly over time–sometimes so slowly that you never will have to put any maintenance effort into it in your lifetime.

There’s a lot more that could be said about this equation. What are your thoughts on it? Anybody have any ideas of how Value of Implementation could be numerically calculated, or if it might break down into a set of other numerically-calculable factors? Anything you have to say about it, I’m interested.

-Max

Comments: 24

There’s only one problem with this whole debate: what is privacy, and why would anybody want it? This is rarely defined–most people just seem to assume that “everybody knows” that privacy is, so why would it have to be explained?

Well, I’m not a big fan of “everybody knows.” And in fact, it turns out that privacy actually means two different things, which many people use interchangeably without specifying what they’re actually talking about. So to help clear up some of the debate online, and to hopefully shed some light on how it can all be resolved, here are some clear definitions and discussions of what privacy is, and why people would want it.

Privacy of Space

The first type of privacy is “privacy of space”. This is the ability to control who does and does not enter a particular physical space, probably because you’re in the space and you don’t want certain others in that space. “Enter the space” in that definition includes any method of being able to perceive the space–so, for example, if somebody stands outside the door with their ear pressed to it, they’re violating your privacy. If somebody installs a camera in your room without your consent, they’re violating your privacy.

This form of privacy is not metaphorical. It does not apply to anything other than physical space. It literally means, “I do or do not want you to be perceiving this physical location, and I have the choice and ability to control that.”

The most common reason that we want this form of privacy is that we want to protect somebody or something from harm, most commonly ourselves. This harm can be minor (we don’t want to be annoyed by people walking through our house all the time), it can be purely social (we close the door when we go to the bathroom because we know others don’t want to perceive us going to the bathroom, and we may also not want to be perceived in such a state), or it can be extreme (a man with a mask and a chainsaw should not be in my closet).

One interesting thing about this form of privacy is that we don’t usually consider animals, plants, or material objects to be capable of violating it, even if they enter a space without our permission. It might be annoying if the cat comes in the room when you don’t want it to, but you’re not going to complain that the cat is “violating your privacy”, right?

So, when it comes to computer programs, this is not the form of privacy we’re talking about, since we don’t consider that a computer program being in the same room with us is a violation of our privacy of space. My word processor is not violating my physical privacy of space, even though it’s “in the room” with me, because it does not, itself, perceive. The only exception would be a computer program that was transmitting perceptions (sound or sight) to some location that we didn’t want to send it to–that would be a privacy violation, because someone could perceive our space through it when we didn’t want them to. When it comes to that sort of privacy, violations are pretty pretty cut-and-dry. If a computer program sends perceptions of my space anywhere without my permission, it is absolutely violating my privacy, it’s not useful to me, and it should stop immediately.

But on the Internet, that’s not usually the type of privacy we’re talking about.

Privacy of Information

The second type of privacy is “privacy of information.” This is the ability to control who knows certain things. When we talk about computer programs and the Internet, this is the most common type of privacy we’re talking about.

So why would somebody want privacy of information? Is it just because they’re doing something that they want to hide from others? Is it just for committing crimes or for hiding harmful acts? Well, sometimes it is, yes. There are many people who use the concept of “privacy” to protect themselves from the law or the moral rejection of others. It is probably because of these individuals that the concept of privacy is a muddy subject–as long as it’s unclear quite what “privacy” is, it’s much easier for those who have have committed harmful acts to invoke “privacy” as a defense.

But is that the only reason that somebody would want privacy of information? What about a normal person, who isn’t doing anything harmful–would they ever want to keep certain information private?

Well, there is absolutely a rational reason that people would want privacy of information, and interestingly, it’s the same reason that people want privacy of space:

An individual or group desires privacy of information because they believe that other people knowing that information could or would be more harmful than them not knowing it.

Here’s a very straightforward example: I consider that a criminal knowing my credit card number would be harmful–far more harmful than them not knowing it.

In certain countries, the fact that I read a certain website or talked to certain people on the Internet could get me killed or put in jail. So, in that situation, other people knowing my browser history could be very harmful, no question about it.

Of course, if one kept everything private, one could not live. If you pay for a piece of candy with a quarter, the person receiving that quarter now knows that you had a quarter. They may know that you kept it in a waller, or that you pulled it out of your pants. They probably know what you look like, if you’re not wearing a mask. They most likely also know that you have five fingers, and that you were in their store at a certain time. In short, no matter what you do, in order to live, you must exchange information with other people. The more things you do, the more information you will have to exchange.

In fact, usually, the more information that others know about you, the more helpful they can be. The bank knows all the transactions that I made, so they can help me by creating an online system that shows me my transactions and lets me search them. That information can be seen by bank employees, but I don’t consider that to be potentially harmful enough to outweigh the obvious benefits of the bank having it.

The web browsers that I use know my passwords to certain sites, so they can help me by putting those passwords into the box, saving me some typing. Potentially, somebody could steal that information from my computer, but the chance of that happening is small enough, and the benefit is significant enough, so I consider it acceptable to save my passwords in the browser.

The examples like this go on and on–the appropriate use of information is extremely beneficial. The inappropriate use is what’s harmful.

So who decides what what’s an appropriate use and what’s an inappropriate use? What information should be sent and stored, and what information should be kept private? Well, these are the fundamental questions being asked when people debate privacy issues–who gets to choose whether my knowledge becomes somebody else’s knowledge? Should I be asked before my information is sent, or should I just be given the option to opt-out and delete the information? Is there some information that should never be sent? What information is more important to keep private than other information?

Though this is all far less cut-and-dry than “privacy of space” issues, these questions can generally be answered by the “help vs harm” equation. The basic sort of questions one might want to ask would be:

• Will sending and storing this information harm any users, immediately or potentially? (Remember, “potentially” is pretty broad–what happens if somebody with bad intentions steals that information from you? What happens if somebody buys your company and decides to use that information in a way that you think is bad?)
• Would it help your users more than harm them to take this information?
• Taking all the above into account, should sending this information be optional? (This is largely determined by how broadly it could be harmful to collect the information.)
• If sending the information is optional, should it be opt-out or opt-in? (That is, should it automatically be on, and people have to turn it off if they don’t want to send the info, or should it be off and people have to choose to turn it on?)
• If it’s opt-in, will the feature still be helpful to enough of your users to justify implementing it?

There are some people who will claim that no information should ever be sent or stored about the user, that all privacy options should always be opt-in, and that all information is so potentially harmful that no debate about this can be accepted. That is, frankly, a ridiculous proposition. It’s so obviously untrue that there’s almost no way to argue with it, because it’s such a shocking irrationality. Just like the fact that somehow, liquids could harm somebody (so you can’t bring liquids on an airplane in the USA) it’s true that there are situations in which almost any piece of information could be dangerous. That doesn’t mean that all information is dangerous, though.

My martial artist friends have frequently joked that they shouldn’t be allowed to bring any object on an airplane, because they could kill somebody with any of them. Similarly, given almost any piece of information, somebody could do something harmful with it, somewhere, at some point. If I know you have a quarter in your pocket, I’m sure there’s some situation in which I could use that information to get you in some serious trouble. But that doesn’t make that information realistically harmful, even potentially.

Even the idea of “every single piece of information should be opt-in” is ridiculous. Do you want the web browser to ask you, “May I send this page your IP address?” every time you load a web page? Well, if you’re a spy in a hostile country, maybe you do. But if you’re like most people, that would probably just annoy you–you’d stop using that web browser and switch to another one. And if you are a spy or a resistance fighter, then you probably know how to use Tor to avoid being tracked.

So when we’re talking about privacy, it’s not an issue of “in some incredibly unlikely situation, this information could be very harmful,” it’s an issue of balancing help vs. harm in real-world situations. Real-world situations can be pretty strange and unexpected, but they at least are real, and can be balanced and talked about. Doing so, you can make good decisions about how to protect your users’ privacy–how much information to take, how you inform them about the information you’re taking, and what you do with that information when you have it.

So no, this is not a casual issue or something that we should brush-off and just ignore the dangerous implications of, but it’s also not an extreme unsolvable situation where we have to decide to keep everything private because we can’t make up our minds about it. Privacy is simply something that we should be able to analyze factually, based on real-world situations and data, and come to some practical and useful decision about.

-Max

Comments: 22

What I didn’t know at the time was why programmers did this. It was obvious that they did do it, but why would the software development industry produce so many crazy, complex masses of unreadable code? Why did it keep happening, even when developers should have learned their lesson after their first bad experience? What was it that made programmers not just make bad code, but keep on making bad code?

Well, this was a mystery, but I didn’t worry too much about it at first. Just the revelation that “bad programs are caused entirely by bad programmers”, as simple and obvious as it may seem, was enough to fuel an entire investigation and study into the field of programming, one which had some pretty good results (that’s mostly what I’ve written about on this blog, and it’s also the subject of a book that’s in development). The problem had been defined (bad programmers who create complexity), it seemingly had a solution (describe laws of software design that would prevent this), and that was enough for me.

But it still baffled me that the world’s universities, technical schools, and training programs could turn out such terrible programmers, even with all of the decades of advancement in software development techniques. Sure, a lot of the principles of software design hadn’t been codified, but a lot of good advice was floating around, a lot of it very common. Even if people hadn’t gone to school, didn’t they read any of this advice?

Well, the truth was beyond my imagination, and it took almost five years of working on the Bugzilla Project with a vast number of separate contributors until one day I suddenly realized an appalling fact:

The vast majority (90% or more) of programmers have absolutely no idea what they are doing.

It’s not that they haven’t read about software design (though they likely haven’t). It’s not that the programming languages are too complex (though they are). It’s that the vast majority of programmers didn’t have the first clue what they were really doing. They were just mimicking the mistakes of other programmers–copying code and typing more-or-less meaningless incantations at the machine in the hope that it would behave like they wanted, without any real understanding of the mechanics of the computer, the principles of software design, or the meanings of each individual word and symbol they were typing into the computer.

That is a bold, shocking, and offensive statement, but it has held up in my experience. I have personally reviewed and given feedback on the code of scores of programmers. I have read the code of many others. I have talked to many, many programmers about software development, and I’ve read the writings of hundreds of developers. The number of programmers who really understand what they are doing comprise only about 10% of all the programmers I’ve ever talked to, worked with, or heard about.

In open source, we get the cream of the crop–people who want to program in their spare time. And even then, I’d say only about 20% of open source programmers have a really good handle on what they are doing.

So why is this? What’s the problem? How could there be so many people working in this field who have absolutely no clue what they’re doing?

Well, that sounds a bit like they’re somehow “stupid.” But what is stupidity? People are not stupid simply for not knowing something. There’s a lot of stuff that everybody doesn’t know. That doesn’t make them stupid. That may make them ignorant about certain things, but it doesn’t make them stupid. No, stupidity, real stupidity, is not knowing that you don’t know. Stupid people think they know something when they don’t, or they have no idea that there is something more to know.

This sort of stupidity is something that can be found in nearly every field, and software development is no exception. Many programmers simply don’t know that there could be laws or general guidelines for software development, and so they don’t even go looking for them. At many software companies, there’s no attempt to improve developers’ understanding of the programming language they’re using–perhaps simply because they think that the programmers must “already know it if they were hired to do it”.

Unfortunately, it’s particularly harmful to have this sort of mindset in software development, because there is so much to know if you really want to be good. Anybody who thinks they already know everything (or who has a “blind spot” where they can’t see that there’s more to learn) is having their ability to produce excellent code crippled by a lack of knowledge–knowledge they don’t even know exists and that they don’t even know they lack.

No matter how much you know, there is almost always more to know about any field, and computer programming is no exception. So it’s always wrong to think you know everything.

Sometimes it’s hard to figure out what one should be learning about, though. There’s so much data, where does one start? Well, to help you out, I’ve come up with a few questions you can ask yourself or others to help figure out what areas might need more study:

• Do you know as much as possible about every single word and symbol on every page of code you’re writing?
• Did you read and completely understand the documentation of every single function you’re using?
• Do you have an excellent grasp of the fundamental principles of software development–such a good grasp that you could explain them flawlessly to novice programmers at your organization?
• Do you understand how each component of the computer functions, and how they all work together?
• Do you understand the history of computers, and where they’re going in the future, so that you can understand how your code will function on the computers that will be built in the future?
• Do you know the history of programming languages, so that you can understand how the language you’re using evolved and why it works like it does?
• Do you understand other programming languages, other methods of programming, and other types of computers than the one you’re using, so that you know what the actual best tool for each job is?

From top to bottom, those are the most important things for any programmer to know about the code they’re writing. If you can truthfully answer “yes” to all those questions, then you are an excellent programmer.

It may seem like an overwhelming list. “Wow, the documentation for every single function? Reading that is going to take too long!” Well, you know what else takes a long time? Becoming a good programmer if you don’t read the documentation. You know how long it takes? Forever, because it never happens.

You will never become a good programmer simply by copying other people’s code and praying that it works right for you. But even more importantly, investing time into learning is what it takes to become good. Taking the time now will make you a much faster programmer later. If you spend a lot of time reading up on stuff for the first three months that you’re learning a new technology, you’ll probably be 10 times faster with it for the next 10 years than if you’d just dived into it and then never read anything at all.

I do want to put a certain limiter on that, though–you can’t just read for three months and expect to become a good programmer. First of all, that’s just too boring–nobody wants to just study theory for three months and not get any actual practice in. Very few people would keep up with that for long enough to become programmers at all, let alone good programmers. So I want to point out that understanding comes also from practice, not just from study. But without the study, understanding may never come. So it’s important to balance both the study and the practice of programming.

This is not an attack on any programmer that I’ve worked with personally, or even an attack on any individual programmer at all. I admire almost every programmer I’ve ever known, as a person, and I expect I’d admire the rest were I to meet them, as well. Instead, this is an open invitation to all programmers to open your mind to the thought that there might always be more to know, that both knowledge and practice are the key to skill, and that it’s not shameful at all to not know something–as long as you know that you don’t know it, and take the time to learn it when necessary.

-Max

Comments: 52

It’s not anything complicated, and it’s not something that’s hard to know. It’s not something that you can only do if you’re born with a special talent or a “magical ability to program well.” There is nothing about the nature of the individual that determines whether or not they will become an excellent programmer or a poor one.

There is only one, singular fact:

The better you understand what you are doing, the better you will do it.

Rockstar programmers understand what they are doing far, far better than average or mediocre programmers. And that is it.

All you have to do in order to become an excellent programmer is fully understand what you are doing.

Some may say that they already understand everything. The test is whether or not they can apply it. Do they produce beautifully architected systems that are a joy to maintain? Do they plow through programming problems at a rate almost unimaginable to the average programmer? Do they explain everything clearly and in simple concepts when they are asked for help? Then they are an excellent programmer, and they understand things well.

However, far more commonly than believing that they “know it all”, many programmers (including myself) often feel as though they are locked in an epic battle with an overwhelming sea of information. There is so much to know that one could literally spend the rest of his life studying and still come out with only 90% of all the data there is to know about computers.

The secret weapon in the epic battle, the Excalibur of computing knowledge, is understanding. The better that you understand the most fundamental level of your field, the easier it will be to learn the next level. The better you understand that level, the easier the next one after that will be, and so on. Then once you’ve gone from the most fundamental simplicities to the highest complexities, you can start all over again and find amazingly that there is so much more to know at the very, very bottom.

It seems almost too simple to be true, but it is in fact the case. The path to becoming an excellent programmer is simply full and complete understanding, starting with a profound grasp of the basics and working up to a solid control of the most advanced data available.

I won’t lie to you–it sometimes is a long path. But it is worthwhile. And at the end of it, you may find yourself suddenly the amazing senior engineer who everyone comes to for advice. You may be the incredible programmer who solves everything and is admired by all his peers. You might even come out a “rock star” with millions of dollars and a fantastically successful product. Who knows? I can’t tell you what to do or what to become. I can only point out some information that I’ve found to be truthful and valuable. What you do with it is up to you.

-Max

Comments: 32

I can solve this problem the right way.

Whatever the problem is, there’s always a right way to solve it. The right way can be known, and it can be implemented. The only valid reason ever to not implement something the right way is lack of resources. However, you should always consider that the right way does exist, you are able to solve the problem the right way, and that given enough resources, you would solve the problem the right way.

The “right way” usually means “the way that accounts for all reasonably possible future occurrences, even unknown and unimaginable occurrences.”

A bridge that could stand up to any reasonably possible environmental condition or any reasonably possible amount of traffic without constant maintenance would be built the “right way.”

Software code that maintained its simplicity while providing the flexibility needed for reasonably possible future enhancements would be designed the “right way.”

There are lots of invalid reasons for not solving a problem the right way:

• I don’t know the right way. Often this just requires more understanding or study, to figure out the right way. When I run into this situation, I walk away from the problem for a while, and then often I’ll come up with the solution when I’m just out walking, or the next day when I come back to it. I try not to compromise on something that isn’t the right way just because I don’t know what the right way is yet.
• The group cannot agree on what the right way would be. Sometimes a group of people have argued about what would be the “right way” and the subject has gotten very confused. Groups are not very good at making decisions. As we all know, you don’t design software by committee, and I suspect that “design by committee” in other fields of engineering is just as bad. The solution here is to assign an experienced and trusted engineer who understands the basic laws of the subject you’re working in to determine the right way by himself or herself, probably after carefully studying the existing arguments and collecting relevant information, following standard, valid engineering procedures.
• I am too lazy/tired/hungry/discombobulated to do this the right way, right now. This happens to everybody from time to time. It’s 1 in the morning, you’ve been working on the project for 15 hours straight, and you just need the damn thing to work, right now! Give it a rest, though, and come back later. The world isn’t ending, and the problem will still be here and solvable later. Go to sleep, go eat something, take a walk–do whatever it takes to get into a mental space where you’re willing to solve the problem the right way, and then come back. If you’re in a state where you can’t solve the problem the right way, then it’s really time to take a break. You’re not being delinquent in your duties if you do so–you’re actually correctly taking responsibility for the success of the project by saying “this needs to be done right, and the way to do it right, right now, is to take a break and come back later”.

Mostly, it all just takes the constant and continual belief in yourself that you can solve the problem the right way.

-Max

Comments: 38

Well, some of it’s really obvious. You press a button and the program takes 10 minutes to respond. That sucks pretty bad. You get 100 complaints a week about the UI of a particular page–okay, so that sucks.

Usually there are one or two HUGE things that really suck, and they’re really obvious–those are the things to focus on first, even if they require a tremendous amount of work. For example, before Bugzilla 3.0, Bugzilla had to compile every single library and the entire script it was about to run, every time you loaded a page. This added several seconds to each page load, on slower machines, and at least 1 second on faster machines. So performance was one big obvious thing that sucked about Bugzilla. But even more importantly, the code of Bugzilla sucked. It was being read by everybody–because people were frequently customizing Bugzilla at their company–and it was an unreadable, garbled mess.

Thankfully, both of those problems had the same solution. The performance problem was solved by allowing people to run Bugzilla in a way that would pre-compile all our scripts when the web server started, instead of every time somebody loads a page. And to enable that pre-compiling, we had to do enormous amounts of refactoring. So, we actually ended up handling our performance problem by handling our code problem.

However, it took four major releases (Bugzilla 2.18, 2.20, 2.22, and finally 3.0) to get all this done! We fixed a lot of little issues for each release along the way, too, so each release really did suck less than the previous one. But handling the major issues was a tremendous effort–it wasn’t just something we could code up in one night and have it be done with.

I think sometimes the big issues in a software project don’t get handled because they do require that much work to fix. That doesn’t mean you can ignore them, though. It just means that you have to plan for a long project, and figure out how you can keep getting releases out in the mean time.

After all that was fixed, then we could turn our attention elsewhere, and wow! It turned out that elsewhere, there were still a bunch of things that sucked! All of the sudden, there were a new batch of “totally obvious” things to fix–things that had been there all the time, but were just overshadowed by the previous set of “totally obvious” things.

Now, we could have just gone on like this forever–fixing one set of “totally obvious” problems and then going on to the next set of “totally obvious” problems. But we ran into an issue–what happens when suddenly, you get to the point where there there are fifty “totally obvious” things that need fixing, and you can’t get them all done for one release? Well, that’s when you suddenly need some method of prioritizing what you’re going to fix.

For the Bugzilla Project, there were two things that we did that really helped us prioritize: the Bugzilla Survey and the Bugzilla Usability Study.

With the survey, the most important part was allowing people to respond in free-form text, to questions asked to them personally. That is, I sent the questions from me personally to Bugzilla administrators personally, often customizing the message for their exact situation. And there were no multiple-choice questions, only questions that prompted them to tell us what was bothering them and what they wanted to see. They were actually really happy to get my emails–lots of them thanked me for just doing the survey.

Once they had all responded, I read everything and compiled a list of major issues that were reported–overall a surprisingly small list! We’re focusing on those issues pretty heavily nowadays, and I think it’s making people happier with Bugzilla in general.

With the usability study, surprisingly the most helpful part was when the researchers (who were usability experts) just sat down in front of Bugzilla and pointed out things that violated usability principles. That is, even more valuable than the actual research they did was just their observations as experts, using the standard principles of usability engineering. The fact that they were fresh eyes–people who’d never worked on Bugzilla and thus didn’t just think “well that’s the way it is”–also was important, I think.

So we took all this data, and it really helped us prioritize. However, it’s important that we did the survey and research when we did them and not earlier. Back before we fixed the top few major issues, the usability and survey results would have just been overwhelming to us–they would have pointed out a million things we already knew, or a lot of things that we just didn’t have the time to work on at that point, and we would have had to re-do the survey and research again later, making it all a bunch of wasted time. So we had to wait until we were at the point of asking ourselves, “Okay, what’s most important now?”, and that was when gathering data became tremendously important and incredibly useful.

So overall, I’d say that when you’re trying to make things suck less, first go with what you know are the top few big obvious issues, and handle those, no matter what it takes. Then things will calm down a little, and you’ll have a huge pile of stuff that all needs fixing. That’s when you go and gather data from your users, and work to fix whatever they tell you actually sucks.

-Max

Comments: 4

But eventually, thanks to a bunch of new members in the Bugzilla community, we released Bugzilla 2.18. Hooray! Bells rang, birds sang, and there was much rejoicing.

However, in the space between Bugzilla 2.16 (which was before my time) and Bugzilla 2.18 (which was the first release that I helped get out), something very strange happened–we developed serious competition. All of the sudden there were a bunch of new bug-tracking systems, some of them open-source and some of them not, that people were actually using.

At first it wasn’t too worrisome. Bugzilla was pretty dominant in its field, and it’s hard to lose that kind of position. But as time went on, there was more and more competition, and some people were predicting doom and gloom for Bugzilla. We were a tiny group of completely unpaid volunteers, and some of these competing products were being made by companies whose marketing and development resources absolutely dwarfed us.

And yet, with every release, our download numbers kept going up. And always significantly–30-50% more than the previous release, every time.

And then we hit Bugzilla 3.0, and our download numbers nearly doubled. And they’ve kept going up with every release from there. Nowadays we get over 10 times the number of downloads per release that we did in 2004.

So how did we pull this off? Well, as far as I can tell:

All you have to do to succeed in software is to consistently suck less with every release.

Nobody would say that Bugzilla 2.18 was awesome, but everybody would say that it sucked less than Bugzilla 2.16 did. Bugzilla 2.20 wasn’t perfect, but without a doubt, it sucked less than Bugzilla 2.18. And then Bugzilla 3.0 fixed a whole lot of sucking in Bugzilla, and it got a whole lot more downloads.

Why is it that this worked?

Well, when people are deciding at first what software to use, they have varying criteria. Sometimes they just use what’s presented to them by default on the computer. Sometimes they have a whole list of requirements and they do lots of research and pick the software that has all the features they need. But once they’ve picked a program, they will stick with it unless there is some compelling reason to leave. It’s not like people constantly are looking for new software to replace yours–they only start looking when your software just won’t stop sucking.

As long as you consistently suck less with every release, you will retain most of your users. You’re fixing the things that bother them, so there’s no reason for them to switch away. Even if you didn’t fix everything in this release, if you sucked less, your users will have faith that eventually, the things that bother them will be fixed. New users will find your software, and they’ll stick with it too. And in this way, your user count will increase steadily over time.

You have to get out releases frequently enough that people believe that you really will suck less, of course. If your new release never comes out, then effectively, your current release never stops sucking.

But what happens if you do release frequently, but instead of fixing the things in your software that suck, you just add new features that don’t fix the sucking? Well, eventually the patience of the individual user is going to run out. They’re not going to wait forever for your software to stop sucking.

I remember a particular piece of software that I used every day for years. It had a lot of great features and a nice user interface, but it it would crash two or three times a week. I really liked the software in general, but man, the crashing sucked. I reported a bug about it, and the bug was ignored. I kept using it through 10 new releases, and it still crashed. The upgrades brought lots of new features, but I didn’t care about them. Remember, the feature set only mattered to me when I first picked the software. Now I just needed it to suck less.

But it never did.

So eventually, I went and looked for another piece of software that did the same thing, switched over, and I was happy with that one for a while. But guess what? It had a bug that really sucked. It didn’t happen very often, but when it did, boy was it a problem. But it sucked less than my old software, so I kept using it. Until one day, my patience ran out (after maybe 7 upgrades of the software), and I switched again.

Now I’m using a program that has half the feature set of either of the previous two programs. But, as a user, I’m the happiest I’ve ever been with this type of software. Because you know what? My new program sucks hardly at all. I mean, there are little things about it that suck, but supposedly a new release is coming out soon that will fix some of that sucking, and so I’m okay with it for now.

Would I have guessed this secret of success before I started working on Bugzilla? No. I would have told you the traditional wisdom–that a product succeeds or fails based on its feature set and user interface. But after 5 years on this project, managing our releases and watching our download count, I can tell you from my factual experience this strange thing: all you have to do to succeed as a software project is to suck less with every release. It doesn’t matter how much competition you have, or how many buzzword-friendly features you can cram into your interface. Just suck less, and you’ll succeed.

-Max

Comments: 57

Why do over 75% of Facebook’s users think that the new Facebook UI is bad? Because it makes different things look similar to each other. Nobody can tell if they’re updating their status or writing on somebody else’s wall, because even though the text is slightly different in the box depending on what you’re doing, the box itself looks the same. The new Chat UI (introduced a few days ago) makes idle users look basically identical to active users, except for a tiny icon difference. (It’s also important that different things are different enough, not just a little different, because people often won’t notice little differences.)

This is an easy pitfall for developers to fall into because developers love consistency. Everything should be based on a single framework, in the backend of an application. But that doesn’t mean that everything has to be displayed the same in the UI.

This fact–that different things should look different–is actually true with code, too, but people rarely think about it, because developers are actually pretty good about it. For example, accessing a value of an object should look different than calling a method on it, and in most programs, it does. For example, in Bugzilla’s code, accessing a value on an object looks like $object->value whereas calling a method on the object looks like $object->method(). It’s not all that different, but the () at the end is enough difference for the average programmer to notice “Oh, that’s a method call that does something–it’s not just accessing a value in the object.”

All in all, consistency is really important in both the backend and the frontend of an application. But that doesn’t mean that every single thing should look exactly the same–if we took that to extremes, we’d just have a solid white page, and that doesn’t seem all that usable (frontend) or readable (backend), does it?

Comments: 7

Twitter’s just one of the many proofs that you don’t have to have lots of features to be successful. In fact, many successful apps have fewer features than their less-successful competitors.

Still, you’ve got to have some features. After all, it’d be pretty silly to be programming, otherwise. But how do you decide which features you should have? Is it just up to the Chief Architect’s intuition, or how many users demand that you give them “feature X”? Does whoever shouts the loudest in the development meeting get their feature implemented first?

Well, no, there is a way to decide whether or not you should implement a feature, and it comes out of one of our most basic principles: the purpose of software. This principle (that the purpose of software is “to help people”) isn’t just some fancy-sounding gibberish I made up to make myself happy–I wrote it because it’s something that can actually be really useful to think about in everyday programming, and this question of “Should we implement this feature?” gives us a great opportunity to show how it can be applied.

So, let’s say that the purpose of your software is “to help people learn to type,” and you have to decide, “Should we add New Feature X?” Well, here’s what you can ask yourself about the feature: “Does this help people learn to type?” Simple. If the answer is no, then you don’t implement the feature. Also, “Does this harm people learning to type, or get in the way of learning to type?” If it harms people more than it helps them, then it shouldn’t be implemented.

Now, quite often, you’ll find that you can’t really say whether or not a proposed feature actually helps your users (or helps them more than it hinders them). You can’t stand there with certainty and say, “Yes, this absolutely helps people learn to type.” In that case, just skip it. Don’t implement the feature. There are plenty of features out there in the world where you can be sure that it will help your users–implement those instead. Perhaps some more data will come along in time that will make your decision clearer about this uncertain feature, but until then, you don’t have to implement it. Just wait (or go collect data to make your decision easier).

This whole idea is also very handy for prioritizing features. You ask yourself, “Out of all these features, which ones will most help people learn to type?” That sounds perhaps like an overly-simple suggestion, but I think that for Bugzilla, if we really looked at all our requested features and said, “Which one of these will most help people track bugs?” we’d come up with a pretty good priority list, and probably have some interesting discussions, too.

The higher you set the bar for certainty on these questions, the fewer features you will have, and the simpler your application will be. That is, you could say, “We have to only be kinda certain it will help people,” and that’s your standard (or “bar”) for helpfulness. Or you could say, “We only implement features that we are totally certain will help people,” and you’d probably have an enormously simple and very well-designed application. But it’s a judgment call–sometimes it’s hard to get data, and you have to implement features that you’re only pretty sure will be helpful. That can be the right decision, though, sometimes–different situations require different solutions.

Overall, how you make these decisions is up to you. What really matters is that you ask the questions. That’s one of the keys to keeping your product focused, simple, and polished.

-Max

Comments: 5

Now, some people misunderstood me and said, “Oh, SAR is just another name for MVC.” No, I used MVC as an example of SAR, but SAR is a much, much broader concept than MVC–they are not comparable theories. MVC is a pattern for designing software, whereas SAR (or ISAR) is a statement of the three (or four) components that are present in all software.

The fascinating thing about SAR is that it applies not only to a whole program, but also to any piece of that program. A whole program has a Structure, just as a function or single line of code has a Structure. Same for Action and Results.

Here’s a little more about each of the pieces, and some examples to help explain:

Structure

Here are some examples of things that would be considered “Structure” for the whole program:

• The directory layout of your code.
• All of the classes and how they relate to each other.
• The structure (schema) of the database, if your program uses a database.

  Note here that the actual data in the database isn’t part of the Structure, though. If your program is producing the data and then sticking it into the database, then that’s part of the Result. If the data is sitting in the database and your program is supposed to process it, then that data is part of the Input.

Then an individual class (and I mean a “class” in the object-oriented sense) would also have a Structure:

• The names of methods in the class and the types/names of parameters that they take.
• The names and types of variables (member variables) in the class.

Whether or not a function (or variable) is private or public would also be part of the Structure, because Structure describes what something is (as opposed to what it does or produces), and “private” or “public” are words that describe what something is.

A Structure is sort of “the components of the program” or “the pieces you make the program out of.” Function names and types, variable names and types, classes–these things are all Structure.

Structure just “sits there.” It doesn’t do anything unless there’s some part of your program that uses it. For example, a method doesn’t call itself, it just sits there waiting to be called. A variable doesn’t put data into itself, it just sits there waiting for you to do something with it.

Action

The Action of a whole program is very easy to understand. A tax program “does taxes.” A calculator program “does math.”

An Action is always a verb of some sort. “Calculates.” “Fixes.” “Adds.” “Removes.” Those are actions. Usually they’re a little more descriptive and specific, though, like, “Calculates how much rainfall there will be in Africa next year,” or “Fixes broken hard drives.”

Inside of a class, the Action is the code inside of the methods. That’s all some sort of action–something going on, something happening. In many programming languages, you can also have code outside of any class or function–code that just runs when you start the program. That’s Action.

Results

Every program, every function, and every line of code has some effect. It produces some result.

A Result can always be talked about in the past tense–it’s something that has been done or created. “Calculated rainfall,” or “Fixed hard drives.” In a tax program, we’d call the Result either filed taxes or filled-out tax forms. As you can see, it sounds a lot like the Action, just completed.

You don’t have to describe a Result in the past tense, though. I’m just saying it always can be described that way. For example, in a calculator program, normally we’d call the Result of addition “the sum,” (not past-tense, just a noun) but you could also say that the Result is “added numbers” (which is past-tense). Same thing, just a different way of describing it.

Individual pieces of your program have Results, too. When you call a method or function, it has a very specific Result. It gives you back some data, or it causes some data to be changed.

Whatever your program (or any part of your program) produces, that’s the Result.

ISAR in a Single Line of Code

So, I said that SAR applies to a single line of code, but I didn’t give you any examples. So here’s a single line of code:

x = y + z

y and z in that line are part of the Structure. They’re variables that hold some data. To make an analogy: A jug is a structure that holds water. A variable is a structure that holds data.

The numbers that are stored inside y and z are the Input. That’s the data that we’re doing something with.

+ is an Action: “Add these two numbers.” = is also an Action: “Store the result in x.”

And, of course, the Result is the sum of y and z that gets stored in x. If y is 1 and z is 2, then the Result is the number 3, which gets stored in x. (Also note that x is a itself variable and thus also part of the Structure, but that’s getting pretty technical.)

Wrapping It Up

So anyhow, I hope that explains SAR a bit better. It’s a concept that applies to any kind of programming, whether you’re building a big application or just writing a single-line script. And it’s not something that you have to think about in-depth every time you write a piece of code, but it’s something that can help us analyze and understand a program, particularly when we’re looking at how we can improve its design.

-Max

Comments: 4

In the MVC sense, the Model is the Structure, the Controller is what does the Actions, and the View is the Result. I think the analogy (and the words) Structure, Action, and Results are more widely and accurately applicable to the operation of every program in existence, though, moreso than MVC, though MVC is a perfectly good way of looking at it for GUI applications.

Really, Structure, Action, and Results probably describes almost any machine in existence. A machine has some parts that don’t move, a framework–that’s the structure. Some parts move and do something–that motion is the action. And of course the machine produces something (otherwise we wouldn’t care much about it) so that’s the result.

Computer programs are unusual machines in that they can modify their own structure. However, it’s important that some part of the program be stable, that they “not move” in a logical sense. The way that object classes relate to each other, the names of methods and variables–these are all parts of the structure that usually don’t change while you’re running. (Sometimes you make new classes, methods, or variables while you’re running, but they usually follow some pre-set plan, so there’s still a lot of “not moving” involved.)

When I’m writing software, I usually build the Structure first, then I work on the Actions, and then I work on the displaying of the Result. Some people work backwards from the Results, that’s fine too. Probably the only inadvisable thing to do is to start with the Actions, since it’s kind of confusing to be performing Actions without a Structure and with no defined Result.

There’s so much to this concept that I could probably write a whole book just on this one topic, but I think this is a decent introduction, and I’m sure that given this, you can think of lots of other useful applications of it.

-Max

Comments: 0

When we think about software security, the first question that we ask is, “How many different ways could this program possibly be attacked?” That is, how many “ways in” are there? It’s a bit like asking “How many doors and windows are there on this building?” If your building has 1 exterior door, it’s very easy to protect that door. If it has 1000, it will be impossible to keep the building secure, no matter how good the doors are or how many security guards you have.

So we need to limit the “ways in” to our software to some reasonable number, or it won’t ever be secure. That’s accomplished by making the overall system relatively simple, or breaking it down into very simple and totally separate component parts.

Then, once we’ve limited the ways in, we need to start thinking about “How many different possible attacks are there against each way in?” We limit that by making the ways in themselves very simple. Like a door with only one unique key, instead of a door that can take five different keys, all of which individually will open the door.

Once that’s done, we limit how much damage any attack could do if it got through. For example, in a building, we’d make any given door only allow access to one room.

All of this explains, for example, why Windows is fundamentally flawed and will never be secure, and why UNIX-based systems have a better reputation for security.

Standard UNIX has a very small number of system calls that are used to implement the vast majority of all UNIX programs out there. (Even the extended list is only about 140 system calls, though most of those are never used by the average program.) Each system call is extremely specific and does one very limited thing.

Windows, on the other hand, has a ridiculous set of system calls that are confusing, take too many arguments, and do too much.

Going up to a higher level in the system, the Windows API is massive and complex. It’s a strange beast that controls both the OS and the GUI. There’s really no equivalent thing in UNIX (because the OS and the GUI are separate), but we can at least compare parts of it. Here’s The Windows Logging API. Here’s the Linux Logging API. There’s just no comparison. It’s like a joke. There’s so many “ways in” to any part of Windows that it will never be fundamentally secure.

You might say, “Well, I haven’t had a virus on my Windows machine in a long time.” That’s not what I’m talking about–I’m talking about fundamental security. In order to have a secure Windows machine, you have to have a firewall that asks you every time a program wants to make an outbound connection. You have to have a spyware scanner. You have to have antivirus software that slows down your computer by as much as 2000%. If Windows was secure, you wouldn’t need those things.

When we design our own systems, keeping them simple is the only real guarantee of security. We keep each “way in” to the system as simple as possible, and we never add more “ways in” than we absolutely need. These are compatible things, too, because the simpler each “way in” is, the fewer we’ll actually need. That may not make sense until you think about it this way: If all actions on the system can be reduced to, say, 13 fundamental function calls, then the user can do everything with those 13 calls, even if they’re not very powerful individually. If instead we only let them do 100 different specific tasks, and don’t allow them to use the 13 fundamental calls, we have to add a new function for every specific task.

There are lots of other “ways in” to a program than just its public API, too. How the user interface interacts with the backend–that involves various “ways in”. Can we access this program’s internal structure from another program? That would be another “way in.” There’s lots of ways to apply this principle.

Any way you slice it, though, the best way to get real security in things is simplicity. We shouldn’t have to put a small army in front of our software just to keep it secure. It should just fundamentally have so few “ways in” that it doesn’t need the protection, and those “ways in” should be so streamlined and simple that they’re impossible to exploit.

-Max

Comments: 14

But what is it that makes all that stuff a computer? Why do we look at it and go, “Oh yeah, that’s a computer,” as opposed to, say, “Oh, that’s just a TV,” or “That’s where I keep the leprechauns at night.”?

Some people try to define the word “computer” just by saying “it’s got such and such parts and they all work this way,” but that’s like saying “airplanes have two wings and jet engines.” It’s true, but I could build an airplane that didn’t have two wings or jet engines. The way something works is not a definition for that thing.

Others try to define it mathematically, but that can also be somewhat limiting, because then only the devices that fit into your mathematical scheme are computers, and there are multiple mathematical models that would all be considered “computers.”

So I turned to the dictionary. That was fun for me–I’m a dictionary fanatic. I’ve got lots of great dictionaries, and there are even more online. The Compact Oxford English Dictionary had the best definition, as it turned out.. I was very happy with it at first, but when I started to think about it, it didn’t quite work. For example, it calls computers “an electronic device,” and we know that computers can be built without electronics.

So I worked to come up with a definition of my own. Strangely enough, the key question that it boiled down to was “Why is a player piano not a computer?” It “processes information” by playing notes from its roll. If you gave it an etching machine, it could “store information” back on to the roll. But despite all that, it’s clearly not a computer. What is a computer doing that is fundamentally different from a player piano, that a player piano could never do?

After about two years, I finally came up with an answer that was both simple and all-encompassing. A computer is:

Any piece of matter which can carry out symbolic instructions and compare data in assistance of a human goal.

And that, my friends, is really it. My only thought left is whether I should say “a series of symbolic instructions” to more clearly differentiate it from a calculator. What do you think?

-Max

Comments: 35

10. Dudes at Silicon Valley parties will think you’re, like, cool.

    Impress people you don’t know! You can say, “I work on an open source project,” and nod your head like you’re cool. But no, more seriously, about 50% of all the people I meet at Silicon Valley parties are totally amazed that I’m one of the primary developers of Bugzilla, something that they use every day.

    This doesn’t work so well with the ladies, though, usually. “Yeah, I’m an extremely intense programmer, oh yeah.”

    No, but more seriously: How many girls did you see at OSCON? If anybody has a scheme to get more girls involved or interested in open source, the whole open-source world (the current girls included, I’m sure) would be thrilled.

9. It totally has nothing to do with whether or not you’re a “hottie.” Just be yourself, man, just be yourself.

   Open Source is definitely one place where you’ll be respected for your intelligence and ability, not how expensive your clothes are or how much you paid for your haircut. Nobody cares what you look like. We only care how good your ideas are.

8. It’s hella sick when you’re interviewing!

   When you’re interviewing and you worked on some open-source project, it’s completely valid job experience. Sometimes it even makes you more valuable than normal job experience, if the project is well-known. Also, if you worked on open-source in your spare time, that shows the kind of passion for software engineering that employers are really looking for.

7. You can, like, totally freelance. Seriously? Ya, like totally seriously.

   There’s a lot of need for contractors for certain open-source projects. If you become a prominent contributor to a project and get your name on its Consultants List, then you can make a living doing consulting for people who use the software!

6. Dude, we’ll like, laugh at your jokes and stuff, dude.

   I have laughed harder while listening to some conversations on IRC than I have at almost anything else in the world. There’s something about shared experience and understanding that makes things much funnier than jokes that “everybody” is supposed to get. And where else can you make jokes about programming languages and have multiple people actually get them?

5. You can say how, like, stuff goes and people will actually listen to you. Whoa.

   When you write a feature, to a large degree you’re the one who’s going to decide how it works! Don’t like how a particular program works? Well, maybe you could have been the one who wrote that feature instead! Don’t like some documentation? What if you had written it instead?

   And if you don’t like how something works now, the more you contribute to an open-source project, the more say you’ll have in fixing that thing. Have an itch to just make things work right? Open Source is the place to be.

4. You learn, like, so much stuff, like seriously.

   Are you looking for something new, some way to expand your horizons and learn something new instead of just mechanically doing the same thing over and over at work? No matter what your interest is, there’s going to be some open-source project out there that uses the things you want to learn about. And it won’t just be some tiny project just for yourself, but something that people really use!

3. You can like, belong to something, and stuff. That’s some cosmic stuff, man.

   When you contribute to an open-source project, you’re not just a cog in a great machine, or just a worker at a job. You become part of a larger community, with its own in-jokes, culture, and people.

   I used to think that was just some marketing gibberish, but it’s really true. It may not be the best way to find people who you can “hang out” with every day, but you’ll get to know a lot of new people and become part of a group in a very definite way.

   It’s even more true if you go to conventions like OSCON or the more specific ones for the various open-source projects, where you can meet and hang out with lots of other developers “In Real Life”, most of whom are really great (and intelligent) people.

2. You get to feel like a revolutionary (a revolutionary nerd, but…hey, s’all good, s’all good).

   Yeah! Down with The Man! We don’t need any stinkin’ proprietary software!

   You can even get to protest things, like software patents! You’ll be almost as cool as your parents were in the 60′s. Kind of.

   No, but seriously: Open Source is still a relatively new thing in the world, and you can be a part of blazing its trail. It’s not the “normal” way to do things quite just yet, so if you like being a little different than the swarming masses, open source is the place to be.

1. It seriously helps out, man.

   Working on open-source software helps out a lot of people:

   • The people who use the software. Millions of people use open-source software around the world every day to do their job, handle their problems, or just have fun! You could be affecting the lives of all those people.
   • The people who write the software. Open-source projects almost all really want your assistance! If a project is at all popular, they probably have more feature requests than they can handle. Don’t come to an open-source project and say, “This is what I’m going to do for you,” but do come and say, “How can I help you guys out?” We all need some help, and competent help is much appreciated.
   • All the people who won’t have to write the software that you write. Sometimes you just want to download a program to do a task for you. You don’t want to have to write a program for everything you want to do. You’re saving the time of millions of users and computer programmers, by creating something that everybody can use and modify!

And that’s my list! Hope you had totally chill time, brah.

-Max

Comments: 30

Everybody wants to “innovate.” The news talks about “who’s being the most innovative.” Marketing for companies insists that they are “innovating.”

Except actually, it’s not innovation that leads to success. It’s execution.

It doesn’t matter how good or how new my idea is. It matters how well I carry it out in the real world.

Now, our history books worship the inventors, not the executors. We are taught all about the people who invent new things, come up with new ideas, and plough new trails. But look around you in present time and in the recent past, and you’ll see that the most successful people are the ones who carried out the idea really well, not the people who came up with the idea.

Elvis didn’t invent rock and roll. Ford didn’t invent the automobile or the assembly line. Apple didn’t invent the GUI. Webster didn’t invent dictionaries. Maytag didn’t invent the washing machine. Google didn’t invent web searching. I could go on and on and on.

Granted, sometimes the innovator also is an excellent executor (Alexander Graham Bell being an example), but usually that’s not the case. Most inventors don’t turn out to be the most successful people in their field (or even successful at all).

So stop worrying about “coming up with something new.” You don’t have to do that. You just have to execute an already existing idea really, really well. You can add your own flair to it, maybe, or fix it up a little, but you don’t have to have something brand new.

There are so many examples that prove this that it’s hard not to see one if you move your eyes anywhere. Just look, you’ll see.

Now, I’m not saying that people shouldn’t innovate. You should! It’s fun, and it advances the whole human race a tiny step every time you do. But it’s not the path to long-term success for you or for any group you belong to. That’s all in execution.

-Max

Comments: 39

One of the features of this web browser is its JavaScript engine, called v8, which is designed for performance.

Designing for performance is something that Google does often. Now, designing for performance usually leads to complexity. So, being a major supporter of software simplicity, I’m opposed, in a theoretical sense, to designing for performance.

However, Google is in an interesting situation. Essentially, we live in the Bronze Age of computing (or perhaps the Silicon Age, as I suspect future historians may call this period of history). Our computers are primitive, compared to what we are likely to have 50 to 100 (or 1000!) years from now. That may seem hard to believe, but it’s always hard to imagine the far future. Google is operating on a level far exceeding our current hardware technology, really, and so their design methods unfortunately can’t live in a theoretical fairy-land where hardware is always “good enough.” (However, I personally like to live in that land, when possible, because hardware will improve as time goes on–an important fact to understand if you’re going to have a long-lived software project).

What is it about our computers that makes them so primitive? Well, usually I don’t go about predicting the future in this blog, or even talking about too many specifics, because I want to keep things generally applicable (and also because the future is hard to predict, particularly the far future). But I will talk about some of my thoughts here on this, and you can agree with them or not, as you please.

Our Current Computers

First, you have to understand that to me, anything that could be running this web browser right now, that could be showing me my desktop, and that I could be typing into right now–I’d consider that a computer. It actually doesn’t matter what it’s doing underneath, or how it works. A good, offhand definition then of a computer would be:

Any piece of matter which can carry out symbolic instructions and compare data in assistance of a human goal.

Currently computers do this using math, which they do with transistors, digitally. The word “digital” comes from “digit”, a word meaning, basically, “fingers or toes.” For most people, your fingers and toes are separate, individual items. They don’t blend into each other. “Digitally” means, basically, “done with separate, individual numbers.” You know, like 1, 2, 3, 4–not 1.1, 1.2, 1.3. People (normally) have 1 or 2 fingers, not 1.5 fingers.

Said another way, current computers change from one fixed state to another, very fast. They follow instructions that change their state. I don’t really care if we say that the state is “transistor one is on, transistor two is off, transistor three is on…” or “Bob has a cat, Mary has a dog, Jim has a cat…” it’s all a description of a state. What we care about ultimately is the total current state. If there are 1,000,000 possible states (and there are far, far, far more in a current computer), then we can say that “we are at state 10,456″ and that’s all we really need to know.

The problem with current computers is Moore’s Law. We don’t need computers that are twice as fast. We need computers that are about a million times faster than the ones we have. We need computers so fast that software engineers never have to worry about performance ever again, and can just design their code to be the sanest, most maintainable thing possible. With that kind of performance, we could design almost any software imaginable.

The problem is that with Moore’s Law, we’re not going to get computers 1000 times faster for about 20 years. We’re going to get to 1,000,000 times faster in about 40 years. And there’s a chance that the laws of physics will stop us dead in our tracks before that point, anyhow.

Future Computers

So, let’s stick with the idea of a machine that changes states for the near future, because I can’t think of any other clever way to make a computer that would follow my definition from above. There are three problems, then:

1. How many states can we represent?
2. How many physical resources does it require to represent all those states (including space, power, etc.)?
3. How quickly can we change between states?

And then “How many states can we represent at once?” might also be a good question–we’re seeing this come up more and more with dual-core and quad-core processors (and other technologies before that, but I don’t want to assume that everybody reading my blog is an expert in hardware architecture, and I don’t want to explain those technologies).

So the ideal answers are:

1. We can represent an infinite number of states.
2. It requires no physical resources to represent them.
3. We can change between them without time.

And then also “We can represent an infinite number of different states at once.”

Currently we theoretically could represent an infinite number of states, we’d just have to add more transistors to our chip. So really, the question becomes, “How many states can we represent with how many physical resources?” Currently we can fit two states into 32 nanometers. (That’s one transistor.)

My suspicion is that the future is not in fitting two states into a continually smaller space, but in fitting a near-infinite number of states into a slightly larger space. Electricity and other force waves can be “on” or “off”, but they also have lots of other properties, many of which are sufficient to represent an infinity (or near-infinity). Frequency of any wave represents an infinity, for example–you can have a 1 Hz wave, a 1.1 Hz wave, a 1.15 Hz wave, a 1.151 Hz wave, etc. So, that basically answers Question 1 ideally–you can have an infinite number of states, you just have to have some device which is sufficiently small, in which a wave can have its properties modified and measured by electronics, optics, or some other such technology.

You’ll notice that we’ve also conveniently answered our bonus question, because we can represent quite a few different states at once, once each individual component of our system can represent an infinity all by itself.

If we want to look a bit further into the future, our second question can be answered by the fact that waves take up essentially no space (only the medium that they vibrate takes up space). Our understanding of physics is not (as far as I know) currently good enough to create structures out of pure force just yet, but such structures would come quite close to taking up “no physical resources.”

And beyond that (how we get the state changes to happen without time), I have no idea. That question may be unanswerable, and may only be resolvable by changing computers to being something other than mathematical devices. (That is, not be involved with states at all, but some other method of following instructions and comparing data.) But the better our components become, the closer we can get to “no time.”

The Roundup

So there’s my thoughts for the day on the future of computing. Sometimes designing software for performance is a necessary evil (but really only in the case where it’s an extreme issue, like with Google’s products, or the great new need for speed in JavaScript nowadays, or in other low-level places), but I hope that future changes in the fundamental architecture of computers will obsolete that necessity.

-Max

Comments: 14

My policy on projects that I control is that we never add a feature unless the design can support it simply. This drives some people crazy, notably people who have no concept of the future. They start to foam at the mouth and say things like, “We can’t wait! This feature is so important!” or “Just put it in now and we’ll just clean it up later!” They don’t realize that this is their normal attitude. They’re going to say the same thing about the next feature. If you give in to them, then all of your code will be poorly designed and much too complex. It’ll be Frankenstein’s monster, jammed together out of broken parts. And just like the friendly green giant, it’ll be big, ugly, unstable, and harmful to your health.

Adding a tiny little piece and refactoring it afterward is fine. Landing a huge feature that the architecture can’t support and then trying to clean it up afterward is a terrible task. Size matters.

The worst situation, however, is when you let people keep adding features with no design for months or years, and then one day you wake up and realize that something is not right. Now you have to fix your whole codebase. This is a terrible task, because just like adding a new feature, it can’t be done all at once, unless you want to re-write. If you want to start doing things the right way, you have to start doing things the right way. And that means that you have to fix the design piece by piece, in simple steps. That usually requires months or years of effort–totally wasted effort, because you should have just designed from the start. You should have thought about the future.

If your project lacks a strict design, and it continues to grow, then you will eventually end up over your head in complexity. I understand that this is hard for some people to imagine. Some folks can’t imagine that there is a future beyond lunch. Other folks just haven’t had enough experience to understand how complex things can get. And I understand that there can be a corporate culture that says, “Oh, we just hack in new features, and we should do things the right way, but we can’t because blah blah blah.” But one day your project will fail. And no matter how many reasons you can give for that failure, it won’t change the fact that your project failed.

Often, when you’ve done your design right, there’s not a whole lot of credit that comes your way. Catastrophic failures in design are big and noticeable, small increments of work toward a good design are invisible to people who aren’t intimately connected with the code. So this can make it difficult–handling a big failure gets you a lot of thanks, preventing one in the first place, well, nobody noticed.

So I’ll congratulate you myself. Did you design from the start? That was awesome. You absolutely did the right thing. Have you started designing now? Well, you should have started earlier, but congratulations on starting to move in the right direction. Your users and fellow developers will see the benefits–working software, on-time releases, and a clear, understandable codebase. Will they know how much work it took to get it that way? Maybe not. But that’s OK. Sometimes doing things the right way is really its own reward.

-Max

Comments: 8

Imagine that you are building a structure out of lead bars. The final structure will look like this:

   |
_|_|_|_
   |
   |
   |

You have to build the structure and put it up at a certain location, so that people can use it (for this example, we don’t care what they need it for, but assume that they need it for some specific reason).

The lead bars represent the individual pieces of your software. Putting it up at the location is like putting your software into production (or sending it out to your users). Everything else should be fairly clear as to how it translates to software, if you think about it. You don’t have to translate everything to software in your mind as you read, though. Everything should be quite clear if you just imagine that you really are just building a structure out of lead bars.

The Wrong Way

Imagine that you were building this all by yourself, and that you had to make the bars out of raw metal. Here’s the wrong way to build it:

1. Make one tall lead bar, and lay it flat on the ground in your workshop:

   |
   |
   |
   |
   |

2. Cut a hole through the tall bar, and measure that hole.
3. Make a new bar that will fit through that hole:

   _____

4. Put that new bar through the hole and weld them together permanently:

     |
   __|__
     |
     |
     |

5. Cut two holes in the horizontal bar, measure them, and make two new lead bars that will fit in those individual holes:

   |   |

6. Insert the two bars into the horizontal bar, and weld them together permanently:

      |
   _|_|_|_
      |
      |
      |

7. With a forklift, put this into a truck to move it to the location where it’s supposed to be (It’s too heavy to move by yourself.)
8. With a pulley, make the construction stand upright and put it into the ground.
9. Discover that it won’t stay up by itself, but if you put some blocks next to it as an emergency solution, it doesn’t fall over:

      |
   _|_|_|_
      |
     _|_
    | | |

10. Three days later, watch the structure fall over and break because the blocks aren’t actually a permanent solution.
11. Unfortunately, part of the horizontal bar has snapped, and you have to fix it. This is difficult because the bars are all welded together, so you can’t easily take out the bar and replace it with another one. You either have to build a whole new structure or weld together the broken bar. Welding the broken halves together creates a weak bond, but it’s cheaper than building a whole new structure, so you just weld them.
12. Put stronger blocks next to the structure to keep it up.
13. Next week, the weather breaks the welded bars. Weld them back together again.
14. In six days, watch the structure fall over because blocks are not a permanent solution.
15. Repeat the last few steps until you run out of money or time.

Analysis of The Wrong Way

So, what was good about the above process? Well, it did allow one person to successfully complete a structure. In software terms, that one person “made something that works.” It also created a lot of work for one person, which is good if that one person wanted a lot of work.

What was bad about it?

• The bars all had to be made in sequence, individually.
• Problems with the final structure (that it wouldn’t stay up) were only discovered after it was entirely built and in place.
• When problems were discovered, they were just “quick fixed” without planning for the future.
• It took enormous effort to move the completed structure into place.
• If we ever had to change the configuration of the bars, we couldn’t, because they’re welded together. We’d have to build a whole new structure.
• The completed structure requires frequent attention.

And I’m sure we could come up with other faults. This whole analogy (including the parts below) could be analyzed all day.

Bringing It To a Group

The biggest problem with the Wrong Way process is that it wouldn’t work at all if there were multiple people working on the project (as there usually are in real-world software projects). The main problem is that you had to measure all the holes before you built a bar, so everything had to be done by one person, in order.

There are, generally, two ways to solve this problem:

1. Write a specification for the sizes of all the individual holes beforehand, and then spread out the work of making all the different bars for each hole.

   This is problematic because one person has to write this specification, and if this were a large project (imagine thousands of holes instead of just three or four), it would take a lot of time. Nobody else on the team can be working until the specification is completed. The spec could be full of mistakes–there are as many chances for mistakes as there are holes specified, so if there are thousands of holes, that’s a lot of chances for errors to be made.

2. Just say, “All bar holes will always be the same size and in the same places on the bars. Bars can be screwed together.” Then set everybody to making bars with standardized holes (or go buy them from the store).

   That is simple, and it gets everybody working at once. Because you’ve standardized your bars, you’ve lost a little flexibility in dealing with any special cases that come up (maybe a half-width hole would be more useful in some part of the structure). However, you should be able to build a decent structure entirely with standard holes, so that’s not too much of a problem. And when you have a standard, you can make specific exceptions in some places more easily than if things are not standardized.

   Of course, with this method it is very important that you do a little research to pick a good hole size and good bars.

The Right Way

So, what would our process look like for many people all using standardized bars that screw together? (This is the right way to build something.)

1. Have your team all go build (or buy) their individual bars. You can have as many people working simultaneously as there are bars in the structure.
2. Have them test their individual bars to make sure that they won’t break.
3. Have them carry their individual bars to the place where the structure needs to be.
4. Put the first bar into the ground, standing upright:

   |

5. Push on the first bar from all angles to see if it is going to fall over.
6. Screw in a second bar to the first one:

   |
   |

7. Test the complete structure now, only to find that it’s not strong enough to stand by itself.
8. Attach unbreakable steel ropes to the sides of the structure, like so:

     /|\
    / | \

   These ropes should be able to withstand anything within reason, or even well beyond reason.

9. Test it again and find out that it now can stay up no matter how hard you push on it.
10. Add a third bar, and put new ropes on so that it looks like this:

       /|\
      //|\\
     // | \\

11. Remove the lower ropes:

       /|\
      / | \
     /  |  \

    (Anybody who’s been involved in refactoring Bugzilla can remember doing a lot of things that sound just like these last two steps. )

12. Test it again.
13. Continue these steps until you have a completed structure:

          |
       _|_|_|_
      /   |   \
     /    |    \
    /     |     \

14. When a pipe breaks in three months, figure out what was wrong with that pipe, fix the problem, and replace it with a new pipe that fits into the same holes. The structure is just as strong as it was before.
15. Continue the above process until you no longer have to pay attention to the structure and it just stays up all by itself.
16. Adjust the structure as necessary for the changing requirements of the users of the structure, which is easy because the holes are all standardized.

So, did you notice that we followed all the Laws Of Software Design?

• We thought about the future. We did that for the entire process, but we particularly did it when we put on unbreakable steel ropes that would last no matter what happened in the future.

  Also note that we didn’t try to predict the future, we just followed our principles so that no matter what happened, our structure was going to stay together and be easy to build.

• We allowed for change by screwing the bars together instead of welding them. We also put standardized holes in all the bars, even if we didn’t need them, in case we needed to add more bars in the future.
• In every step of creating the structure, we kept our changes small and tested everything as we went. Creating each individual bar was a small task, and we put them together in small steps.
• And of course, the most important decision we made was to keep it simple by making all the holes consistent and standard, and keeping each piece small and simple.

Whether you are one person or a thousand, whether your project is 10 lines of code or 10 million, this process will work.

-Max

Comments: 2

Here’s the slides from my talk (also available in PDF Format).

-Max

Comments: 0

I’ll be talking on Thursday, July 24, at 3:25pm, in either room E143 or E144 at the Oregon Convention Center.

I’ll tell you basically everything I know and have learned about software design, and then how it applies to Open Source software, all in about 45 minutes. Should be fun and interesting!

-Max

Comments: 0

Bugs most commonly come from somebody’s failure to reduce complexity. Less commonly, they come from the programmer’s misunderstanding of something that was actually simple.

Other than typos, I’m pretty sure that those two things are the source of all bugs, though I haven’t yet done extensive research to prove it.

When something is complex, it’s far too easy to misuse it. If there’s a black box with millions of unlabeled buttons on it, and 16 of them blow up the world, somebody’s going to blow up the world. Similarly, in programming, if you can’t easily understand the documentation of a language, or the actual language itself, you’re going to mis-use it somehow.

There’s no right way to use a box with millions of unlabeled buttons, really. You could never figure it out, and even if you wanted to read the 1000-page manual, you probably couldn’t remember the whole thing well enough to use the box correctly. Similarly, if you make anything complex enough, people are more likely to use it wrongly than to use it correctly. If you have 50, 100, or 1000 of these complex parts all put together, they’ll never work right, no matter how brilliant an engineer puts them together.

So do you start to see here where bugs come from? Every time you added some complexity, somebody (and “somebody” could even be you, yourself) was more likely to mis-use your complex code. Every time it wasn’t crystal clear exactly what should be done and how your code should be used, somebody could have made a mistake. Then you put your code together with some other code, and there was another chance for mistakes or mis-use. Then we put more pieces together, etc.

Often, this sort of situation happens: the hardware designer made the hardware really complicated. So it had to have a complicated assembly language. This made the programming language and the compiler really complicated. By the time you got on the scene, you had no hope of writing bug-free code without ingenious testing and design. And if your design was less than perfect, well…suddenly you have lots of bugs.

This is also a matter of understanding the viewpoint of other programmers. After all, something might be simple to you, but it might be complex to somebody who isn’t you.

If you want to understand the viewpoint of somebody who doesn’t know anything about your code, find the documentation of a library that you’ve never used, and read it.

Also, find some code you’ve never read, and read it. Try to understand not just the individual lines, but what the whole program is doing and how you would modify it if you had to. That’s the same experience people are having reading your code. You might notice that the complexity doesn’t have to get very high before it becomes frustrating to read other people’s code.

Now, once in a while, something is really simple, and the programmer just misunderstood it. That’s another thing to watch for. If you catch a programmer explaining something to you in a way that makes no sense, perhaps that programmer misunderstood something somewhere along the line. Of course, if the thing he was studying was extremely complex, he had basically no hope of fully understanding it without a PhD in that thing.

So these two things are very closely related. When you write code, it’s partially your responsibility that the programmer who reads your code in the future understands it, and understands it easily. Now, he could have some critical misunderstanding—maybe he never understood what “if” meant. That’s not your responsibility. Your responsibility is writing clear code, with the expectation that the future programmer reading your code understands the basics of programming and the language you’re using.

So, there are a few interesting rules that you can get out of this one:

The simpler your code is, the fewer bugs you will have.

Always work to simplify everything about your program.

-Max

Comments: 9

But really, when we say “bug” what exactly do we mean?

Here’s the precise definition of what constitutes a bug. Either:

1. The program did not behave according to the programmer’s intentions.
   or
   
2. The programmer’s intentions did not fulfill common and reasonable user expectations.

So usually, as long as the program is doing what the programmer intended it to do, it’s working correctly. Sometimes what the programmer intended it to do is totally surprising to a user and causes him some problem, so that’s a bug.

Anything else is a new feature. That is, if the program does exactly what was intended in exactly the expected fashion, but it doesn’t do enough, that means it needs a new “feature.” That’s the difference between the definition of “feature” and “bug.”

Note that hardware can have bugs too. The programmer’s intention is rarely “the computer now explodes.” So if the programmer writes a program and the computer explodes, that’s probably a bug in the hardware. There can be other, less dramatic bugs in the hardware, too.

Essentially, anything that causes the programmer’s intentions to not be fully carried out can be considered a bug, unless the programmer is trying to make the computer do something it wasn’t designed to do. For example, if the programmer tells the computer “take over the world” and it wasn’t designed to be able to take over the world, then the computer would need a new “take over the world” feature. That wouldn’t be a bug.

With hardware, you also have to think about the hardware designer’s intentions, and common and reasonable programmer expectations. At that level, software programmers are actually the main “users”, and hardware designers are the people whose intentions we care about. (Of course, we also care about the normal user’s expectations, especially for hardware that users interact with directly like printers, monitors, keyboards, etc.)

-Max

Comments: 4

Well, okay, maybe it’s not all opinion. There must be some way to tell a good technology from a bad one (besides looking back after five years of development and saying, “Wow, we really shouldn’t have decided to base our product off of Microsoft Bob.”)

When I’m evaluating a technology for inclusion in one of my projects, I look particularly at the technology’s survival potential, interoperability, and attention to quality.

By “survival potential” I mean “how long is this software going to continue to be maintained?” If you get stuck with libraries or some dependency that becomes obsolete, you’re really in for some trouble. You can get some idea of the survival potential of the software by seeing how often they’ve released recently. Also, how responsive are they to bug reports? Do they have a mailing list that’s very active with users and developers? Are there lots of people online talking about this technology? If a technology has a lot of momentum now, you can be fairly sure that it’s not going to die any time soon.

By “interoperability” I mean, “Does this system use some standard that would allow us to switch to a different system if we want, in the future?” For example, some database systems support standard SQL very well. Some other ones aren’t as good about that–they have strange behaviors that don’t agree with the standard, or they just don’t support it at all. Using a database with standard-SQL support would allow you to switch to any other standard-SQL database in the future, whereas using a less-standard database would lock you in.

The final one, “attention to quality,” is more of a subjective measurement, but the idea is to see if the product is getting better in its recent releases. If it’s open-source, check if they’re refactoring and cleaning up the code base. Is it becoming easier to use or more complex? Do the people who maintain the technology actually care about the quality of their product, or are they just code monkeys working for the pay? Are there are a lot of serious security vulnerabilities in the software that have been published lately?

Lock-in is a worse problem when you’re using proprietary software than when your dependencies are open-source. Proprietary vendors can just stop supporting something (or go out of business), whereas in the open-source world anybody is free to pick up a codebase and maintain it if the original maintainers drop it. Granted, open-source projects are just as likely to fall out of existence as proprietary projects are, but at least there’s the hope (particularly if the project was very popular) that somebody will come around to maintain it again, with open-source.

Proprietary vendors also have a monetary incentive to lock you into their technologies–even their free technologies. Once you start using their free technologies, they can sell you services and products related to them. Open-source vendors usually lack that monetary incentive, and so are more likely to be “friendly” when you want to switch away from them. This isn’t always true (some proprietary vendors beat the pants off of open-source vendors in terms of interoperability, and some open-source products will still lock you in), but it’s enough the case that I automatically think “let’s use open-source” when I’m looking for interoperability.

There are other aspects to look at when you’re choosing a technology, and some of it really is opinion. Some people like the way Ruby looks better than the way Python looks. That’s a valid reason to choose a technology sometimes–if you just like some technology more than another one, and everything else seems equal according to the criteria above, go with the one that makes you happy. After all, you’re the one who’s going to be using it–your opinion matters! All I’ve tried to do here is give some universal guidelines that can be used to weed out the definitely bad choices, and the rest is up to your personal research, needs, and desires.

-Max

Comments: 4

There are some things about the future that you do not know.

Obviously it’d be ideal if we were all-knowing and could perfectly predict every consequence of every decision we’ll ever make. But that’s impossible. In fact, it’s so far from possible that if you predict more than half of the consequences of your engineering decisions, you’re godlike or just really lucky.

Let’s take an example outside of the realm of programming: CDs, which were designed in 1979 to replace cassette tapes as the primary method of listening to music. Who could have predicted that 20 years later, DVDs would be made the same size and shape so that manufacturers could make CD/DVD drives for computers? Did anybody imagine the problems of spinning a CD fifty times faster than it was supposed to be spun, for reading in a CD-ROM drive?

This is why, in engineering, we have “guiding principles.” There are certain rules that, when we follow them, keep things working well no matter what happens in the future.

In the case of software, I’ve summed up the most basic guiding principles as the Laws Of Software. The very first (and most important) law is that the future is more important than the present (because it’s very big and the present is very small). But here’s the thing to know about that: that doesn’t mean you have to predict everything about the future. Instead, it explains why you should be making decisions according to the laws of software–because those laws make for good future software, no matter what that future is.

Sometimes it can be hard to understand why you should be stupid, dumb simple, if you’re only thinking about the present. And it can be impossible to predict exactly why or how simplicity will help you in the future. I can tell you that it will help, and you’ll be glad you did it, and if you don’t believe me (which you are welcome to do–your mind is yours!) you’re going to end up in mess of trouble somewhere down the line, in a future you can’t predict.

I’m not singling you out, here. You’re not dumb. It’s just that nobody can predict the total future of a piece of software, except to know that certain decisions now will make that future better, and that other decisions will make that future worse. We know this from decades of experience in the software development industry–certain basic principles invariably lead us in the right direction, and certain “anti-patterns“, if followed consistently, invariably lead us into trouble.

To be a successful software developer, you have to be willing to not know some things about the future, and trust that following your principles is the right thing to do. Sometimes it can take years to see that you were right, but it’s a glorious day when you finally realize that your product still exists because you made the right decision three years ago, when you couldn’t even prove to anybody that it was the right decision, but forged ahead anyway with your “excessive simplicity” because you knew it was the right thing to do.

-Max

Comments: 5

Registration for FOSSCoach is free, but is limited to 100 people, so sign up if you want to come.

-Max

Comments: 0

“We started working on this project five years ago, and the technology we were using/making was modern then, but it’s obsolete now. Things keep getting more and more complex with this obsolete technology, so it keeps getting less and less likely that we’ll ever finish the project. But if we re-write, we could be here for another five years!”

Another popular one is: “We can’t develop fast enough to keep up with modern user needs.” Or, “While we were developing, Google wrote a product better than ours much faster than us.”

When I hear things like that, the first thing I ask myself is “How did that happen? Why did it take so long for them to finish their product that they ran into that problem?”

The answer lies in complexity. The more complex a task is, the harder it is to complete it. So you start out with something simple that can be completed in one month. Then you add complexity, and the task will take three months. Then you take each piece of that and make it more complex, and the task will take nine months.

Complexity builds on complexity–it’s not just a linear thing. It’s not like, “We have ten features, and so adding one more will only add 10% more time.” No, one new feature will have to be coordinated with all ten of your existing features. So if just the feature itself takes 10 hours of coding time, there will be another hour of coding time for that feature interacting with each other feature. The more features there are, the higher the cost gets of adding a feature.

Some projects start out with such a complex set of requirements that they never get a first version out. If you’re in this situation, you should just trim features. Don’t shoot for the moon in your first release–get out something that works and make it work better over time.

There are other ways to add complexity than just adding features, too. The most common other ways are:

• Expanding the purpose of the software. Generally, just don’t ever do that. Your marketing droids might be drooling over the idea of “making a single piece of software that does your taxes and cooks dinner”, but you should be screaming as loud as you can whenever any suggestion like that comes near your desk. Stick to your purpose–your software just has to do what it does well, and you will succeed, if that purpose is something people need.
• Adding programmers. Yes, that’s right–adding more people to the team adds complexity, it does not make things simpler. Remember The Mythical Man Month? What he says in there is true because of the complexity equation I explained higher up in this article–if you have ten programmers, adding an eleventh means spending time to groove in that one programmer, plus the time to groove in the existing ten programmers to the new guy, plus the time spent by the new guy interacting with the existing ten programmers.
• Change things: Any time you change something, you’re adding complexity. Whether it’s a requirement, a design, or just a piece of code, you’re introducing the possibility of bugs, the time required to implement the change, the time required to validate that the new change works with all the other pieces of the software, the time required to decide upon the change, and the time required to track the change, and the time required to test the change. Each change builds on the last in terms of all this complexity, so the more you change, the more and more time each new change is going to take. It’s still important to make certain changes, but you should be making informed decisions about it, not just changing everything on a whim.
• Lock-In to bad technologies. This is where you make a bad decision about your backend or libraries and then are stuck with it for a long time because you’re so dependent on it. Obviously “bad technology” is very subjective, but sometimes there are obvious good choices and obvious bad ones. For example, if you need an embedded scripting language, Lua is generally considered to be a good choice, and “write our own” would probably be one of the worse choices, depending on the situation. It’s definitely relative–it’s not like there’s only one good choice and all the rest are bad, but some choices might make your life easier than others.
• Poor design or no design. Basically, this just means “a failure to plan for change.” Things are going to change, and that requires design work, to maintain simplicity while the project grows. Failing to do this can introduce massive complexity very fast, because suddenly each new feature quadruples the complexity of the code instead of just adding a little bit to the complexity.
• Re-inventing the wheel. For example, if you invent your own protocol when a perfectly good one exists, you’re going to be spending a lot of time working on the protocol, when you could just be working on your software. You should basically never have any huge invented-in-house dependency, like a webserver, a protocol, or a major library, unless that is your product.

The thing about all of these is that they’re insidious. Most of them only do long-term damage–something you won’t see for a year or more. So when somebody proposes them, often they sound harmless! And even when you start implementing them, maybe they seem fine. But as time goes on–and particularly as more and more of these stack up–the complexity becomes more apparent and grows and grows and grows, until you’re another victim of that ever-so-common horror story: “The Never-Shipping Product.” (Which is nowhere near as cool as The Neverending Story, believe me.)

-Max

Comments: 1

Why has The Sims sold 100 million copies, while Second Life, an ostensibly much more flexible and powerful universe, only has about 2 million active users? They look pretty similar, and you might guess at first glance that they’d have somewhat similar audiences. But, although 2 million users is nothing to scoff at, 100 million absolutely trounces it. So why the big difference?

Well, of course, The Sims has EA Games behind them, who have a massive distribution channel and a lot of marketing power, but the Internet buzz and general promotion of Second Life is pretty good too, so although EA has the edge, that doesn’t explain a 50-to-1 difference in sales. There must be something actually different about the products themselves.

Well, at first glance, The Sims is very user-friendly and Second Life is (from what I’ve heard) hard to use. The Sims does a limited scope of things very well, and Second Life does an unlimited number of things through a difficult interface, with mediocre results.

But fundamentally, why is it that products like The Sims succeed so much more than things like Second Life? And why does The Sims have a better interface, why do people want to play The Sims more than they want to play Second Life? Well:

It’s easy to make a system that does something specific, and hard to make a system that does everything.

When something is easy to make, you can spend a lot more time focusing on the little details–the polish. When something is hard to make, you spend all your efforts just making it work, and there’s no time left to sand off the rough edges. Specific solutions allow you to handle a problem with a level of grace, efficiency, and quality that could never be achieved by a generic, do-it-all solution.

A great example is the Colossus computer, built in 1943 to break encrypted German radio messages in World War II. That was all the Colossus did–it was fundamentally incapable of doing any other task. However, it would have taken a 5 MHz general-purpose computer to break that code at the same rate as the Colossus–a level of speed that desktop computers didn’t reach until the 1970′s, 30 years later.

This is something that I have been trying to get across to programmers for quite some time–you don’t need to solve all the world’s problems with one piece of code, you only need to solve the problem you’re solving. Whether you’re designing a whole system or just a tiny piece, your code doesn’t need to do any more than is called for by the known requirements. Sure, keep it extendable for the future–that’s part of making a quality solution. But all solutions should be specific solutions to known problems.

The Sims satisfies a specific, known need–the desire of people to play out specific types of fantasy lives. It succeeds because it’s not infinitely flexible. Second Life is a nightmare of complexity because it tries to be “everything to everyone”–a situation in which you almost always end up being not enough to anyone.

When people have problems with complexities or confusions in their software, I encourage them to be The Sims. Be Notepad. Be the Google main page. Don’t try to anticipate every need in the world, just solve the ones you know exist. Everything tends to fall into place when you really know your requirements and just go to solve them.

-Max

Comments: 4

For example, it’s very difficult to make a car move if it has square wheels. You’re going to be spending lots and lots of time figuring out how to make the car work, when really it should just have round wheels.

Any time there’s an “unsolvable complexity” in your program, it’s because there’s something fundamentally wrong with it. If the problem is “unsolvable” at one level, maybe you should back up and look at what’s underlying the problem. Maybe you put square wheels on the car, and now you’re trying to figure out how to make it go fast.

Programmers actually do this quite often. For example, “I have this terribly messy code, now it’s really complex to add a new feature!” Well, your fundamental problem there is the that code is messy. Clean it up, make the already-existing code simple, and suddenly adding the new feature will be simple.

What Problem Are You Trying To Solve?

If somebody comes up to you and says something like, “How do I make this pony fly to the moon?”, the question you need to ask is, “What problem are you trying to solve?” You’ll find out that they really need to collect gray rocks. Why they thought they had to fly to the moon, and use a pony to do it, only they know. People do get confused like this.

So when things get complex, back up and you look at the problem that you’re trying to solve. Take a really big step back. You are allowed to question everything. Maybe you thought that adding 2 and 2 was the only way to get 4, and you didn’t think about adding 1 and 3 instead, or just skipping the addition entirely and just putting “4” there. The “problem” is “How do I get 4?” Any method of solving that problem is acceptable, so figure out what the best method would be, for the situation that you’re in.

Discard your assumptions. Really look at the problem that you’re trying to solve, and think about the simplest way to solve that problem. Not “How do I solve this problem using my current code?” Not “How did Professor Bob solve this problem in his program?” No, just how, in general, in a perfect world, should that problem be solved? From there, you might see how your code needs to be re-worked. Then you can re-work your code. Then you can solve the problem.

-Max

Comments: 18

For the frontpage of codesimplicity.com, I think that’s pretty useful–it makes it a lot easier to browse the various articles. But I was wondering, for those who read this in an aggregator or via RSS–for the long posts, do you prefer getting only the short version (with the “more…” link) or would you rather just have the whole post?

Update: It looks like, from the comments, that full text in the feed is the clear winner! That’s actually my preference too, when reading sites, but I usually am not reading posts as long as the ones I write. I’ve installed and activated the Full Text Feed plugin for WordPress, so in the future the feeds will contain the full text even though the front page and email notifications will still be cut by the “more” link.

-Max

Comments: 13

In the US, it’s probably a symptom of a general cultural problem–if an American can’t see an instant result from something, they think it doesn’t work. This leads to fast food, french fries, and fat people. The healthy way to eat (protein and vegetables) has a delayed effect on the body (you don’t get the energy for over an hour), and the bad way to eat (endless carbohydrates without nutritional value) has an instant result–immediate energy.

Software is always a long-term process. I wrote the first version of VCI in about three weeks, and that was insanely fast. Any actual application (VCI’s just a library) takes months or years of person-hours, even if you keep it small. So you’d think that organizations would be far-sighted about their development strategies, right?

Unfortunately, it just doesn’t happen. Competitor X comes out with “Shiny New Feature” and The Company says “we must have Shiny New Feature RIGHT NOW!” That’s not a long-term winning strategy, that’s just short-sighted panic. If you have users, they’re not all going to get up and go away in the next five minutes just because somebody else has one feature that you don’t. You should be looking at trends of how many users you’re gaining or losing, not just responding mindlessly to the immediate environment.

So what’s a good long-term strategy? Well, refactoring your code so that you will still be able to add features in the future, that’s a good one. Or spending some extra time putting some polish on your features and UI so that when the product is released, users are actually happy with it. Not adding features that you don’t want to maintain, if they’re not important enough–that’s another one.

Remember that Mozilla did poorly for years, only to finally start gaining dominance in a market that Netscape had lost, because they had a long-term plan. Granted, Mozilla made some decisions early on that caused some things to take longer than they should have, but they still won out in the long term, despite failing in the short term.

Of course, it can be hard to convince people that your long-term plan is right, sometimes, because it takes so long to show results! When I started refactoring Bugzilla about four years ago, there was pretty constant resistance, particularly when I would review patches and say, “You need to wait for the new architecture before this can go in,” or “This needs to be fixed to not be spaghetti code.” But once the refactoring really got rolling (after about two and a half years), it suddenly became way easier to add new features and nearly all the developers became big supporters of refactoring.

I read so much “advice” on “how to run your software business” that just focuses on instant gratification–what you can get done right now. “Add features!” “Get millions of dollars instantly from VCs!” Unfortunately, the way the universe seems to work is that you can destroy something in an instant, but it takes time to create something. So in reality, the closer you get to “instant gratification”, the closer you get to destruction of your product, your business, and your future.

If you want a good plan, pick one that admits that creation takes time. It doesn’t have to take forever, but it’s never instant.

-Max

Comments: 11

Never “fix” anything unless it’s a problem, and you have evidence showing that the problem really exists.

Of course, most of us know this as “If it ain’t broken, don’t fix it.” However, these wise words are frequently ignored, because many developers don’t have a good understanding of what “broken” means. Often, developers just imagine that users have a problem with something, and start fixing it. Or they go off and develop features that don’t solve anybody’s problem. This is far, far more common than you might think.

So that’s why I say you should have evidence that there’s a problem. Without that evidence, you could just be fixing things that aren’t actually problems, and if you go around fixing things that aren’t broken, you’re going to break things. And not only could you be generating bugs, but you’re wasting your time and adding complexity to your program for no reason. You need evidence that there’s a problem, before you start coming up with a solution.

What do I mean by “evidence”? Well, five users report that when they push the red button, your program explodes. Okay, that’s good enough for me! Or you push the red button, and you notice that the program explodes.

However, just because a user reports something doesn’t mean it’s a problem. Sometimes a user just didn’t realize that your program had some feature already, and so asked you to implement something else silly. For example, you write a program that sorts a list of words alphabetically, and a user asks you to add a feature that sorts a list of letters alphabetically. Your program already does that. (Actually, it already does more than that. This is often the case, with this sort of confused request.) So in this case, the user thought there was a problem, but there wasn’t even really a problem, even though he could maybe present “evidence” that he couldn’t sort a list of letters. (He just didn’t realize that he should use the word-sorting feature.)

Note that if you get a lot of requests like the above, it probably means that users can’t easily figure out how to find what they need. There’s some complexity that needs to be reduced on your end.

Finally, sometimes a user will report that there’s a bug, but actually that’s the program behaving exactly as you intended it to. In this case, it’s a matter of “majority rules.” If a significant number of users think that the behavior is a bug, it’s a bug. If only a tiny minority (like one or two) think it’s a bug, it’s not a bug.

The most famous error in this area is what we call “premature optimization”. That is, some developers seem to like to “make things go fast”. But they do it before they know that it’s slow! This is like a charity sending food to rich people. “We just wanted to help people!” Right—illogical, isn’t it? They’re solving a problem that doesn’t exist.

The only parts of your program where you should be concerned about speed are the exact parts that you can show cause a real performance problem for the users. The rest of the code, the primary concern is simplicity, not “make things go fast”.

There are infinite ways of violating this rule, but the way to follow it is so simple: just get real evidence that the problem is valid before you fix it.

-Max

Comments: 0