1. It is more important to reduce the Effort of Maintenance than it is to reduce the Effort of Implementation.
2. The Effort of Maintenance is proportional to the complexity of the system.

And that is pretty much it. If you all you knew about software design were those two principles and the purpose of software, you could evolve every other general principle of software development.

-Max

Comments: 6

• Potential Value of Implementation (V_i for short): How “valuable” could it be to implement this? For example, if we add something to the program that could directly prevent somebody from dying, that’s very valuable. If it simply might prevent a future typo in a single error message, that’s hardly valuable at all.

  The “value” can be for the end user or for other programmers. When we’re talking about design decisions that affect only code, and not the end user, flexibility is one of the major values–how important could this flexibility be?

  The potential value is separate from how likely it is that the situation will occur where you will need it. That’s the next issue.

• Probability of Value (P_v for short): What is the chance that this value will be, in fact, realized by an end user (for a feature or functional change) or by another developer (for some design decision)? If we’re adding in code flexibility to allow for potential contact with extraterrestrial apes, that’s not a very probable occurrence. If we’re adding in a feature that is immediately useful to every single user, that’s 100% probability. (The number of users that a feature will be useful to is also part of the Probability of Value.)
• Effort of Implementation (E_i for short): How hard will it be to implement this? This is a one-time cost–the immediate difficulty of performing the work required to create this thing the very first time. This would probably be measured in person-hours.
• Effort of Maintenance (E_m for short): How much effort will it require to maintain this in the future? (This includes any effort added to maintaining the entire program by implementing this.) Will this complicate maintenance for the whole system? This is an amount that increases over time. Similar to Effort of Implementation, this would be most likely measured in person-hours.

What we’re trying to determine is the Desirability Of Implementation (D for short). This answers the questions “Is this something we should do or not?” and “What should the priority of implementing this be?”

The simplest form of the equation is:

D = (P_v * V_i) / (E_i + E_m)

Or, in English:

The Desirability of Implementation is directly proportional to the Probability of Value and the Potential Value of Implementation, and inversely proportional to the total effort, consisting of the Effort of Implementation plus the Effort of Maintenance.

However, there is a critical factor missing from the simple form of this equation: time. What we actually want to know is the limit of this equation as time approaches infinity, and that gives us the true Desirability of Implementation. So let’s look at this from a logical standpoint:

The Effort of Implementation is a one-time cost, and never changes, so is mostly unaffected by time.

The Value of Implementation may increase or decrease over time, depending on the feature. It’s not predictable, and so we can assume for the sake of this equation that it is a static value that does not change with time (though if that’s not the case in your situation, keep this factor in mind as well). One could even consider that the Effort of Maintenance is actually “the effort required to maintain this exact level of Value,” so that the Value would indeed remain totally fixed over time.

The Probability of Value, being a probability, approaches 1 (100%) as time goes to infinity.

The Effort of Maintenance, being based on time, approaches infinity as time goes to infinity.

At first glance, that might sound as though design is hopeless, because maintenance becomes infinite effort–an amount that no Potential Value could surpass–and it seems like every possibility must be accounted for, because given infinite time the probability seems to indicate that every possibility will occur. Those are not true statements, though, because you have to think about the rate at which both of those items increase. If the fundamental effort of maintenance is very small, then even as time goes on, it will remain small. You could say that there is a “coefficient of maintenance” on any design decision or feature, and that that determines how rapidly maintenance effort will accumulate over time. As far as the Probability of Value goes, if it is a very tiny number, it may remain tiny until thousands or millions of years have passed–so if the Effort of Maintenance increases at a great rate, then it will easily outstrip the Probability of Value and the Desirability of Implementation will approach zero as time approaches infinity.

What this equation actually tells us is that the most important factors to balance, in terms of time, are probability of value vs. effort of maintenance. If the probability of value is high and the effort of maintenance is low, the desirability is then dependent only upon the Potential Value of Implementation vs. the Effort of Implementation–a decision that a product manager can easily make. If the probability of value is low and the effort of maintenance is high, the only justification for implementation would be a near-infinite Potential Value of Implementation.

This interestingly indicates why small improvements in programming languages and development frameworks result in such enormous changes in the resulting products–because tiny reductions in the Effort of Maintenance can make tremendous changes in the Desirability of Implementation. Features that otherwise would be thrown away by a product manager as impossible become part of the basic design plan. Polishing the UI becomes more desirable, because it requires less effort for both implementation and maintenance.

And finally, I think that this communicates most exactly and truthfully why simplicity is so important–because simplicity is what determines the “coefficient of maintenance” that I talked about above. The effort involved in maintaining simple code increases very slowly over time–sometimes so slowly that you never will have to put any maintenance effort into it in your lifetime.

There’s a lot more that could be said about this equation. What are your thoughts on it? Anybody have any ideas of how Value of Implementation could be numerically calculated, or if it might break down into a set of other numerically-calculable factors? Anything you have to say about it, I’m interested.

-Max

Comments: 15

There’s only one problem with this whole debate: what is privacy, and why would anybody want it? This is rarely defined–most people just seem to assume that “everybody knows” that privacy is, so why would it have to be explained?

Well, I’m not a big fan of “everybody knows.” And in fact, it turns out that privacy actually means two different things, which many people use interchangeably without specifying what they’re actually talking about. So to help clear up some of the debate online, and to hopefully shed some light on how it can all be resolved, here are some clear definitions and discussions of what privacy is, and why people would want it.

Privacy of Space

The first type of privacy is “privacy of space”. This is the ability to control who does and does not enter a particular physical space, probably because you’re in the space and you don’t want certain others in that space. “Enter the space” in that definition includes any method of being able to perceive the space–so, for example, if somebody stands outside the door with their ear pressed to it, they’re violating your privacy. If somebody installs a camera in your room without your consent, they’re violating your privacy.

This form of privacy is not metaphorical. It does not apply to anything other than physical space. It literally means, “I do or do not want you to be perceiving this physical location, and I have the choice and ability to control that.”

The most common reason that we want this form of privacy is that we want to protect somebody or something from harm, most commonly ourselves. This harm can be minor (we don’t want to be annoyed by people walking through our house all the time), it can be purely social (we close the door when we go to the bathroom because we know others don’t want to perceive us going to the bathroom, and we may also not want to be perceived in such a state), or it can be extreme (a man with a mask and a chainsaw should not be in my closet).

One interesting thing about this form of privacy is that we don’t usually consider animals, plants, or material objects to be capable of violating it, even if they enter a space without our permission. It might be annoying if the cat comes in the room when you don’t want it to, but you’re not going to complain that the cat is “violating your privacy”, right?

So, when it comes to computer programs, this is not the form of privacy we’re talking about, since we don’t consider that a computer program being in the same room with us is a violation of our privacy of space. My word processor is not violating my physical privacy of space, even though it’s “in the room” with me, because it does not, itself, perceive. The only exception would be a computer program that was transmitting perceptions (sound or sight) to some location that we didn’t want to send it to–that would be a privacy violation, because could perceive our space through it when we didn’t want them to. When it comes to that sort of privacy, violations are pretty pretty cut-and-dry. If a computer program sends perceptions of my space anywhere without my permission, it is absolutely violating my privacy, it’s not useful to me, and it should stop immediately.

But on the Internet, that’s not usually the type of privacy we’re talking about.

Privacy of Information

The second type of privacy is “privacy of information.” This is the ability to control who knows certain things. When we talk about computer programs and the Internet, this is the most common type of privacy we’re talking about.

So why would somebody want privacy of information? Is it just because they’re doing something that they want to hide from others? Is it just for committing crimes or for hiding harmful acts? Well, sometimes it is, yes. There are many people who use the concept of “privacy” to protect themselves from the law or the moral rejection of others. It is probably because of these individuals that the concept of privacy is a muddy subject–as long as it’s unclear quite what “privacy” is, it’s much easier for those who have have committed harmful acts to invoke “privacy” as a defense.

But is that the only reason that somebody would want privacy of information? What about a normal person, who isn’t doing anything harmful–would they ever want to certain information private?

Well, there is absolutely a rational reason that people would want privacy of information, and interestingly, it’s the same reason that people want privacy of space:

An individual or group desires privacy of information because they believe that other people knowing that information could or would be more harmful than them not knowing it.

Here’s a very straightforward example: I consider that a criminal knowing my credit card number would be harmful–far more harmful than them not knowing it.

In certain countries, the fact that I read a certain website or talked to certain people on the Internet could get me killed or put in jail. So, in that situation, other people knowing my browser history could be very harmful, no question about it.

Of course, if one kept everything private, one could not live. If you pay for a piece of candy with a quarter, the person receiving that quarter now knows that you had a quarter. They may know that you kept it in a waller, or that you pulled it out of your pants. They probably know what you look like, if you’re not wearing a mask. They most likely also know that you have five fingers, and that you were in their store at a certain time. In short, no matter what you do, in order to live, you must exchange information with other people. The more things you do, the more information you will have to exchange.

In fact, usually, the more information that others know about you, the more helpful they can be. The bank knows all the transactions that I made, so they can help me by creating an online system that shows me my transactions and lets me search them. That information can be seen by bank employees, but I don’t consider that to be potentially harmful enough to outweigh the obvious benefits of the bank having it.

The web browsers that I use know my passwords to certain sites, so they can help me by putting those passwords into the box, saving me some typing. Potentially, somebody could steal that information from my computer, but the chance of that happening is small enough, and the benefit is significant enough, so I consider it acceptable to save my passwords in the browser.

The examples like this go on and on–the appropriate use of information is extremely beneficial. The inappropriate use is what’s harmful.

So who decides what what’s an appropriate use and what’s an inappropriate use? What information should be sent and stored, and what information should be kept private? Well, these are the fundamental questions being asked when people debate privacy issues–who gets to choose whether my knowledge becomes somebody else’s knowledge? Should I be asked before my information is sent, or should I just be given the option to opt-out and delete the information? Is there some information that should never be sent? What information is more important to keep private than other information?

Though this is all far less cut-and-dry than “privacy of space” issues, these questions can generally be answered by the “help vs harm” equation. The basic sort of questions one might want to ask would be:

• Will sending and storing this information harm any users, immediately or potentially? (Remember, “potentially” is pretty broad–what happens if somebody with bad intentions steals that information from you? What happens if somebody buys your company and decides to use that information in a way that you think is bad?)
• Would it help your users more than harm them to take this information?
• Taking all the above into account, should sending this information be optional? (This is largely determined by how broadly it could be harmful to collect the information.)
• If sending the information is optional, should it be opt-out or opt-in? (That is, should it automatically be on, and people have to turn it off if they don’t want to send the info, or should it be off and people have to choose to turn it on?)
• If it’s opt-in, will the feature still be helpful to enough of your users to justify implementing it?

There are some people who will claim that no information should ever be sent or stored about the user, that all privacy options should always be opt-in, and that all information is so potentially harmful that no debate about this can be accepted. That is, frankly, a ridiculous proposition. It’s so obviously untrue that there’s almost no way to argue with it, because it’s such a shocking irrationality. Just like the fact that somehow, liquids could harm somebody (so you can’t bring liquids on an airplane in the USA) it’s true that there are situations in which almost any piece of information could be dangerous. That doesn’t mean that all information is dangerous, though.

My martial artist friends have frequently joked that they shouldn’t be allowed to bring any object on an airplane, because they could kill somebody with any of them. Similarly, given almost any piece of information, somebody could do something harmful with it, somewhere, at some point. If I know you have a quarter in your pocket, I’m sure there’s some situation in which I could use that information to get you in some serious trouble. But that doesn’t make that information realistically harmful, even potentially.

Even the idea of “every single piece of information should be opt-in” is ridiculous. Do you want the web browser to ask you, “May I send this page your IP address?” every time you load a web page? Well, if you’re a spy in a hostile country, maybe you do. But if you’re like most people, that would probably just annoy you–you’d stop using that web browser and switch to another one. And if you are a spy or a resistance fighter, then you probably know how to use Tor to avoid being tracked.

So when we’re talking about privacy, it’s not an issue of “in some incredibly unlikely situation, this information could be very harmful,” it’s an issue of balancing help vs. harm in real-world situations. Real-world situations can be pretty strange and unexpected, but they at least are real, and can be balanced and talked about. Doing so, you can make good decisions about how to protect your users’ privacy–how much information to take, how you inform them about the information you’re taking, and what you do with that information when you have it.

So no, this is not a casual issue or something that we should brush-off and just ignore the dangerous implications of, but it’s also not an extreme unsolvable situation where we have to decide to keep everything private because we can’t make up our minds about it. Privacy is simply something that we should be able to analyze factually, based on real-world situations and data, and come to some practical and useful decision about.

-Max

Comments: 22

What I didn’t know at the time was why programmers did this. It was obvious that they did do it, but why would the software development industry produce so many crazy, complex masses of unreadable code? Why did it keep happening, even when developers should have learned their lesson after their first bad experience? What was it that made programmers not just make bad code, but keep on making bad code?

Well, this was a mystery, but I didn’t worry too much about it at first. Just the revelation that “bad programs are caused entirely by bad programmers”, as simple and obvious as it may seem, was enough to fuel an entire investigation and study into the field of programming, one which had some pretty good results (that’s mostly what I’ve written about on this blog, and it’s also the subject of a book that’s in development). The problem had been defined (bad programmers who create complexity), it seemingly had a solution (describe laws of software design that would prevent this), and that was enough for me.

But it still baffled me that the world’s universities, technical schools, and training programs could turn out such terrible programmers, even with all of the decades of advancement in software development techniques. Sure, a lot of the principles of software design hadn’t been codified, but a lot of good advice was floating around, a lot of it very common. Even if people hadn’t gone to school, didn’t they read any of this advice?

Well, the truth was beyond my imagination, and it took almost five years of working on the Bugzilla Project with a vast number of separate contributors until one day I suddenly realized an appalling fact:

The vast majority (90% or more) of programmers have absolutely no idea what they are doing.

It’s not that they haven’t read about software design (though they likely haven’t). It’s not that the programming languages are too complex (though they are). It’s that the vast majority of programmers didn’t have the first clue what they were really doing. They were just mimicking the mistakes of other programmers–copying code and typing more-or-less meaningless incantations at the machine in the hope that it would behave like they wanted, without any real understanding of the mechanics of the computer, the principles of software design, or the meanings of each individual word and symbol they were typing into the computer.

That is a bold, shocking, and offensive statement, but it has held up in my experience. I have personally reviewed and given feedback on the code of scores of programmers. I have read the code of many others. I have talked to many, many programmers about software development, and I’ve read the writings of hundreds of developers. The number of programmers who really understand what they are doing comprise only about 10% of all the programmers I’ve ever talked to, worked with, or heard about.

In open source, we get the cream of the crop–people who want to program in their spare time. And even then, I’d say only about 20% of open source programmers have a really good handle on what they are doing.

So why is this? What’s the problem? How could there be so many people working in this field who have absolutely no clue what they’re doing?

Well, that sounds a bit like they’re somehow “stupid.” But what is stupidity? People are not stupid simply for not knowing something. There’s a lot of stuff that everybody doesn’t know. That doesn’t make them stupid. That may make them ignorant about certain things, but it doesn’t make them stupid. No, stupidity, real stupidity, is not knowing that you don’t know. Stupid people think they know something when they don’t, or they have no idea that there is something more to know.

This sort of stupidity is something that can be found in nearly every field, and software development is no exception. Many programmers simply don’t know that there could be laws or general guidelines for software development, and so they don’t even go looking for them. At many software companies, there’s no attempt to improve developers’ understanding of the programming language they’re using–perhaps simply because they think that the programmers must “already know it if they were hired to do it”.

Unfortunately, it’s particularly harmful to have this sort of mindset in software development, because there is so much to know if you really want to be good. Anybody who thinks they already know everything (or who has a “blind spot” where they can’t see that there’s more to learn) is having their ability to produce excellent code crippled by a lack of knowledge–knowledge they don’t even know exists and that they don’t even know they lack.

No matter how much you know, there is almost always more to know about any field, and computer programming is no exception. So it’s always wrong to think you know everything.

Sometimes it’s hard to figure out what one should be learning about, though. There’s so much data, where does one start? Well, to help you out, I’ve come up with a few questions you can ask yourself or others to help figure out what areas might need more study:

• Do you know as much as possible about every single word and symbol on every page of code you’re writing?
• Did you read and completely understand the documentation of every single function you’re using?
• Do you have an excellent grasp of the fundamental principles of software development–such a good grasp that you could explain them flawlessly to novice programmers at your organization?
• Do you understand how each component of the computer functions, and how they all work together?
• Do you understand the history of computers, and where they’re going in the future, so that you can understand how your code will function on the computers that will be built in the future?
• Do you know the history of programming languages, so that you can understand how the language you’re using evolved and why it works like it does?
• Do you understand other programming languages, other methods of programming, and other types of computers than the one you’re using, so that you know what the actual best tool for each job is?

From top to bottom, those are the most important things for any programmer to know about the code they’re writing. If you can truthfully answer “yes” to all those questions, then you are an excellent programmer.

It may seem like an overwhelming list. “Wow, the documentation for every single function? Reading that is going to take too long!” Well, you know what else takes a long time? Becoming a good programmer if you don’t read the documentation. You know how long it takes? Forever, because it never happens.

You will never become a good programmer simply by copying other people’s code and praying that it works right for you. But even more importantly, investing time into learning is what it takes to become good. Taking the time now will make you a much faster programmer later. If you spend a lot of time reading up on stuff for the first three months that you’re learning a new technology, you’ll probably be 10 times faster with it for the next 10 years than if you’d just dived into it and then never read anything at all.

I do want to put a certain limiter on that, though–you can’t just read for three months and expect to become a good programmer. First of all, that’s just too boring–nobody wants to just study theory for three months and not get any actual practice in. Very few people would keep up with that for long enough to become programmers at all, let alone good programmers. So I want to point out that understanding comes also from practice, not just from study. But without the study, understanding may never come. So it’s important to balance both the study and the practice of programming.

This is not an attack on any programmer that I’ve worked with personally, or even an attack on any individual programmer at all. I admire almost every programmer I’ve ever known, as a person, and I expect I’d admire the rest were I to meet them, as well. Instead, this is an open invitation to all programmers to open your mind to the thought that there might always be more to know, that both knowledge and practice are the key to skill, and that it’s not shameful at all to not know something–as long as you know that you don’t know it, and take the time to learn it when necessary.

-Max

Comments: 47

It’s not anything complicated, and it’s not something that’s hard to know. It’s not something that you can only do if you’re born with a special talent or a “magical ability to program well.” There is nothing about the nature of the individual that determines whether or not they will become an excellent programmer or a poor one.

There is only one, singular fact:

The better you understand what you are doing, the better you will do it.

Rockstar programmers understand what they are doing far, far better than average or mediocre programmers. And that is it.

All you have to do in order to become an excellent programmer is fully understand what you are doing.

Some may say that they already understand everything. The test is whether or not they can apply it. Do they produce beautifully architected systems that are a joy to maintain? Do they plow through programming problems at a rate almost unimaginable to the average programmer? Do they explain everything clearly and in simple concepts when they are asked for help? Then they are an excellent programmer, and they understand things well.

However, far more commonly than believing that they “know it all”, many programmers (including myself) often feel as though they are locked in an epic battle with an overwhelming sea of information. There is so much to know that one could literally spend the rest of his life studying and still come out with only 90% of all the data there is to know about computers.

The secret weapon in the epic battle, the Excalibur of computing knowledge, is understanding. The better that you understand the most fundamental level of your field, the easier it will be to learn the next level. The better you understand that level, the easier the next one after that will be, and so on. Then once you’ve gone from the most fundamental simplicities to the highest complexities, you can start all over again and find amazingly that there is so much more to know at the very, very bottom.

It seems almost too simple to be true, but it is in fact the case. The path to becoming an excellent programmer is simply full and complete understanding, starting with a profound grasp of the basics and working up to a solid control of the most advanced data available.

I won’t lie to you–it sometimes is a long path. But it is worthwhile. And at the end of it, you may find yourself suddenly the amazing senior engineer who everyone comes to for advice. You may be the incredible programmer who solves everything and is admired by all his peers. You might even come out a “rock star” with millions of dollars and a fantastically successful product. Who knows? I can’t tell you what to do or what to become. I can only point out some information that I’ve found to be truthful and valuable. What you do with it is up to you.

-Max

Comments: 12

I can solve this problem the right way.

Whatever the problem is, there’s always a right way to solve it. The right way can be known, and it can be implemented. The only valid reason ever to not implement something the right way is lack of resources. However, you should always consider that the right way does exist, you are able to solve the problem the right way, and that given enough resources, you would solve the problem the right way.

The “right way” usually means “the way that accounts for all reasonably possible future occurrences, even unknown and unimaginable occurrences.”

A bridge that could stand up to any reasonably possible environmental condition or any reasonably possible amount of traffic without constant maintenance would be built the “right way.”

Software code that maintained its simplicity while providing the flexibility needed for reasonably possible future enhancements would be designed the “right way.”

There are lots of invalid reasons for not solving a problem the right way:

• I don’t know the right way. Often this just requires more understanding or study, to figure out the right way. When I run into this situation, I walk way from the problem for a while, and then often I’ll come up with the solution when I’m just out walking, or the next day when I come back to it. I try not to compromise on something that isn’t the right way just because I don’t know what the right way is yet.
• The group cannot agree on what the right way would be. Sometimes a group of people have argued about what would be the “right way” and the subject has gotten very confused. Groups are not very good at making decisions. As we all know, you don’t design software by committee, and I suspect that “design by committee” in other fields of engineering is just as bad. The solution here is to assign an experienced and trusted engineer who understands the basic laws of the subject you’re working in to determine the right way by himself or herself, probably after carefully studying the existing arguments and collecting relevant information, following standard, valid engineering procedures.
• I am too lazy/tired/hungry/discombobulated to do this the right way, right now. This happens to everybody from time to time. It’s 1 in the morning, you’ve been working on the project for 15 hours straight, and you just need the damn thing to work, right now! Give it a rest, though, and come back later. The world isn’t ending, and the problem will still be here and solvable later. Go to sleep, go eat something, take a walk–do whatever it takes to get into a mental space where you’re willing to solve the problem the right way, and then come back. If you’re in a state where you can’t solve the problem the right way, then it’s really time to take a break. You’re not being delinquent in your duties if you do so–you’re actually correctly taking responsibility for the success of the project by saying “this needs to be done right, and the way to do it right, right now, is to take a break and come back later”.

Mostly, it all just takes the constant and continual belief in yourself that you can solve the problem the right way.

-Max

Comments: 37

Well, some of it’s really obvious. You press a button and the program takes 10 minutes to respond. That sucks pretty bad. You get 100 complaints a week about the UI of a particular page–okay, so that sucks.

Usually there are one or two HUGE things that really suck, and they’re really obvious–those are the things to focus on first, even if they require a tremendous amount of work. For example, before Bugzilla 3.0, Bugzilla had to compile every single library and the entire script it was about to run, every time you loaded a page. This added several seconds to each page load, on slower machines, and at least 1 second on faster machines. So performance was one big obvious thing that sucked about Bugzilla. But even more importantly, the code of Bugzilla sucked. It was being read by everybody–because people were frequently customizing Bugzilla at their company–and it was an unreadable, garbled mess.

Thankfully, both of those problems had the same solution. The performance problem was solved by allowing people to run Bugzilla in a way that would pre-compile all our scripts when the web server started, instead of every time somebody loads a page. And to enable that pre-compiling, we had to do enormous amounts of refactoring. So, we actually ended up handling our performance problem by handling our code problem.

However, it took four major releases (Bugzilla 2.18, 2.20, 2.22, and finally 3.0) to get all this done! We fixed a lot of little issues for each release along the way, too, so each release really did suck less than the previous one. But handling the major issues was a tremendous effort–it wasn’t just something we could code up in one night and have it be done with.

I think sometimes the big issues in a software project don’t get handled because they do require that much work to fix. That doesn’t mean you can ignore them, though. It just means that you have to plan for a long project, and figure out how you can keep getting releases out in the mean time.

After all that was fixed, then we could turn our attention elsewhere, and wow! It turned out that elsewhere, there were still a bunch of things that sucked! All of the sudden, there were a new batch of “totally obvious” things to fix–things that had been there all the time, but were just overshadowed by the previous set of “totally obvious” things.

Now, we could have just gone on like this forever–fixing one set of “totally obvious” problems and then going on to the next set of “totally obvious” problems. But we ran into an issue–what happens when suddenly, you get to the point where there there are fifty “totally obvious” things that need fixing, and you can’t get them all done for one release? Well, that’s when you suddenly need some method of prioritizing what you’re going to fix.

For the Bugzilla Project, there were two things that we did that really helped us prioritize: the Bugzilla Survey and the Bugzilla Usability Study.

With the survey, the most important part was allowing people to respond in free-form text, to questions asked to them personally. That is, I sent the questions from me personally to Bugzilla administrators personally, often customizing the message for their exact situation. And there were no multiple-choice questions, only questions that prompted them to tell us what was bothering them and what they wanted to see. They were actually really happy to get my emails–lots of them thanked me for just doing the survey.

Once they had all responded, I read everything and compiled a list of major issues that were reported–overall a surprisingly small list! We’re focusing on those issues pretty heavily nowadays, and I think it’s making people happier with Bugzilla in general.

With the usability study, surprisingly the most helpful part was when the researchers (who were usability experts) just sat down in front of Bugzilla and pointed out things that violated usability principles. That is, even more valuable than the actual research they did was just their observations as experts, using the standard principles of usability engineering. The fact that they were fresh eyes–people who’d never worked on Bugzilla and thus didn’t just think “well that’s the way it is”–also was important, I think.

So we took all this data, and it really helped us prioritize. However, it’s important that we did the survey and research when we did them and not earlier. Back before we fixed the top few major issues, the usability and survey results would have just been overwhelming to us–they would have pointed out a million things we already knew, or a lot of things that we just didn’t have the time to work on at that point, and we would have had to re-do the survey and research again later, making it all a bunch of wasted time. So we had to wait until we were at the point of asking ourselves, “Okay, what’s most important now?”, and that was when gathering data became tremendously important and incredibly useful.

So overall, I’d say that when you’re trying to make things suck less, first go with what you know are the top few big obvious issues, and handle those, no matter what it takes. Then things will calm down a little, and you’ll have a huge pile of stuff that all needs fixing. That’s when you go and gather data from your users, and work to fix whatever they tell you actually sucks.

-Max

Comments: 4

But eventually, thanks to a bunch of new members in the Bugzilla community, we released Bugzilla 2.18. Hooray! Bells rang, birds sang, and there was much rejoicing.

However, in the space between Bugzilla 2.16 (which was before my time) and Bugzilla 2.18 (which was the first release that I helped get out), something very strange happened–we developed serious competition. All of the sudden there were a bunch of new bug-tracking systems, some of them open-source and some of them not, that people were actually using.

At first it wasn’t too worrisome. Bugzilla was pretty dominant in its field, and it’s hard to lose that kind of position. But as time went on, there was more and more competition, and some people were predicting doom and gloom for Bugzilla. We were a tiny group of completely unpaid volunteers, and some of these competing products were being made by companies whose marketing and development resources absolutely dwarfed us.

And yet, with every release, our download numbers kept going up. And always significantly–30-50% more than the previous release, every time.

And then we hit Bugzilla 3.0, and our download numbers nearly doubled. And they’ve kept going up with every release from there. Nowadays we get over 10 times the number of downloads per release that we did in 2004.

So how did we pull this off? Well, as far as I can tell:

All you have to do to succeed in software is to consistently suck less with every release.

Nobody would say that Bugzilla 2.18 was awesome, but everybody would say that it sucked less than Bugzilla 2.16 did. Bugzilla 2.20 wasn’t perfect, but without a doubt, it sucked less than Bugzilla 2.18. And then Bugzilla 3.0 fixed a whole lot of sucking in Bugzilla, and it got a whole lot more downloads.

Why is it that this worked?

Well, when people are deciding at first what software to use, they have varying criteria. Sometimes they just use what’s presented to them by default on the computer. Sometimes they have a whole list of requirements and they do lots of research and pick the software that has all the features they need. But once they’ve picked a program, they will stick with it unless there is some compelling reason to leave. It’s not like people constantly are looking for new software to replace yours–they only start looking when your software just won’t stop sucking.

As long as you consistently suck less with every release, you will retain most of your users. You’re fixing the things that bother them, so there’s no reason for them to switch away. Even if you didn’t fix everything in this release, if you sucked less, your users will have faith that eventually, the things that bother them will be fixed. New users will find your software, and they’ll stick with it too. And in this way, your user count will increase steadily over time.

You have to get out releases frequently enough that people believe that you really will suck less, of course. If your new release never comes out, then effectively, your current release never stops sucking.

But what happens if you do release frequently, but instead of fixing the things in your software that suck, you just add new features that don’t fix the sucking? Well, eventually the patience of the individual user is going to run out. They’re not going to wait forever for your software to stop sucking.

I remember a particular piece of software that I used every day for years. It had a lot of great features and a nice user interface, but it it would crash two or three times a week. I really liked the software in general, but man, the crashing sucked. I reported a bug about it, and the bug was ignored. I kept using it through 10 new releases, and it still crashed. The upgrades brought lots of new features, but I didn’t care about them. Remember, the feature set only mattered to me when I first picked the software. Now I just needed it to suck less.

But it never did.

So eventually, I went and looked for another piece of software that did the same thing, switched over, and I was happy with that one for a while. But guess what? It had a bug that really sucked. It didn’t happen very often, but when it did, boy was it a problem. But it sucked less than my old software, so I kept using it. Until one day, my patience ran out (after maybe 7 upgrades of the software), and I switched again.

Now I’m using a program that has half the feature set of either of the previous two programs. But, as a user, I’m the happiest I’ve ever been with this type of software. Because you know what? My new program sucks hardly at all. I mean, there are little things about it that suck, but supposedly a new release is coming out soon that will fix some of that sucking, and so I’m okay with it for now.

Would I have guessed this secret of success before I started working on Bugzilla? No. I would have told you the traditional wisdom–that a product succeeds or fails based on its feature set and user interface. But after 5 years on this project, managing our releases and watching our download count, I can tell you from my factual experience this strange thing: all you have to do to succeed as a software project is to suck less with every release. It doesn’t matter how much competition you have, or how many buzzword-friendly features you can cram into your interface. Just suck less, and you’ll succeed.

-Max

Comments: 57

Why do over 75% of Facebook’s users think that the new Facebook UI is bad? Because it makes different things look similar to each other. Nobody can tell if they’re updating their status or writing on somebody else’s wall, because even though the text is slightly different in the box depending on what you’re doing, the box itself looks the same. The new Chat UI (introduced a few days ago) makes idle users look basically identical to active users, except for a tiny icon difference. (It’s also important that different things are different enough, not just a little different, because people often won’t notice little differences.)

This is an easy pitfall for developers to fall into because developers love consistency. Everything should be based on a single framework, in the backend of an application. But that doesn’t mean that everything has to be displayed the same in the UI.

This fact–that different things should look different–is actually true with code, too, but people rarely think about it, because developers are actually pretty good about it. For example, accessing a value of an object should look different than calling a method on it, and in most programs, it does. For example, in Bugzilla’s code, accessing a value on an object looks like $object->value whereas calling a method on the object looks like $object->method(). It’s not all that different, but the () at the end is enough difference for the average programmer to notice “Oh, that’s a method call that does something–it’s not just accessing a value in the object.”

All in all, consistency is really important in both the backend and the frontend of an application. But that doesn’t mean that every single thing should look exactly the same–if we took that to extremes, we’d just have a solid white page, and that doesn’t seem all that usable (frontend) or readable (backend), does it?

Comments: 7

Twitter’s just one of the many proofs that you don’t have to have lots of features to be successful. In fact, many successful apps have fewer features than their less-successful competitors.

Still, you’ve got to have some features. After all, it’d be pretty silly to be programming, otherwise. But how do you decide which features you should have? Is it just up to the Chief Architect’s intuition, or how many users demand that you give them “feature X”? Does whoever shouts the loudest in the development meeting get their feature implemented first?

Well, no, there is a way to decide whether or not you should implement a feature, and it comes out of one of our most basic principles: the purpose of software. This principle (that the purpose of software is “to help people”) isn’t just some fancy-sounding gibberish I made up to make myself happy–I wrote it because it’s something that can actually be really useful to think about in everyday programming, and this question of “Should we implement this feature?” gives us a great opportunity to show how it can be applied.

So, let’s say that the purpose of your software is “to help people learn to type,” and you have to decide, “Should we add New Feature X?” Well, here’s what you can ask yourself about the feature: “Does this help people learn to type?” Simple. If the answer is no, then you don’t implement the feature. Also, “Does this harm people learning to type, or get in the way of learning to type?” If it harms people more than it helps them, then it shouldn’t be implemented.

Now, quite often, you’ll find that you can’t really say whether or not a proposed feature actually helps your users (or helps them more than it hinders them). You can’t stand there with certainty and say, “Yes, this absolutely helps people learn to type.” In that case, just skip it. Don’t implement the feature. There are plenty of features out there in the world where you can be sure that it will help your users–implement those instead. Perhaps some more data will come along in time that will make your decision clearer about this uncertain feature, but until then, you don’t have to implement it. Just wait (or go collect data to make your decision easier).

This whole idea is also very handy for prioritizing features. You ask yourself, “Out of all these features, which ones will most help people learn to type?” That sounds perhaps like an overly-simple suggestion, but I think that for Bugzilla, if we really looked at all our requested features and said, “Which one of these will most help people track bugs?” we’d come up with a pretty good priority list, and probably have some interesting discussions, too.

The higher you set the bar for certainty on these questions, the fewer features you will have, and the simpler your application will be. That is, you could say, “We have to only be kinda certain it will help people,” and that’s your standard (or “bar”) for helpfulness. Or you could say, “We only implement features that we are totally certain will help people,” and you’d probably have an enormously simple and very well-designed application. But it’s a judgment call–sometimes it’s hard to get data, and you have to implement features that you’re only pretty sure will be helpful. That can be the right decision, though, sometimes–different situations require different solutions.

Overall, how you make these decisions is up to you. What really matters is that you ask the questions. That’s the one of the keys to keeping your product focused, simple, and polished.

-Max

Comments: 5

Now, some people misunderstood me and said, “Oh, SAR is just another name for MVC.” No, I used MVC as an example of SAR, but SAR is a much, much broader concept than MVC–they are not comparable theories. MVC is a pattern for designing software, whereas SAR (or ISAR) is a statement of the three (or four) components that are present in all software.

The fascinating thing about SAR is that it applies not only to a whole program, but also to any piece of that program. A whole program has a Structure, just as a function or single line of code has a Structure. Same for Action and Results.

Here’s a little more about each of the pieces, and some examples to help explain:

Structure

Here are some examples of things that would be considered “Structure” for the whole program:

• The directory layout of your code.
• All of the classes and how they relate to each other.
• The structure (schema) of the database, if your program uses a database.

  Note here that the actual data in the database isn’t part of the Structure, though. If your program is producing the data and then sticking it into the database, then that’s part of the Result. If the data is sitting in the database and your program is supposed to process it, then that data is part of the Input.

Then an individual class (and I mean a “class” in the object-oriented sense) would also have a Structure:

• The names of methods in the class and the types/names of parameters that they take.
• The names and types of variables (member variables) in the class.

Whether or not a function (or variable) is private or public would also be part of the Structure, because Structure describes what something is (as opposed to what it does or produces), and “private” or “public” are words that describe what something is.

A Structure is sort of “the components of the program” or “the pieces you make the program out of.” Function names and types, variable names and types, classes–these things are all Structure.

Structure just “sits there.” It doesn’t do anything unless there’s some part of your program that uses it. For example, a method doesn’t call itself, it just sits there waiting to be called. A variable doesn’t put data into itself, it just sits there waiting for you to do something with it.

Action

The Action of a whole program is very easy to understand. A tax program “does taxes.” A calculator program “does math.”

An Action is always a verb of some sort. “Calculates.” “Fixes.” “Adds.” “Removes.” Those are actions. Usually they’re a little more descriptive and specific, though, like, “Calculates how much rainfall there will be in Africa next year,” or “Fixes broken hard drives.”

Inside of a class, the Action is the code inside of the methods. That’s all some sort of action–something going on, something happening. In many programming languages, you can also have code outside of any class or function–code that just runs when you start the program. That’s Action.

Results

Every program, every function, and every line of code has some effect. It produces some result.

A Result can always be talked about in the past tense–it’s something that has been done or created. “Calculated rainfall,” or “Fixed hard drives.” In a tax program, we’d call the Result either filed taxes or filled-out tax forms. As you can see, it sounds a lot like the Action, just completed.

You don’t have to describe a Result in the past tense, though. I’m just saying it always can be described that way. For example, in a calculator program, normally we’d call the Result of addition “the sum,” (not past-tense, just a noun) but you could also say that the Result is “added numbers” (which is past-tense). Same thing, just a different way of describing it.

Individual pieces of your program have Results, too. When you call a method or function, it has a very specific Result. It gives you back some data, or it causes some data to be changed.

Whatever your program (or any part of your program) produces, that’s the Result.

ISAR in a Single Line of Code

So, I said that SAR applies to a single line of code, but I didn’t give you any examples. So here’s a single line of code:

x = y + z

y and z in that line are part of the Structure. They’re variables that hold some data. To make an analogy: A jug is a structure that holds water. A variable is a structure that holds data.

The numbers that are stored inside y and z are the Input. That’s the data that we’re doing something with.

+ is an Action: “Add these two numbers.” = is also an Action: “Store the result in x.”

And, of course, the Result is the sum of y and z that gets stored in x. If y is 1 and z is 2, then the Result is the number 3, which gets stored in x. (Also note that x is a itself variable and thus also part of the Structure, but that’s getting pretty technical.)

Wrapping It Up

So anyhow, I hope that explains SAR a bit better. It’s a concept that applies to any kind of programming, whether you’re building a big application or just writing a single-line script. And it’s not something that you have to think about in-depth every time you write a piece of code, but it’s something that can help us analyze and understand a program, particularly when we’re looking at how we can improve its design.

-Max

Comments: 4

In the MVC sense, the Model is the Structure, the Controller is what does the Actions, and the View is the Result. I think the analogy (and the words) Structure, Action, and Results are more widely and accurately applicable to the operation of every program in existence, though, moreso than MVC, though MVC is a perfectly good way of looking at it for GUI applications.

Really, Structure, Action, and Results probably describes almost any machine in existence. A machine has some parts that don’t move, a framework–that’s the structure. Some parts move and do something–that motion is the action. And of course the machine produces something (otherwise we wouldn’t care much about it) so that’s the result.

Computer programs are unusual machines in that they can modify their own structure. However, it’s important that some part of the program be stable, that they “not move” in a logical sense. The way that object classes relate to each other, the names of methods and variables–these are all parts of the structure that usually don’t change while you’re running. (Sometimes you make new classes, methods, or variables while you’re running, but they usually follow some pre-set plan, so there’s still a lot of “not moving” involved.)

When I’m writing software, I usually build the Structure first, then I work on the Actions, and then I work on the displaying of the Result. Some people work backwards from the Results, that’s fine too. Probably the only inadvisable thing to do is to start with the Actions, since it’s kind of confusing to be performing Actions without a Structure and with no defined Result.

There’s so much to this concept that I could probably write a whole book just on this one topic, but I think this is a decent introduction, and I’m sure that given this, you can think of lots of other useful applications of it.

-Max

Comments: 0

When we think about software security, the first question that we ask is, “How many different ways could this program possibly be attacked?” That is, how many “ways in” are there? It’s a bit like asking “How many doors and windows are there on this building?” If your building has 1 exterior door, it’s very easy to protect that door. If it has 1000, it will be impossible to keep the building secure, no matter how good the doors are or how many security guards you have.

So we need to limit the “ways in” to our software to some reasonable number, or it won’t ever be secure. That’s accomplished by making the overall system relatively simple, or breaking it down into very simple and totally separate component parts.

Then, once we’ve limited the ways in, we need to start thinking about “How many different possible attacks are there against each way in?” We limit that by making the ways in themselves very simple. Like a door with only one unique key, instead of a door that can take five different keys, all of which individually will open the door.

Once that’s done, we limit how much damage any attack could do if it got through. For example, in a building, we’d make any given door only allow access to one room.

All of this explains, for example, why Windows is fundamentally flawed and will never be secure, and why UNIX-based systems have a better reputation for security.

Standard UNIX has a very small number of system calls that are used to implement the vast majority of all UNIX programs out there. (Even the extended list is only about 140 system calls, though most of those are never used by the average program.) Each system call is extremely specific and does one very limited thing.

Windows, on the other hand, has a ridiculous set of system calls that are confusing, take too many arguments, and do too much.

Going up to a higher level in the system, the Windows API is massive and complex. It’s a strange beast that controls both the OS and the GUI. There’s really no equivalent thing in UNIX (because the OS and the GUI are separate), but we can at least compare parts of it. Here’s The Windows Logging API. Here’s the Linux Logging API. There’s just no comparison. It’s like a joke. There’s so many “ways in” to any part of Windows that it will never be fundamentally secure.

You might say, “Well, I haven’t had a virus on my Windows machine in a long time.” That’s not what I’m talking about–I’m talking about fundamental security. In order to have a secure Windows machine, you have to have a firewall that asks you every time a program wants to make an outbound connection. You have to have a spyware scanner. You have to have antivirus software that slows down your computer by as much as 2000%. If Windows was secure, you wouldn’t need those things.

When we design our own systems, keeping them simple is the only real guarantee of security. We keep each “way in” to the system as simple as possible, and we never add more “ways in” than we absolutely need. These are compatible things, too, because the simpler each “way in” is, the fewer we’ll actually need. That may not make sense until you think about it this way: If all actions on the system can be reduced to, say, 13 fundamental function calls, then the user can do everything with those 13 calls, even if they’re not very powerful individually. If instead we only let them do 100 different specific tasks, and don’t allow them to use the 13 fundamental calls, we have to add a new function for every specific task.

There are lots of other “ways in” to a program than just its public API, too. How the user interface interacts with the backend–that involves various “ways in”. Can we access this program’s internal structure from another program? That would be another “way in.” There’s lots of ways to apply this principle.

Any way you slice it, though, the best way to get real security in things is simplicity. We shouldn’t have to put a small army in front of our software just to keep it secure. It should just fundamentally have so few “ways in” that it doesn’t need the protection, and those “ways in” should be so streamlined and simple that they’re impossible to exploit.

-Max

Comments: 14

But what is it that makes all that stuff a computer? Why do we look at it and go, “Oh yeah, that’s a computer,” as opposed to, say, “Oh, that’s just a TV,” or “That’s where I keep the leprechauns at night.”?

Some people try to define the word “computer” just by saying “it’s got such and such parts and they all work this way,” but that’s like saying “airplanes have two wings and jet engines.” It’s true, but I could build an airplane that didn’t have two wings or jet engines. The way something works is not a definition for that thing.

Others try to define it mathematically, but that can also be somewhat limiting, because then only the devices that fit into your mathematical scheme are computers, and there are multiple mathematical models that would all be considered “computers.”

So I turned to the dictionary. That was fun for me–I’m a dictionary fanatic. I’ve got lots of great dictionaries, and there are even more online. The Compact Oxford English Dictionary had the best definition, as it turned out.. I was very happy with it at first, but when I started to think about it, it didn’t quite work. For example, it calls computers “an electronic device,” and we know that computers can be built without electronics.

So I worked to come up with a definition of my own. Strangely enough, the key question that it boiled down to was “Why is a player piano not a computer?” It “processes information” by playing notes from its roll. If you gave it an etching machine, it could “store information” back on to the roll. But despite all that, it’s clearly not a computer. What is a computer doing that is fundamentally different from a player piano, that a player piano could never do?

After about two years, I finally came up with an answer that was both simple and all-encompassing. A computer is:

Any piece of matter which can carry out symbolic instructions and compare data in assistance of a human goal.

And that, my friends, is really it. My only thought left is whether I should say “a series of symbolic instructions” to more clearly differentiate it from a calculator. What do you think?

-Max

Comments: 34

10. Dudes at Silicon Valley parties will think you’re, like, cool.

    Impress people you don’t know! You can say, “I work on an open source project,” and nod your head like you’re cool. But no, more seriously, about 50% of all the people I meet at Silicon Valley parties are totally amazed that I’m one of the primary developers of Bugzilla, something that they use every day.

    This doesn’t work so well with the ladies, though, usually. “Yeah, I’m an extremely intense programmer, oh yeah.”

    No, but more seriously: How many girls did you see at OSCON? If anybody has a scheme to get more girls involved or interested in open source, the whole open-source world (the current girls included, I’m sure) would be thrilled.

9. It totally has nothing to do with whether or not you’re a “hottie.” Just be yourself, man, just be yourself.

   Open Source is definitely one place where you’ll be respected for your intelligence and ability, not how expensive your clothes are or how much you paid for your haircut. Nobody cares what you look like. We only care how good your ideas are.

8. It’s hella sick when you’re interviewing!

   When you’re interviewing and you worked on some open-source project, it’s completely valid job experience. Sometimes it even makes you more valuable than normal job experience, if the project is well-known. Also, if you worked on open-source in your spare time, that shows the kind of passion for software engineering that employers are really looking for.

7. You can, like, totally freelance. Seriously? Ya, like totally seriously.

   There’s a lot of need for contractors for certain open-source projects. If you become a prominent contributor to a project and get your name on its Consultants List, then you can make a living doing consulting for people who use the software!

6. Dude, we’ll like, laugh at your jokes and stuff, dude.

   I have laughed harder while listening to some conversations on IRC than I have at almost anything else in the world. There’s something about shared experience and understanding that makes things much funnier than jokes that “everybody” is supposed to get. And where else can you make jokes about programming languages and have multiple people actually get them?

5. You can say how, like, stuff goes and people will actually listen to you. Whoa.

   When you write a feature, to a large degree you’re the one who’s going to decide how it works! Don’t like how a particular program works? Well, maybe you could have been the one who wrote that feature instead! Don’t like some documentation? What if you had written it instead?

   And if you don’t like how something works now, the more you contribute to an open-source project, the more say you’ll have in fixing that thing. Have an itch to just make things work right? Open Source is the place to be.

4. You learn, like, so much stuff, like seriously.

   Are you looking for something new, some way to expand your horizons and learn something new instead of just mechanically doing the same thing over and over at work? No matter what your interest is, there’s going to be some open-source project out there that uses the things you want to learn about. And it won’t just be some tiny project just for yourself, but something that people really use!

3. You can like, belong to something, and stuff. That’s some cosmic stuff, man.

   When you contribute to an open-source project, you’re not just a cog in a great machine, or just a worker at a job. You become part of a larger community, with its own in-jokes, culture, and people.

   I used to think that was just some marketing gibberish, but it’s really true. It may not be the best way to find people who you can “hang out” with every day, but you’ll get to know a lot of new people and become part of a group in a very definite way.

   It’s even more true if you go to conventions like OSCON or the more specific ones for the various open-source projects, where you can meet and hang out with lots of other developers “In Real Life”, most of whom are really great (and intelligent) people.

2. You get to feel like a revolutionary (a revolutionary nerd, but…hey, s’all good, s’all good).

   Yeah! Down with The Man! We don’t need any stinkin’ proprietary software!

   You can even get to protest things, like software patents! You’ll be almost as cool as your parents were in the 60’s. Kind of.

   No, but seriously: Open Source is still a relatively new thing in the world, and you can be a part of blazing its trail. It’s not the “normal” way to do things quite just yet, so if you like being a little different than the swarming masses, open source is the place to be.

1. It seriously helps out, man.

   Working on open-source software helps out a lot of people:

   • The people who use the software. Millions of people use open-source software around the world every day to do their job, handle their problems, or just have fun! You could be affecting the lives of all those people.
   • The people who write the software. Open-source projects almost all really want your assistance! If a project is at all popular, they probably have more feature requests than they can handle. Don’t come to an open-source project and say, “This is what I’m going to do for you,” but do come and say, “How can I help you guys out?” We all need some help, and competent help is much appreciated.
   • All the people who won’t have to write the software that you write. Sometimes you just want to download a program to do a task for you. You don’t want to have to write a program for everything you want to do. You’re saving the time of millions of users and computer programmers, by creating something that everybody can use and modify!

And that’s my list! Hope you had totally chill time, brah.

-Max

Comments: 30

Everybody wants to “innovate.” The news talks about “who’s being the most innovative.” Marketing for companies insists that they are “innovating.”

Except actually, it’s not innovation that leads to success. It’s execution.

It doesn’t matter how good or how new my idea is. It matters how well I carry it out in the real world.

Now, our history books worship the inventors, not the executors. We are taught all about the people who invent new things, come up with new ideas, and plough new trails. But look around you in present time and in the recent past, and you’ll see that the most successful people are the ones who carried out the idea really well, not the people who came up with the idea.

Elvis didn’t invent rock and roll. Ford didn’t invent the automobile or the assembly line. Apple didn’t invent the GUI. Webster didn’t invent dictionaries. Maytag didn’t invent the washing machine. Google didn’t invent web searching. I could go on and on and on.

Granted, sometimes the innovator also is an excellent executor (Alexander Graham Bell being an example), but usually that’s not the case. Most inventors don’t turn out to be the most successful people in their field (or even successful at all).

So stop worrying about “coming up with something new.” You don’t have to do that. You just have to execute an already existing idea really, really well. You can add your own flair to it, maybe, or fix it up a little, but you don’t have to have something brand new.

There are so many examples that prove this that it’s hard not to see one if you move your eyes anywhere. Just look, you’ll see.

Now, I’m not saying that people shouldn’t innovate. You should! It’s fun, and it advances the whole human race a tiny step every time you do. But it’s not the path to long-term success for you or for any group you belong to. That’s all in execution.

-Max

Comments: 39

One of the features of this web browser is its JavaScript engine, called v8, which is designed for performance.

Designing for performance is something that Google does often. Now, designing for performance usually leads to complexity. So, being a major supporter of software simplicity, I’m opposed, in a theoretical sense, to designing for performance.

However, Google is in an interesting situation. Essentially, we live in the Bronze Age of computing (or perhaps the Silicon Age, as I suspect future historians may call this period of history). Our computers are primitive, compared to what we are likely to have 50 to 100 (or 1000!) years from now. That may seem hard to believe, but it’s always hard to imagine the far future. Google is operating on a level far exceeding our current hardware technology, really, and so their design methods unfortunately can’t live in a theoretical fairy-land where hardware is always “good enough.” (However, I personally like to live in that land, when possible, because hardware will improve as time goes on–an important fact to understand if you’re going to have a long-lived software project).

What is it about our computers that makes them so primitive? Well, usually I don’t go about predicting the future in this blog, or even talking about too many specifics, because I want to keep things generally applicable (and also because the future is hard to predict, particularly the far future). But I will talk about some of my thoughts here on this, and you can agree with them or not, as you please.

Our Current Computers

First, you have to understand that to me, anything that could be running this web browser right now, that could be showing me my desktop, and that I could be typing into right now–I’d consider that a computer. It actually doesn’t matter what it’s doing underneath, or how it works. A good, offhand definition then of a computer would be:

Any piece of matter which can carry out symbolic instructions and compare data in assistance of a human goal.

Currently computers do this using math, which they do with transistors, digitally. The word “digital” comes from “digit”, a word meaning, basically, “fingers or toes.” For most people, your fingers and toes are separate, individual items. They don’t blend into each other. “Digitally” means, basically, “done with separate, individual numbers.” You know, like 1, 2, 3, 4–not 1.1, 1.2, 1.3. People (normally) have 1 or 2 fingers, not 1.5 fingers.

Said another way, current computers change from one fixed state to another, very fast. They follow instructions that change their state. I don’t really care if we say that the state is “transistor one is on, transistor two is off, transistor three is on…” or “Bob has a cat, Mary has a dog, Jim has a cat…” it’s all a description of a state. What we care about ultimately is the total current state. If there are 1,000,000 possible states (and there are far, far, far more in a current computer), then we can say that “we are at state 10,456″ and that’s all we really need to know.

The problem with current computers is Moore’s Law. We don’t need computers that are twice as fast. We need computers that are about a million times faster than the ones we have. We need computers so fast that software engineers never have to worry about performance ever again, and can just design their code to be the sanest, most maintainable thing possible. With that kind of performance, we could design almost any software imaginable.

The problem is that with Moore’s Law, we’re not going to get computers 1000 times faster for about 20 years. We’re going to get to 1,000,000 times faster in about 40 years. And there’s a chance that the laws of physics will stop us dead in our tracks before that point, anyhow.

Future Computers

So, let’s stick with the idea of a machine that changes states for the near future, because I can’t think of any other clever way to make a computer that would follow my definition from above. There are three problems, then:

1. How many states can we represent?
2. How many physical resources does it require to represent all those states (including space, power, etc.)?
3. How quickly can we change between states?

And then “How many states can we represent at once?” might also be a good question–we’re seeing this come up more and more with dual-core and quad-core processors (and other technologies before that, but I don’t want to assume that everybody reading my blog is an expert in hardware architecture, and I don’t want to explain those technologies).

So the ideal answers are:

1. We can represent an infinite number of states.
2. It requires no physical resources to represent them.
3. We can change between them without time.

And then also “We can represent an infinite number of different states at once.”

Currently we theoretically could represent an infinite number of states, we’d just have to add more transistors to our chip. So really, the question becomes, “How many states can we represent with how many physical resources?” Currently we can fit two states into 32 nanometers. (That’s one transistor.)

My suspicion is that the future is not in fitting two states into a continually smaller space, but in fitting a near-infinite number of states into a slightly larger space. Electricity and other force waves can be “on” or “off”, but they also have lots of other properties, many of which are sufficient to represent an infinity (or near-infinity). Frequency of any wave represents an infinity, for example–you can have a 1 Hz wave, a 1.1 Hz wave, a 1.15 Hz wave, a 1.151 Hz wave, etc. So, that basically answers Question 1 ideally–you can have an infinite number of states, you just have to have some device which is sufficiently small, in which a wave can have its properties modified and measured by electronics, optics, or some other such technology.

You’ll notice that we’ve also conveniently answered our bonus question, because we can represent quite a few different states at once, once each individual component of our system can represent an infinity all by itself.

If we want to look a bit further into the future, our second question can be answered by the fact that waves take up essentially no space (only the medium that they vibrate takes up space). Our understanding of physics is not (as far as I know) currently good enough to create structures out of pure force just yet, but such structures would come quite close to taking up “no physical resources.”

And beyond that (how we get the state changes to happen without time), I have no idea. That question may be unanswerable, and may only be resolvable by changing computers to being something other than mathematical devices. (That is, not be involved with states at all, but some other method of following instructions and comparing data.) But the better our components become, the closer we can get to “no time.”

The Roundup

So there’s my thoughts for the day on the future of computing. Sometimes designing software for performance is a necessary evil (but really only in the case where it’s an extreme issue, like with Google’s products, or the great new need for speed in JavaScript nowadays, or in other low-level places), but I hope that future changes in the fundamental architecture of computers will obsolete that necessity.

-Max

Comments: 14

My policy on projects that I control is that we never add a feature unless the design can support it simply. This drives some people crazy, notably people who have no concept of the future. They start to foam at the mouth and say things like, “We can’t wait! This feature is so important!” or “Just put it in now and we’ll just clean it up later!” They don’t realize that this is their normal attitude. They’re going to say the same thing about the next feature. If you give in to them, then all of your code will be poorly designed and much too complex. It’ll be Frankenstein’s monster, jammed together out of broken parts. And just like the friendly green giant, it’ll be big, ugly, unstable, and harmful to your health.

Adding a tiny little piece and refactoring it afterward is fine. Landing a huge feature that the architecture can’t support and then trying to clean it up afterward is a terrible task. Size matters.

The worst situation, however, is when you let people keep adding features with no design for months or years, and then one day you wake up and realize that something is not right. Now you have to fix your whole codebase. This is a terrible task, because just like adding a new feature, it can’t be done all at once, unless you want to re-write. If you want to start doing things the right way, you have to start doing things the right way. And that means that you have to fix the design piece by piece, in simple steps. That usually requires months or years of effort–totally wasted effort, because you should have just designed from the start. You should have thought about the future.

In your project lacks a strict design, and it continues to grow, then you will eventually end up over your head in complexity. I understand that this is hard for some people to imagine. Some folks can’t imagine that there is a future beyond lunch. Other folks just haven’t had enough experience to understand how complex things can get. And I understand that there can be a corporate culture that says, “Oh, we just hack in new features, and we should do things the right way, but we can’t because blah blah blah.” But one day your project will fail. And no matter how many reasons you can give for that failure, it won’t change the fact that your project failed.

Often, when you’ve done your design right, there’s not a whole lot of credit that comes your way. Catastrophic failures in design are big and noticeable, small increments of work toward a good design are invisible to people who aren’t intimately connected with the code. So this can make it difficult–handling a big failure gets you a lot of thanks, preventing one in the first place, well, nobody noticed.

So I’ll congratulate you myself. Did you design from the start? That was awesome. You absolutely did the right thing. Have you started designing now? Well, you should have started earlier, but congratulations on starting to move in the right direction. Your users and fellow developers will see the benefits–working software, on-time releases, and a clear, understandable codebase. Will they know how much work it took to get it that way? Maybe not. But that’s OK. Sometimes doing things the right way is really its own reward.

-Max

Comments: 8

Imagine that you are building a structure out of lead bars. The final structure will look like this:

   |
_|_|_|_
   |
   |
   |

You have to build the structure and put it up at a certain location, so that people can use it (for this example, we don’t care what they need it for, but assume that they need it for some specific reason).

The lead bars represent the individual pieces of your software. Putting it up at the location is like putting your software into production (or sending it out to your users). Everything else should be fairly clear as to how it translates to software, if you think about it. You don’t have to translate everything to software in your mind as you read, though. Everything should be quite clear if you just imagine that you really are just building a structure out of lead bars.

The Wrong Way

Imagine that you were building this all by yourself, and that you had to make the bars out of raw metal. Here’s the wrong way to build it:

1. Make one tall lead bar, and lay it flat on the ground in your workshop:

   |
   |
   |
   |
   |

2. Cut a hole through the tall bar, and measure that hole.
3. Make a new bar that will fit through that hole:

   _____

4. Put that new bar through the hole and weld them together permanently:

     |
   __|__
     |
     |
     |

5. Cut two holes in the horizontal bar, measure them, and make two new lead bars that will fit in those individual holes:

   |   |

6. Insert the two bars into the horizontal bar, and weld them together permanently:

      |
   _|_|_|_
      |
      |
      |

7. With a forklift, put this into a truck to move it to the location where it’s supposed to be (It’s too heavy to move by yourself.)
8. With a pulley, make the construction stand upright and put it into the ground.
9. Discover that it won’t stay up by itself, but if you put some blocks next to it as an emergency solution, it doesn’t fall over:

      |
   _|_|_|_
      |
     _|_
    | | |

10. Three days later, watch the structure fall over and break because the blocks aren’t actually a permanent solution.
11. Unfortunately, part of the horizontal bar has snapped, and you have to fix it. This is difficult because the bars are all welded together, so you can’t easily take out the bar and replace it with another one. You either have to build a whole new structure or weld together the broken bar. Welding the broken halves together creates a weak bond, but it’s cheaper than building a whole new structure, so you just weld them.
12. Put stronger blocks next to the structure to keep it up.
13. Next week, the weather breaks the welded bars. Weld them back together again.
14. In six days, watch the structure fall over because blocks are not a permanent solution.
15. Repeat the last few steps until you run out of money or time.

Analysis of The Wrong Way

So, what was good about the above process? Well, it did allow one person to successfully complete a structure. In software terms, that one person “made something that works.” It also created a lot of work for one person, which is good if that one person wanted a lot of work.

What was bad about it?

• The bars all had to be made in sequence, individually.
• Problems with the final structure (that it wouldn’t stay up) were only discovered after it was entirely built and in place.
• When problems were discovered, they were just “quick fixed” without planning for the future.
• It took enormous effort to move the completed structure into place.
• If we ever had to change the configuration of the bars, we couldn’t, because they’re welded together. We’d have to build a whole new structure.
• The completed structure requires frequent attention.

And I’m sure we could come up with other faults. This whole analogy (including the parts below) could be analyzed all day.

Bringing It To a Group

The biggest problem with the Wrong Way process is that it wouldn’t work at all if there were multiple people working on the project (as there usually are in real-world software projects). The main problem is that you had to measure all the holes before you built a bar, so everything had to be done by one person, in order.

There are, generally, two ways to solve this problem:

1. Write a specification for the sizes of all the individual holes beforehand, and then spread out the work of making all the different bars for each hole.

   This is problematic because one person has to write this specification, and if this were a large project (imagine thousands of holes instead of just three or four), it would take a lot of time. Nobody else on the team can be working until the specification is completed. The spec could be full of mistakes–there are as many chances for mistakes as there are holes specified, so if there are thousands of holes, that’s a lot of chances for errors to be made.

2. Just say, “All bar holes will always be the same size and in the same places on the bars. Bars can be screwed together.” Then set everybody to making bars with standardized holes (or go buy them from the store).

   That is simple, and it gets everybody working at once. Because you’ve standardized your bars, you’ve lost a little flexibility in dealing with any special cases that come up (maybe a half-width hole would be more useful in some part of the structure). However, you should be able to build a decent structure entirely with standard holes, so that’s not too much of a problem. And when you have a standard, you can make specific exceptions in some places more easily than if things are not standardized.

   Of course, with this method it is very important that you do a little research to pick a good hole size and good bars.

The Right Way

So, what would our process look like for many people all using standardized bars that screw together? (This is the right way to build something.)

1. Have your team all go build (or buy) their individual bars. You can have as many people working simultaneously as there are bars in the structure.
2. Have them test their individual bars to make sure that they won’t break.
3. Have them carry their individual bars to the place where the structure needs to be.
4. Put the first bar into the ground, standing upright:

   |

5. Push on the first bar from all angles to see if it is going to fall over.
6. Screw in a second bar to the first one:

   |
   |

7. Test the complete structure now, only to find that it’s not strong enough to stand by itself.
8. Attach unbreakable steel ropes to the sides of the structure, like so:

     /|\
    / | \

   These ropes should be able to withstand anything within reason, or even well beyond reason.

9. Test it again and find out that it now can stay up no matter how hard you push on it.
10. Add a third bar, and put new ropes on so that it looks like this:

       /|\
      //|\\
     // | \\

11. Remove the lower ropes:

       /|\
      / | \
     /  |  \

    (Anybody who’s been involved in refactoring Bugzilla can remember doing a lot of things that sound just like these last two steps. )

12. Test it again.
13. Continue these steps until you have a completed structure:

          |
       _|_|_|_
      /   |   \
     /    |    \
    /     |     \

14. When a pipe breaks in three months, figure out what was wrong with that pipe, fix the problem, and replace it with a new pipe that fits into the same holes. The structure is just as strong as it was before.
15. Continue the above process until you no longer have to pay attention to the structure and it just stays up all by itself.
16. Adjust the structure as necessary for the changing requirements of the users of the structure, which is easy because the holes are all standardized.

So, did you notice that we followed all the Laws Of Software Design?

• We thought about the future. We did that for the entire process, but we particularly did it when we put on unbreakable steel ropes that would last no matter what happened in the future.

  Also note that we didn’t try to predict the future, we just followed our principles so that no matter what happened, our structure was going to stay together and be easy to build.

• We allowed for change by screwing the bars together instead of welding them. We also put standardized holes in all the bars, even if we didn’t need them, in case we needed to add more bars in the future.
• In every step of creating the structure, we kept our changes small and tested everything as we went. Creating each individual bar was a small task, and we put them together in small steps.
• And of course, the most important decision we made was to keep it simple by making all the holes consistent and standard, and keeping each piece small and simple.

Whether you are one person or a thousand, whether your project is 10 lines of code or 10 million, this process will work.

-Max

Comments: 2

I’ll be talking on Thursday, July 24, at 3:25pm, in either room E143 or E144 at the Oregon Convention Center.

I’ll tell you basically everything I know and have learned about software design, and then how it applies to Open Source software, all in about 45 minutes. Should be fun and interesting!

-Max

Comments: 0

But really, when we say “bug” what exactly do we mean?

Here’s the precise definition of what constitutes a bug. Either:

1. The program did not behave according to the programmer’s intentions.
   or
   
2. The programmer’s intentions did not fulfill common and reasonable user expectations.

So usually, as long as the program is doing what the programmer intended it to do, it’s working correctly. Sometimes what the programmer intended it to do is totally surprising to a user and causes him some problem, so that’s a bug.

Anything else is a new feature. That is, if the program does exactly what was intended in exactly the expected fashion, but it doesn’t do enough, that means it needs a new “feature.” That’s the difference between the definition of “feature” and “bug.”

Note that hardware can have bugs too. The programmer’s intention is rarely “the computer now explodes.” So if the programmer writes a program and the computer explodes, that’s probably a bug in the hardware. There can be other, less dramatic bugs in the hardware, too.

Essentially, anything that causes the programmer’s intentions to not be fully carried out can be considered a bug, unless the programmer is trying to make the computer do something it wasn’t designed to do. For example, if the programmer tells the computer “take over the world” and it wasn’t designed to be able to take over the world, then the computer would need a new “take over the world” feature. That wouldn’t be a bug.

With hardware, you also have to think about the hardware designer’s intentions, and common and reasonable programmer expectations. At that level, software programmers are actually the main “users”, and hardware designers are the people whose intentions we care about. (Of course, we also care about the normal user’s expectations, especially for hardware that users interact with directly like printers, monitors, keyboards, etc.)

-Max

Comments: 4

Well, okay, maybe it’s not all opinion. There must be some way to tell a good technology from a bad one (besides looking back after five years of development and saying, “Wow, we really shouldn’t have decided to base our product off of Microsoft Bob.”)

When I’m evaluating a technology for inclusion in one of my projects, I look particularly at the technology’s survival potential, interoperability, and attention to quality.

By “survival potential” I mean “how long is this software going to continue to be maintained?” If you get stuck with libraries or some dependency that becomes obsolete, you’re really in for some trouble. You can get some idea of the survival potential of the software by seeing how often they’ve released recently. Also, how responsive are they to bug reports? Do they have a mailing list that’s very active with users and developers? Are there lots of people online talking about this technology? If a technology has a lot of momentum now, you can be fairly sure that it’s not going to die any time soon.

By “interoperability” I mean, “Does this system use some standard that would allow us to switch to a different system if we want, in the future?” For example, some database systems support standard SQL very well. Some other ones aren’t as good about that–they have strange behaviors that don’t agree with the standard, or they just don’t support it at all. Using a database with standard-SQL support would allow you to switch to any other standard-SQL database in the future, whereas using a less-standard database would lock you in.

The final one, “attention to quality,” is more of a subjective measurement, but the idea is to see if the product is getting better in its recent releases. If it’s open-source, check if they’re refactoring and cleaning up the code base. Is it becoming easier to use or more complex? Do the people who maintain the technology actually care about the quality of their product, or are they just code monkeys working for the pay? Are there are a lot of serious security vulnerabilities in the software that have been published lately?

Lock-in is a worse problem when you’re using proprietary software than when your dependencies are open-source. Proprietary vendors can just stop supporting something (or go out of business), whereas in the open-source world anybody is free to pick up a codebase and maintain it if the original maintainers drop it. Granted, open-source projects are just as likely to fall out of existence as proprietary projects are, but at least there’s the hope (particularly if the project was very popular) that somebody will come around to maintain it again, with open-source.

Proprietary vendors also have a monetary incentive to lock you into their technologies–even their free technologies. Once you start using their free technologies, they can sell you services and products related to them. Open-source vendors usually lack that monetary incentive, and so are more likely to be “friendly” when you want to switch away from them. This isn’t always true (some proprietary vendors beat the pants off of open-source vendors in terms of interoperability, and some open-source products will still lock you in), but it’s enough the case that I automatically think “let’s use open-source” when I’m looking for interoperability.

There are other aspects to look at when you’re choosing a technology, and some of it really is opinion. Some people like the way Ruby looks better than the way Python looks. That’s a valid reason to choose a technology sometimes–if you just like some technology more than another one, and everything else seems equal according to the criteria above, go with the one that makes you happy. After all, you’re the one who’s going to be using it–your opinion matters! All I’ve tried to do here is give some universal guidelines that can be used to weed out the definitely bad choices, and the rest is up to your personal research, needs, and desires.

-Max

Comments: 3

There are some things about the future that you do not know.

Obviously it’d be ideal if we were all-knowing and could perfectly predict every consequence of every decision we’ll ever make. But that’s impossible. In fact, it’s so far from possible that if you predict more than half of the consequences of your engineering decisions, you’re godlike or just really lucky.

Let’s take an example outside of the realm of programming: CDs, which were designed in 1979 to replace cassette tapes as the primary method of listening to music. Who could have predicted that 20 years later, DVDs would be made the same size and shape so that manufacturers could make CD/DVD drives for computers? Did anybody imagine the problems of spinning a CD fifty times faster than it was supposed to be spun, for reading in a CD-ROM drive?

This is why, in engineering, we have “guiding principles.” There are certain rules that, when we follow them, keep things working well no matter what happens in the future.

In the case of software, I’ve summed up the most basic guiding principles as the Laws Of Software. The very first (and most important) law is that the future is more important than the present (because it’s very big and the present is very small). But here’s the thing to know about that: that doesn’t mean you have to predict everything about the future. Instead, it explains why you should be making decisions according to the laws of software–because those laws make for good future software, no matter what that future is.

Sometimes it can be hard to understand why you should be stupid, dumb simple, if you’re only thinking about the present. And it can be impossible to predict exactly why or how simplicity will help you in the future. I can tell you that it will help, and you’ll be glad you did it, and if you don’t believe me (which you are welcome to do–your mind is yours!) you’re going to end up in mess of trouble somewhere down the line, in a future you can’t predict.

I’m not singling you out, here. You’re not dumb. It’s just that nobody can predict the total future of a piece of software, except to know that certain decisions now will make that future better, and that other decisions will make that future worse. We know this from decades of experience in the software development industry–certain basic principles invariably lead us in the right direction, and certain “anti-patterns“, if followed consistently, invariably lead us into trouble.

To be a successful software developer, you have to be willing to not know some things about the future, and trust that following your principles is the right thing to do. Sometimes it can take years to see that you were right, but it’s a glorious day when you finally realize that your product still exists because you made the right decision three years ago, when you couldn’t even prove to anybody that it was the right decision, but forged ahead anyway with your “excessive simplicity” because you knew it was the right thing to do.

-Max

Comments: 5

“We started working on this project five years ago, and the technology we were using/making was modern then, but it’s obsolete now. Things keep getting more and more complex with this obsolete technology, so it keeps getting less and less likely that we’ll ever finish the project. But if we re-write, we could be here for another five years!”

Another popular one is: “We can’t develop fast enough to keep up with modern user needs.” Or, “While we were developing, Google wrote a product better than ours much faster than us.”

When I hear things like that, the first thing I ask myself is “How did that happen? Why did it take so long for them to finish their product that they ran into that problem?”

The answer lies in complexity. The more complex a task is, the harder it is to complete it. So you start out with something simple that can be completed in one month. Then you add complexity, and the task will take three months. Then you take each piece of that and make it more complex, and the task will take nine months.

Complexity builds on complexity–it’s not just a linear thing. It’s not like, “We have ten features, and so adding one more will only add 10% more time.” No, one new feature will have to be coordinated with all ten of your existing features. So if just the feature itself takes 10 hours of coding time, there will be another hour of coding time for that feature interacting with each other feature. The more features there are, the higher the cost gets of adding a feature.

Some projects start out with such a complex set of requirements that they never get a first version out. If you’re in this situation, you should just trim features. Don’t shoot for the moon in your first release–get out something that works and make it work better over time.

There are other ways to add complexity than just adding features, too. The most common other ways are:

• Expanding the purpose of the software. Generally, just don’t ever do that. Your marketing droids might be drooling over the idea of “making a single piece of software that does your taxes and cooks dinner”, but you should be screaming as loud as you can whenever any suggestion like that comes near your desk. Stick to your purpose–your software just has to do what it does well, and you will succeed, if that purpose is something people need.
• Adding programmers. Yes, that’s right–adding more people to the team adds complexity, it does not make things simpler. Remember The Mythical Man Month? What he says in there is true because of the complexity equation I explained higher up in this article–if you have ten programmers, adding an eleventh means spending time to groove in that one programmer, plus the time to groove in the existing ten programmers to the new guy, plus the time spent by the new guy interacting with the existing ten programmers.
• Change things: Any time you change something, you’re adding complexity. Whether it’s a requirement, a design, or just a piece of code, you’re introducing the possibility of bugs, the time required to implement the change, the time required to validate that the new change works with all the other pieces of the software, the time required to decide upon the change, and the time required to track the change, and the time required to test the change. Each change builds on the last in terms of all this complexity, so the more you change, the more and more time each new change is going to take. It’s still important to make certain changes, but you should be making informed decisions about it, not just changing everything on a whim.
• Lock-In to bad technologies. This is where you make a bad decision about your backend or libraries and then are stuck with it for a long time because you’re so dependent on it. Obviously “bad technology” is very subjective, but sometimes there are obvious good choices and obvious bad ones. For example, if you need an embedded scripting language, Lua is generally considered to be a good choice, and “write our own” would probably be one of the worse choices, depending on the situation. It’s definitely relative–it’s not like there’s only one good choice and all the rest are bad, but some choices might make your life easier than others.
• Poor design or no design. Basically, this just means “a failure to plan for change.” Things are going to change, and that requires design work, to maintain simplicity while the project grows. Failing to do this can introduce massive complexity very fast, because suddenly each new feature quadruples the complexity of the code instead of just adding a little bit to the complexity.
• Re-inventing the wheel. For example, if you invent your own protocol when a perfectly good one exists, you’re going to be spending a lot of time working on the protocol, when you could just be working on your software. You should basically never have any huge invented-in-house dependency, like a webserver, a protocol, or a major library, unless that is your product.

The thing about all of these is that they’re insidious. Most of them only do long-term damage–something you won’t see for a year or more. So when somebody proposes them, often they sound harmless! And even when you start implementing them, maybe they seem fine. But as time goes on–and particularly as more and more of these stack up–the complexity becomes more apparent and grows and grows and grows, until you’re another victim of that ever-so-common horror story: “The Never-Shipping Product.” (Which is nowhere near as cool as The Neverending Story, believe me.)

-Max

Comments: 1