So, after my last post, a few people asked, “Okay, but how do you figure out what sucks?”
Well, some of it’s really obvious. You press a button and the program takes 10 minutes to respond. That sucks pretty bad. You get 100 complaints a week about the UI of a particular page–okay, so that sucks.
Usually there are one or two HUGE things that really suck, and they’re really obvious–those are the things to focus on first, even if they require a tremendous amount of work. For example, before Bugzilla 3.0, Bugzilla had to compile every single library and the entire script it was about to run, every time you loaded a page. This added several seconds to each page load, on slower machines, and at least 1 second on faster machines. So performance was one big obvious thing that sucked about Bugzilla. But even more importantly, the code of Bugzilla sucked. It was being read by everybody–because people were frequently customizing Bugzilla at their company–and it was an unreadable, garbled mess.
Thankfully, both of those problems had the same solution. The performance problem was solved by allowing people to run Bugzilla in a way that would pre-compile all our scripts when the web server started, instead of every time somebody loads a page. And to enable that pre-compiling, we had to do enormous amounts of refactoring. So, we actually ended up handling our performance problem by handling our code problem.
However, it took four major releases (Bugzilla 2.18, 2.20, 2.22, and finally 3.0) to get all this done! We fixed a lot of little issues for each release along the way, too, so each release really did suck less than the previous one. But handling the major issues was a tremendous effort–it wasn’t just something we could code up in one night and have it be done with.
I think sometimes the big issues in a software project don’t get handled because they do require that much work to fix. That doesn’t mean you can ignore them, though. It just means that you have to plan for a long project, and figure out how you can keep getting releases out in the mean time.
After all that was fixed, then we could turn our attention elsewhere, and wow! It turned out that elsewhere, there were still a bunch of things that sucked! All of the sudden, there were a new batch of “totally obvious” things to fix–things that had been there all the time, but were just overshadowed by the previous set of “totally obvious” things.
Now, we could have just gone on like this forever–fixing one set of “totally obvious” problems and then going on to the next set of “totally obvious” problems. But we ran into an issue–what happens when suddenly, you get to the point where there there are fifty “totally obvious” things that need fixing, and you can’t get them all done for one release? Well, that’s when you suddenly need some method of prioritizing what you’re going to fix.
With the survey, the most important part was allowing people to respond in free-form text, to questions asked to them personally. That is, I sent the questions from me personally to Bugzilla administrators personally, often customizing the message for their exact situation. And there were no multiple-choice questions, only questions that prompted them to tell us what was bothering them and what they wanted to see. They were actually really happy to get my emails–lots of them thanked me for just doing the survey.
Once they had all responded, I read everything and compiled a list of major issues that were reported–overall a surprisingly small list! We’re focusing on those issues pretty heavily nowadays, and I think it’s making people happier with Bugzilla in general.
With the usability study, surprisingly the most helpful part was when the researchers (who were usability experts) just sat down in front of Bugzilla and pointed out things that violated usability principles. That is, even more valuable than the actual research they did was just their observations as experts, using the standard principles of usability engineering. The fact that they were fresh eyes–people who’d never worked on Bugzilla and thus didn’t just think “well that’s the way it is”–also was important, I think.
So we took all this data, and it really helped us prioritize. However, it’s important that we did the survey and research when we did them and not earlier. Back before we fixed the top few major issues, the usability and survey results would have just been overwhelming to us–they would have pointed out a million things we already knew, or a lot of things that we just didn’t have the time to work on at that point, and we would have had to re-do the survey and research again later, making it all a bunch of wasted time. So we had to wait until we were at the point of asking ourselves, “Okay, what’s most important now?”, and that was when gathering data became tremendously important and incredibly useful.
So overall, I’d say that when you’re trying to make things suck less, first go with what you know are the top few big obvious issues, and handle those, no matter what it takes. Then things will calm down a little, and you’ll have a huge pile of stuff that all needs fixing. That’s when you go and gather data from your users, and work to fix whatever they tell you actually sucks.