What's bugging me? I'll tell you what's bugging me. It's when my dashboard dials stop working right.
My first car was a 1978 Datsun B210 sedan that I bought when I was in graduate school. It was a very simple vehicle with nothing fancy on it. But it did have a few, um, quirks. I bought it in the bitterly cold winter of 1983 and promptly spun it in a 360 degree circle in an intersection in Loveland, Colorado late at night after visiting with my then-fiance (and now wife) Deb. Almost all of my interesting car stories come from that car, whom we had dubbed "Elliot."
One time I hopped in Elliot and the gas gauge had moved from near empty to three-fourths full. I was terrified that the fuel level indicator had broken, and I actually had no idea how much gas I really had in the car.
Fortunately, Deb had been nice and had just put some gas in the car and hadn't told me. But it did make for a moment of panic.
I believe that the right indicators can be a good way to measure a software project's health. But rather than having 50 different indicators measuring all kinds of things, I like having a very small set of really good ones and asking questions when they are warranted.
So you can imagine my shear panic when I saw one of my favorite indicators go non-linear during this development cycle of the Yocto Project. Here is a snapshot of the weighted defect density for the v1.3 development cycle:
(You can find the current version on our project wiki in the QA Project page).
This is such a simple metric - take the number of open bugs and multiply them by their severity. As we get close to a release, we should see the line slope towards some mutually acceptable level, which in this case is the dotted green line.
It has been clear for some weeks that we would not be hitting that goal. What the heck is going on?
- Did we goof up how we were counting?
- Was there a change in the way the metric was calculated?
- Or (shudder, shudder) is the software really that much buggier? A total failure if we're trying to fix a lot of bugs in this release.
My first thought was that we were counting bugs against both the current development effort (v1.3) plus bugs that were in v1.2 and earlier. The answer is yes, but were were always doing that. The red line is a measure of just the bugs found against v1.3. As you can see, this is still pretty non-linear.
After a lot of analysis by Song Liu and others, I believe I have an answer:
- With operating system products from Mentor Graphics and Wind River now with the Yocto Project as its upstream, we have a lot more large-scale commercial usages of the project. This means we have a lot more people banging on the code in more creative ways and breaking some things we never saw before. I believe this means that the bugs were always there, we just didn't know it.
- Mid-cycle, we added a bunch of new engineers to the project, many of them doing QA work. This meant that often they were submitting "pilot error" bugs or bugs that were duplicates of ones already there.
- As we analyzed the numbers more deeply, we found that we really had achieved our goal of higher quality as measured by bugs fixed.
So I'm satisfied that we are headed the right direction on the Yocto Project v1.3 release and we can only get better from here. Now we just need to reset our "goals" so our gauges work well again.