M2, Moneyball, and a new way to create software

Cross-posted from my blog on the Yocto Project website. Check us out there and the other things we're doing on Embedded Linux.

I sat down this morning to jot down a few words about the latest Yocto Project development milestone, which has the very homely name of "M2". We pause at this time to branch our code, run a full pass of our QA suite and make sure we are on track with major features. (This is all good by the way).

The developers working on the Yocto Project are a very interesting bunch - we have people who have worked in open source projects for their entire careers and others who come from the world of product development. It's fascinating to see these worlds intermix as we try to do the right thing to make Linux the  best choice for embedded development.

Now before you think I'm totally confused on this point, let me assure you, I am not. The Yocto Project is not a product. It is an open source project, which will form the upstream for products, ranging from devices to board support packages to operating systems from the like of Mentor Graphics and Wind River Systems. To be a stable basis for these products, we take seriously the need to track the health of our bits as we develop them.

You can track project health with all kinds of metrics and dashboards and charts. Often it comes down to the experience and intuition of the project leaders to figure things out.

In 2001, the Oakland Athletics baseball team was eliminated from the playoffs by the New York Yankees, a team with three times the money available for players. How could the Athletics (or A's as they are called) ever hope to beat a team with so much money to spend? The tale is told in Moneyball, nominated for the 2011 Best Picture Oscar.

Now, I will confess, I am not such a fan of baseball, and I have a hard time caring about such things. What drew me in was the way players are traditionally chosen in baseball. Professional scouts, who are quite experienced in baseball, will evaluate a player based on everything from their statistics to how pretty their girlfriend is.

How in the world could you evaluate a player on whether they had an attractive girlfriend? This was part of the intuition the scouts would use to indicate a player's confidence.

The A's general manager tries a different tactic. Could you apply economic theory and create a formula that would boil down all of the metrics for a player and create a single number to evaluate them? And can the movie's makers take such a dry topic and make it interesting, as Aaron Sorkin did with The Social Network, 2010's Best Picture.

Well, you can judge that last bit for yourself. I thought it was very good (and beautifully photographed as well).

How about the Yocto Project? Can we as leaders boil all those statistics down to a single number to tell us the health of the project? I'd like to think we do.

Meet the Weighted Defect Density:

 

WW02 weighted defect density.JPG

This is just a snapshot I grabbed from the end of https://wiki.yoctoproject.org/wiki/Yocto_Bug_Trend - you can see all kinds of other statistics in there as well. But this is the one I look at first when I want to know how we're doing. TO compute it, we eliminate the bugs which are not defects (there are enhancements and features tracked in Bugzilla as well) and then weight the open bugs by their severity and track this number over time.

This single trend chart gives us a lot of insight into the project. It helps us to ask more questions about what is going on, to drill into other data and to potentially change course. Do we think we're going to hit our goals for the release? Do we need to stop development work and focus people on bug fixing for a while? Maybe we need to stop testing so much and work on fixing bugs. Or maybe the line is lower than we expect it to be and we should be doing more testing.

Using a Weighted Defect Density in a project is not a new idea[1]. I first heard about it and used it way back in the early 1990s. But it has proven to be a good indicator of Yocto Project health and helped us make informed decisions.

In Moneyball, the hero played by Brad Pitt says his ultimate goal is to change the way the game of baseball is played. He sounds like he could be on the Yocto Project. We're trying to change the way the world develops devices. Time will tell if we're right.

 

[1] Robert Baetke, an associate at Sequent Computer Systems learned about the Weighted Defect Density metric at a talk given about how Boeing developed the 787 aircraft. He brought the tool to Sequent and employed in there, and I am very grateful for his contribution to my way of thinking about development.

For more complete information about compiler optimizations, see our Optimization Notice.