Why Linux people lust after DTrace

By David Stewart (Intel) (138 posts) on May 15, 2007 at 12:25 pm

Back in 2002, I was managing the team that worked with Oracle on the engineering side.  This was when they were transitioning from developing the database on Solaris to Linux, and we were helping them with this transition.  Of the various requests their devs made, one of the first was "when can we get DTrace on Linux?" 

I was a bit confused, since I had never heard of DTrace! 

Fast forward to 2007, and I sat listening to Bryan Cantrill, the inventor of DTrace show some examples of the power of this little tool which comes with Solaris. 

The recent example they used was Twitter.  Remember my recent comments about this new micro-blogging application?  Well it turns out that the gang at Twitter implemented their system using Ruby on Rails, the new and sexy rapid website programming system.  After the article in the New York Times, Twitter has grown in popularity to be the biggest application of Ruby on Rails.  And of course, they were having performance problems. 

Fortunately for Twitter, they implemented it on Solaris, and thus DTrace to the rescue.  Within a few hours, they had identified an issue with how often deep stack back traces were being taken, they get a fix from the Ruby guys and had a 30% performance boost.  Pretty snaz, yeah? 

No wonder I saw DTrace listed as one of the top 10 reasons Solaris is better than Linux.

Categories: Open Source

Comments (27)

May 15, 2007 2:33 PM PDT

Josh Bancroft (Intel)
Total Points:
3,492
Status Points:
2,992
Brown Belt
 Great story, David. I'm an avid Twitter user, and anything that helps them improve performance and uptime is great in my book. ;-)
May 16, 2007 7:18 AM PDT


blogstar		 Of course if you actually have your 5 nines, "improving uptime" hardly merits a mention.

At this point, I'm not sure if twitter has 2 nines, let alone more, and has been plagued for a fortnight with 503's, offline messages, and null pages.
May 16, 2007 7:18 AM PDT


z		 Isn't there KTrace for Linux?
May 16, 2007 7:35 AM PDT


Brian Moyles		 Linux has its own DTrace-alike coming along, and some would argue that it's better than DTrace in some ways.
http://sourceware.org/systemtap/
May 16, 2007 8:25 AM PDT


S		 Solaris has had very slow startup times for a process. Historically thats been one reason to avoid it.
May 16, 2007 9:39 AM PDT


dazza		 Efforts are being made to port DTrace to FreeBSD. I think it's scheduled for release with FreeBSD 7 http://people.freebsd.org/~jb/dtrace/
May 16, 2007 10:47 AM PDT

Josh Bancroft (Intel)
Total Points:
3,492
Status Points:
2,992
Brown Belt
 Oh, cool! It looks like DTrace is coming to Mac OS X Leopard, with a cool GUI. It's called Xray, and it's part of the XCode 3 IDE that's coming when Leopard launches.

http://www.apple.com/macosx/leopard/xcode.html

Can't wait to try it out! :-)
May 16, 2007 10:53 AM PDT

David Stewart (Intel)
Total Points:
9,435
Status Points:
9,435
Black Belt
 Yes, well of course if you LIKE the GUI interface, it's good to have one. I think I heard about one that's Solaris-based as well.

Lack of a GUI doesn't phase most *nix developers. In fact they prefer the command line interface for development when they can.

I could write a bunch about the GUI vs non-GUI approach for development. A lot seems to be based on where you got your start as a developer.
May 16, 2007 11:21 AM PDT


David		 Try SystemTap: http://sourceware.org/systemtap/
May 17, 2007 2:36 AM PDT


Oliver Herold		 http://people.freebsd.org/~jb/dtrace/

MacOS, FreeBSD ... where is the problem with Linux? Some silly license-war?
May 17, 2007 5:47 AM PDT


Tet		 SystemTap is the Linux equivalent of DTrace. True, it's not yet quite as mature as DTrace, but it gives you 90% of what most people want, and it's getting better all the time...
May 17, 2007 8:41 AM PDT

David Stewart (Intel)
Total Points:
9,435
Status Points:
9,435
Black Belt
 Comment from Oliver Herold: "MacOS, FreeBSD ... where is the problem with Linux? Some silly license-war?"

*I* don't have any problem with Linux. I think Linux is great and my hat is off to Linus and the Community for pulling it off.

Sun did choose to license the Solaris kernel under CDDL rather than GPL.

In brief: they felt that GPL wouldn't allow them to protect the IP rights of some of the code suppliers to the kernel and BSD wasn't protective enough of the rights of the Community. So they compromised between the two.

So actually there are some advantages of Solaris over Linux. Dtrace is one of them.
May 17, 2007 11:46 PM PDT


mangoo		 Have you checked SystemTap on Linux? It's more or less an equivalent of DTrace (some will argue it's less, some - it's more).

SystemTap website: http://sourceware.org/systemtap/

Some usage examples:

http://sourceware.org/systemtap/wiki/WarStories
http://sourceware.org/systemtap/wiki/ScriptsTools
May 18, 2007 4:17 AM PDT


Alok		 Just use /usr/bin/strace and grok the system calls etc. Write a perl script to parse the log output to get required details. More work, but good enough in most cases.
May 18, 2007 1:30 PM PDT


Bryan Cantrill		 Glad that you enjoyed the talk, Dave. In retrospect, I got a bit fooled by the "CommunityOne" moniker into thinking that the presentations were more informal than they in fact were. So if you had never seen DTrace before I started demo'ing some of our newer features, it might have been a bit of a wild ride. ;)

To address some of the issues that have come up in the comments here: yes, DTrace has been ported to FreeBSD and MacOS X Leopard -- which we're naturally very excited about. (The thought of potentially having DTrace on my phone is almost too much excitement to bear.)

In terms of SystemTap and DTrace, I would refer interested parties to Stephen O'Grady's blog entry on the topic (http://redmonk.com/sogrady/2006/04/07/linux-responds-to-dtrace-systemtap-on-tap/).

As for the license issues, I will only comment that the issue here is with the GPL, not with the CDDL -- as far as CDDL is concerned, you can drop DTrace into Linux and off you go...
May 18, 2007 1:32 PM PDT


Bryan Cantrill		 Whoops -- no HTML syntax apparently; here's the DTrace/SystemTap link:

http://redmonk.com/sogrady/2006/04/07/linux-responds-to-dtrace-systemtap-on-tap
May 20, 2007 6:07 AM PDT


Steve		 The problem with DTrace that it only help to the developers of crap applications who tries to push the debugging to the system administrators.
Poor sysadmins then have to use DTrace to figure out why that crap application doesn't work as expected.
It's not a cool application, it only something that a well designed system like Linux should avoid to implement.

No wonder, Solaris needs to have such a tool..
May 21, 2007 8:49 AM PDT

David Stewart (Intel)
Total Points:
9,435
Status Points:
9,435
Black Belt
 Hmmm ... methinks someone is trying to bait me! :-)

Having been in the real world a few years, I can say for sure that people's intentions often don' reflect reality. The reality is that there are plenty of applications which started life as a prototype, but which went on to become the production system without being rewritten.

Most often, these systems go into production and later on as more load comes in, there are strange performance problems or bugs, and there is no good way to instrument live code on a production system to see what's up.

This is where DTrace, built into Solaris, really saves some people's bacon.
May 21, 2007 2:47 PM PDT

Michael Shadle (Intel)
Total Points:
373
Status Points:
323
Green Belt
 ZFS is another big advantage Solaris has. A couple BSD's are working on implementing that as well, as well as OS X (supposedly) - so it looks like as far as licensing is concerned, the BSD types go quite well with the CDDL types.
May 21, 2007 3:01 PM PDT

David Stewart (Intel)
Total Points:
9,435
Status Points:
9,435
Black Belt
 Yes, BSD is compatible with CDDL licensing.

We could probably do a whole 'nother thread on ZFS and it's pros and cons. I have heard nothing but high praise for ZFS from those who are familiar with it, though I have also heard from some Linux people that it violates some architectural principle for them in terms of layering of abstractions.
May 21, 2007 4:13 PM PDT

Michael Shadle (Intel)
Total Points:
373
Status Points:
323
Green Belt
 I think that might be just another excuse to get away from trying to port a ZFS clone (since right now it looks like ZFS won't be supported in Linux officially due to licensing [perhaps] and definately due to how it handles some kernel-level stuff)

At this point I could care less about who has a better license, etc. I want more ZFS style options out there, having inline data redundancy/checksumming and corruption tolerance with the ability to scale and be device agnostic is fantastic. It would almost be enough by itself for me to switch to OS X as my desktop OS. Worrying about data corruption (which has happened to me a lot lately it seems) is something that I would like to do without.
August 2, 2007 2:10 PM PDT


James Dickens		 http://uadmin.blogspot.com/2006/05/what-is-dtrace.html is a document I wrote over a year ago in May of 2006. It has numerous examples of what DTrace is and how it has been used to solve problems by both DTrace designers and its users on production and non-production machines.

If Systemtap is so great, where is a similar document showing how Systemtap has been used to solve problems in both production and non-production systems?

At this point, the people saying that Systemtap is equal or even better would be the equivalent getting advice on which beef is the best from vegans and the sales people trying to sell you a cow.

In short "Where's the Beef!"
October 28, 2007 2:07 PM PDT


parga		 i am looking for carrier in linux but after linux i want some thing which give me good post in MNC so please just reply me that after working in the linux then which course is better to get my carrier better and safe
April 2, 2008 10:15 PM PDT


Adam		 Good story, David http://artimoza.ru/ !!! I'm realy think that Solaris feature that Linux....
May 4, 2008 3:24 AM PDT


paul fox		 dtrace for linux does exist - i am busy getting it to work. i have the kernel drivers and userland dtrace binary compiled and loaded. i am slowly working my way through the features and mechanisms, and hope to announce more widely when i feel its ready for prime time (at the moment its like a car jacked-on-wheels!)
June 4, 2009 9:43 AM PDT

Prasanna PK
Total Points:
60
Status Points:
10
Green Belt
 Josh had mentioned about Dtrace on Mac called Instruments / Xray, is proprietary license.
June 5, 2009 8:34 AM PDT

TC2009074
 good one..

Trackbacks (9)


Leave a comment  

To obtain technical support, please go to Software Support.
Name (required)*

Email (required; will not be displayed on this page)*

Your URL (optional)


Comment*