First, the quote I was referencing from Darwin was the following: "A part developed in any species in an extraordinary degree or manner ... tends to be highly variable."  If you replaces species with computer language, operating system, library, etc., I think that you get a very apt statement of what we see going on in the educational and developer communities today with respect to parallelism.

First, what did Darwin mean?  It took me quite a while the first time I read that section of the book to puzzle through it.  In effect, what it is saying is that if there has been a substantial development to take advantage of some new niche, that that act of change and development will inevitably result in a lot of variations and choices that take advantage of that niche.  It takes a long time for the dust to settle after that big effort so that there are distinct winners and losers.

This is where we come back to parallelism.  Although parallel computing has been around for a long time, it was a very limited (ecological) niche.  Now that there's a big market for growth in it, we see a lot of innovation and variations, and from an educational perspective that's very hard.  We can't just teach our students pthreads, or MPI, or OpenMP, or TBB, or CUDA, or any of the other "old" models.  There are many languages, libraries, OSes, application frameworks, and so on that have made it across the 'parallel' threshold.  Sorting out what's "good" in some absolute or even relative sense is going to take some time... more time than we have to wait before adapting our curricula.

So we need to write the curricula to acknowledge that lack of a single known "best way", even to embrace it.  Our students are the ones that will be making the decisions in the years to come as to what types of parallel architectures and languages make the most sense in industry and academia -- we need to focus on giving them the tools to make those decisions, rather than the tools to just program them.  Diversity is good -- it means this parallel thing is something big and important.  And we can have impact.

This is all prompted by a series of comments I've gotten over the summer from several different sources, complaining that when I was discussing parallelism in education that sometimes I was talking about concurrency instead.  Not to be overly harsh on those who put forward those complaints, but they seemed to be stuck in a time loop  where there were some very real differences between the tools and programming models that one would use depending on the type of problem.  If fork/exec solves your issues, then your problem is likely concurrent.  If it doesn't, or if you have to do something funky once you've used fork/exec like used SysV-style shared memory, then you're writing parallel code.  However... those distinctions don't make as much sense today.

So what happened?  Time.  And some really good thinking on what it means to be parallel in the first place.  And a complete retooling of the way operating systems and languages deal with shared state between processes and/or threads and/or tasks.  And development of much more complete programming infrastructures for dealing with parallelism.  With copy-on-write and read-copy-update techniques (already inside your favorite OS), template- and library-based inherent parallelism (like TBB or math libraries), novel models with implicit parallelism (like Map-Reduce), and language extensions for parallelism (like OpenMP), we're operating in a very different *software* world from where we were 25 years ago, let alone the hardware changes.

So let's lay this to rest.  Concurrency is still a useful term in the same way that phlegmatic is -- useful descriptions, but not diagnostic states in and of themselves.  Modern medicine has moved beyond worrying about bleeding out the ill humors... I believe that we in CS can move beyond concerns of separating our "parallel" from our "concurrent" as well.

I also have a section where we go through and look at alternative ways to write a simple data-parallel code -- the core of which comes from work that Michael Wrinn and his team put together. Nothing makes the trade-off between ease of programming vs ease of implementation clear like having to look at a set of alternatives.  (or paging through multiple pages of pthreads code vs fitting the whole OpenMP code on one screen.)

As I tell class, everyone knows that the integral from 0 to 1 of 4/1+x^2 is pi, and pi is just 3.1415926535897932384626 and a bit. (yes, that's from memory.  And yes, it was a *very* long study hall my freshman year in high school)  Since they've survived Calc 1, as well, the idea of a Riemann sum should also be old hat.  So we take a look at what that simple numerical integration code looks like in pthreads, OpenMP, CUDA, MPI, and so on.

The core pseudo-C looks like this:

dx = 1.0/1000000
for ( i=0; i<1000000; i++) {
x=(i+.5)*dx;
sum += 4/(1 + x*x);
}
pi = dx*sum;

I'm working on adding a couple of more during this break -- maybe adding a SysV shared memory IPC version, maybe brush up on UPC.  But I open up the call -- anyone want to submit an example in OpenCL, Ct, Concurrent Collections, X10, ... ?

This is good stuff... and much of it is still in there in some form.  But Steven Parker of NVidia mentioned in a panel discussion at SC that, before he left academia,  he was doing something similar in one of his classes. Out of curiousity, he did an informal poll of many of his colleagues, and found that they, too, were trying to "be good" to their students by covering parallelism... and almost all of them were covering the exact same pthreads lectures.  Depending on your institution's choice of languages, you could equally well substitute java threads or windows threads in there, too.

(shameless plug -- you can watch the panel here: http://software.intel.com/en-us/videos/preparing-the-world-for-ubiquitous-parallelism-at-sc09-part-1/)

So, as important as it is to transition from no parallelism to having some parallelism in your curriculum, it's also very important that you make sure that you balance that with some depth and breadth of exposure.  So, concretely, I took the pthreads section (which, incidentally, mirrored what was in both their sophomore- & senior-level class) and tried to deconstruct it.  I grabbed a copy of the slides from the sophomore class, replayed a subset of them (the simple banker's mutex example), and then asked them to explain how the OS could be offering this service.

We (eventually, with some professorial prodding) dug all the way down to discussions of atomic instructions, and the cooperation between the memory & cpu that they require.  I then challenged them, as they walked out the door, to think about what happens in multicore systems where it may no longer be practical to think about locking the whole memory bus -- i.e. what happens to mutexes (and the whole pthreads model) if compare-and-swap dies?

The point here is to emphasize that parallelism as we in the community are used to defining it is a good starting point... but isn't necessarily sufficient for where our students need to go.  As useful as the multi-threaded web server is as a project (for example), it's not likely to be typical of the way students have to think about parallelism in the future in order to deal with multicore chips.  So we have to stretch... find some ways to channel new challenges and opportunities into the classroom, even if we don't know which way the technological ball will end up bouncing.

If you have tracked this for a while, you find that there is always a set of articles about higher education right around this time of year in the press -- I think it's a matter of those mid-career journalists reacting to their 1st -> nth child going off the college each August/September.  Or their reaction to writing the tuition check.  This year, the buzz has been very heavy over the use of online tools.  Even here inside Georgia Tech there's been an interesting tit-for-tat argument among professors, each side citing studies on whether video lectures, blogs, tweets, etc. enhance or distract from the learning process.

On the one side -- yes, there is a revolution in content delivery that is on-going, and higher education will have to react.  And there is good evidence that ignoring this makes for poorer student results as well as being a poor pedagogical (and economic) choice.  Some studies have even shown that using social media (in a very broad definition) can have a strong positive impact.  This is what journalists tend to pick up on -- after all, if you can just put your kids in front of youtube and get them a university education, isn't that a much better choice?  (For your pocket books at least...)

The other side fires back with their own studies, though, and those tend not to get heard as much.  There's evidence, they say (and I'm just passing this along, since I haven't done any demographic studies myself), that the higher student grades tend to be because the retention rates in the heavily web-driven courses drops way down.  Students who would get A's if you just stood up front and hummed the theme to "Shaft" all day will stick it out, but the students that really need the benefit of the full educational experience walk away unsatisfied, thereby raising the average grades.  So, the traditionalists argue, the whole things should be ignored.

As with many things in life, I think that the truth lies somewhere in the middle.  I was involved with an experiment with full online lectures, online daily quizzes, coupled with small weekly recitation sessions back over 10 years ago.  Much has changed in the interim, but I think some of the core observations are the same -- students really liked having the resources available, but they procrastinate unless they have a driver function like "show up at 12:00 on Mondays, Wednesdays, and Fridays".  The online quizzes were nice because they could be that forcing function.  But the lectures tended to be a mess -- students wouldn't keep up, and so the servers would get completely slammed by them trying to watch 3 weeks of lectures in the 24 hours before the next test.  And, of course, students just could parse that much material in such a short time, so their grades and long-term retention of information suffered.

So anyone have any thoughts?  Are brick-and-mortar universities doomed?  Or, more importantly, are they missing out on something they *could* be doing for their students and alumni?

We're right in the middle of talking about synchronization and how you go about building semaphores or mutexes at the kernel level, so it's pretty natural that we talk about parallelism in pretty much every lecture at the moment.  But it's amazing how many times it comes up even when you aren't dealing with explicitly parallel topics -- library design, memory management, file systems.  Not to mention the interesting issues that shared vs partitioned L3 caches, NUMA, and other multicore-related architecture trends cause for scheduling algorithms, paging table management, etc.

For those of you who are just starting down the road of adding parallelism, I have a very simple suggestion that I like doing at least once a week (in some guise or another).  Just stop somewhere in your lecture, look out at the class, and ask "Do you think this will still make sense in two years?  Will the next-next generation of multicore chips still do things this way?"  Sometimes the answer's yes, sometimes it's no... but the discussion is almost always interesting.

I'll try to lay out some of the specific things I've done in my class in subsequent entries -- little modules and small self-contained units that try to weave multicore/parallel topics into the curriculum in bite-sized chunks.  Questions, suggestions, and contributions are very welcome!

The semester's finally over, and I now have some time to write up the various things that have been ruminating in the back of my mind.  Tom's earlier blog post resonates nicely with one of those... namely, what is it that I, as an educator, could measure that would let me tell if students did or did not understand multicore.  (There's a more general analog, too... what should your interviewers focus on when talking to the next generation of developers?)  Think of this as being a concept inventory for multicore computing -- what are the basic things about multicore that every student should just know.

For comparison, the sorts of things that show up in, say, a 1st year Physics concept inventory would be conservation of energy, conservation of momentum, short circuit as a special case of Kirchoff's Laws, etc.  You should be able to throw up a picture of three light bulbs with a short circuit across the third, and the students should immediately be able to tell you whether the 3rd light bulb would light up or not.

What's the corresponding set of ideas/concepts/building blocks for multicore computing?  There are a few of the bits that we bring over whole hog from HPC & traditional parallelism -- Amdahl's Law.  The price of synchronization.  But... what else?  I have colleagues here who preach the line that multicore is just about parallel performance... but I think that that sells the current revolution short. Multicore programming is about more than just traditional concepts of parallelism, if for no other reason than that we have to now deal with a lot of algorithms and situations that I/we have never had to deal with on the big HPC machines.  (You want to put what on my cable box?)

Here's a short (non-authoritative) list of some of the things I've been thinking about -- give me another cup of coffee, and I could probably double or triple it, but what fun would that be?  What are your suggestions?

• Amdahl's Law (look at the whole problem)
• Gustafson's Law (bigger problems may be more parallel problems)
• Lock synchronization  (sometimes you have to check on your neighbors)
• Atomic primitives (you have to start somewhere)
• Power is constrained (why are we going multicore in the first place?)
• Planning for power (Can you arrange code to allow for altered power states?)
• Memory has location (multiple layers of cache & NUMA access to main memory)
• Threads vs Tasks (What is the right state to carry around?)

Multicore breaks a fundamental link in how we prepare our current and future developers – teach them to break a problem down into pieces and find a nice logical progression to solve each individual piece sequentially. A normal CS curriculum gets around to telling people about the idea of concurrent execution only as they have one foot out the door (senior-level systems courses, if then). Unfortunately, the current and forth-coming generations of multicore don't allow us that luxury.

The laptop that students lug into class will be multicore – two today, four to eight in the very near future. And the performance of those cores is increasingly tied to the parallel architecture – the shared cache infrastructure, OS optimization for thread teaming, and so on. Even when teaching would reasonably call for single threaded execution, performance can be difficult to assess without teaching students how to go through the extra steps to pin processes to processors, etc.

At Georgia Tech we've been trying to tackle this by trying to integrate bits of multi-core through out the curriculum – introduced gently into the entry classes as "my isn't it interesting when that happens", and getting increasingly more focused as time goes on. This admittedly means we have to forgo teaching some things to make space in the curriculum, but so far it has been surprisingly little. You can keep track of some of our evolving efforts at http://www.cc.gatech.edu/cercs/multicore. This approach also has the pleasant advantage of following through on a bit of wisdom on teaching passed down to me by my father, "If it's worth saying once, it's worth saying at least three times." I think multicore fits in the "at least three" category – I'd love to hear what others may think!