The "distributed" in the workshop title refers to the "distributed development", as in multi-location teams, not "distributed software", as in, say, grid applications. Nevertheless, the organizers considered parallel software development (as in "parallel software") to be so important they invited me to deliver four afternoon sessions focusing on the topic, so I flew to Beijing with a laptop full of powerpoint erudition on threading building blocks (generics, concurrent containers, scalable memory allocators) and their goodness in realizing classical algorithms -- old favorites like convolution, prefix scan, and quicksort. This somewhat stodgy preparation survived, like all battle plans, about one minute into the actual engagement. My normal workshop audience is professors, or sometimes working professionals, but here before me were  actual university students,  mostly the undergraduate variety. Not since my own grad school days had I been before such an audience, which had faded in memory to faint rumor. Ok, time for a new plan.

Luckily, the connection from campus (in Beijing) to Intel's Manycore Testing Lab (several 32-core systems in Dupont, Washington) was good, so we proceeded with coding exercises, starting with a parallel "hello, world" and moving up from there. As the week progressed, we also commandeered local systems to install and build parallel versions of "Destroy the Castle", and Intel game demo. The emphasis on coding turned out to be key, and it didn't hurt to have a game demo to break up the calculations of pi.  Key learning for me (a "duh", in retrospect): keep students busy, with solvable puzzles. Hmm, maybe this approach can work for audiences of professors, too?

Last night's episode of House played on this notion, with a patient soliciting diagnostic information from the online crowd. As dramatized, it turned out that one particular response was pivotal -- but the rest were irrelevant, or worse.

Recently, preparing to entertain a large, very technical audience on the topic of Intel Threading Building Blocks, I connected my laptop to the auditorium projector, only to find the image wavering, in that slow subtle crawl toward seasickness; clearly, we could not proceed without remedy. The crowd voiced several suggestions, but one, in particular, was pivotal: "Ground loop". Huh - out here in the wild? I'd thought of those, if at all, as eccentric creatures lounging on the grass in a Maurice Sendak illustration. But that was answer, and pulling the laptop power plug cleared the problem.

In neither case, fictional nor anecdotal, was there any "crowd wisdom"; rather, it was the single, correct contribution which mattered. I'm inclined to think this is how it normally goes, the "crowd" merit being an increase in sample size, raising the odds of finding that one smart individual. (Of course, I'd also dismissed ground loops as college lab contrivances.)

As we read and contribute to our online communities, here at ISN and elsewhere, are we looking for crowd wisdom, or for those occasional nuggets of individual brilliance?

In 2008, you'd meet suprise (Parallel computing is already here?), complacency (Parallel computing might be important in the future) and skepticism (Parallel computing is always important in the future). In 2009, I heard none of this; comments were more like, What's the recommended model? Do we introduce this early or late? Where are the textbooks? and (spoken softly, when no one was looking), How do I learn this myself?

Preparing Students for Ubiquitous Parallelism: Profs Wittman (Purdue), Harvey (Berkeley) and Murphy (Contra Costa) descibe their approaches (I'm listening, on the right).

Thanks to organizer Dan Ernst (Wisconsin), I had the privilege of participating in the panel discussion of this blog entry's title: Preparing Students for Ubiquitous Parallelism. The other panelists, pictured here, represented some of the parallel curriculum efforts at Purdue, Berkeley, Contra Costa Community College, and also the SC Education workshop series (which specifically address the question, how do professors themselves learn this stuff). We heard about parallelism taught in the first semester, using Scheme, in the first year (as an experiment), using Java threads, and to adjacent disciplines such as science and engineering, using C on clusters. Audience members related their own experiments and challenges, many pivoting around the inertia of university curriculum change, and (again) scarcity of introductory texts.

On textbooks, I'll offer this: parallel systems are the only ones that current students will ever see, in their careers. That ubiquity impacts nearly all traditional CS topics, well beyond programming fundamentals. We don't need "intro to parallel" textbooks, but instead introductory books which incorporate parallelism as a routine component -- changes will come, for example, in the implementation of classical algorithms, and in the design choices for data structures. Authors: an opportunity!

1. Finding concurrency in a program - how to help programmers “think parallel”?
2. Scheduling tasks at the right granularity onto the processors of a parallel machine.
3. The data locality problem: associating data with tasks and doing it in a way that our target audience will be able to use correctly.
4. Scalability support in hardware: bandwidth and latencies to memory plus interconnects between processing elements.
5. Scalability support in software: libraries, scalable algorithms, and adaptive runtimes to map high level software onto platform details.
6. Synchronization constructs (and protocols) that enable programmers write programs free from deadlock and race conditions.
7. Tools, API’s and methodologies to support the debugging process.
8. Error recovery and support for fault tolerance.
9. Support for good software engineering practices: composability, incremental parallelism, and code reuse.
10. Support for portable performance. What are the right models (or abstractions) so programmers can write code once and expect it to execute well on the important parallel platforms?

Reactions, please! Some things to consider:

• We make no mention of power consumption - is that reasonable? How would the power challenge be phrased, and where would it slot into the list?
   
• Data locality (#3) is intimately tied to scalability (#4). As systems become heterogeneous, with an assortment of different latencies among elements, how can we reconcile this tuning to portability (#10)?
   
• Are we underestimating the importance of composability, so innocently tucked into "good software engineering" (#9)? And by the way, how would you teach that concept to university students, and in which course?

Lots more -- let's pick it up in 2009. Happy New Year to all!

• this shift to parallel systems is disruptive; does industry really expect academia to invest all the effort needed to revamp its comfortably-established approach to teaching computational sciences?
  Yes, and it's very much in our interest to help as effectively as possible. The SC 08 BoF (and related followups at the show) demonstrated consistent motivation among industry stalwarts: weighing in with Intel were NVidia, Sun, AMD and Microsoft, all alert to the importance of this educational shift.
   
• isn't this parallelism thing just another passing fad?
  No. (pace, Donald Knuth)
   
• isn't parallelism an old story, well-mined by the HPC crowd?
  Partly true. Certain algorithms in specialized fields do indeed run successfully on massively parallel systems, and some of the learnings there may find broader application.
   
• ok, so when should we introduce concurrency?
  As soon as we introduce computing -- it's a normal, ubiquitous part of the discipline. This is not the same as pounding intro students with pthreads.
   
• but what do we take out of the curriculum, to make room for parallel programming?
  Wrong question. Traditional topics survive, but get revamped to comprehend concurrency. Lots more work, but what a terrific opening to overhaul the whole approach (one panelist asserted the curriculum had "ossified in the 1970s". And didn't I hear something about declining enrollments?)
   
• there are so many models -- SIMD, SPMD, threads, messages, etc -- which one to choose?
  Whichever conveys key concepts, and affords hands-on practice. At more advanced levels, offer vividly contrasting approaches (concurrency models in declarative vs imperative, say, or shared state vs shared-none).
   
• some schools are onto this already, and share their course content; could someone please provide guidance: what fits where, how good is it, who else uses it, etc?
  Excellent idea! Expect more along these lines in 2009.

We were guided by Ralph Johnson, coauthor of the original bible of this regime, Design Patterns: Elements of Reusable Object-Oriented Software, as well as Kurt Keutzer and Tim Mattson (coauthor, Patterns for Parallel Programming).

Ok, so...why is this interesting? Consider: the present state of parallel programming looks a lot like that of object-oriented programming in the early 1990s -- a small number of practioners understand and use it, but the overall profession looks on with anxiety. The Design Patterns approach helped to shove OO into the mainstream; could it do so again for parallel programming?

To test this premise, a project has been launched to build up an integrated, coherent set of patterns useful for exposing high-level constructs to concurrency -- a pattern language, as opposed to a collection of patterns. Kurt and Tim have proposed a layer of structural and computational patterns, integrating to one or more an underlying layers of concurrency patterns. We presented a detailed snapshot of this view at the recent ICCAD; slides from that tutorial are here: Architecting Parallel Software.

This is going to take a lot of savvy and effort to get right, so the project has been opened up for viewing and commentary by all:
Berkeley Pattern Language for Parallel Programming.
Participation welcome!

At SC 08 last week, in several forums, I heard variations on this comment: Ok, so there's content out there, how do I know it's any good? And whether it's suitable for my (students') needs? Steve Heller, from Sun, offered an intriguing solution: could we introduce a citation practice, analogous to that used in research publications, where curriculum content sources are credited? As in research, the more-cited content could be presumed better, or at least more useful. I like this suggestion a lot, but wonder about the timeframe and infrastructure challenges involved to pull this off.

I posed this later to Scott Apeland, Director of Intel's Developer Network, and he immediately yanked me from the 19th century to the 21st: wouldn't a web tool handle this? Yes...duh. I check Digg every day, but simply hadn't made the connection. All content is passed around by web, so why not have the community rank things there, as well? There are enough "digg-like" tools available (Pligg, PHPDug, etc) that a specialized ranking of parallel-computing course content could be created and maintained. Thoughts? Anyone already doing this??

Dan Garcia gave a quick overview of the UC Berkeley approach; they really *do* get the new kids coding a version of MapReduce - in the first school term. They do so in the language Scheme, which presents two distinct advantages: it levels the playing field, taking care of the issue of the students' broad range of prior coding experience (no one knows Scheme, apparently), and also, a functional language like Scheme is a very natural way to express MapReduce (4 lines of code, in fact). However -- and I view this as very important -- the curriculum quickly progresses from this academic level through Java and then to C, bringing in the corresponding concurrency model along the way. Students thus, in one school year, see the range of abstractions, right down to levels which expose the underlying architecture, with access to performance details. (Dan's slides are posted: Infusing Concurrency into the Intro CS Undergraduate Curricula).

Tom Murphy reminded us that not all software practitioners arrive from computer science departments; in fact, a great deal of coding goes on in application domains such as physical sciences, biology, engineering, and even social sciences, where performance is a key requirement -- could we please keep them in mind?

Charles Leiserson (from MIT) posted to us that performance motivates parallelism, but performance treatments have been evaporating from CS training over the past decade.

Which brings us to the recurrent theme: performance. The audience, mainly from industry, certainly picked it up; several identified themselves as hiring managers, and lamented the general ignorance of performance and architecture details. At least one of them said he prefers to interview only EE graduates - for software jobs - since CS students typically do not bring what his company needs (the industries represented here were quite varied: search engine, medical instruments, cluster consulting etc).

I wonder if the academic computing universe is splitting into two camps: those where students deal directly with architecture, low-level languages, concurrency, and performance, and those where students stay at a higher level of abstraction (typically expressed with Java or Python)?
And, if that division is true, is that ok with everyone?

One of these is a panel discussion, Academic Community Multi-core Programming Roundtable, where we'll address the positioning of multi-core (more broadly, parallel) computing in undergraduate CS courses. We are fortunate to have two panelists with uniquely broad points of view, each from his own school, and from national-level efforts:

Dan Garcia  teaches at UC Berkeley, and has developed curricula to teach concurrency to *introductory* students, in the first 3 CS courses; topics range from a functional implementation of MapReduce, right through pthreads and MPI. Part of these UCB efforts are documented in the paper, Infusing Parallelism into Introductory Computer Science Curriculum using MapReduce. Beyond his immediate campus, Dan is also active in education: he is a member of the ACM Education Board, and has presented every year at SIGCSE since 2001.

Tom Murphy teaches at Contra Costa College, where he chairs the department and covers a range of courses in theory and practice. In addition to this role, Tom is seriously involved, as organizer and instructor, with the SC Education Outreach (a part of NCSI, the National Computational Science Institute), delivering summer workshops in parallel computing to college professors.

Our third panelist is your humble blog author here, Michael Wrinn. I'll join this distinguished cast as a voice from industry, where I've enjoyed a role in recent years of working with academia on fostering parallel computing in education. Scot Buck, Intel's Director of University Relations for Intel Research, will provide referreeing services, and keep things (more or less) civil.

While coming from different points on the computing education spectrum, the panelists are all concurrency "evangelists", in the sense that we practice and teach parallel programming, and suspect that everyone else should too. To me, a CS degree based solely on sequential, von Neumann techniques is starting to resemble a degree in CS history: interesting, somewhat important, but ignorant of the key design of modern computing platforms. Of course, there are other points of view -- no less than Donald Knuth states: "I won’t be surprised at all if the whole multithreading idea turns out to be a flop...a pipe dream".

So, my academic friends, where do you stand on this matter, and how does that influence your teaching content? How soon should it be introduced, and how much is enough? How important is the choice of implementation model?

Please join us at this IDF Panel discussion - to challenge us, to share your experiences, plans or frustrations, and to let us know what we could to do help, in contending with the computing switch to concurrent platforms. There's a free, 3-day IDF pass available for professors interested in attending; I look forward to seeing you there!

I had the privilege of delivering a talk there, "Confronting Manycore", hammering again on the theme of getting concurrency into undergraduate curricula worldwide. After all, even laptops now have multiple cores, so what's the point of a continued focus on sequential approaches? The hardware now available is not limited to sequential execution, so why limit students' outlook to that outmoded constraint?

One response we hear frequently to this shift is, Ok, so we've got to teach concurrency; which of our essential CS topics do you propose we leave out? In the ITiCSE talk, I made the argument that this is a false choice; it is not required to choose "either sequential or parallel", but rather, to keep teaching those valuable CS topics in the context of concurrency. This got things moving, with audience members responding to one another on the points.

Two particular comments struck me: even after an algorithm is distributed in parallel, code executing on each core is, itself, sequential. Fair enough. Clay Breshears touches on this topic in his own recent blog, Something Serial This Way Comes, making the case for retention of serial-optimization skills. Ok, let's concede that sequential topics stay with us, as a subset of the overall CS portfolio.

The other: there are concurrency-specific topics which do not have counterparts, or obvious mappings, to the sequential CS topics. One example cited was thread safety: important to teach, now, but necessarily displacing some traditional topic. Thus, it was asserted, it really is a zero-sum game, and decisions must be made as to what of the historic material gets dumped. Suggestions welcome!

As expected, this covered the CUDA programming model in detail -- handling host-device communication ("device" being the NVidia processor, G80 in this case), C syntax extensions, memory model, performance considerations, etc. Though the format was lecture-only (i.e., no labs), the day was worthwhile and informative. Two different lecturers tag-teamed most of the day, and a 3rd concluded with a series of case studies. The instructor choices were somewhat surprising...

The NVidia instructor was their chief scientist, David Kirk. I'd seen his name frequently in the context of CUDA university workshops, so was not surprised to see him at DAC. But the company's chief scientist? Wouldn't this be like Intel sending Justin Rattner to deliver, besides the keynote, a workshop on, say, Nehalem performance tuning? So, ok, this is at least a mild surprise. NVidia is taking this evangelism and training thing *very* seriously.

Another lecturer was Wen-mei Hwu, professor from Univ of Illinois. Hwu is a well-regarded figure in parallel computing, so it was fun to finally meet him, hear him speak. Hwu is also, come to think of it, Co-Director of Intel's "Universal Parallel Computing Research Center", UPCRC, at Illinois. Hmm, an Intel-sponsored Director instructing at an NVidia event? Given recent and very public barbs tossed each way from the two companies, this was at least a bit surprising.

The third speaker was Damir Jamsek, from IBM, with case studies on power grid analysis, and small circuit simulation (this was a "design automation" conference, after all). There was some detailed analysis on performance, including the impact of loop unrolling and memory management, as well as a peek at some issues in porting sparse linear solvers to this kind of platform. IBM is also, come to think of it, purveyor of a competing platform called Cell, so here was another surprise.

Ahead of this tutorial, I'd wondered how NVidia staff might react to an Intel student in their class, so in the morning break I introduced myself, pointing to the "Intel" on my conference badge. Kirk was both calm and cordial; no problem at all. Given that two of the three instructors held affiliations with competitors, I suppose there would be no difficulties over provenance of the students!

Though initially surprising, this approach -- sharing of parallel-computing insight among companies -- is profoundly right. All of us will survive to the extent that software developers adapt to the new concurrency universe, and it's in our collective interest that we bring about that adaptation as quickly as possible.

Given that context, it was startling to hear a panelist at IPDPS (in Miami, a couple of weeks ago) assert that parallel-processing topics have been disappearing from CS curricula in recent years. As anecdotal evidence, he pointed out the topic’s removal in the 2^nd edition of Introduction to Algorithms (the elegant tome otherwise known by its authors’ initials, CLRS).

Could this possibly be true? Before rushing off to blog, I had to check. First, find an old copy of CLRS, oops, CLR in those days. Sure enough, there was a whole chapter called “Algorithms for parallel computers” which had gone missing in the current edition. Hmm, perhaps parallel concepts were infused into the overall approach? After a couple of long evenings getting reacquainted with CLRS, I had to conclude: no, it’s simply disappeared.

But perhaps this example is a weird one-off, a singular anomaly offset by broader trends? After all, schools like UC Berkeley introduce concurrency in the freshman course, and we here at ISC work with many schools on this topic. Further, googling the obvious keywords pops over a million hits, so things would appear to be, reassuringly, busy.

No. The panelist’s observation is accurate. If anything, the situation is worse than described. Here are some unhappy details:

Way back in 1995, IEEE Computer published an article, Parallel Computing in the Undergraduate Curriculum, by profs at Colgate. As stated in the abstract, “The author describes how parallel computing can be integrated into courses throughout the computer science undergraduate curriculum.”

Good stuff. So, what is Colgate doing now, 13 years later? There is one course on the topic, at the advanced level only, and it is NOT required for the CS major.

Still 1995: the First Wellesley Forum on Parallel Computing Curricula. Brown university presents a paper called Integrating Parallelism into the First Theory Course

Good stuff. So, what is Brown doing now, 13 years later? They list a single grad course, offered occasionally -- btw NOT this year.

Forward a couple years, to ITiCSE ’97. George Washington University presents a paper, Concurrent programming CAN be introduced into the lower-level undergraduate curriculum. They even refer back to ACM standardization efforts in this area, from 1991.

Good stuff. So what is GW doing now, 11 years later? At last, something: in one required course, called Software Paradigms, we read among the topics: “concurrent software design paradigms and patterns”. Excellent. I’m going to contact this prof, Michael Feldman, maybe buy him a beer – uh, hold that thought, Feldman is listed as retired…

One more. Forward to the OOPSLA 1998 Educators’ Symposium, where Loyola of Chicago is arguing: “that a computer-science curriculum should introduce the principles of concurrent programming in an integrated, coherent, and application-independent fashion early in the major.”

Good stuff. So, what is Loyola doing now, 10 years later? Sigh. One advanced class, NOT required for the CS major. No evidence of concurrency in other courses.

The pattern is monotonously repetitive. Forward-looking academics looked at the parallel computing challenge a decade ago, announced they would champion a solution, and saw their efforts, for the most part, defeated. What happened??

I’m headed to ITiCSE ’08, in Madrid this summer, to tilt at this windmill myself. Those of us looking to retire the now-obsolete sequential assumptions, baked into 50 years of CS curricula, do enjoy one serious advantage not available to those courageous predecessors: multicore computing platforms are now the norm. Recognizing that reality, let’s make the adjustment time short.

1. The tone, compared to a typical academic conference, was remarkably congenial. Technical conferences, in my experience, tend to be combative (everyone's chasing the same grant money, after all), a modern-day descendant of medieval jousts: bring a sharp lance (make clever points in your own talk), and look for advantage (challenge the other speakers at theirs). In contrast, SIGCSE attendees were relaxed, cordial, and constructive -- demeanors all refreshing, even startling.

2. Manycore/concurrency concerns may be on the curriculum radar, but only as a distant speck. You could count the number of concurrency talks or workshops on a single hand, and I observed an alarming lack of urgency to adapt to the retirement of single-core architectures. One speaker talked of changing things when 32-core chips showed up "in about ten years". Sorry, some of these are already here (GPGPU); many more will enter the mainstream before current students graduate.

To this second point, it was gratifying to see the crowded attendance at the manycore talk, where I worked on raising the anxiety level (manycore programming is here, be very afraid). This shift, serial to multicore to manycore, is going to be an interesting if bumpy ride for all, and we'll want to keep sharing information in that congenial and open spirit of SIGCSE.