Parallelism Education Workshop @ SC 2011 -- An open invitation


It’s that time again -- I and my colleagues from the Educational Alliance for a Parallel Future (EAPF) which included Adobe, AMD, Intel, Microsoft and a host of other academic and industry partners are happily out challenging the status quo again!  We are running a session on the trials, tribulations, and possibilities in teaching about parallelism and other advanced architecture capabilities at the SuperComputing conference next week. For those of you who are going to be there in person, please join us! 

The schedule link is here:
Parallelism, the Cloud, & the Tools of the Future for the next generation of practitioners

This is part of a continuing series of panels, and follows on from the very successful panel at IDF this past September that my partner in crime, Paul Steinberg, described in an earlier post. For SC this year, we have a sizzling panel to open up the discussion, with Dick Brown (St. Olaf College), Kevin Goldsmith (Adobe), Ben Gaster (AMD), Simon Mcintosh-Smith(Bristol), and Michael McCool (Intel) leading the charge.

Unlike normal panels, where the audience has to ask questions from the dark auditorium of the speakers up at the podium, for the “question” portion of these meetings we have the panelists disperse out into the crowd for small group discussions.  We have a number of additional luminaries joining in for these breakouts, including Michael Wrinn (Intel), Sushil Prasad (GSU/IEEE TCPP), Steve Teixera (Microsoft), James Reinders (Intel), Tim Matson (Intel), Charlie Peck (Earlham College), and Dan Ernst (Cray).

I have the pleasure of chairing the session and getting the discussion moving.  One of the things that we all agree on is that parallelism, virtualization, deep memory hierarchies, and cloud computing are here to stay.  All of the communities that have been investigating and developing these technologies separately have a lot to learn from each other.  We  agree that we need  a steady state in the future.  When you go to asking how we should be training people today for that future, however, the gloves come off.

Just to get the conversation started, here are some of the key position points from the various contributors.  Tell us what you think in the comments section below, and we’ll be sure that they are included in the discussions in Seattle next week.  We’ll also post follow-up summaries for everyone to continue the discussions -- we need a whole community of developers, educators and hardware innovators to chime in if we’re going to keep ecosystem healthy and moving forward!

In alphabetical order:

Dick Brown
The workforce-development buck stops with college and university professors, who must design and implement classroom, lab, and project experiences intended to give undergraduates the knowledge, experience, and attitudes they need to start their careers. As a professor, I can’t teach everything.  But I might need to:  our students might need anything, especially in times of computational revolution such as the present.
To prepare them for their unforeseeable careers, I want students to learn at least something about high-level and low-level, HPC and cloud/mobile, heterogeneous and homogeneous and serial, etc., and to learn how to go deeper and broader on their own.   Although research in the 80’s and 90’s provides a starting point for curriculum, teaching parallelism is decades behind teaching sequential computing.  Some of today’s most popular tools are so close to hardware that they feel like assembly language.

    • Parallel (and/or distributed and/or heterogeneous) computing belongs in practically every undergraduate CS course.  Henceforth, we do a disservice to our students if we let them think sequential computing is enough.

    • Incrementally adding a day or two of parallelism in each course, including hands-on exercises, is a feasible and sufficient strategy for rapidly introducing breadth in parallelism throughout the curriculum.  A spiral approach to key issues adds depth.

    • Applications beyond CS with hands-on experience of speedup will increase student interest in parallel computing.

    • Design patterns deliver encapsulated expertise to students.  Identify them whenever parallelism appears, and structure problem solving with them in upper-level courses.

Daniel Ernst
At conferences like SC, it's interesting as a trained computer scientist to watch computational scientists work. Of particular interest, these researchers - very few of which have any formal computer science training - live and breathe parallelism. Many of the education, outreach, and training efforts they sponsor (such as NCSI workshops and SC Education and Broader Engagement Programs) focus on bootstrapping people into reasoning about parallel codes (and writing them) with most of the participants having only basic procedural programming experience. The results have largely been very good.

Among HPC practitioners, there is no fear of parallelism - of the coordination involved, of resource scheduling/allocation problems, and of reasoning about performance. If these "novices" can clearly handle reasoning about parallelism - what are Computer Science faculty afraid of? Is it time for members of the scientific community to take a more active role in bringing the 'old' HPC technology of parallelism to the 'new' audience of Computer Science faculty and students?

Benedict Gaster:
Programming is hard and many people argue that parallel programming is harder still. My personal view is that, like so many things, if one late comes to parallel programming and more generally concurrency, then it will be hard to adapt from the ingrained sequential view but this does not need to be the case. If we teach students parallel and concurrency concepts as an afterthought,  continue to leave it as an optional computer science module or two, then yes we will fail to educate future generations, but if we adopt it as a core, integrate it into all aspects of schooling, then all future developers will handle it as they handle Python, Haskell, C++, Java or whatever today.

Concurrency and parallelism is everywhere, we do it all the time in our lives... my 6 year old daughter programs MIT's Scratch, an explicitly parallel programming model, and if someone this young can, then I take exception when a dean or professor, who should know better, say it is too hard, or what about other parts of course, or what should we drop? Stop it! This is what you are paid to answer, think out of the box, do something that will benefit us all.

Kevin Goldsmith:
The team I manage is building a single, modern, software product. A few years ago, that would have meant a desktop application written primarily in C++, mostly likely single-threaded. Today, it means software that runs on the desktop, but also on mobile devices and in the cloud. Working in my organization are developers who write shaders for the GPU, developers who write SSE (both x86 and ARM), developers using distributed computing techniques on EC2 and threads everywhere throughout the clients and server code. We write code in C, C++, ObjC, assembly, Lua, Java, C#, Perl, Python, Ruby and GLSL. We leverage Grand Central Dispatch, pThreads, TBB and boost threads.  How many of the technologies that we use today in professional software development existed when we went to school? Nearly none. How many will still be used in a few years from now? Who knows.  The reason we can continue to work in the field is that our education was grounded not just in programming techniques for the technology of the time, but also in computer architecture, operating systems, and programming languages (high level, low level and domain-specific).

Learning GPGPU was much easier for me because I could understand the architecture of graphics processors. I was able to understand Java's garbage collection because I understood how memory management worked in C.  I chose TBB over Grand Central Dispatch to solve a specific threading problem because I could evaluate both technologies given my experience
with pThreads.

We're doing students a disservice if we teach them the concepts using high-level abstractions or only teach them a single programming language. Having an understanding of computer architecture is also critical to a computer science education.

These fundamentals of computer science do not necessarily need to be broken out into their own classes. They can and should be integrated throughout the curriculum. Threading should be part of every course. It is a critical part of modern software development. Different courses should use different programming languages to give students exposure to different programming models.

If I was a Dean of Computer Science somewhere, I¹d look to creating a curriculum where parallel programming using higher-level abstractions was part of the introductory courses using something like C++11, OpenMP or TBB. Mid-level requirements would include some computer architecture instruction. Specifically, how computer architecture maps to the software that runs on top of it. This may also include some lower level instruction in things like pThreads, Race conditions, lock-free programming or even GPU or heterogenous programming techniques using OpenCL. In later courses focused more on software engineering, specific areas like graphics, or
larger projects: I¹d encourage the students to use whichever tools they found most appropriate to the tasks at hand. This might even include very high level proprietary abstractions like DirectCompute or C++AMP as long as the students could make the tradeoffs intelligently because of their understanding of the area from previous courses.

Tim Matson
As educators, we are providing a service to our students.  For a small fraction of our students, we are preparing them for pursuit of  a Ph.D.  That’s fine and we all love to work with those students.  But the majority of students in a computer science major are seeking a degree that will help them build and sustain fulfilling and stable careers.  To that end, it is our duty to help them acquire the tools they need to succeed.  And to do that, we need to teach them what they will need professionally 5, 10 and 20 years down the road.

The trends in computing are crystal clear.  High level programming languages such as Python, Ruby, and Matlab have made domain-specialists into programmers.  If the physicists, chemists, biologists, game-artists, etc. are all creating their own code, what do we need computer scientists for?  We need them to be the experts of the low level tools that support the Python, Ruby and Matlab programmers.  We need them to write assembly code to optimize essential kernels.  We need them to master how algorithms map onto hardware.  We need them to create code transformation tools that take high level code and map it onto hardware.  We need them to understand floating point arithmetic and how round-off error propagates through numerical algorithms.

In my humble opinion, to teach a computer scientist Java or Python or any functional language as their primary programming language is “criminally negligent”.   OK, “criminally” is the wrong word here.  Let’s say is “professionally negligent”.   Computer Science education needs to get back to the basics and stay there.  It needs to build a foundation in C and assembly code layered on top of a solid background in computer architecture.  On top of that, they need computer arithmetic and numerical analysis skills so they can support the end-user programmers of the future.

Or we can ignore this and educate computer scientists unsuited to the jobs of the coming decades.  And the center of the world in computing will shift to countries that get their act together and educate the workforce industry needs.  You know … all a worker needs is a modem and phone line.  Don’t you love our global economy?

Michael McCool:
Software developers do not have to be computer architects, but they do need to have a clear mental model of how computer systems operate, and that includes everything from networks to operating system stacks to cache behaviour.   At the same time, they need a high-level framework, of ideas and of tools, to manage the complexity.   One of these tools should be programming languages.   Programming languages can be seen as an integrated set of abstractions.   Abstractions are necessary (programming everything in machine language is interesting but unproductive), but should be seen as automation of well-understood implementation strategies, not information hiding: white boxes vs. black boxes.   Regarding new languages vs. augmenting additional languages: there have actually been a host of new parallel programming languages lately... and they all look like C.   I think we can do better.  There is an opportunity to extend the C++ standard to better support parallelism.   We should consider carefully how to do this in a way that benefits the largest number of developers.   Regardling HPC vs. mobile: while at opposite ends of the scale, these two communities have much in common and could, and should, learn from each other.   I think we should seek out and foster the development of technologies that can be deployed across the entire scale of computing.

Simon McIntosh-Smith:

    • Every CS student needs a thorough grounding in parallel programming. It's now ubiquitous and has already become an essential (and valuable) skill.

    • Being good at parallel programming means being good at optimising performance, and that requires the ability to understand how the hardware works, from the chip level up to the system level. Trying to teach parallelism only at a high-level of abstraction doesn't work because of this.

    • Focusing on proprietary technologies is wrong, even if they're technically superior to more standard, open alternatives. It leaves many students with too narrow a view of what the right answer is. We're seeing a big example of this right now in the latest heterogeneous many-core developments.

James Reinders:

    • We need to learn to teach parallel programming without relying on teaching parallel computer architecture first.  (Coming in conjunction with our new book – watch in Spring)

    • Adding new restrictions in high level programming to match weaknesses in target hardware is a big step backwards, and a waste of time.  (Available only in my talks today, expect a blog in the future)

For more complete information about compiler optimizations, see our Optimization Notice.


dan_ernst's picture

Absolutely, Dick. We spend most of our time pushing that incentive train, and it’s true we need to encourage the importance of the topic to CS people.

In this case, at a place like SC where we get to “preach to the choir” a bit more, I think it is worth taking a moment to talk about how “the choir” might be enablers for that professional development. (Particularly in the “give them ready-to-use, low-"cost" ways to change” department)

Dick Brown's picture


I suspect CS faculty who have fears are just as afraid of facing other areas that may be outside of their own, not just parallelism and computational science. Both they and the "fearless" faculty can't invest in everything, and will spend their limited professional development time on what they can least do without knowing, and/or what they most care about learning. I'm sure this kind of choice needs to be made in other professions, too. The HPC folks without CS learn you mention have pushed through because they needed it and wanted it.

So, some ways to get more CS faculty to acquire and teach more parallelism is to make the entry bar worth climbing for them: make the need more compelling; give them incentives to change; give them ready-to-use, low-"cost" (in terms of professional development energy) ways to change. The experience of HPC practitioners could be a great asset for helping faculty make changes, but it can't be enough unless faculty are ready to invest their energy more in this than in 99 other things.


Paul Steinberg (Intel)'s picture

I'll be there, of course, and am looking forward to continuing the conversation.

Add a Comment

Have a technical question? Visit our forums. Have site or software product issues? Contact support.