4,391 Posts served

10,718 Conversations started

Some Views from the 22nd International Parallel & Distributed Processing Symposium

By Robert Chesebrough (Intel) (13 posts) on April 15, 2008 at 6:30 pm

Greetings from the 22nd International Parallel & Distributed Processing Symposium in Miami!

I had the privilege of discussing the future of parallel programming with a number of distinguished luminaries in parallel computing! The topics discussed in side hall discussions and informal lunch table chats were varied and dynamic. Given that this is the first day of the conference for the broader set of technical sessions (the NSF track actually started on Sunday) I have no doubt that there will be more intersting discussions throughout the rest of the week. Let me start my recap of the conference with one lunchtime discussion I had on day one.

Today I had the privilege of talking to Jack Dennis, Professor Emeritus of Computer Science and Engineering at MIT. Over cold cut sandwiches and iced-tea, we had two very interesting discussions. The first regards Professor Dennis’ Fresh Breeze Project at MIT and the second regards his view of how to prepare undergraduate students for the many core world we are now entering.

Fresh Breeze is multi-core chip design which supports composable parallel programming. Composable means that parallel programs should be able to be used as components in still larger parallel programs without a lot of fuss and without a detailed knowledge of the implementation details of the smaller parallel program. Composable in this context implies that resources, such as processor allocation & memory, should be flexible and able to be reassigned according to the current needs of computations. This project adheres to six important principles for supporting modular software construction with a view towards composability of parallel programs:

1. Information Hiding Principle: The user of a module must not need to know anything about the internal mechanism of the module to make effective use of it.
2. Invariant Behavior Principle: The functional behavior of a module must be independent of the site or context from which it is invoked.
3. Data Generality Principle: The interface to a module must be capable of passing any data object an application may require.
4. Secure Arguments Principle: The interface to a module must not allow side-effects on arguments supplied to the interface.
5. Recursive Construction Principle: A program constructed from modules must be usable as a component in building larger programs or modules.
6. System Resource Management Principle: Resource management for program modules must be performed by the computer system and not by individual program modules.

Professor Dennis holds that his new multi-core design would be key to building modular parallel programs. For more details – see Prof Dennis’ website - http://csg.csail.mit.edu/Users/dennis/.

The second topic we bantered around the table was how academia could prepare undergraduate CS students for the burgeoning parallel computational world that has been thrust upon them (and us). On this topic – I asked Professor Dennis this question – "If there were a handful of changes that academia could make to prepare undergraduates for this many core transition – what would they be"? This was his response:

1. Teach undergraduates a functional language – yes functional as in LISP – which avoids synchronization hazards. MIT’s been teaching LISP for years. Another choice might be Erlang as it side steps synchronization hazards, is a simple language to learn and has been designed for high performance use in the communications industry
2. Teach undergraduates MPI as in Message passing Interface. This paradigm has been used successfully in large scale parallel programs already.
3. Give them a blended HW/SW computer architecture course, including multicore arhcitecture – expose them to cache architecture so they understand performance
4. Expose them to hazards like synchronization hazards and data races so they can avoid these hazards when they implement parallel designs
5. Revisit object oriented programming methodology, which is great for sequential programs, but is not well suited for structured parallel programming. A program can easily be written which follows OOP but can still exhibit a data race.
6. Teach students the concepts of atomicity, determinacy, and the difference between them. Atomicity is the concept that all or none of a transaction must be completed. Determinacy deals with achieving a correct result regardless of the order in which threads execute. Atomicity does not imply determinacy.

Professor Dennis – thanks for lending me your sage advice on how to steer our curriculum efforts!

So now I ask YOU!

If there were three (or five) changes that academia could make to prepare undergraduates for this many core transition – what would they be? What is YOUR response?

Bob C

Categories: Academic, Events, Parallel Programming, Software Tools
Tags: Curriculum, Erlang, Functional Programming Language, MPI

For more complete information about compiler optimizations, see our Optimization Notice.

Comments (9)

April 16, 2008 12:49 PM PDT Josh Bancroft (Intel) Total Points: 4,584 Community Manager

Great post, Bob. I don't work a lot with academia, but thinking back to the CS classes I had when I was in school, and then looking forward to the future of muticore (which is the biggest fundamental shift in computing since the x86 architecture was introduced, IMO), I definitely see the need to educate future (and current!) programmers on what they need to know. And I'm glad that Intel is doing something to help provide materials and help to curriculum planners and schools. That way, we all win! :-)

April 16, 2008 2:31 PM PDT Clay Breshears (Intel) Total Points: 30,126 Black Belt

In the curriculum recommendations, I'm not sure how Prof. Dennis would resolve his #1 and #4. Also, how would teaching a functional programming language as the initial foray into programming be beneficial to the future employability of the student? I'm not sure how widely functional languages have pentrated the software industry, but I don't think it's very deep. Even if students in CS are expected to come in to university with some programming skill/knowledge, turning then around 180 degrees with a functional language doesn't seem very useful. If I can assume that incoming freshman have C/C++ programming experiences (and if not, teach a remedial section to get them up to snuff) I would offer a parallel programming course as the initial programming course. That is, teach CS101 as if this is the first programming language course a student has ever taken (with respect to the topics covered and their order), but introduce and emphasize parallel programming design, implementation, correctness, and performance concepts. Since students are already familiar with programming, there would be no reason to dwell on things like data types, arrays, conditional execution, loops, functions, objects, etc. However, how those data and coding structures are affected by concurrency would be new and vital material.

April 16, 2008 8:55 PM PDT BobC

That s great question! In fairness to Dr Dennis - I had painted a backdrop before I had asked these questions of Dr Dennis and I'll explain that backdrop below. Also - he told me early on that he could not make a firm commitment to responding to questions on my blog. I asked Dr Dennis if he would like to post a reply - but he kindly declined. There are too many other matters to attend to, and he could not afford to add another. As he said - "I have offered an opinion and let's leave it at that". I respect his position here and want to thank him once again for his time during the conference. So let me tackle the quesiton based on the discussion I had with him and with a handful others here at the conference this week - and by the way - there is NO consensus answer to my "handful" questions so far at this conference. The backdrop I painted for Dr Dennis was this: in ISC, we are tyring to prepare students for a manycore world by influencing universities to inlcude more prallelism in their computing curriculums. As part of that backdrop, I posited that since we (Intel) don't know all the answers to what the future of parallel computing will look like - I proposed that we plant a handful of seeds of possibilities and let the flowers grow. Meaning, that as far as ISC is concerned - we cannot back any one parallel methodology at the exclusion of another - but rather we should offer training around "possible eventualities". For example, the course materials we collect from Universities, and that we create and that we post in this Acadmia community should span perhaps MPI, OpenMP, explicit threading, TBB, etc but beyond these things there may be other topics - functional languages, extensions to existing languages, etc With this backdrop painted - I asked Dr Dennis his thoughts on what should be taught. This was a lunchtable conversation and not a true curriculum planning session - but none the less - Dr Denis did not hesitate to mention functional languages as one of the first things that should be taught. I don't think he was implying that a functional language is the magic bullet or one size fits all solution to bringing new developers to the discipline. The reasons he articulted for teaching a functional language are that: 1) it is a model for parallel prgramming that avoids the sunchroonization hazards associated with threading - data races etc. 2) version of parallel languages are already being taught in many schools already so there s a low barrier to entry 3) Examples of functional languages can already be found in industry I would argue that there will still be a need to teach students about threading (but not with as high a priority). If you teach them about threading, you have to teach them about the associated hazards. This does not contradict that we should also teach them a functional language. Anyway thats my view! bob c

April 16, 2008 9:59 PM PDT BobC

More discussion on functional languages On other fronts here at IPDPS we talked to Keshav Pingali from the University of Texas, and Vivek Sarkar from Rice University who were panelists here Wednesday night. These panelists discussed how the current parallel computation community could pass their learnings on to a new generation of develpoers just starting out in multicore. After the panel discussion, Michael Wrinn & I were able to get their take on essentially the same question I asked of Dr Dennis. They indicated that there still is some value in teaching functional programming to the new crop of developers. Keshav in particular indicated that his early exposure to functional languages has influenced his coding style in positive ways - years later. They acknowledged that some implentations of functional languages have been poor performers on sequential machines in the past and so they have not really caught on in industry (Erlang is the excetion to this statement) and also they cited some difficulties with cache locality & dealing with multidmensional arrays from a developers perspective. However, they also indicated that several groups have been adding extensions to these lanuages, I think Haskell was mentioned, to allow for improved handling of arrays and to address locaility issues. So here are my musings after havng these discussions. Warning controvesial road ahead - use caution when entering - Here's a controversial perspective on appropriateness of functional languages as we move into many core land. Maybe the locaility issues that caused functional languages to be discounted wont be so significant in a time when we are utilizing hundreds of cores. Why? In sequential programming days and in early days of multicore - 2 cores could possibily give you a 2X speedup. But the tax we pay for poor cache locality might be hypothetically ~10X - the number thrown around here in our conversations with these disinguished professors from Texas. So any language that could gain you 2X by parallelism - but lose you 10X thru poor locality probably would die a natural death. BUT - improve cache locality fo these languages and apply to many more cores. Suppose in future we have 100 cores and that compilers for the functional languages get a second look and get needed optimizations so that we reduce cache locality issues to be a 50% penalty rather than 10X (50% penalty means we run at 1.5X speedup). Now assume that we approach 100X due to parallelism but pay 1.5X tax. Thats a net gain ~66X speedup. Compiler writers might be persuaded to optimize a functional language if there was a decent pay off. 66X might do it. If the industry had frozen in a past instant of time at only 2 cores and if compiler writers at that time thought that they might be able to reduce locality penalty to 50% would it have been a worthwhile endeavor? 2X gain/1.5X loss 1.3X net - that a 30% gain in performance - not so clear that compiler writers would be enthusiastic to do all the work required because the potential pay off would not be that large. But maybe for 66X gain thay may consider it. Caveat - I know I am assuming no overhead here and we likley wont scale perfectly or anywhere near it at 100 cores etc. But if there are no synchronization hazards - and we reduce the locality penalties maybe these things will scale better than we imagine - does anyone have a quantitaive way to address this? Quick - can someone whip out an equation from Hannesy & Patterson? My point is that perhaps we should keep an open mind ot the possiblity of functional languages and hope that the "economy scale" and clever compiler writers might be able to bring a solution. The payoff ultimately would be fundamentally easier approaches to parallelism. Who knows? bobc

April 18, 2008 12:05 PM PDT Bev Bachmayer

Hey Bob, this is a great discussion. Thanks for giving us an update on the conference. I am looking forward to your next conversation

April 22, 2008 8:56 AM PDT BobC

4/22/08 - Summary of Katherine Yelick's Panelist discussion (from 4/16/08 panel) One of the highlights of this 22nd anual IPDPS confernce was the Wednesday night panelist discussion. The discussion probed the general topic of what the current parallel programming experts (eps IPDPS faculty & researchers) can teach to a new generation who will just now be cutting their teeth on MC processors and growing up in a non sequential programming landscape. The discussion was recorded for IEEE TV but I have not seen it posted yet on either the IPDPS site or the IEEE TV site. So for lack of being able to review the panelists discussion again in detail I have relied on my cryptic notes, scribbled furiosuly during the discussion. The panel consisted of several of the "earth movers" - those prominent professors or researchers in parallel computing - who have participated in massively parallel computing or more appropriately, shaped massively parallel computing landscape. I want to cover each of the panelists opinions about the future, ut must start somehwere - so let me begin with Katerhine Yelick's presentation. Kathy Yelick is NERSC Division Director, Lawrence Berkeley National Lab and EECS Department, University of California at Berkeley. She received her Bachelors (1985), Masters (1985), and PhD (1991) degrees in Electrical Engineering and Computer Science from the Massachusetts Institute of Technology. Her research interests include parallel computing, memory hierarchy optimizations, programming languages and compilers. In this panel discussion, Kathy laid out a path whereby CS students would likley be split into two types of scpecialties: 1) an efficiency layer - Performance oriented parallel programmers who understand cache locality and lower level parallelism mexhanics and who write abtraction layers & libraries upon which the rest of developer community stands 2) a productivity layer - who use the lower level parallel abstractions and libraires to and who focus on application domain In Katherine's model - she woul dhope that ~10% of the developers who have to be deeply involved in the efficiency layer, while the majority (90%) of developers would be trained as appplication domain folks Katherine stated that it would be a disaster if the split between efficiency programmers and application programmers urns out to be more like 50/50% rather than the 10/90% split proposed and believes universities should begin prepping CS students along these two disciplines and hopefully - the numeric proportion will fall to the 10/90. She laid a case for tackling large manycore application going forward with Parallel Global Addressible Systems (PGAS) languages - like Unified parallel C (UPC), Titanium, CAF etc as opposed to continued effrtos in MPI. They case she laid out was similar or better performance with PGAS with substantially fewer lines of source code. It turns out UPC was discussed frequently in the sessions I attended and I am onsidering wether Intel Software College should add a module on UPC or Titanium etc for our University program - Thoughts? She said there are gaps in current training at the undergraduate level that need correction. Essentailly these gaps are 1) there is currently a lack of deep understanding of performance & algorithmic complexity, 2) there is currently a lack of of understanding of algorithms 3) there is a lack of sophisticated understanding of concurrency, synchronization, non determinism, load balance etc - presumably - these gaps are less of an issue for the application domain folks but are of critical importance ot efficiency programmers of the future. This is an area where many of our Intel Software college already has some material in the pipeline and its was good to get implicit validation that our plans align with the Berkely vision. Katherine roughly laid out the Berkley approach, where they have have begun teaching parallelsim to more entry level programming students. They have incorporated cluster computing fundamentals into the introductory computer science curriculum at UC Berkeley. In that course, they have developed coursework and programming problems centered around Google's MapReduce. They used a language called Scheme to write and run MapReduce programs and can tackle parallel problems on a cluster using a special interface. Students can begin addressing data-parallel problems about two thirds of the way into this course. They are targeting more concurrency in their systems course and now have a capstone course for seniors who will be moving off into the efficiency programming type of endevours. At the higher level, they will be focusing on 13 Berkley "motif" or what has been called the Berkely 13 drwarfs. These are 13 categories of applications that have similar target kernels underlying their operation. Efficienntly parallelize the kernels and you solve a whole class of applications in that share that kernel or motif. This Berkley approach is one that Intel Software College is pursuing to help seed our faculty training. We are working with a couple of the Berkley faculty teaching the Berkley motifs exploration course together with design patterns. We plan to incorporate findings and lablets and instructor materials from this course to our wiki in the June time frame. As part of those postings we will also be posting materials we have created or materials we have recieved on other parallel techniques: Design Patterns, MPI, functional language examples (Erlang), Software Transactional Memory, etc. So next time I'll recap a few more of the panelist discussions and and how they may impact our curriculum postings. If you have examples of what YOU are teaching in YOUR undergraduate courses to teach parallelism - I'd lovce to hear from you - please respond to my blog or better yet - post your materials for use by many universities to our wiki! bye for now

July 7, 2008 4:12 AM PDT M.R.Archana Jenis

Hi sir Plz tell me abt the thread test

December 16, 2008 11:37 PM PST M.R.Archana of IT Student

Hi sir, I Want a complete detail abt intel processing.

January 13, 2009 4:39 AM PST Mike

Sorry, Where I can read some views from the next international symposium? http://www.comfort-software.com/

Trackbacks (1)

• Intel® Software Network Blogs » Functional Languages versus threading
  April 22, 2008 9:26 AM PDT

Leave a comment