Do We Need A(nother) Parallel Programming Language?

By Clay Breshears (Intel) (123 posts) on February 23, 2007 at 4:22 pm

There seems to be a lot of buzz  that has been generated by the article "The Problem with Threads," by UC Berkeley Professor Edward A. Lee, which appeared in the May 2006 issue of IEEE Computer.  After reading the article, this Thread Monkey agrees with the conclusions, but he's not sure how soon we might be able to realize this dream.

In a complementary piece, Alexandra Weber Morales categorizes OpenMP as a "niche solution for supercomputers."  She later claims that commercial software developers are clamoring for new programming languages that support concurrency.  I've heard this from other sources, so I'm inclined to believe this, too. 

However, Ms. Morales has illustrated why we're still working with low-level threads or slightly higher threading abstracts like OpenMP and Threading Building Blocks.  There are six different languages (Erlang, Haskell, E, Split-C, Cilk and Smalltalk) cited as potential solutions for those commercial developers.  Besides these, there is also Ada, Co-Array Fortran, Unified Parallel C (UPC), Titanium, High Performance Fortran (HPF), ZPL, and Occam, just off the top of my head.  There must be at least a dozen or two others I've never even heard about.  (Okay, so maybe none of the Fortran variants would be useful for game programming.  I'm just trying to keep in the good graces of Doctor Fortran.)

Recently, the DARPA HPCS program has been looking for a parallel programming language.  There are three research efforts currently working to meet the DARPA requirements: X10 from IBM, Chapel from Cray, and Fortress from Sun.  Each of these projects has draft language specifications out and one or two have systems that can be used to test the language.  Granted, these languages are targeted at the High Performance Computing (HPC) and technical computing space that utilize large clusters processing nodes.  Still, if the programs can scale to 10000 processors, why couldn't they be used for 80 cores?

Addressing the title question, I must say "Yes, we could really use a parallel programming language."  One that overcomes all the things bad about multithreaded programming, please.  Which will it be?  Just by reading through to this point you've now heard of sixteen different possibilities.  And then there are the threading models and libraries we're using today.

Won't my status as a Thread Monkey disappear when parallel and concurrent programming languages are in use, you ask?  Probably.  By then, though, I'll have become a Parallel Giraffe or a Concurrent Penguin or something like that, and still retain my stable of groupies and sycophants.  More likely, I'll have retired (especially if the Intel stock price gets close to its early 2000 levels).

The one thing advocates of an end-all, be-all, higher-level parallel programming language haven't seemed to address is that we've got too many choices right now.  Before there was MPI, each parallel hardware vendor had its own message-passing syntax; before there was OpenMP, each shared-memory parallel hardware vendor had its own threading syntax.  We're getting close to a tipping point similar to the ones that brought about MPI and OpenMP with regards to concurrent programming languages. 

totteranimationsmall1.gifUnfortunately, the 800-pound gorilla sitting on one end of the teeter-totter is the billions of lines of C, C++, Java, and Fortran code.  Who is going to be willing to translate their 2-million line application into a new language and then write in all the new coding features to implement parallelism?  (Personal note to current Thread Monkeys willing to be the future Parallel Giraffes: Here's another activity that can translate into years of job security.)

So, for now, I'm content to teach people about how to thread, how to avoid the problems common with multithreaded code, and how to write code to scale the number of threads and the application execution with the increasing number of cores.  It may not be the most elegant solution, but it works and it's what we've got for right now.

Categories: Parallel Prog. & Multi-Core

Comments (13)

February 26, 2007 3:11 PM PST


Shiv		 While you are correct in pointing out that millions of lines of existing code needs to be considered, you are forgetting something. It is possible to do gradual evolution of a language as opposed to revolutionary "throw away everything" upgrades. C++ has been doing this for a while now. We've had (then) fancy features like exceptions, templates and namespaces added without taking anything away from existing code. You can opt-in for those when needed. You should read Bjarne Stroustrups explanation of how C++ evovled in D&E (Design and Evolution of C++). Even C# has seen its share or evolution from 1.0 to 2.0 (ironically adding things like templates called generics). So it is entirely possible that when we are faced with mega multi core CPUs on each desktop and the thread programming state of affair sucks, then direct language thread support will make it into mainstream languages for ease of programming. You could do container/collection classes and decent error handling before templates/exceptions, just that it was gunkier, more error prone and problematic for new comers. I firmly believe that such things will make it into the language at some point in time. After all templates/exception like concepts have been in many languages before they came to mainstream languages. Its all a question of the right time and the right set of circumstances in confluence to make it all happen.
February 28, 2007 2:19 PM PST

Clay Breshears (Intel)
Total Points:
14,983
Status Points:
14,983
Black Belt
 I think that you're right. It will have to be a gradual transition of moving from what we have today to something that supports a parallel programming model (because of the billions of lines of legacy code). The description of Fortress, from Sun, looks a lot like Java in it's syntax (surprise, surprise). I expect the other DARPA projects are going to be similar to current programming languages.

From the paper you can read describing the Chapel language, there are (at least) three fundamental programmer responsibilities in writing concurrent and parallel code: 1) Identify the parallelism, 2) Synchronize thread executions, and 3) Distribute data (global or local). A good programming method/language will allow these three things to be done. The easier each of these is, the better the language will be, and, I predict, the more readily accepted, too. Explicit thread APIs cover the three responsibilities, but the details can be complex and error-prone. OpenMP and TBB cover the details and are easier to use, but the flexibility of these are limited.

No one is going to invent a universally acceptable new parallel programming language this year or next or even in the next decade. The existing langauges, and the ones yet to be invented, that support parallelism will serve to test out ideas about how to best handle the three responsibilities of the parallel programmer. Little by little, the best of these ideas will find their way into C++ or Java or C# or even Fortran or any other established language in wide use (COBOL, anyone?).
March 7, 2007 8:45 AM PST


AlexO		 I‘ve read the article "The Problem with Threads", and quite frankly do not get it. Sounds like one more theological discussion , based on likes and dislikes. Though use of big words and formulas masterfully mask it, what it seems to argue is that since multithreading is difficult we need something (new language) that will magically solve all the problems of concurred programming. Probably same way Java "solved" all the memory -management problems :).
March 9, 2007 3:07 PM PST


Khubaib		 I think that we should not involve programmer in the 3 tasks that you have identified. That would be too much to ask for from a programmer, who has to concentrate hard on the abstract and complex problem, and actually program it. Thus, my view of future parallel programming paradigm is that programmer tells "something" about his program. This something is information about, e.g. ordered and unordered loops, some information about the data structures that he is using etc (obviously more domain knowledge in good). Then its the responsibility of compiler and run-time system to figure out parallelism in the correct way. Traditionally this has been the case, and is still true: abstract away more details from programmer, to realize more complex computing systems. I think the same trend should continue. This would require adding more layers of software and hardware management between programmer and bare architecture (whatever it is), and yeah, these layers will interact more with each other, compiler with run-time system, and run-time with architecture...
March 20, 2007 8:19 AM PDT

Clay Breshears (Intel)
Total Points:
14,983
Status Points:
14,983
Black Belt
 AlexO - I would agree with you, for the most part. The article was difficult for me to read and understand, even with my CS background. (Maybe things have advanced more than I would have thought since I've been out of academia.) The discussion is quite theoretical, and I still can't say I understand all the fine points. The gist of the argument is that threads are very dangerous to program with because of the nondeterministic nature of their execution. Lee is proposing (or advocating the use of) a more deterministic model of concurrent programming. Then add nondeterminism as needed. A new programming language would be one way to ensure that these "safer" ideas would be available.
March 20, 2007 8:34 AM PDT

Clay Breshears (Intel)
Total Points:
14,983
Status Points:
14,983
Black Belt
 Khubaib - I would agree with 2/3 of what you have said. I still think programmers would still need to perform task #1 - Identify the parallelism. In fact, 'telling "something"' to the compiler is just that. In OpenMP, you insert pragmas to the code, the compiler adds the stuff needed to implement the threading according to the pragmas. Even in OpenMP, you still need to synchronize threads or add clauses to locate data items in such a way as to remove data races and other problems.

As far as doing things all automatically by the compiler, you've identified a research topics that has been open for almost 40 years. We still have no complete solution that can handle an arbitrary code, but there have been specific cases in code that can be parallelized automatically. The last two tasks could be done by the compiler or runtime support, but, for now, we're still stuck with the requirement of having these done by the programmer.

New languages might be able to ease the burden of the programmer to need to worry about all the little niggly details that they need to worry about now. Other abstractions (like OpenMP and TBB) might be stopgap measures or pre-cursors to what we'll be using 10, 15, or 20 years from now.
May 3, 2007 8:29 AM PDT


randY		 You are missing something.

i just got back from the future and every Large Scale Computer Application runs on it's own personalize Language. Yhe jobs are in writing the programs needed for each Company. The Code is protected from any one seeing it except the hand full of programers. Then of course they are killed after the project is complete. (to protect the code:)

This is all done to do One thing [protection of Information]
and Yes programable Chips are the future also. The code in the cpu is most important!
May 3, 2007 12:50 PM PDT

Clay Breshears (Intel)
Total Points:
14,983
Status Points:
14,983
Black Belt
 Welcome back, randY. How did you avoid the killer robots, nanoviruses, and mutant cyborg rabbits?

We're almost at a point where each industry is developing their own programming language or variation. I've heard that some game developers are not impressed with the current state of parallel programming and have contemplated developing their own parallel programming language(s). It's not much of a leap to imagine coprporate programming languages and, then, application specific languages. (Maintenance of future source code must be a bear with all the rotting corpses of past programmers smelling up the cubicles.)
May 17, 2007 12:22 PM PDT


Michael McCool		 It is in fact possible to implement a high-level interface for parallel programming "inside" C++ without adding *any* new features to C++.

The idea is to use an API to describe the computation, then have a staged compiler (implemented inside a library) generate the code and manage its execution. Such an approach makes it possible to get very high performance *without* changing C++. With the right programming model (I favor deterministic SPMD data-parallelism) it is also possible to write code that focuses on the algorithm but then can run on any number of cores, or even on accelerators, heterogeneous cores, or different kinds of processors, all without worrying about race conditions and the like.

At RapidMind, we have done this, and in addition to scaling over multiple cores, we can also use "extra" parallelism to vectorize over SIMD units and prefetch data (processors actually have multiple levels of parallelism; to get maximum performance, you have to exploit them ALL). We've gotten an 8x speedup on an Intel Quad-core with this approach, a 2x performance advantage per core. In fact, the same application can also be run using a GPU as an accelerator, where we get over a 30x speedup.

One problem with extending C++ directly is that C++ has a lot of overhead and historical baggage, like pointer aliasing, that make it hard to generate really tight code. The staged compilation approach bypasses this issue completely, but you can still use all the modularity constructs of C++ to keep things organized.
May 17, 2007 8:12 PM PDT


Sam Post		 Clay,

I couldn't help but crack up at your description of the rotting corpses... it's bad enough just with the current stock of languages common in the industry. And within each language you still have to account for variations in style - don't even get me started about perl...

I can see why game programmers would want their own specific language... heck, OpenGL / GLSL adds a host of data structures and native types that make pixel / vertex shaders more intuitive and let you focus on the specific problem. GLSL is a great example of a language targeted to a specific problem domain... but at least it has C-like syntax so the learning curve is relatively small. Even then, I think there is a general trend of divergence - people keep inventing new languages & formats and writing a pile of spaghetti for someone else to maintain. shudder.

I can't imagine companies actually implementing their own languages... or perhaps the thought is too scary for this coder's head to latch on to.
June 11, 2007 2:29 PM PDT

Clay Breshears (Intel)
Total Points:
14,983
Status Points:
14,983
Black Belt
 Companies might not implement their own language, but Intel Fellow Shekhar Borkar is attributed with the comment: "specific tasks could have their own optimized languages." This is the synopsis from the CNET article (http://news.com.com/Intel+Software+needs+to+heed++Moores+Law.....l?tag=item), so I don't have the direct quote that was the source of this summation. We already have languages that are designed for specific uses or industries (COBOL for business, Fortran for scientific computation, LISP for AI). I don't think it's much of a leap to see more languages tuned to more specific tasks.
February 28, 2008 5:23 AM PST


David Biesack		 Note: the "DARPA HPCS" link in this article is not longer valid. Try:

http://www.darpa.mil/IPTO/programs/hpcs/hpcs.asp
February 29, 2008 1:59 PM PST

Clay Breshears (Intel)
Total Points:
14,983
Status Points:
14,983
Black Belt
 Thanks, David.

Trackbacks (0)


Leave a comment  

To obtain technical support, please go to Software Support.
Name (required)*

Email (required; will not be displayed on this page)*

Your URL (optional)


Comment*