"Developing for Terascale-on-a-Chip" is a series of three articles on looking at the future of programming for many-core parallel processors that run in the teraflop and above range – terascale-on-a-chip. This is the first article in the series.

Intel®—and the entire processor industry—have been chanting a parallel processing mantra for several years. It’s only going to get more intense. The company now produces 4-core processors, and with Moore’s law as a guide, we should expect 128 cores in ten years, 256 in twelve, 512 in fourteen, and over one thousand by 2023. Of course, that’s all theoretical. The reality is that Intel built an experimental 80-core processor called Polaris and tested it last year using a parallel application with message passing. Polaris delivered over one single-precision teraflop (and is capable of 2 TFs) for a Stencil benchmark from a single chip that dissipated only 62 watts—terascale-on-a-chip. That’s a first.

Intel has been delivering firsts in parallel computing for many years. They were the first commercial company to ship a massively parallel processing (MPP) supercomputer in 1985 and, in 1996, the first to deliver a machine that ran over one teraflop (ASCII Option Red with over 9000 processors). In 2006, they were the first to deliver a quad-core processor for the general-purpose computing space. Intel’s experience in parallel computing has a long history.

When Intel talks about terascale-on-a-chip, they mean more than the multi-core architecture currently shipping. They’re talking about a near future where applications can run on hundreds of cores, processing terabytes of data per second using a single processor.

The future terascale processor will be quite different from today’s CPUs. Today’s processors are multi-core, consisting of a few high-performance, power-efficient, megalithic (fat) cores with multiple advanced caches communicating over one or more system buses.

Tomorrow’s terascale processor will be “many-core,” with a lot of smaller, low-power cores. Its architecture will likely be heterogeneous in some cases, integrating fat cores with a variety of many smaller cores; advanced, shared caches; special-purpose accelerators (network offload, XML acceleration, graphics processors, etc.); and even on-chip, hierarchical memory arrays – all interconnected by a high-speed network on a chip. Unlike today’s processors, failure of some of the cores in a terascale processor will not cause the entire processor to fail. One might compare it to a fault-tolerant cluster of machines on silicon.

The whole architecture concept is designed to continue increasing processor performance—into the terabits per second rate and beyond—without increasing the chip’s power requirements.

The result is an extremely power-efficient, parallel processor with many cores that can support a lot more concurrency. And that has profound impact on a programmer.

For years, developers have reaped the benefits of the hardware evolution described by Moore’s Law without having to do much to their code. Improving the performance of their code simply required running it on the next-generation processors.

To see continued performance improvements on many-core processors, developers will have to think, from the ground up, in terms of highly parallel applications. They will need to expose more concurrency in their design problem and express a lot more concurrency in their code.

Fortunately, programmers will be able to utilize the same tools they do for threading today – OpenMP*, MPI*, and Pthreads* – in their applications for many-core processors. Plus, they will be able to take advantage of parallel programming tools, like Intel® Thread Checker, and Intel® Thread Profiler. Intel developers are also working on new programming models to meet the needs of a new generation of parallel programmers, such as Intel® Threading Building Blocks. There is a lot of information on the Multi-Core Developer Community on these topics. But programming for terascale-on-a-chip will demand even more.

Benefiting from terascale-on-a-chip comes down to Amdahl’s law and the serial fraction of the program (you can learn more about Amdahl’s law in “Nuts and Bolts of Multithreading”). With an infinite amount of parallel capacity, Amdahl’s law says that your best speedup will be the inverse of the code’s serial fraction. For example, a program that is 25 percent sequential will run a theoretical maximum of four times faster on infinitely parallel hardware

Another way of looking at it is if you want maximum speedup from a target machine of four cores, you only need to parallelize 75 percent of your code. This means that, with multi-core, finding loops and other obviously threadable functions and threading them for four or eight cores can deliver significantly better throughput performance on a multi-core machine. Leaving the rest of the application sequential doesn’t degrade performance for a system with a few cores. For 64 cores, it’s an entirely different matter.

To maximize speedup on 64 or more cores requires making the code 99 percent, or higher, concurrent. Getting to this level of concurrency is challenging. You can’t do this by taking your serial algorithm and sprinkling the code with Pthreads or OpenMP constructs. You need to rethink the problem in terms of a well-designed parallel algorithm. You have to get creative with “wetware,” the stuff that goes on behind your eyes and between your ears. You have to go back to the original problem and consider it from a whole new perspective—that of a many-core, parallel computing environment—to expose every bit of concurrency you can find.

This process takes work. Experienced parallel programmers will tell you that getting the serial fraction to 10 percent is difficult—but possible. And there are no automated tools (yet) that can replace the kind of intelligent creativity required.

Starting from the problem requires having knowledge (or access to someone with it) of the domain, the problem, algorithm types, and the kinds of data structures the problem depends on.

From there you can design a parallel algorithm with maximum concurrency in mind, which you can then express in your code. Methods to do this are discussed in article 2.