Best way to build large applications in parallel

Best way to build large applications in parallel

Portrait de Henry Gabb (Intel)
Henry Gabb (Intel)
mer, 18/08/2004 - 21:46


Hello everyone,
At the HPC site where I worked before joining Intel, I used a program called pmake to build applications in parallel. Pmake worked on distributed-memory parallel systems. One of the proprietary UNIX systems also had a regularmake program with a -P option for parallel builds.

What is the best way to build large applications in parallel on a Linux cluster? A Google search on 'pmake' turned up a couple of candidates: Parallel Make and Berkeley Pmake. Does anyone have other recommendations?

Thanks,
Henry

RSS Haut
4 posts / 0 nouveau(x)
Dernière contribution
Reportez-vous à notre Notice d'optimisation pour plus d'informations sur les choix et l'optimisation des performances dans les produits logiciels Intel.
Portrait de Clay Breshears (Intel)
Clay Breshears (Intel)
mar, 24/08/2004 - 15:55



Henry -

The GNU make utility allows for parallel compilation with the "-j" flag.

gmake -j 2 MyApp will spawn off two threads to divide up the building of multiple files and libraries. Unfortunately, this only works within a single node, not across a cluster.

--clay

Message Edited by ClayB on 08-24-2004 10:55 AM

Haut
Portrait de jolsson
jolsson
mer, 15/09/2004 - 00:02

I'm also interested in this, but on Windows. We are using the cons
build system, which has a pcons flavor http://www.baldmt.com/parallel-cons/.
We are not using pcons yet, and also might need a better cluster file system
than just a share on windows.

There is also the newer http://www.scons.org/build system that supports parallel build on Linux and Windows.

I think there has been cluster versions of either cons or scons, but I cannot
find them now.

cons/scons have the advantage over make that dependencies are automatically generated, including what libs to build before what
execiutables and what source code needs to be generated before
compilation starts. It also handles changes using MD5 checksums
which means that things will rebuild properly even if time stamp
of file doesn't change or goes backwards (like if you undo a version
control checkout).

Jonas

Haut
Portrait de Community Admin
Community Admin
mer, 15/09/2004 - 03:57

I believe you are looking for:

http://distcc.samba.org/

It is very easy to setup and use.

RPMS for RHEL or CentOS can be found at:

http://dag.wieers.com/packages/distcc/

Haut

Connectez-vous pour laisser un commentaire.