Best way to build large applications in parallel

Best way to build large applications in parallel

Imagen de Henry Gabb (Intel)


Hello everyone,
At the HPC site where I worked before joining Intel, I used a program called pmake to build applications in parallel. Pmake worked on distributed-memory parallel systems. One of the proprietary UNIX systems also had a regularmake program with a -P option for parallel builds.

What is the best way to build large applications in parallel on a Linux cluster? A Google search on 'pmake' turned up a couple of candidates: Parallel Make and Berkeley Pmake. Does anyone have other recommendations?

Thanks,
Henry

publicaciones de 4 / 0 nuevos
Último envío
Para obtener más información sobre las optimizaciones del compilador, consulte el aviso sobre la optimización.
Imagen de Clay Breshears (Intel)



Henry -

The GNU make utility allows for parallel compilation with the "-j" flag.

gmake -j 2 MyApp will spawn off two threads to divide up the building of multiple files and libraries. Unfortunately, this only works within a single node, not across a cluster.

--clay

Message Edited by ClayB on 08-24-2004 10:55 AM

Imagen de jolsson

I'm also interested in this, but on Windows. We are using the cons
build system, which has a pcons flavor http://www.baldmt.com/parallel-cons/.
We are not using pcons yet, and also might need a better cluster file system
than just a share on windows.

There is also the newer http://www.scons.org/build system that supports parallel build on Linux and Windows.

I think there has been cluster versions of either cons or scons, but I cannot
find them now.

cons/scons have the advantage over make that dependencies are automatically generated, including what libs to build before what
execiutables and what source code needs to be generated before
compilation starts. It also handles changes using MD5 checksums
which means that things will rebuild properly even if time stamp
of file doesn't change or goes backwards (like if you undo a version
control checkout).

Jonas

Imagen de Community Admin

I believe you are looking for:

http://distcc.samba.org/

It is very easy to setup and use.

RPMS for RHEL or CentOS can be found at:

http://dag.wieers.com/packages/distcc/

Inicie sesión para dejar un comentario.