Introduction

Distcc is a small software used to increase the computation power by using other computers on a network. In fact we can distribute the compilation phase between more hosts in order to reduce the compilation time.

You can find the source here: http://code.google.com/p/distcc/downloads/list

Our scenario

In our configuration we have two machines, 192.168.1.4 and 192.168.1.5. We will use these two host to compile the kernel 3.3.0-rc4.

First of all, we have to choose the client machine, the one that execute the "distcc" command. In our case 192.168.1.4 is the client.

The server machine (one or many) runs the daemon. If we don't use ssh, we have to start "distccd" first in order to accept distribuited compiling request.

The IP address of the server is 192.168.1.5.

gcc and glibc

I suggest to install the same version of gcc and glibc on both server and client. This to avoid problems during the compilation phase.

Set up the distcc user

Since we want to compile our program without the need to log in as root, we have to create a user named "distcc":

This is what the /etc/ passwd file looks like:

root@lightstar:/etc/distcc/#cat /etc/passwd | grep distcc
distcc:x:1002:100:,,,:/etc/distcc/:/bin/bash

You can create the user on both client and server.

Distcc on the client

The host with the IP 192.168.1.4 is the client so we will execute the command "distcc".

On this machine, we have to specify the other hosts that will participating to the compilation process.

We have mainly three choices:

• export the environment variable DISTCC_HOSTS
• export the environment variable DISTCC_DIR and create the file "hosts" in this directory
• create the /etc/distcc/hosts file

We choose the third option.

Compiling with SSH

Distcc offers the possibility to execute the compilation process through an ssh connection. This can slow down a bit the transfer of data between the hosts but the advantages (especially if we think that there aren't authentication mechanisms in distcc) are relevant.

To use ssh we have simply to write something like this in the configuration file:

user@host

Compression with LZO

We also have the possibility to compress the data across the network thanks to the "lzo" option.

We can add this option after the host specification:

user@host,lzo

Authentication with keys

If you want to use SSH you have to set up authentication with keys first. This to avoid the request for a password during the compilation process.

On the client machine, log in as "distcc" and generate the keys:

distcc@lightstar:~$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/etc/distcc/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /etc/distcc/.ssh/id_rsa.
Your public key has been saved in /etc/distcc/.ssh/id_rsa.pub.
The key fingerprint is:
3c:a6:b0:de:e1:bd:34:13:98:b8:86:ea:b9:a1:99:ee distcc@lightstar
The key's randomart image is:
+--
[ RSA 2048]----+
|                 |
|                 |
|                 |
|     . +         |
|    o o S        |
|   . + o o       |
| .. + o +        |
|.+oo o + o&
amp;nbsp;      |
|OE. . o o.       |
+-----------------+

Now we have to copy the public key to the servers.

In our case the client 192.168.1.4 will log in on 192.168.1.5 by using SSH keys so we need to copy the keys from 192.168.1.4 to the other server.

You can use this command:

distcc@darkstar:~$ ssh-copy-id distcc@192.168.1.5
distcc@192.168.1.5's password: 
Now try logging into the machine, with "ssh 'distcc@192.168.1.5'", and check in:

  ~/.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting

On the server it will be created an "authorized_keys" file with the public key of 192.168.1.4.

Now when you will log in from the client to the server no password will be asked:

distcc@lightstar:~$ ssh 192.168.1.5
Last login: Sun Feb 26 16:16:43 2012 from darkstar
Linux 3.2.2-smp.
distcc@lightstar:~$ 

Configuration file on the client

At the end, our "hosts" file on 192.168.1.4 will looks like this:

#distcc config file
localhost/4 root@192.168.1.5/4,lzo

The syntax is quite simple. We specify the host, for example localhost and after the "/" the limit of jobs that this server can run.

On the servers

Normally we would run "distccd" on the server in order to accept incoming connection, but in our case we use ssh so we don't need "distccd".

If you don't use ssh you can start the distcc daemon as follow:

root@lightstar:~$ distccd --allow 192.168.1.0/24 --daemon

If you use SSH you have only to set up authentication with keys as explained above.

Let the compilation start

When you are done with the configuration phase, you can try to compile something.

In our case we will compile the kernel 3.3.0-rc4.

To do this simply use make and add at the end the definition of the environment variable CC to specify which compiler use:

distcc@darkstar:/usr/src/linux$ make -j5 CC=distcc
  SYSTBL  arch/x86/syscalls/../include/generated/asm/syscalls_32.h
  CHK     include/linux/version.h
  HOSTCC  scripts/basic/fixdep
  CHK     include/generated/utsrelease.h
  HOSTCC  scripts/kallsyms
  CC      scripts/mod/empty.o
  HOSTCC  scripts/pnmtologo
  HOSTCC  scripts/conmakehash
  HOSTCC  scripts/mod/mk_elfconfig
  MKELF   scripts/mod/elfconfig.h
  HOSTCC  scripts/mod/file2alias.o
  HOSTCC  scripts/mod/modpost.o
  HOSTCC  scripts/bin2c
  HOSTCC  scripts/mod/sumversion.o
  CC      kernel/bounds.s
  GEN     include/generated/bounds.h
  CC      arch/x86/kernel/asm-offsets.s
  HOSTLD  scripts/mod/modpost
  GEN     include/generated/asm-offsets.h
  CALL    scripts/checksyscalls.sh
  CC      init/main.o
  CHK     include/generated/compile.h
  LD      usr/built-in.o
  CC      kernel/fork.o
  AS   ;  arch/x86/crypto/aes-i586-asm_32.o
  CC      arch/x86/kernel/process_32.o
  UPD     include/generated/compile.h
  CC      kernel/exec_domain.o
  CC      arch/x86/crypto/aes_glue.o
  AS      arch/x86/crypto/twofish-i586-asm_32.o
  CC      arch/x86/kernel/signal.o
  CC      arch/x86/crypto/twofish_glue.o
  CC      init/do_mounts.o
  CC [M]  arch/x86/crypto/crc32c-intel.o
  CC      init/do_mounts_rd.o
  LD      arch/x86/crypto/aes-i586.o
  LD      arch/x86/crypto/twofish-i586.o
  LD      arch/x86/crypto/built-in.o
  CC      kernel/panic.o
  CC      kernel/printk.o 
.....
.....

The monitor

You can monitor the compilation by using distccmon-text or distccmon-gnome.

Here we've an example of distccmon-text:

bash-4.1$ distccmon-text 
 27981  Compile     mprotect.c                               192.168.1.5[0]
 27996  Compile     wait.c                                      192.168.1.5[1]
 27969  Compile     mmap.c                                   localhost[0]
 28004  Compile     i387.c                                      localhost[1]
 27988  Compile     fs-writeback.c                           localhost[2]