Fast forward to now, and FreeBSD has a tool called poudriere, which can compile a set of packages with exactly the options needed and publish them as a FreeBSD package repository, from which any FreeBSD machine can then use to download the binary packages from and install them via pkg. It's a bit like starting your own Linux distro, picking just the software and compile options you need and distributing them as pre-built binary packages.
Finally I took the time to set up a FreeBSD build machine running poudriere (in a virtual machine) and can much more easily perform updates on the SolarNetwork infrastructure. There was just one major stumbling block along the way: I didn't know how to get pourdriere to use the specific version of Clang I needed. There is plenty of information online about setting up poudriere, but I wasn't able to find information online about getting it to use a custom compiler toolchain. After some trial and error, here's how I finally ended up accomplishing it:
Create toolchain package repository
Poudriere works with FreeBSD jails to manage package repositories. Each package distribution uses its own jail with its own configuration such as what compiler options to use and which packages to compile. The first task is to create a package repository with the toolchain packages needed, in my case this is provided by the devel/llvm39 package. This toolchain repository can then be installed in other poudriere build jails to serve as their compiler.Once poudriere was installed and configured properly, the steps looked like this:
# Create jail poudriere jail -c -j toolchain_103x64 -v 10.3-RELEASE mkdir /usr/local/etc/poudriere.d/toolchain_103x64-options # Create port list (for this jail, just the toolchain needed, devel/llvm39) echo 'devel/llvm39' >/usr/local/etc/poudriere.d/toolchain-port-list # Update to latest (each time build) poudriere jail -u -j toolchain_103x64 poudriere ports -u -p HEAD # Configure options poudriere options -j toolchain_103x64 -p HEAD \ -f /usr/local/etc/poudriere.d/toolchain-port-list # Build packages poudriere bulk -j toolchain_103x64 -p HEAD \ -f /usr/local/etc/poudriere.d/toolchain-port-list
After quite some time (llvm takes a terribly long time to compile!) the toolchain packages were built and I had nginx configured to serve them up via HTTP.
Create target system package repository
Now it was time to build the packages for a specific target system. In this case I am using the example of building a Postgres 9.6 based database server system, but the steps are the same for any system.First, I created the system's poudriere jail:
# Create jail poudriere jail -c -j postgres96_103x64 -v 10.3-RELEASE # Create port list for packages needed echo 'databases/postgresql96-server' \ >/usr/local/etc/poudriere.d/postgres96-port-list echo 'databases/postgresql96-contrib' \ >>/usr/local/etc/poudriere.d/postgres96-port-list echo 'databases/postgresql-plv8js' \ >>/usr/local/etc/poudriere.d/postgres96-port-list # Configure options poudriere options -j postgres96 -p HEAD \ -f /usr/local/etc/poudriere.d/postgres96-port-list
Second, install the llvm39 toolchain, using the custom toolchain repository:
# chroot into the build jail chroot /usr/local/poudriere/jails/postgres96_103x64 # enable dns resolution for the build server (if DNS names to be used) echo 'nameserver 192.168.1.1' > /etc/resolv.conf # Copy /usr/local/etc/ssl/certs/poudriere.cert from HOST # to /usr/local/etc/ssl/certs/poudriere.cert in JAIL mkdir -p /usr/local/etc/ssl/certs # manually copy poudriere.cert here
Then I configured pkg to use the toolchain repository via a /usr/local/etc/pkg/repos/poudriere.conf file:
poudriere: { url: "http://poudriere/packages/toolchain_103x64-HEAD/", mirror_type: "http", signature_type: "pubkey", pubkey: "/usr/local/etc/ssl/certs/poudriere.cert", enabled: yes, priority: 100 }
The URL in this configuration resolves to the directory where poudriere build the packages, served by nginx. Next I installed the toolchain, explicitly telling pkg to use this repository:
pkg update pkg install -r poudriere llvm39 # clean up and exit the chroot rm /etc/resolv.conf exit
Now I can configure poudriere to use the toolchain by creating a /usr/local/etc/poudriere.d/postgres96_103x64-make.conf file with content like this:
# Use clang
CC=clang39
CXX=clang++39
CPP=clang-cpp39
DEFAULT_VERSIONS+=pgsql=9.6 ssl=openssl
The next step is what took me the longest to figure out, probably because I had not studied how poudriere works with ZFS very carefully. It turns out poudriere makes a snapshot of the jail named clean, and then clones that snapshot each time it performs a build. So all I needed to do was re-create that snapshot:
# Recreate snapshot for build
zfs destroy zpoud/poudriere/jails/postgres96_103x64@clean
zfs snapshot zpoud/poudriere/jails/postgres96_103x64@clean
Finally, the build can begin normally, and the custom toolchain will be used:
# Build packages
poudriere bulk -j postgres96_103x64 -p HEAD \
-f /usr/local/etc/poudriere.d/postgres96-port-list
Update target system to use poudriere repository
Once the system's build is complete, it is possible to configure pkg on that system to use the toolchain repository via a /usr/local/etc/pkg/repos/poudriere.conf file:poudriere: { url: "http://poudriere/packages/postgres96_103x64-HEAD/", mirror_type: "http", signature_type: "pubkey", pubkey: "/usr/local/etc/ssl/certs/poudriere.cert", enabled: yes, priority: 100 }
Then I copied the certificate from the build host to the file as configured above. I no longer want to use the default FreeBSD packages on this system, so I created a /usr/local/etc/pkg/repos/freebsd.conf file to disable it, with the following content:
FreeBSD: { enabled: no }
Done! Now, after running pkg update, all packages will install from the poudriere repository, and I no longer need to compile the software on the system itself.
