The below describes building Firedancer from scratch and running it
optimized on a stock GCP
n2-standard-80 instance with a stock GCP
RHEL8.5 image. For reference, this instance is a dual socket 20
physical core Intel Cascade Lake at 2.8 GHz with hyperthreading enabled
-> 2 NUMA nodes total, 80 logical cores total).
Setup for other reasonably modern x86_64 architecture hosts (reasonable modern here AVX2 support, this includes most Intel architectures since ~mid-2013 and AMD architectures since ~mid-2015) running reasonable modern Linux-ish distributions is expected to be very similar. Though it is possible to run Firedancer to run on older hosts, it is not supported and the below assumes a reasonably modern host.
While the Firedancer applications are meant to run from a normal user
account, tuning a host to run Firedancer optimally requires a some
administrative operations (should not be any more than the existing
validator). Though it is possible to run Firedancer without any of
these tunings, it is not recommended and the below assumes the user has
Log into the host and configure user environment to taste (e.g. install favorite editors / code development environment, etc). This is not specific to Firedancer but note hosts like this have very minimal installs on first login.
Install standard development tools.
$ sudo dnf groupinstall development
As mentioned above, the minimal installs are missing even basic development tools. The
developmentgroup includes such things as the stock
gcccompiler, build tools like
make, version control systems like
git, etc. Firedancer likely can use other tool chains / compilers (e.g.
clang) but this is not routinely tested currently.
Install additional dependencies.
$ sudo dnf install numactl-devel hwloc
To simplify install and Firedancer tries to have virtually no external dependencies that aren't readily nearly universally prepackaged available in stock Linux distributions.
hwlocare not included in
developmentby default above unfortunately but they are widely available pre-packaged. They provide APIs used to help implement various NUMA optimizations on the host.
Configure the host for high performance by allowing users to lock pages in memory and increase the scheduler priority of performance critical user threads. As superuser (e.g.
sudo su -), add the following lines to
* - memlock unlimited * - nice -20 * - rtprio unlimited
(The user might only opt to be more restrictiv if desired, e.g. only allow Firedancer users to do this.) Recommend logging out and then logging back in again after making these changes.
ulimit -acan be used to tell if the new user limits have taken effect. E.g., as a regular user, the following would indicate that the changes are in effect:
$ ulimit -a ... snip ... scheduling priority (-e) 40 ... snip ... max locked memory (kbytes, -l) unlimited ... snip ... real-time priority (-r) unlimited ... snip ...
Get Firedancer. E.g.:
$ git clone https://github.com/firedancer-io/firedancer.git firedancer
will make a directory in the current directory called firedancer and copy of the current head-of-tree code base into that directory.
Build Firedancer. E.g. From the directory where firedancer was checked out:
This will do a parallel incremental build using all non-isolated cores and should be reasonably quick even when done from scratch (less than a minute). The default machine target will be
MACHINE=rh8_x86_64(details of this machine can be found in
config/rh8_x86_64.mk). The build results will be in the relative directory
makehas many powers; run
make helpfor more info. If building on a system with lots of isolated cores, see
Reserve host resources for application usage. E.g.:
$ sudo build/rh8/x86_64/bin/fd_shmem_cfg \ alloc 8 gigantic 0 \ alloc 8 gigantic 1 \ alloc 512 huge 0 \ alloc 512 huge 1
will reserve 8 2GiB pages on numa node 0, 8 2GiB pages on numa node 1, 512 2MiB pages on numa node 0, and 512 2MiB pages on numa node 1 on the host for application use (assuming the host in fact has enough free contiguous physical DRAM availability). Adjust this as necessary for the number of cores, system DRAM availabiity, application mix, application configurations, etc.
fd_shmem_cfghas many powers. Run
fd_shmem_cfg helpfor more info.
Create an appropriately permissioned sandbox for managing shared memory data structures. E.g.:
$ sudo build/rh8/x86_64/bin/fd_shmem_cfg init 0700 [USER] ""
[USER]is the user that will run Firedancer applications. Multiple sandboxes with different permissions, users, groups can / coexist simultaneously (the
""will be use default group of