CPU real-time scheduling

This page describes how to configure options related to CPU scheduling and priority.

Barefoot supports the sched_setscheduler() and sched_setaffinity() system calls that make it possible to choose the scheduling algorithm, priority and CPUs for Barefoot processes. These system calls are available on platforms such as Linux and FreeBSD. SunOS has a related system call called processor_bind() which is currently not supported.

Using this functionality can have an effect on the performance and behavior of Barefoot. Especially in usage scenarios where there are other resource intensive applications running on the same machine as Barefoot there might be benefits from using this functionality.

The real-time functionality in Barefoot falls into two groups: scheduling/priority and CPU-affinity (processor binding). Separate keywords exist for controlling each one. Common for either group is that the options are set globally and apply to all connections, but separate values can be set for the different process types in Barefoot.

The different operations performed by Barefoot is spread among four different process types:

Mother Accepts connections from clients and forwards clients between the other three Barefoot process types.
Negotiate Handles ACL-lookup for TCP clients.
Request Opens connections to external hosts.
IO ACL-lookup for UDP clients and I/O for both UDP and TCP clients.

It is possible to set different real-time values for each process type, but it is generally not necessary to consider having different setting for each process type. The settings of the mother, negotiate, and request processes affect the session setup time. We recommend using the same settings for all these process types or leaving them unchanged. After completion of session setup, a client session will be managed by an io process, and the real-time settings of io processes will potentially affect the throughput and latency of the data passing through Barefoot.

Real-time scheduling/priority control

The scheduling algorithm and priority to use for the different Barefoot process types can be specified in the Barefoot server configuration file using the following syntax:

cpu.schedule.PROCTYPE: SCHED/PRIORITY

The above values are defined as follows:

PROCTYPE: Barefoot process type, which can be one of the following: mother, negotiate, request or io. Setting a value for the io process type is likely to have the largest impact on performance; setting values for the other processes types will only have an impact on the initial session establishment time.
SCHED: Real-time scheduling algorithm to use, with the following values usually being available: fifo, rr, other. Note that other is the standard, non-realtime, scheduling algorithm.
PRIORITY: A numeric value giving the process' scheduling priority.

Note that root privileges might be required to enable real-time scheduling.

The following shows an example where real-time scheduling is enabled for all Barefoot process types:

#when root access is needed to enable real-time scheduling
user.privileged: root

#set real-time scheduling (round-robin):
cpu.schedule.mother:    rr/10
cpu.schedule.negotiate: rr/10
cpu.schedule.request:   rr/10
cpu.schedule.io:        rr/15

We recommend that cpu affinity (see below) is used when real-time scheduling is enabled.

CPU affinity control

CPU affinity/proccess binding makes it possible to specify which CPUs the different Barefoot processes should use, via the sched_setaffinity() system call. This functionality can be used to eliminate the possibility of Barefoot consuming all available CPU resources if real-time scheduling is enabled, by making sure that there is at least one CPU that will not be used by Barefoot. The syntax is as follows:

cpu.mask.PROCTYPE: CPUIDS

The above values are defined as follows:

PROCTYPE: Barefoot process type, which can be one of the following: mother, negotiate, request or io.
CPUIDS: A space-separated list of CPUs to which processes of the specified type should bind. CPUs are identified numerically, starting with the value 0 for the first CPU/core.

The following shows an example where CPU affinity is specified for all Barefoot process types:

#run io processes on CPUs 1 + 2, all other Barefoot processes on CPU 0.
#Assuming there are four CPUs on the machine, this leaves CPU #3 free 
#for other, non-Barefoot related, tasks.
cpu.mask.mother:    0
cpu.mask.negotiate: 0
cpu.mask.request:   0
cpu.mask.io:        1 2