Use of Barefootd for temporary service transition

Moving can be a lot of work, but it can also be an opportunity to solve problems that are not commonly encountered during the daily grind, and when Inferno Nettverk recently moved to new and slightly bigger offices across the street from the old offices, one of the more interesting aspects was handling the transfer of our servers from the old location to the new one.

When planning the move it was possible to divide the servers and the services running on them into two groups; services that could easily be handled by the backup server at our office in another country, and services that could not. Most of our public facing services fell into the first category. For example, stateless web servers that only serve static pages and SMTP servers could easily be moved with no or only minor changes to DNS. These services were handled by the backup server during the move and then moved back to the main servers after the move had been completed and all machines were successfully running at the new location.

Then there were the services that could not simply be moved to the backup server for various reasons, which included servers for services such as a stateful web server and a login server. These were services that primarily are used internally, so the servers being down would not inconvenience our customers, but they were still being actively used by our own employees, so it was still desirable to limit the downtime as much as possible.

Preparations

The move also included a change of Internet provider, so at a minimum, it would be required to change the IP address of all public services and the machines would need to be moved physically from the old to the new office location.

Both locations are fairly close physically, so optimistically the time from a server being taken down at the old location, to it being reconnected and started at the new location, could perhaps be limited to around 30 minutes if no unexpected problems were encountered. More time consuming would be the time to update the DNS information, which at best would take several hours to propagate around the world.

A downtime of many hours would be problematic for these services, making it necessary to find a way to avoid any downtime or disruption in relation to the change in DNS information, and as luck would have it, we have a software solution that could be used for this purpose.

Barefoot is a generic port bouncer developed by Inferno Nettverk, that can be used to redirect TCP and UDP traffic between different IP addresses and port numbers. By using Barefoot it would be possible to make the services accessible both by their old address and by their new address simultaneously.

Transition server

The plan that was eventually set in motion involved a temporary firewall/router at the new location, which was realized using an OpenBSD machine with Barefoot installed. The Barefoot/OpenBSD machine was temporarily configured with the new IP addresses of all public facing servers.

Barefoot was then configured to accept traffic on all the new IP addresses and service ports that the services would use after the move, and temporarily forward all traffic to the addresses/port numbers that the services were still running on at our old office location.

This made it possible to connect to the services on both their old addresses (directly, as before) and the new addresses (via Barefoot, which would forward the traffic, invisible to the user, to the old addresses), allowing the DNS information to be updated without any time pressure.

As the new DNS information propagated around the world, clients gradually moved from using the old addresses to the new addresses, while the services were still actually hosted on the old site, using the old addresses.

Another benefit with this approach was that it simplified moving the router/firewall. The configuration of the transition server was not time critical and one service at a time could be added and tested. When forwarding of all services had been fully added on the transition server, we also had a running configuration that could serve as documentation of the changes that would later need to be made to the router used at the old site to get it running on the new network.

Machine move

After sufficient time had passed for DNS information to have updated, the server machines were moved physically to the new location and connected to the Internet via the transition server, one machine at a time. As each service was enabled on the new site, the new IP address of the service was decoupled from the transition server, making requests go directly to the service, and not via the Barefoot server.

Eventually all services were running on the new network, with only minimal downtime having been experienced by each service, primarily from the time required to actually move the machine physically between the two locations.

The final unit to be configured was the firewall/router from the old site, which was moved to the new network with the services still running via the transition server. Based on the configuration on the transition firewall/router, the router from the old site was configured to work on the new network.

The final step was then to reconnect the servers to have the traffic go via the router and not via the transition firewall/router. Since the configuration of the server machines would be identical in both cases, this simply involved moving the Ethernet cable of each server from one switch to another. After all machines had been reconnected and the operation of the services running on the machines had been verified, the transition server could be shut down.

Post-move analysis

The approach chosen to reduce service downtime during the move was based on traffic redirection using the Barefoot port bouncer. By using a transition server at the new site, it became simple to prepare the new site for the server machines and downtime was reduced to the time required to actually disconnect, move, and reconnect the machines.

The move went well, with no major problems encountered. Most of the problems encountered were related to the transition server not having the full set of functionality offered by the router at the old site. This resulted in some inconvenience due to some missing DNS information on internal hosts, but nothing that was difficult to work around.

A possible improvement that might be considered should the need to move arise again in the future is to start on the new site with a router that is configured identically to the router on the old site, and then gradually adapt it as necessary for the new site. The benefit of the chosen approach was that it resulted in a simple configuration that very clearly documented the changes that it was necessary to perform, and it would have been more work to duplicate all services that were available internally in the network on the original site. However, in the end, the chosen approach worked well enough and made the transition fairly simple since it could be done in small incremental steps.

Conclusion

We were able to move multiple servers with the only downtime being the time required to physically move the machines. An alternative approach would have been to add a backup server for each service that could handle client requests while the machines were being moved, but for services where adding redundancy would be difficult, or where some limited downtime can be accepted, the chosen approach worked well.

Even though all services got new IP-addresses at the new office location, by using Barefoot to forward data between the new and old IP-addresses, no downtime was caused waiting for DNS info to propagate around the world.

Using Barefoot during service transition between machines not only worked well, it was also satisfactory to find a solution to a problem that needed to be solved in software that we had developed ourselves.

Example configuration

To forward traffic with Barefoot in a way similar to what is described above, a configuration like the following can be used:

external: eth1 #interface for outgoing traffic

#forward traffic received at 10.0.0.1 (new site address), port http
#to 172.16.0.1 (old site address)
client pass { from: 0.0.0.0/0 to: 10.0.0.1 port = http
              bounce to: 172.16.0.1 port = http
              protocol: tcp
              log: connect disconnect error
}

Assuming 10.0.0.1 corresponds to the new IP address of a web server, and 172.16.0.1 the old address, the above rule will forward all HTTP requests received on the new address to the old server.

Using Barefoot like this made it possible to keep all such mappings in a single file and it was simple to see when traffic started being forwarded by looking at the Barefoot log file.

Add comment:

To add a public comment to this article, please use the form below.

For information on how we process and store information submitted via this form, please see our privacy policy for blog comments.