Doing cool stuff with OpenBSD rdomains

2014-05-08 - Doing cool stuff with OpenBSD rdomains


What I did the first time:

I manage some internally routed IP ranges at work for what we call the project network. All projects consists of a few servers and clients that all use the same local IP range ( The first plan was to put NAT routers in front of all projects and binat several addresses, but then we would have to by equipment for every project (not in out budget). Since we use VMware for everything anyway, simply giving each machine a second network card with their routed IP was much easier.

We bought a Cisco ASA 5510 and received two Class C networks from our Company IT department ( and The Company internal routers now said these could be found at (the address of our Cisco ASA in the office LAN).

Now we could not just connect all the different projects to the same subnet. Mainly because they all like to use the same DNS and NetBIOS names, and because our control systems software would "freak out" if it could see all the other systems on the same subnet.

We split our /24 subnets down to /28 to keep the projects separated (Ie. became .etc), and connected each of these with separate VLANs in our project network switches (Cisco Catalyst). Then made virtual interfaces in the ASA to each VLAN.

Now we could sit in our offices and use remote desktop to addresses in the project network. After a while we found that this had it's own set of problems. All projects use our company Control system software(which is basically what we sell) and it did not like this new "fansy" interface. It started complaining of license violations and other weird things because it could now see it's clients on two separate networks.

The first workaround was that we had to take the license servers of the routed networks to that they stopped nagging. Having this extra interface gave us other problems as well, like clients registering with the wrong IP in the project DNS server, and having the interfaces appear in the wrong binding order would be a constant thing to fix.

Later we got a different kind of problem; not enough IP's. I asked Company IT for another subnet, which they gave... reluctantly (but only half aka. /25). They could not understand how we had manage to use all the IPs in the subnets we had. But my allocation excel sheet said differently.

But that got me wondering. Had we really used all the IP's? Short answer.. No. Long answer: We had allocated all the IP's but we did not use them. Sub-netting had cost us a few addresses, but that did not count for much. But the nature of keeping the systems separate with subnets had limited the network capacity to 32 system (one per subnet). This is extremely wasteful since each subnet has 13 valid host IP's and on average a system might use 7-8 IP's.

What I did the second time:

When I finally got the next subnet from company IT ( I wanted to do something different. I wanted NAT for all the projects, so they could live in their own little bubble. But I needed the NAT equipment to be free (I have no budget, and OpenBSD is my goto router OS). But I can not have one OpenBSD virtual machine for each project because that would take to much resources away from the actual project virtual machines. So I need to use one OpenBSD router for all projects.

My new OpenBSD 5.5 router with PF(tj) connects to it's two clients (allan and kris) on the "inside" both using the same IPs. It is connected to the ASA on the "outside". The ASA routing table refer everything .207 to the OpenBSD machine.

Since tj is a virtual machine, there is no problem for me to add one interface for each project to it.

em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,NOINET6> rdomain 1 mtu 1500
        lladdr 00:50:56:96:65:36
        priority: 0
        media: Ethernet autoselect (1000baseT full-duplex,master)
        status: active
        inet netmask 0xfffffc00 broadcast
em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,NOINET6> rdomain 2 mtu 1500
        lladdr 00:50:56:96:65:39
        priority: 0
        media: Ethernet autoselect (1000baseT full-duplex,master)
        status: active
        inet netmask 0xfffffc00 broadcast

Each interface is in a separate rdomain (see that bit on the end of the first line). And each rdomain has itself as default route.

root at tj# route -T 1 add default
add net default: gateway
root at tj# route -T 2 add default
add net default: gateway

This won't actually route anything, but there needs to be an entry there. The reason for that is that packets are evaluated for valid routes before they get to pf. So something needs to match or the packet is dropped before it can be changed in pf.conf

pass in
pass out

## rdomain 1
pass in on egress inet from any to rdr-to rtable 1
pass on em1 inet from to ! binat-to rtable 0

## rdomain 2
pass in on egress inet from any to rdr-to rtable 2
pass on em2 inet from to ! binat-to rtable 0

Edit 22.05.2014: I had a bug here in the pf.conf where I used nat-to insted of binat-to. That makes it work only from the outside to the inside, but not the other way around.

TADA! repeat similar rules for each interface and rdomain and magic happens.