“If I have two networks attached to my server, how do I make sure traffic that comes into an interface goes out the same interface?”
I have found this question has come up a fair amount, from fellow techs and from customers. It was always answered with a murky explanation skirting around “advanced Linux routing” and an short explanation of a “better” or “more standard” way. I knew there must be a better way, and sure enough there is - Source Policy Routing with iproute2. This is just one way to do it, and I am sure there are other ways as well (marking traffic, for example).
First add two tables into /etc/iproute2/rt_tables with your favourite text editor, one for each network. This is creating two new routing tables which we can work on.
Next up, we will assign each network and default gateway to each of our new routing tables (network10 and network20):
Then, we need to make sure our main routing table is aware of our networks
Finally, we need to add a routing rule to say if traffic is coming from one of our networks to use a specific routing table.
Loading at Boot
I am admittedly not too happy with this next bit and I think it could be done a lot better, but currently I use a script in /etc/network/if-pre-up.d/ to facilitate loading of these routes at boot. Please contact me if you ever make a better version of this, I would love to use it!