Multiple interfaces do not look like something complex to support in distributed storage, but it took us quite a while to get it right.
Mainly because Elliptics has a route table which is used by server and client nodes to get info about other nodes and to connect to them. If we increase this table, it means more connections, more stat/ping/heartbeat traffic, but mainly becase of confusion for client – what address should it use when multiple addresses have been received in route table? What if there is no route to some of them? What it some node has 3 addresses and others only one?
I decided to fix it the most confusing way ever possible. But yet very effective.
Let’s suppose there are 3 server nodes with the following addresses:
srv1: addr = 126.96.36.199:1025:2 10.10.10.10:1025:2 100.100.100.100:1025:2 srv2: addr = 188.8.131.52:1025:2 184.108.40.206:1025:2 220.127.116.11:1025:2 srv3: addr = 18.104.22.168:1025:2 22.214.171.124:1025:2 300.300.300.300:1025:2
Server node will be listening on 0.0.0.0 address actually, but its route table will contain appropriate above addresses.
When client connects to one of them, let’s say to srv2 – and client chose 126.96.36.199 address. Client doesn’t know which other addresses it received (above addresses is the whole route table which lives on every server node after they are connected to each other) should be used to connect to other nodes.
Should it be 188.8.131.52 for srv1 or 300.300.300.300 for srv3? If client connected to all addreses, which one should be used to actually send commands?
So we force admin to group addresses which belong to the same ‘logical route table’. So, when client connects to 184.108.40.206, it will receive whole route table, but will use 10.10.10.10 for srv1 and 220.127.116.11 for srv3. How to achieve that? Route table ID must be a monotonically increased number, so above table should be rewritten as following:
srv1: addr = 18.104.22.168:1025:2-0 10.10.10.10:1025:2-1 100.100.100.100:1025:2-2 srv2: addr = 22.214.171.124:1025:2-0 126.96.36.199:1025:2-1 188.8.131.52:1025:2-2 srv3: addr = 184.108.40.206:1025:2-0 220.127.116.11:1025:2-1 300.300.300.300:1025:2-2
0, 1 and 2 above are those ‘logical route table’ ids.
When client connects to 10.10.10.10 he knows which addresses are in that ‘logical route table’, in this example it is 18.104.22.168 and 22.214.171.124. Client will connect to those addresses and will use them to send commands to the cluster.
If other client connects to address 126.96.36.199 in above example, it will receive whole route table, but will only use ‘logical route table’ with ID being equal to 2 (that’s what is written in srv2′s config for 188.8.131.52 address). Thus it will connect to 100.100.100.100 and 300.300.300.300.
This allows to use fast but lossy links for data recovery and somewhat slower, but reliable links for common client serving. Or consider video streaming service – clients connect to many serving nodes using ‘external’ addresses, while internal interfaces (usually there are at least 2 of them on server motherboards) are used for compressed/processed/(re)encoded clips to be (re)uploaded from appropriate servers.
Available in 2.22 version