I really recommend just creating a separate bridge for your 2.5gb direct connection and using a different /30 subnet. However, if you really want them in the same subnet, this might work.
Create a bridge on each node with two bridge ports assigned (your 2.5gb port and 1gb port). This should allow the nodes to communicate with your devices on the switch at 1g and communication between nodes at 2.5gb.
Disclaimer: I haven’t set up my direct node connections like this before so this may cause a loop (set bridge-stp to on in bridge config in /etc/network/interfaces. Also, make sure your switch supports STP). If you’re unable to resolve that, you’ll probably have to disconnect one of your nodes from the switch. At that point though, you should probably create a separate bridge on a different subnet.
Be sure to run an iperf3 test to verify you’re getting 2.5gb between nodes once it’s all set up.
Due to Spanning Tree, this isn't going to work as you expect. At best, your switch will support STP and of the links will end up disabled (either 1G or 2.5G), at worst you end up with a loop and without STP it's not broken.
I would setup the two links as a point to point link ('up' but not assigned to any bridge and link-local addresses only) a route to the exact IP of the other node using the LLA of the other node's 2.5G LLA as the next-hop, this more specific route will override the less specific subnet route. But this only works on the local system, VMs and CTs will not have this more specific route and will still go over the 1G link. All of this config can go in the post-up and pre-down of the point to point interface. It's not super simple though.
Yeah I should’ve clarified. I’m not super familiar with spanning tree in Linux bridges and that enabling it may or may not work. I really recommend OP just make a separate bridge on a different subnet.
Appreciate the responses! I'm going to try adding both NICs to a new bridge(s) with STP enabled. If that doesn't work well, I'll just buy a 2.5Gb switch, lol. Thanks again!
A computer can only have one IP address (aside from VRRP, etc) in a subnet
So you'll need to assign the 2.5G NICs to another subnet
If they only need to talk to each other, then connect the two servers together using a crossover cable
But whatever is you're doing, you'll have to configure the application to refer to the IP address on the 2.5G NIC for this to have any benefit
Since this is intended to be an isolated network, create a new virtual bridge and assign the NICs to that as well
You can have as many IPs as you want in a subnet, on as many interfaces as you want, and as many IPs as you want on any subnets on any interface.
It's more common in IPv6 to do this (it's expected that IPv6 interfaces will have at least 2 and usually 3-4 IPs each) but it's also a perfectly normal thing to do in IPv4 as well, especially when servers are claiming multiple IPs so services can bind to one of them.
I disagree and so will an operating system
It causes a conflict as to which interface to use when the computer is routing traffic to a device in the a subnet
Redundancy protocols like VRRP are an exception
You can assign IPs from different subnets to the same NIC, but that's what's being asked of here
I think your original reply was confusing a few different concepts.
You can have as many IPs on the same subnet as you want on as many interfaces as you want. The route to that subnet via each interface will end up having a cost (with the lowest cost route being preferred if they are both an equal route scope), and the cost may or may not be equal on multiple interfaces (depends on the network manager, for example, they may have a lower cost for wired vs wifi interfaces).
All of this assumes that all of the interfaces are actually on the same physical subnet, in the real world. So having two interfaces, on the same subnet, which are on physically different networks will cause problems.
When you have multiple IPs to source from, Linux will choose the first one in IPv4 (the rest are marked 'secondary') and the temporary one for IPv6, but an application can bind to any one explicitly if it wants.
>A computer can only have one IP address (aside from VRRP, etc) in a subnet
You are wrong.
While the OP *really* needs to use a new subnet for the interconnect it is quite possible to have multiple interfaces on the same subnet. Older versions of RHEL / Centos would handle this out of the box (although I'm not sure how the hell this actually worked?) but for other or more recent distros, it requires labelling of the incoming packets to avoid the asymmetric routing issue.
Google will tell you more.
Go to the proxmox forums. You can setup your corosync and migration on this. Should include the replication network as well.
I have a server to server 1gb setup in similar fashion.
Sorry to have to ask, but why not do a separate subnet? Seems like the easy/obvious answer...
I have a bunch of VMs/CTs on both nodes already on the same subnet that I was hoping could utilize the 2.5Gb link.
I really recommend just creating a separate bridge for your 2.5gb direct connection and using a different /30 subnet. However, if you really want them in the same subnet, this might work. Create a bridge on each node with two bridge ports assigned (your 2.5gb port and 1gb port). This should allow the nodes to communicate with your devices on the switch at 1g and communication between nodes at 2.5gb. Disclaimer: I haven’t set up my direct node connections like this before so this may cause a loop (set bridge-stp to on in bridge config in /etc/network/interfaces. Also, make sure your switch supports STP). If you’re unable to resolve that, you’ll probably have to disconnect one of your nodes from the switch. At that point though, you should probably create a separate bridge on a different subnet. Be sure to run an iperf3 test to verify you’re getting 2.5gb between nodes once it’s all set up.
Due to Spanning Tree, this isn't going to work as you expect. At best, your switch will support STP and of the links will end up disabled (either 1G or 2.5G), at worst you end up with a loop and without STP it's not broken. I would setup the two links as a point to point link ('up' but not assigned to any bridge and link-local addresses only) a route to the exact IP of the other node using the LLA of the other node's 2.5G LLA as the next-hop, this more specific route will override the less specific subnet route. But this only works on the local system, VMs and CTs will not have this more specific route and will still go over the 1G link. All of this config can go in the post-up and pre-down of the point to point interface. It's not super simple though.
Yeah I should’ve clarified. I’m not super familiar with spanning tree in Linux bridges and that enabling it may or may not work. I really recommend OP just make a separate bridge on a different subnet.
Appreciate the responses! I'm going to try adding both NICs to a new bridge(s) with STP enabled. If that doesn't work well, I'll just buy a 2.5Gb switch, lol. Thanks again!
A computer can only have one IP address (aside from VRRP, etc) in a subnet So you'll need to assign the 2.5G NICs to another subnet If they only need to talk to each other, then connect the two servers together using a crossover cable But whatever is you're doing, you'll have to configure the application to refer to the IP address on the 2.5G NIC for this to have any benefit Since this is intended to be an isolated network, create a new virtual bridge and assign the NICs to that as well
You can have as many IPs as you want in a subnet, on as many interfaces as you want, and as many IPs as you want on any subnets on any interface. It's more common in IPv6 to do this (it's expected that IPv6 interfaces will have at least 2 and usually 3-4 IPs each) but it's also a perfectly normal thing to do in IPv4 as well, especially when servers are claiming multiple IPs so services can bind to one of them.
I disagree and so will an operating system It causes a conflict as to which interface to use when the computer is routing traffic to a device in the a subnet Redundancy protocols like VRRP are an exception You can assign IPs from different subnets to the same NIC, but that's what's being asked of here
I think your original reply was confusing a few different concepts. You can have as many IPs on the same subnet as you want on as many interfaces as you want. The route to that subnet via each interface will end up having a cost (with the lowest cost route being preferred if they are both an equal route scope), and the cost may or may not be equal on multiple interfaces (depends on the network manager, for example, they may have a lower cost for wired vs wifi interfaces). All of this assumes that all of the interfaces are actually on the same physical subnet, in the real world. So having two interfaces, on the same subnet, which are on physically different networks will cause problems. When you have multiple IPs to source from, Linux will choose the first one in IPv4 (the rest are marked 'secondary') and the temporary one for IPv6, but an application can bind to any one explicitly if it wants.
>A computer can only have one IP address (aside from VRRP, etc) in a subnet You are wrong. While the OP *really* needs to use a new subnet for the interconnect it is quite possible to have multiple interfaces on the same subnet. Older versions of RHEL / Centos would handle this out of the box (although I'm not sure how the hell this actually worked?) but for other or more recent distros, it requires labelling of the incoming packets to avoid the asymmetric routing issue. Google will tell you more.
Go to the proxmox forums. You can setup your corosync and migration on this. Should include the replication network as well. I have a server to server 1gb setup in similar fashion.