Paul Monaghan wrote:

Okay, perhaps what I am trying to do can't work for what ever reason but here it is. I've setup ipvs rules as follows:

TCP 209.226.95.146:8080 wlc
       -> 10.0.0.1:8001         Local   1      1          0
       -> 10.0.0.1:8000         Local   1      0          0

Wensong

For the LocalNode feature, the load balancer just lets the packets pass to the upper layer (up to service daemon) to process the request. So, the port number of the local service must be equal to that of the virtual service, otherwise it won't work. Maybe I should add some code to check whether the port number of local service is equal to that of virtual service, if not, reject it to avoid such an error.

i.e. you can't use localnode with LVS-NAT and have it rewrite ports. You can't have a request coming to port VIP:80 and have it serviced by a demon listening on 127.0.0.1:81.

If you want to do this, you could try instead

$ipchains -A input -j REDIRECT 81 -d 192.168.1.12 80 -p tcp

Requests to 192.168.1.12:80 will go to 192.168.1.12:81 (more info is available).

Pablo Ares paresd (at) airtel (dot) net 19 Jul 2004

I have a configuration with only two machines that act both as directors and realservers (Localnode). With a Localnode configuration you can't do port redirection/rewrite independently of the forwarding method (DR, TUN or NAT). I need port redirection because I want to offer a Virtual HTTP Service on port 80, and map this service to two realservers running Tomcat on port 8080 with an unprivileged account. I tried this iptables DNAT rule in the PREROUTING CHAIN.

iptables -t nat -A PREROUTING -p tcp -d VIP --dport 80 -j DNAT --to VIP:8080

This rule functions well for the traffic that is mapped to the local realserver, but the traffic that goes to the other realserver returns with source port 8080 to client (which causes a Reset of TCP connection by client). I probed this configuration with LVS/NAT and LVS/DR with forward_shared (source martians) patch. Is it possible to do port redirection in a Localnode environment?

ratz

If I understand you correctly, the other RS is a physically different machine, right? You need someone to do a port mapping for you on your back-path. First idea:

[Internet] ----> eth0[director/node1]eth1 -----> eth0[node2]

Two DNAT rules:

iptables -t nat -A PREROUTING -i eth0 -p tcp -d $VIP --dport 80 \
                 -j DNAT --to $VIP:8080
iptables -t nat -A POSTROUTING -i eth0 -p tcp -d $CIP -s $RIP \
                  --sport 8080 -j SNAT --to-source $IP_of_eth0:80

The "problem" is that netfilter maintains a template table which is used to lookup the n-tuple corresponding to your initial connection attempt which was port-redirected. Of course the source port of the outgoing packet is then not known which gives you little to know option of back mapping the port. What you could do is have a tcp forwarding tool on a local socket on node2 which redirects traffic to the local socket on port 8080. There are other possibilities, however I'm not sure if I understand your current setup correctly.

The problem is solved. I applied NFCT patch http://www.ssi.bg/~ja/nfct/ and I add a second iptables rule in the POSTROUTING chain.

iptables -t nat -A POSTROUTING -p tcp -s $VIP--sport 8080 -j SNAT --to-source $VIP:80

Mikel Ruiz Echeverria Jun 07, 2005

I would like to balance the service over two real instances running all on the same machine. I have tried ldirectord.cf

virtual=158.227.82.39:8090 real=158.227.82.39:8091 real=158.227.82.39:8092

And I also have tried:

virtual=158.227.82.39:8090 real=127.0.0.1:8091 real=127.0.0.1:8092

but when I start ldirectord, I get:

Starting ldirectord Error [] reading file /etc/ha.d/conf/ldirectord.cf at line 14: invalid address for real server (wrong format).

It seems like IPs of realservers could not be the same as the Director Server's one. Must Director and realservers run on different machines to work properly with LVS?

Horms

Unfortunately, what you are trying to do is not possible, and here is why:

When you set up a real server that is on the same machine as LVS, it uses a special forwarding mechanism called Local. It uses this regardless of weather you asked for Masq, Route or Tun. You can't ask for it, it just knows if the address is local and sets it. You can however observe it using ipvsadm -L.

The reason for this is running packages that are going to be delivered to a local process through Masq, Route or Tun has overhead and in most cases makes very little sense.

However, the downside is that the Local forwarding mechanism (like Route and Tun, but) unlike Masq does not allow port-mapping. That is, your port 8090 packets will stay as port 8090 packets. So in a nutshell IPVS translates your configuration to.

virtual=158.227.82.39:8090
         real=158.227.82.39:8090
         real=158.227.82.39:8090

Which obiously isn't going to work because you have a duplicate entry, and that is what the error message you are getting is trying to say. Well, thats what it should be trying to say, looks like there might be a bit of a bug in ldirectord somewhere, but that doesn't change the fact that IPVS can't do what you want to do.

I believe an easy solution to this problem would be to deliver the packets to different addresses rather than different ports. Something like the following might just work.

virtual=158.227.82.39:8090
         real=127.0.0.1:8090
         real=127.0.0.2:8090

A longer term solution would be to fix up the way the Local delivery mechanism works. But this would likely be quite tricky, and certainly increase its current complexity - its basically a NULL opp at the moment.

With LVS-NAT you can rewrite ports (see Re-mapping ports with LVS-NAT). However, LVS-DR and LVS-Tun just forward the packets to the realserver without rewriting the ports.

Note

You could write code to rewrite the ports before the packet left the director, if you wanted to. However, the replies from the realserver go directly to the client and do not return through the director. For the client to receive a reply from the correct source port, then the realserver would have to rewrite the ports for the reply. There doesn't seem to be enough demand for rewritten ports with LVS-DR or LVS-Tun, that anyone has bothered to write the code.

You can still enter a port for the realserver with ipvsadm -r in the same way that you can for LVS-NAT. However with LVS-DR and LVS-Tun, the port is silently ignored. This leads people to mistakenly think that they can rewrite the ports with LVS-DR. It would be better if ipvsadm disallowed a port with the -r option, or at least gave a warning and exited with a non-zero error code.

Horms will have a fix out for the next releases of ipvsadm.

Horms horms (at) verge (dot) net (dot) au 21 May 2004

ipvsadm has code to change the port if it doesn't match. Below the service on the realserver is entered as 10.0.0.3:200 but is added to the ipvsadm table as 10.0.0.3:100. However no warning is generated.

# ipvsadm -C # ipvsadm -A -t 10.0.0.1:100 # ipvsadm -a -t 10.0.0.1:100 -r 10.0.0.1 # ipvsadm -a -t 10.0.0.1:100 -r 10.0.0.2:100 # ipvsadm -a -t 10.0.0.1:100 -r 10.0.0.3:200 # ipvsadm -L -n IP Virtual Server version 1.0.10 (size=65536) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.0.0.1:100 wlc -> 10.0.0.3:100 Route 1 0 0 -> 10.0.0.2:100 Route 1 0 0 -> 10.0.0.1:100 Route 1 0 0

I don't think it is a good idea to disallow specifying the port. It may break some peoples scripts, and seems uneccessary. Here's a possible patch.

--- ipvsadm-1.21/ipvsadm.c.dist 2004-05-21 11:13:53.000000000 +0900 +++ ipvsadm-1.21/ipvsadm.c 2004-05-21 11:37:53.000000000 +0900 @@ -904,10 +904,18 @@ * if the IP_VS_CONN_F_TUNNEL or IP_VS_CONN_F_DROUTE is set. * Don't worry about this if fwmark is used. */ - if (!urule.vfwmark && + if (!urule.vfwmark && urule.dport != urule.vport && (urule.conn_flags == IP_VS_CONN_F_TUNNEL - || urule.conn_flags == IP_VS_CONN_F_DROUTE)) + || urule.conn_flags == IP_VS_CONN_F_DROUTE)) { + fprintf(stderr, "Warning: " + "Real-Server port must be the same as the " + "virtual-service port for\n" + " direct routing or tunnelling\n" + " Real-Server port has been changed " + "from %u to %u\n", + ntohs(urule.dport), ntohs(urule.vport)); urule.dport = urule.vport; + } /* try to insmod the ip_vs_ftp module if service is for * port 21 if IP_VS_CONN_F_MASQ is used. */

Now when you enter a mismatched port (e.g. the same 10.0.0.3:200 service on the realserver) you get a warning.

# ipvsadm -C # ipvsadm -A -t 10.0.0.1:100 # ipvsadm -a -t 10.0.0.1:100 -r 10.0.0.1 # ipvsadm -a -t 10.0.0.1:100 -r 10.0.0.2:100 # ipvsadm -a -t 10.0.0.1:100 -r 10.0.0.3:200 Warning: Real-Server port must be the same as the virtual-service port for direct routing or tunnelling Real-Server port has been changed from 200 to 100 # ipvsadm -L -n IP Virtual Server version 1.0.10 (size=65536) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.0.0.1:100 wlc -> 10.0.0.3:100 Route 1 0 0 -> 10.0.0.2:100 Route 1 0 0 -> 10.0.0.1:100 Route 1 0 0

You can still rewrite the ports at the realserver with iptables allowing the realserver to listen on another port.

Francois JEANMOUGIN Francois (dot) JEANMOUGIN (at) 123multimedia (dot) com 04 Mar 2004

if you need to rewrite ports for LVS-DR or LVS-Tun, just use

realserver:/# /sbin/iptables -t nat -A PREROUTING -d VIP -p tcp -m tcp --dport 80 -j DNAT --to-destination VIP:9999

Here the client connects to VIP:80 on the director, the realserver is listening on VIP:9999. It works for me for tomcat standalone servers.

Ryan P Linn Oct 01, 2003

I'm currently using a setup where I have individual webservers which are using port based virtual hosts in apache. For instance, I have port 5678 and 5679 which map to ports 80 and 443 on a virtual host. I'm currently using a commercial solution to schedule these hosts and keep them persistant together, however I'm hoping to switch these over to my LVS-DR box.

It appears that the fwmark group is what I would want to do to keep people going to both ports persistant, but from the documentation it didn't appear that you could do port mapping while doing fwmarks. I was wondering if anyone had done this and if they could share how they made it work if they had. This would be for a shopping cart type application where switching between port "80" and "443" were necessary for security, but because the application uses php sessions it has to go back to the same server each time. It appears very easy to do if they were actually listening on port 80 and 443 but since they're not I'm very confused about the correct way to configure this.

Horms horms (at) verge (dot) net (dot) au 02 Oct 2003

The short answer is that you can't, using LVS. But I wonder if it might be possible to change the destination port using nefilter before or after the packets hit LVS. Alternatively it would be possible to modify LVS to do this, the main issue in my mind would be working out a sane way to configure it.