This came from a posting by Jacob Reif Jacob (dot) Rief (at) Tiscover (dot) com 25 Apr 2003.

It is common to run multiple websites (Jacob has 100s) on the same IP, using name based http to differentiate the websites. Sometimes webdesigners use some kind of include-function to include content from one website into another, by means of server-side-includes. (see http://www.php.net/manual/en/function.require.php) using http-subrequests. The include requires a client process running on the webserver, to make a request to a different website on the same IP. If the website is running on an LVS, then the realservers need to be able to make a request to the VIP. For LVS-DR and LVS-Tun this is no problem: the realserver has the VIP (and the services presented on that IP), so requests by http clients running on the realserver to the VIP, will be answered locally.

For LVS-NAT, the services are all running on the RIP (remember, there is no IP with the VIP on realservers for LVS-NAT). Here's what happens when the client on the realserver requests a page at VIP:80

realserver_1 makes a request to VIP:80, which goes to the director. The director demasquerades (rewrites) dst_addr from VIP to RIP_2. realserver_2 then services the request and fires off a reply packet with src_addr=RIP_2, dst_addr=RIP_1. This goes to realserver_1 directly (rather than being masqueraded through the director), but realserver_1 refuses the packet because it expected a reply from VIP and not from RIP_2.

           +-------------+
           |     VIP     |
           |  director   |
           +-------------+
            ^           |
            |           |req
            |req        v
  +-------------+     +-------------+
  |   RIP_1     |<--- |   RIP_2     |
  |  Realserver | ans |  Realserver |
  |  = client   | wer |  = server   |
  +-------------+     +-------------+

Here are the current attempts at solutions to the problem, or you can go straight to Jacob's solution

Jacob's solution: The solution proposed here does not put that extra load onto the director. However each realserver always contacts itself (which isn't a problem). Put the following entry into each realserver. Now the realservers can access the httpd on RIP as if it were on VIP.

realserver#  iptables -t nat -A OUTPUT -p tcp -d $VIP --dport 80 -j DNAT --to ${RIP}:80

Carlos Lozano clozano (at) andago (dot) com 02 Jul 2004

We have a machine that must be both a client and director. The two problems to solve are

I have written a ip_vs_core.c.diff (http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/files/ip_vs_core.c.diff) patch for 2.4.26 using IPVS-NAT. It works correctly in my testcase. The schema is:

External client ---> IPVS:443 --> Local:443 ---> IPVS:80 ---> RealServer

The problem happens when Local:443 goes to localIPVS:80, because the packet is discarded by the next lines in ip_vs_core.c:

if (skb->pkt_type != PACKET_HOST) || skb->dev == &loopback_dev) { 
        IP_VS_DBG(12, "packet type=%d proto=%d daddr=%d.%d.%d.%d ignored\n",
                       skb->pkt_type,
                       iph->protocol,
                       NIPQUAD(iph->daddr));
        return NF_ACCEPT;
}  

Ratz

Why do you need this? Seems like a replication of mod_proxy/mod_rewrite. Your patch obviously makes it work but I wonder if such a functionality is really needed.

We are using it like an ssl accelerator. The first ipvs (443) sends the request to localhost:443 or to a different director, and the second ipvs(80), distributes the traffic in the realservers.

Ext. client --> IPVS:443 --> Local:443 --> IPVS:80 --> RealServer1
                         |-> Director2:443         |-> RealServer2

In the first case, it is a scheme "external machine client+director", but in the second case it is a "client+director in the same machine". This part of the patch only solves the output packet, the return is handled by the second part of the patch. (what is really a bad hack)

For a mini-HOWTO on using this patch see https_on_localnode. Matt Venn has tested it, it works using the local IP of the director, but not 127.0.0.1.

	Note
	Graeme came up with the original idea, Rob Wilson proposed a solution that didn't quite work, Graeme fixed it and then Judd saw an easier solution for the case of only one VIP. I've somewhat mashed the history in my write-up (sorry).

Graeme Fowler is looking for a solution for realservers that can't use iptables

Graeme Fowler graeme (at) graemef (dot) net 11 Feb 2005

After a long day spent tracing packets through the LVS and netfilter trail whilst trying to do cleverness with policy routing using the iproute2 package, I can condense quite a lot of reading (and trial, mainly followed by error!) down as follows:

Conclusions: mixing policy routing and LVS sounds like a great idea, and probably is if you're using LVS-DR or LVS-TUN. Just with LVS-NAT, it's a no-go (for me at the moment, anyway).

Graeme Fowler graeme (at) graemef (dot) net 2005/03/11

Solved... was Re: LVS-NAT: realserver as client (new thread, same subject!)

I've solved it - in as far as a proof of concept goes in testing. It's yet to be used under load though; however I can't see any specific problems ahead once I move it into production.

The solution of type "4" above involves a "classic" LVS-NAT cluster as follows. Nomenclature after DIP/RIP/VIP classification is "e" for external (ie. public address space), "i" for internal (ie. RFC1918 address space) and numbers to delimit machines.

Director: External NIC eth0 - DIPe, VIP1e
          Internal NIC eth1 - DIPi

Realserver 1: Internal NIC eth1 - RIP1

Realserver 2: Internal NIC eth1 - RIP2

In normal (or "classic" as referred to above) LVS-NAT, the director has a virtual server configured on VIP1e to NAT requests into RIP1 and RIP2. Under these circumstances, as discussed in great length in several threads in Jan/Feb (and many times before), a request from a realserver to a VIP will not work, because:

src         dst
RIP1 SYN -> VIP1e
RIP1 SYN -> RIP2  (or RIP1, doesn't matter)
RIP2 ACK -> RIP1

at this point the connection never completes because the ACK comes from an unexpected source (RIP2 rather than VIP1e), so RIP1 drops the packet and continues sending SYN packets until the application times out. We need a way to "catch" this part of the connection and make sure that the packets don't get dropped. As it turns out, the hypothesis I put forward a month ago works well (rather to my surprise!), and involves both netfilter (iptables) to mangle the "client" packets with an fwmark, and the use of LVS-DR to process them.

What I now have (simplified somewhat, this assumes a single service is being load balanced in a very small cluster):

Director: External NIC eth0 - DIPe, VIP1e
          Internal NIC eth1 - DIPi

Realserver 1: Internal NIC eth1 - RIP1
              Loopback adapter lo:0 - VIP1e

Realserver 2: Internal NIC eth1 - RIP2
              Loopback adapter lo:0 - VIP1e

The on the director:

/sbin/iptables -t mangle -I PREROUTING -p tcp -i eth1 \
   -s $RIP_NETWORK_PREFIX -d $VIP1e --dport $PORT \
   -j MARK --set-mark $MARKVALUE

and we need a corresponding entry in the LVS tables for this. I'm using keepalived to manage it; yours may be different, but in a nutshell you need a virtual server on $MARKVALUE rather than an IP, using LVS-DR, pointing back to RIP1 and RIP2. Instead of me spamming configs, here's the ipvsadm -Ln output:

director# ipvsadm -Ln
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port   Forward Weight ActiveConn InActConn

FWM  92 wlc
  -> $RIP1:$PORT           Route  100    0          0
  -> $RIP2:$PORT           Route  100    0          0

(empty connection table right now)

...and believe it or not, that's it. Obviously the more VIPs you have, the more complex it gets but it's all about repeating the appropriate config with different RIP/VIP/mark values.

For ease of use I make the hexadecimal mark value match the last octet of the IP address on the VIP; it makes for easier reading when tracking stats and so on.

I've not addressed any problems with random ARP problems yet because they haven't yet occurred in testing; and one major bonus point is that if a connection is attempted from (ooh, let's say, without giving too much away) a server-side include on a virtual host on a realserver to another virtualhost on the same VIP, then it'll get handled locally as long as Apache (in my case) is configured appropriately.

An interesting, and useful, side-effect of this scheme is that when a realserver wants to connect to a VIP which it is handling, it'll connect to itself - which reduces greatly the amount of traffic traversing the RS -> Director -> RS network and means that the amount of actual load-balancing is reduced too.

Rob Wilson rewilson () gmail ! com 2005-08-09

We have an LVS server for testing which is handling 2 VIPs through LVS-NAT (using keepalived). Each of the VIPs currently points to 1 real server - it's a one realserver LVS - just in testing phase at the moment. Both real-servers are on the same internal network.

VIP1 -> Realserver1 
VIP2 -> Realserver2 

We'd now like Realserver2 to be able to connect to Realserver1 via VIP1. I was able to accomplish this following the solution provided by Graeme Fowler: http://www.in-addr.de/pipermail/lvs-users/2005-March/013517.html However, external connections to VIP1 no longer work while that solution is in place. Dropping the lo:0 interface assigned to VIP1 on Realserver1 fixes this, but then breaks Realserver2 from connecting.

Graeme Fowler graeme () graemef ! net 2005-08-10

Are you doing your testing from clients on the same LAN as the VIP, by any chance? Have you set the netmask on the lo:0 VIP address on the realservers to 255.255.255.255? I can see that making it a /24 mask - 255.255.255.0 - might result in the realservers thinking that the client is actually local to them, thus dropping the packets.

Rob Wilson rewilson () gmail ! com 2005-08-10

That's exactly it. I was hoping it was something daft I misconfigured, so.. wish granted :) It works perfectly now. Thanks for your help (and coming up with the idea in the first place!).

Judd Bourgeois simishag (at) gmail (dot) com 19 Jan 2006

I am running LVS-NAT, where the director has two NICs (and two networks). The VIP is on the inside of the director (in the RIP network) (Joe - this functions as a two network LVS-NAT). Some of my web sites proxy to "themselves" within a page (proxy, PRPC, includes, etc.). The symptom is that the proxy functionality breaks. The real server does a DNS lookup for the remote site, gets back the VIP, and hangs waiting for a response.

Previously I solved this problem by putting the site names and 127.0.0.1 in /etc/hosts (as mentioned in this section and in indexing), but after reading the FAQ more carefully tonight, I solved it by simply adding the VIP as a dummy interface on all of the realservers. This appears to be addressed in Graeme's solution, but he runs an extra iptables command on the director. Is this really necessary? Won't any packets originating on the real servers and destined for the VIP be handled by the dummy interface on the real server, without being put on the wire?

It all appears to work fine and has the added nice effect of forcing each realserver to proxy to itself when necessary.

Graeme Fowler graeme (at) graemef (dot) net 1/20/06

What you've suggested is the "single VIP" case of the above idea. It worked for me, it seems to have worked for Rob Wilson, so casting aside the fact that you might have multiple VIPs frontending multiple realserver clusters (as is my case) I can't see any reason why you shouldn't just go for it.

Judd Bourgeois simishag (at) gmail (dot) com 20 Jan 2006

Right. In fact, after reading your solution again, I think your solution is the more useful general case, where there may be an arbitrary number of VIPs, RIPs, and groupings of real servers (which I don't need right now, but I've realized I will need it down the road). I have some Alteons that call these real server groups, not sure what the LVS equivalent is, but here's a short illustration.

Assume 1 director, 3 VIPs, 4 RIPs on 4 real servers. Assume we have real server groups (RG) RG1 (RIP1-2), RG2 (RIP3-4), RG3 (RIP1-4). VIP1 goes to RG1, VIP2 goes to RG2, VIP3 goes to RG3.

In my solution, servers in RG1 can simply put VIP1 and VIP3 on dummy interfaces, but for proxy requests they will only be able to talk to themselves. They will not be able to talk to VIP2. All servers should be able to talk to VIP3. Your solution solves this by using fwmark.

This is a fairly common problem with NAT in general that I have to deal with a lot. Basically, the NAT box will not apply NAT rules for traffic originating and terminating on the NAT box. I recall that one workaround for this is to use the OUTPUT chain, I can't find the rules at present but it seemed to work ok.

Ratz 21 Jan 2006

There is no LVS equivalent of "real server groups". But I think Alteon (Nortel) only has this feature for adminstrative reasons, so you can assign a group by its identifier to a VIP. What I would love to see with LVS is the VSR approach and a proper and working imlementation of VRRP or CARP. I've just recently set up a 2208 switch using one VSRs and 2 VIRs, doing failover when either the link or the DGW is not reachable anymore. The sexy thing about this setup is that you don't need to fiddle around with arp problems and you don't need to have NAT, so balancing schedulers can get meaningful L7 information. Alteon's groups are just an administrative layer with an identifier. We could add such a layer in ipvsadm and the IPVS code, however what benefit do you see in such an approach?

One problem I see with the Alteon approach is that if you add a RS to a group, to my avail it can only pertain to one RG. This is a bit suboptimal if you want to use RS as spillover servers on top of their normal functionality. Regarding your example, I'd like to say, that RG1 is a spillover group for RG3. You can specify (IIRC) a spare server of each RG in AltheOS, however not cross-RG wise. Correct me if I'm wrong, please.

Judd

Graeme's solution solves this by using fwmark.

Yes, fwmark solves almost all problems

Graeme Fowler graeme (at) graemef (dot) net 21 Jan 2006

Judd doesn't need fwmark, because in a single VIP LVS-NAT, with that VIP assigned locally on the realservers on a dummy interface (or loopback alias), the realservers will always answer requests for the VIP locally.

In a two-VIP case (the simplest multiple), if you have two "groups" [0] of realservers, then the director becomes involved by virtue of it being the default gateway for the realservers. At the point the director gets involved you need some way of determining which interface your traffic is on, and segregation via fwmark seems the most elegant way to achieve this (given the known and predictable failure of realservers as clients in LVS-NAT). I know I struggled for months before realising that I could, in effect, combine the use of NAT via an external interface for my real clients, and DR via an internal interface for my "realservers as clients".

[0] I use the word groups in quotes and advisedly, since it appears that Alteon use that in their setup terminology from previous posts.