Retinix
News
About Retinix
Articles
Projects
Links

Subnetting, Proxy Arp and TFTP

Background

Elsewhere on this site are notes about setting up diskless X terminals from old PCs. All these experiments were done with the BOOTP/TFTP/NFS server on the same subnet as the diskless client. This all worked fine.

But I now have a subnetted home network and was eager to try to set up a diskless client on a different subnet. I had heaps of trouble with this and I made a lot of important discoveries along the way ... and hence I've put them into this article.

Home network diagram

                   Internet
                       |
     +-----------------+--------------------+
     |               ppp0                   |
     |                                      |
     |                  G                   |
     |                                      |
     |  eth0(10.0.0.5)      eth1(10.0.1.5)  |
     +-------+---------------------+--------+
             |                     |
          +--+                     +-------------+
          |                                      |
+---------+---------+                 +----------+--------+
|  eth0(10.0.0.1    |                 |  eth0(?.?.?.?)    |
|                   |                 |                   |
|         S         |                 |          C        |
|                   |                 |                   |
+-------------------+                 +-------------------+

G is the Gateway machine, S is the Server and C is our diskless Client. G is acting as a router and an IP Masquerading gateway to the internet through the PPP connection. On the G gateway, the two ethernet ports are configured as:

   eth0   IP=10.0.0.5  BROADCAST=10.0.0.255    NETMASK=255.255.255.0
   eth1   IP=10.0.1.5  BROADCAST=10.0.1.255    NETMASK=255.255.255.0
Its just a simple class C subnet design(??). The whole reason I have two ethernet subnets is that I was too cheap to buy a hub. Each subnet is just a crossover cable directly connected to 'S' and 'C'.

The 'S' server has its ethernet card configured as follows:

   eth0   IP=10.0.0.1  BROADCAST=10.0.0.255    NETMASK=255.255.255.0
          DEFAULT GATEWAY=10.0.0.5
Obviously, anything not meant for the 10.0.0.x network is meant to go through the default gateway.

The 'C' client machine actually does have a hard disk and is often booted up in Windows and works just fine with the following ethernet config:

   eth0   IP=10.0.1.1  BROADCAST=10.0.1.255    NETMASK=255.255.255.0
          DEFAULT GATEWAY=10.0.1.5

Under this setup, the masquerading on the 'G' gateway seemed to work fine, with both 'S' and 'C' able to access the internet correctly. The two machines could also connect to each other. I just assumed this was an offshoot of turning on ip_forward'ing on the 'G' gateway.

The Problem

All I wanted to do was to boot up 'C' as a diskless client using 'S' as the TFTP/NFS server. The BOOTP server was also on 'S' and I knew that BOOTP broadcasts would not be passed through the 'G' machine at all ... unless I helped it. At the time I was also mucking around the dhcp, so I started up the DHCP relay on 'G' so that it would forward any DHCP/BOOTP requests onto 'S'. So all I did on 'G' was:
   dhcrelay 10.0.0.1
I could now reboot the 'C' machine from an Etherboot floppy. I could see it sending the BOOTP request and getting a response, But it would not download the kernel by TFTP.

Here is the relevant bootptab entry for our diskless client:

.client:\
   hd=/tftpboot:vm=auto:ip=10.0.1.16:\
   :sa=10.0.0.1:sm=255.255.255.0:ht=ethernet:ha=00a02406636a:\
   :gw=10.0.1.5:bf=xterm1.boot3:rp="/tftpboot/10.0.0.5":
The sa= indicates the TFTP server to get the kernel from. The IP address, default gateway should all be OK. I then tried booting off a Bootable Linux CD on the 'C' machine to try a manual tftp. It also did not work.

I then started doing some packet tracing on the 'G' gateway. Using tcpdump, I could see the tftp UDP packet come in from 10.0.1.16 (the 'C' client) on the eth1 interface and then go out again on the eth0 interface to 'S'. 'S' then responded, but the packet didn't seem to go out the eth1 interface.

The solution

In those packet traces I noticed something odd about the port numbers used in some of the UDP packets. The 'G' gateway was actually changing the source port of the UDP packet on the way through. What was happening was that the 'G' gateway was performing masquerading on connections between the 10.0.0.x and 10.0.1.x subnets. This was not a problem for TCP based protocols like telnet and ftp, but UDP ones don't quite work the same.

My Masquerading rule on the 'G' gateway was effectively this:

   ipchains -A forward -j MASQ -s 10.0.0.0/16 -d any/0
i.e. Anything from 10.0.x.y goingto anywhere gets masqueraded. Thats generally what you want for masquerading, but it also meant that packets going from 10.0.1.x to 10.0.0.x would also get masqueraded. I changed the masquerade rule to:
   ipchains -A forward -j MASQ -s 10.0.0.0/16 -d ! 10.0.0.0/16
This allowed the two internal machines to connect to the internet OK, but now they couldn't talk to each other, basically because the default 'forward' policy was to REJECT. I added in another forward rule:
   ipchains -A forward -s 10.0.0.0/16 -d 10.0.0.0/16 -j ACCEPT
This allowed packets to be forwarded anywhere within the 10.0.x.y networks with the important difference that the packets are NOT masqueraded.

Proxy Arp

My home network design is an example of being cheap and using available components. Having seperate subnets does have some advantages, but it would simplify things if the server and client machines above actually thought they were on the same subnet. In fact they can for most purposes.

Lets revisit my home network design, but this time, we'll assume that the 'C' machine is a running machine with a proper IP address:

                   Internet
                       |
     +-----------------+--------------------+
     |               ppp0                   |
     |                                      |
     |                  G                   |
     |                                      |
     |  eth0(10.0.0.5)      eth1(10.0.1.5)  |
     +-------+---------------------+--------+
             |                     |
          +--+                     +-------------+
          |                                      |
+---------+---------+                 +----------+--------+
|  eth0(10.0.0.1)   |                 |  eth0(10.0.1.1)   |
|                   |                 |                   |
|         S         |                 |          C        |
|                   |                 |                   |
+-------------------+                 +-------------------+
The S and C machines both have a subnet mask of 255.255.255.0 and use the corresponding '10.0.x.5' address on the 'G' box as their default gateway. Sounds sensible.

Proxyarp is a method of tricking IP to do things its not meant to. Some background: When a machine on an IP network first attempts to contact another host that it knows is on the same subnet as itself (ie it knows through the netmask that its on the same subnet), then it sends out an ARP request. This is basically a 'Who is IP address 10.0.0.3?'. The machine that has that IP address then responds including its own MAC address. From then on, packets to '10.0.0.3' are sent with the MAC address of that machine that responded.

ProxyARP means to respond to ARP queries that aren't for the IP address in question. In my setup, this means getting the 'G' system to respond to ARP requests that arent for 10.0.0.5 or 10.0.1.5, but for 10.0.0.1 and 10.0.1.1.

To give you the basics:

  • 'G' maintains the same config info for eth0 and eth1. ie to use a subnet mask of 255.255.255.0, but we turn on proxyarp for both these interfaces (more on this later).
  • Both 'S' and 'C' have their netmasks changed to 255.255.0.0. They can keep the same default router. But you can see that by changing to the netmask to this value, both 'S' and 'C' would think they are on the same subnet.
So what happens when 'S' tries to send a packet to 'C'?:
  • 'S' initially would send an ARP request saying 'Who is IP 10.0.1.1?'.
  • 'G' has proxyarp configured on eth0 so it will say it is the machine that has that IP address.
  • 'S' now sends the packet to the eth0 interface on 'G'. Because of proxyarp, 'G' passes the packet through its eth1 interface out to 'C' and voila the packet gets to its destination. The reverse happens for packets coming back.

Turning on proxyarp

For linux 2.2.x and beyond there is a configurable parameter in /proc.
   eth0   /proc/sys/net/ipv4/conf/eth0/proxy_arp
   eth1   /proc/sys/net/ipv4/conf/eth1/proxy_arp

   or for all interfaces

   all    /proc/sys/net/ipv4/conf/all/proxy_arp
You just need to write a "1" to it to turn it on or a "0" to turn it off:
   echo "1" >/proc/sys/net/ipv4/conf/eth0/proxy_arp
   echo "1" >/proc/sys/net/ipv4/conf/eth1/proxy_arp

Jun 04,2001

1