Using IP over IP tunnels for fun and profit[1]

So you've got your node running, and promptly found that there are no nodes you can reach in your area. Well don't despair! This handy guide will show you how to connect to the consume network over the Internet.

N.B. The examples in this guide are written for NetBSD, and FreeBSD with Bruces's if_gre kernel module. I don't have any Linux systems to hand, so I can't give examples for them. If you want to use Linux, then searching on google for "gre tunnel linux" will go a long way towards answering all your questions ;-)

You will need:

• Two node/router/computer's, one at each end of the link (hopefully you already have this).
• These machine should have public ip's (i.e. real, Internet IP addresses, Not 10.something). (But see here if you're behind NAT).
• A 10.x.y.z/30 subnet from the consume ip allocation of one of the ends of the link.
• Communication between both ends of the link (both at the human and Internet level).
• Functioning GRE drivers for your OS.

The first thing to do is to fill in this table:

Then you'll have all the info you need to configure the tunnel. (The Hostname s aren't really needed, but help humans stay sane).

1. Firstly make contact with the person who runs the machine at the other end of the tunnel you want to create. Point them at this document if they don't know what your talking about and make sure that they are happy to do this.
2. Arrange a mutually convenient time to set up the tunnel.
3. Arrange a mutually convenient method of communicating with each other (mobile, IRC email, yelling, etc.) (it might be a good idea to do this at a consume meet, then you can ask people for help etc).
4. decide who's address allocation the /30 is going to come from.
5. Swap external IP's
6. ping and traceroute to each other's ip.
7. Now check that you've got the right version of the GRE drivers for your OS (see below).
8. Now decide who gets which of the two ip's in the /30
9. You should now have all the bits of the above table filled in.
10. Compare your tables to make sure they are the same (but reversed, i.e. you should have the opposite info in the columns for 'my end' and 'the other end').

actually configuring things

FreeBSD

If your doing a tunnel between a NetBSD and FreeBSD machine read the NetBSD section of the driver bits section to find out how to work around the MTU mismatch.

1. Check that the if_gre kernel module is loaded:

   # kldstat
   Id Refs Address    Size     Name
    1    4 0xc0100000 2c1af8   kernel
    3    1 0xc13d4000 6000     ipfw.ko
    4    1 0xc1439000 14000    linux.ko
    5    1 0xc14ab000 4000     if_gre.ko
   # 
   

2. If it's not there load it:

   # kldload /path/to/if_gre.ko
   

   Then check again with kldstat to make sure it got there.

3. Create the interface:

   # ifconfig gre0 create
   

4. Now configure the tunnel layer:

   # ifconfig gre0 tunnel my.real.ipaddress the.other.ends.real.ipaddress
   

5. Now configure the point-to-point link you just created

   # ifconfig gre0 inet my.address.on.the./30 the.other.ends./30.address netmask 255.255.255.252
   

6. Now ifconfig the interface up. (Not really needed, but it doesn't do any harm)

   # ifconfig gre0 up
   

7. Now wait until the human at the other end of the link has finished configuring there end.
8. Now you should be able to ping across the tunnel:

   # ping -n the.other.ends./30.address
   

9. Thats it!
10. Now add the right commands to your rc chain and check that the tunnel still works after a reboot.

NetBSD

On some versions of NetBSD the gre interface is a cloneable interface so you'll need to create it with 'ifconfig greX create'. On older versions the number of gre interfaces available to the system is determined when you compile your kernel. (There are 2 compiled into GENERIC).

To find out which you've got run 'ifconfig -l'. If gre appears in the output then you've got the older, non cloning interface, and you can ignore any mention of 'ifconfig create'. If there are no gre interfaces mentioned in the output then run 'ifconfig -C'. If gre appears in the output then you've got the cloning type, and you need to create the interfaces with 'ifconfig create'. If, however, it doesn't mention gre then you need to compile support for gre into your kernel (and it's possible that your kernel hasn't been patched with the bug fixes to the gre driver, see the NetBSD section of driver bits section and grab the patch).

1. First configure the *inner* point-to-point interface:

   # ifconfig gre0 inet my.address.on.the./30 the.other.ends./30.address netmask 255.255.255.252 up
   

2. Then configure the tunnel:

   # /usr/sbin/greconfig -i gre0 -v -s 212.25.240.34 -d 217.154.12.2
   

3. Now wait until the human at the other end of the link has finished configuring there end.
4. You should now be able to ping across the tunnel:

   # ping -n the.other.ends./30.address
   

5. Thats it!
6. Now add the right commands to your rc chain and check that the tunnel still works after a reboot.

Other Operating systems

I don't know, but if you work it out, email me and I'll update this document.

Update from Stuart Henderson:

OpenBSD needs a "sysctl -w net.inet.gre.allow=1" and has the same problem with MTU as NetBSD.

A real world example

This is what i did when i created a tunnel between mostly.pointless.net and wlan.dek.spc.org.

So:

• The /30 subnet used was 10.1.12.36/30 (from my allocation).
• The interface name was the same on both ends, gre1. gre0 was already in use on mostly, and gre0 on wlan was sometimes used for testing.
• mostly runs NetBSD, wlan runs FreeBSD.

on mostly i ran:

# ifconfig gre1 inet 10.1.12.37 10.1.12.38 netmask 255.255.255.252 up
# /usr/sbin/greconfig -i gre1 -v -s 212.25.240.34 -d 217.154.12.2

and on wlan:

# kldload if_gre.ko
# sysctl -w net.inet.ip.gre_default_mtu=1450
# ifconfig gre1 create
# ifconfig gre1 tunnel 217.154.12.2 212.25.240.34
# ifconfig gre1 mtu 1450
# ifconfig gre1 inet 10.1.12.38 10.1.12.37 netmask 255.255.255.252
# ifconfig gre1 up

And that was that!

FreeBSD

Bruce has written a gre loadable kernel module for FreeBSD which works fine. It'd only for FreeBSD 4.4 and 4.5 tho (more modern versions may work). You can get it here.

NetBSD

For the long time the NetBSD gre driver had a pretty fundamental bug that broke it with OSPF - it would copy the TTL of the inner packet to the outer packet, and since OSPF multicast packets had a TTL of one the outer packets never got very far.

This bug was fixed in mid Feb 2002, so if your using a recent NetBSD snapshot (1.5.3_ALPHA, or latish 1.5ZA) or above you should be Ok.

If your sources are older than that then apply this patch to /usr/src and you should be Ok. (You will need to recompile sysctl as well. Don't forget to do a 'make includes' before you do anything else).

There is one other thing - Bruces if_gre kld thinks that the MTU of the gre tunnel is 1476, and NetBSD's thinks it's 1450. This is probably a problem with NetBSD, and at some point i need to sit down with the RFC's and work out what's going on. In the meantime however if your creating a tunnel between a FreeBSD and NetBSD machine then you'll need to run (on the FreeBSD machine):

# sysctl -w net.inet.ip.gre_default_mtu=1450 

and when you ifconfig the gre interface you need to:

# ifconfig greX mtu 1450

after you've created it.

GRE aka Generic Routing Encapsulation (defined in RFC 1701 and 1702) is a mechanism for creating a virtual point to point link over another IP network. There are several ip over ip tunneling protocols, we use GRE cos it has the lowest overhead, and hence the largest MTU.

Why use 10/30 rather than the ip unnumbered hack?

Rather than using /30 subnets it's possible to use the same ip each end has on some other interface (Ethernet, wireless etc) and a 255.255.255.255 netmask. This works because each end has a unique ip (as far as the other end is concerned). The only problem with this is that zebra's ospfd selects which interface to send the multicast packets out of on the basis of the interfaces address, so if you have multiple interfaces with the same address address all the OSPF hello packets will go out of the first one zebra's ospfd finds.

This breaks OSPF totally, and thats why we have to use /30's.

Some people with ADSL have NAT done for them by the router installed by their ISP. (these are mostly Flowpoint routers). Now because there aren't anything like port numbers to distinguish one gre tunnel from another (only the ip addresses of either end) and because GRE is stateless NAT gateways can't work out what to do with them unless you tell them.

Unfortunately most ISP's will lock you out of the router you've been provided with so you can't make the changes yourself. Fortunately a way round this has been found:

• How to make a cable .
• more on how to make a cable .
• I don't know exactly what to type to enable gre translation, but i think it's:

  # system addserver 10.x.x.x 47 0
  

  Where 10.x.x.x is the ip of the machine behind the router that you want to be the endpoint of the gre tunnel. Dig around in the Efficient docs for more info.

What if one end has a dynamic IP?

This isn't that hard, but is a bit long winded. (and i haven't written it up or done it yet ;-)) (Hint: Use SSH with a key pair to run a script on the end with a fixed ip from the end with the dynamic ip).

How do i choose who to set up a tunnel with?

Ask around on the mailing lists, or see here for a more hopefully efficient scheme.

I tried to ping the local end of a link under FreeBSD and it times out.

Yeah, it does that, i don't know why. If it bugs you you can always add a static route to the local end to 127.0.0.1:

# route add -host local.ip 127.0.0.1

You'll need to do it in zebra.conf too:

!
ip route local.ip 127.0.0.1
!

[1] IP over IP tunnels may be neither fun nor profitable.