How to configure a globally routable FreeBSD 12 Jail with IPv6, VNET and ZFS

It's holiday season in Sri Lanka. I really got a free time to play with one of my favorite OS FreeBSD. For a long time I had idea to setup a FreeBSD Jail. I went through some online documentations. there was a good article on nixCraft for FreeBSD 11 (it didn't worked for me on FreeBSD 12), so I used that to get idea about initial setup especially ZFS configuration, also I looked at FreeBSD official example directory for jails at /usr/share/examples/jails and my own configuration IPv6 setup to make it globally routable

The steps to create a jail as follows

1. Check the feasibility of setting up a jail with globally routable
2. Install required tools
3. Create a ZFS data set for jails
4. Create the base jail files
5. Configure the base jail and create a new jail files
6. Configure the jail.conf on the host
7. Turn on jail service and start a jail
8. Login to a jail and play with it

Step 1: Check the feasibility of setting up a jail with globally routable

Setting up a jail with VNET require a VIMAGE enabled kernel. FreeBSD 12 has VIMAGE enabled in GENERIC kernel on amd64, alternatively we can build custom kernel with VIMAGE support, more information can be found on /usr/share/examples/jails/README 

check the FreeBSD version by running uname, we are good to continue if the version is 12.0 or later

root@fbsd12j:~ # uname -a
FreeBSD fbsd12j.amdexa.com 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC  amd64

The next important thing is having IPv6 configuration with enough unassigned IPs to assign to our jails. My ISP is not providing native IPv6 static block, so I am using IPv6 tunneling over IPv4 from Hurricane Electric free tunnel broker
My IPv6 configuration as follows

root@twister:/home/melan# ifconfig 
.................
sit0      Link encap:IPv6-in-IPv4  
          inet6 addr: ::127.0.0.1/96 Scope:Unknown
          inet6 addr: ::172.17.0.1/96 Scope:Compat
          inet6 addr: ::172.20.0.2/96 Scope:Compat
          UP RUNNING NOARP  MTU:1480  Metric:1
.................

sit1      Link encap:IPv6-in-IPv4  
          inet6 addr: fe80::ac11:1/64 Scope:Link
          inet6 addr: fe80::ac14:2/64 Scope:Link
          inet6 addr: 2001:XXXX:YYYY:ZZZZ::2/64 Scope:Global
          UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
.................

wlan0     Link encap:Ethernet  HWaddr 00:AA:BB:CC:EE:FF  
          inet addr:172.20.0.2  Bcast:172.20.0.255  Mask:255.255.255.0
          inet6 addr: fe80::00AA:BBCC:feCC:EEFF/64 Scope:Link
          inet6 addr: 2001:YYYY:XXXX::1/64 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
.................  

Apart of IPv6 configuration, my hardware/software configuration as follows

• a laptop running on Ubuntu 16.04.4/ Kernel 4.15.0-29-generic is connector to 4G router via wlan0, so my laptop act as a router for IPv6
• VirtualBox ver 5.2 installed on the system 
• FreeBSD 12 is running on VirtualBox with Bridged adapter to wlan0

Step 2: Install required tools

required tools are available on /usr/src/share/examples/jails directory, so we can add them to default PATH by copying it to /usr/sbin/ directory

root@fbsd12j:~ # cp -v /usr/src/share/examples/jails/{jib,jng} /usr/sbin/
/usr/src/share/examples/jails/jib -> /usr/sbin/jib
/usr/src/share/examples/jails/jng -> /usr/sbin/jng

Step 3: Create a ZFS data set for jails

We can create ZFS datasets for jails and templates.Here I’ve created a FreeBSD 12.0 template (base image) in /jails/fullbasejail

root@fbsd12j:~ # zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot  13.5G  2.52G  11.0G        -         -     1%    18%  1.00x  ONLINE  -
root@fbsd12j:~ # zfs create -o mountpoint=/jails zroot/jails
root@fbsd12j:~ # zfs create zroot/jails/fullbasejail

Step 4: Create the base jail files

We can download required based files from freebsd website and extract to the jail base template location as follows, also we can get those files by mounting a FreeBSD 12 ISO image without downloading it

root@fbsd12j:/tmp # wget https://download.freebsd.org/ftp/releases/amd64/12.0-RELEASE/base.txz --no-check-certificate
root@fbsd12j:/tmp # wget https://download.freebsd.org/ftp/releases/amd64/12.0-RELEASE/lib32.txz --no-check-certificate

root@fbsd12j:/tmp # tar -zxvf /tmp/base.txz -C /jails/fullbasejail
root@fbsd12j:/tmp # tar -zxvf /tmp/lib32.txz -C /jails/fullbasejail

Step 5: Configure the base jail and create a new jail files

First we can run freebsd-update by pointing our jail dir template directory, so that we can ensure we have latest version of the FreeBSD base files.

root@fbsd12j:/tmp # freebsd-update -b /jails/fullbasejail fetch install
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching public key from update1.freebsd.org... done.
Fetching metadata signature for 12.0-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Fetching 2 metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 51 patches.....10....20....30....40....50 done.
Applying patches... done.
Fetching 16 files... done.
...........................
Installing updates... done.

We can verify the update by running following command

root@fbsd12j:/tmp #  freebsd-update -b /jails/fullbasejail IDS
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching metadata signature for 12.0-RELEASE from update4.freebsd.org... done.
Fetching metadata index... done.
Fetching 1 metadata files... done.
Inspecting system... done.

We can take a snapshot of the base template using 'zfs snapshot' command. that snapshot can be cloned or sent/received to create a new datasets directory for a new jail.

root@fbsd12j:/tmp # zfs snapshot zroot/jails/fullbasejail@12.0-RELEASE

We can create a new jail by sending/receiving the fullbasejail@12.0-RELEASE snapshot.

root@fbsd12j:/tmp # zfs send -R zroot/jails/fullbasejail@12.0-RELEASE | zfs receive zroot/jails/jail1

We can fine-tune our new jail's configurations by cd to /jails/jail1 and changing the configuration files

Step 6: Configure the jail.conf on the host

We can use a simple configuration as below to tryout our first jail, this file is totally based on the sample file /usr/share/examples/jails/jail.xxx.conf

jail1 {
        host.hostname = "freebsdjail1.amdexa.com";   # hostname
        path = "/jails/jail1";     # root directory
        exec.clean;
        exec.system_user = "root";
        exec.jail_user = "root";

        vnet;
        vnet.interface = "ng0_jail1";               # vnet interface(s)
        exec.prestart += "jng bridge jail1 igb1";   # bridge interface(s)
        exec.poststop += "jng shutdown jail1";      # destroy interface(s)
 
        # Standard stuff
        exec.start += "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown";
        exec.consolelog = "/var/log/jail_jail1_console.log";
        mount.devfs;          #mount devfs
        allow.raw_sockets;    #allow ping-pong
        devfs_ruleset="5";    #devfs ruleset for this jail
        mount.devfs;
}

Step 7: Turn on jail service and start a jail

We can simply start our newly created jail by running

service jail start jail1

Step 8: Login to the jail and play with it

We can list all jails by using 'jls' and 'jexec <id>' to login to jail

root@fbsd12j:/usr/share/examples/jails # jls
   JID  IP Address      Hostname                      Path
     1                  freebsdjail1        /jails/jail1

root@fbsd12j:/usr/share/examples/jails # jexec 1
root@freebsdjail1:/ #

After login to the jail, we can add 'sshd_enable="YES"' rc.conf to enable ssh login. also we can add globally routable IPv6 congratulation by executing following commands (optionally we can add them to rc.conf persist it permanently ). please note that  2001:XXXX:YYYY:0::1 is the IP of wlan0  interface of my laptop act as a IPv6 router( wlan0 is Bridged with my VirtualBox 'Bridged adapter')

root@freebsdjail1:/ # ifconfig e0b_jail1 inet6 2001:XXXX:YYYY:0::2
root@freebsdjail1:/ # route add -inet6 default 2001:XXXX:YYYY:0::1

We can use google IPv6 nameserver to resolve DNS queries

root@freebsdjail1:/ # cat /etc/resolv.conf 
nameserver 2001:4860:4860::8888

ifconfig output of the jail as follows

lo0: flags=8049<up> metric 0 mtu 16384
 options=680003<rxcsum>
 inet6 ::1 prefixlen 128 
 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
 inet 127.0.0.1 netmask 0xff000000 
 groups: lo 
 nd6 options=21<performnud>
e0b_jail1: flags=8843<up> metric 0 mtu 1500
 options=8<vlan_mtu>
........ 
 inet6 2001:XXXX:YYY:0::2 prefixlen 64 
........
 groups: epair 
 media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
 status: active
 nd6 options=21<performnud>

ping ipv6.google.com to test the connectivity

root@freebsdjail1:/ # ping6 -c4 ipv6.google.com
PING6(56=40+8+8 bytes) 2001:XXXX:YYYY:0::2 --> 2607:f8b0:4000:812::200e
16 bytes from 2607:f8b0:4000:812::200e, icmp_seq=0 hlim=57 time=381.131 ms
16 bytes from 2607:f8b0:4000:812::200e, icmp_seq=1 hlim=57 time=287.536 ms
16 bytes from 2607:f8b0:4000:812::200e, icmp_seq=2 hlim=57 time=352.459 ms
16 bytes from 2607:f8b0:4000:812::200e, icmp_seq=3 hlim=57 time=347.322 ms

--- ipv6.l.google.com ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 287.536/342.112/381.131/34.041 ms

Install python3 and run IPv6 enabled simple HTTP server from the source webservev6.py

root@freebsdjail1:/ # /usr/local/bin/python3.7 simple.py 
Serving HTTP on :: port 8000 ...

Run nmap on a remote host against the jail

root@am-web-us-pa:~# nmap --open -6 2001:XXXX:YYYY:0::2 

Starting Nmap 7.01 ( https://nmap.org ) at 2019-04-17 10:54 EDT
Nmap scan report for 2001:XXXX:YYYY:0::2
Host is up (0.39s latency).
Not shown: 991 closed ports, 7 filtered ports
PORT     STATE SERVICE
22/tcp   open  ssh
8000/tcp open  http-alt

Nmap done: 1 IP address (1 host up) scanned in 33.36 seconds

invokes the web server running on the jail on port 8000

root@am-web-us-pa:~# curl -6 --head http://[2001:XXXX:YYYY:0::2]:8000/ 
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.7.3
Date: Wed, 17 Apr 2019 20:34:43 GMT
Content-type: text/html; charset=utf-8
Content-Length: 1047