User:NeddySeagoon/Pi4 Router

Motivation
The price of electricity in the UK has gone up by a factor of four in the last six months (August, 2022).

I have been running a HP Gen 7 Microserver, hosting various KVMs. I'm just left with my router KVM now. A HP Gen 7 Microserver, with 4x4TB conventional hard drives is overkill to support a single small KVM

Consolidating the router into the Pi4 Stratum 1 Time Server I can switch off the Microserver, saving about 60W on my base load, or about 1.5kW/h per day, or £22.50/month.

Overview
The Raspberry Pi4 is the first Pi with sufficient IO bandwith for a domestic router. Earlier Pis have the USB and Ethernet ports on a single USB2 root hub as thats all the Broadcom CPUs provide. Keep in mind that they were designed as mobile phone chips.

The Pi4 has
 * SDIO 0 - The mmc card
 * SDIO 1 - WiFi
 * 1Gb Ethernet Phy
 * Single lane PCIe - USB Ports (2 x USB3, 2 x USB2)
 * All the usual GPIO

By using the on board Ethernet and the USB3 ports, we can have three 1G ethernet ports. All operating at 1G full duplex concurrently may not go too well.

Hardware Requirements

 * A Pi 4
 * Two or more USB3 to 1Gb adaptors



Observant readers will notice the GPS hat in the image. This PI is doing both routing and GPS time.

Setting Up
Carry out a basic install. Follow the Pi4 Install guide. Use the  profile as the router will be headless.

How it works IPv4 Only
Traditionally, routers have used an all static setup but domestic internet providers are moving more and more to dynamic set up only, even if the setup never changes.

A router is two loosely coupled interacting systems.

The first system is the internet facing client. It uses, to aquire its internet facing settings. This may require PPP support.

The second system is the internal network facing side. This uses to hand out settings to internal systems,  to keep a cache of domain names lookups and a firewall, here  to control what goes in and out.

Required packages

 * A caching DNS resolver,
 * To generate Netfilter rules without having to do it all the hard way.
 * A DHCP server
 * A DHCP client

Optional extra packages

 * For intrusion detection
 * Point-to-Point Protocol support. Required if your ISP delivers your internet using PPP or PPPoE
 * IPv6 Router Advertisement Daemon. Required for automated IPv6 support
 * To support wireguard VPN. If you ever use public WiFi, you need a VPN of some sort.
 * The Pi 4 can be a 2.4GHz band WiF Access Point

Debugging packages
Hopefully, these will not be needed, as it will just work.


 * Reads and writes Ethernet port hardware options.
 * Ethernet packet capture
 * Trace the path along a network.

How it works Additions for IPv6 Only
aquires some allocations from your delegated prefix for your internal subnets and allocates the first address in each delegated /64 to the router interfaces. then advertises the router IPv6 address and hosts perform IPv6 auto configuration.

This does mean that until you do something about it all your internal systems, behind NAT on IPv4, are on the big bad public internet on IPv6.

Voice over IP - Coming Soon
Full fibre is coming to a street cabinet near me. Openreach say March 2023, so the POTS service is going away, to be replaced by a voice service over the fibre. No more pulse dialling, so our real finger in the hole dialling, 1960s phone won't work any more. On the upside, my ISP is offering up to 1Gb downlink over fibre ... for a price.

Its still early days but my Pi router will need to cope with the telephone service too. Somehow I'll need IP to DECT. IP to pulse dialling and real ringing would be nice to keep the old phone alive but that needs 48v DC on the line and 50v AC for the ringer.

Voice over IP to DECT is on the TODO list ... unless its already been done and someone would care to add it.

How It Works IPv6 Only
Think of a router as two systems, a client and a server, with a bridge between them. The client faces the outside world. The ISP. It uses to aquire the upstream settings automatically for both IPv4 and IPv6. This may also require.

With those things in place, is the gate keeper on the bridge, determining which packets that are allowed to cross the bridge.

Static configuration of LAN members is still possible for both IPv4 and IPv6. Static configuration of the ISP facing interface is ISP dependent.

To aquire our setup from our ISP. No special USE settings are required.

To pass out settings to our clients. No special USE settings are required.

Shorewall is not yet keyworded for arm64. Add  to

TODO: After further testing, fix the ::gentoo repo.

Cut down on DNS lookups. No special USE settings are required.

Required if your ISP demands it or you use PPP for other reasons. No special USE settings are required.

Required for IPv6 only, to advertise that we are a IPv6 router to enable clients to auto configure for IPv6.

Kernel Options
Choose IPv6 support if its required. The  profile already provides USE=ipv6 by defaut, so IPv6 support will be included everywhere that its optional.

Add IPv4 Netfilter support.

Add IPv6 Netfilter support.

Putting the Pieces Together - IPv4
The documentation reserved subnets are used for the illustrations that follow.

2001:DB8::/32 Document Prefix

The blocks 192.0.2.0/24 (TEST-NET-1), 198.51.100.0/24 (TEST-NET-2), and 203.0.113.0/24 (TEST-NET-3) are provided for use in documentation.

/etc/dhcpcd.conf
To be added by others - the authors IPv4 setup is static.

/etc/conf.d/net
The setup for connecting to the ISP uses PPPoE in the example below.

modules="iproute2"


 * 1) TODO Map private address ranges to documentation ranges.

config_green="192.168.100.252/24"

config_dmz="192.168.10.252/24"

config_blue="192.168.54.252/24"

config_red="null"
 * 1) Only if your ISP uses PPPoE ##
 * 2) PPPoE will go here

config_ppp0="ppp" link_ppp0="red"

plugins_ppp0="pppoe"
 * 1) Roaring Penguin pppoe but its in the kernel now

dns_servers_ppp0="xxx.xxx.xxx.xxx                 xxx.xxx.yyy.xxx"


 * 1) There may be other settings you want, see /usr/share/doc/openrc-*/net.example.bz2

config_ppp0="dhcp"
 * 1) for IPv6 only

username_ppp0='Your ISP Username' password_ppp0='Your Secret ISP Pass phrase'

postup { # This function could be used, for example, to register with a      # dynamic DNS service. Another possibility would be to      # send/receive mail once the interface is brought up.

${IFACE} == "ppp0" && echo 2 > /proc/sys/net/ipv6/conf/ppp0/accept_ra
 * 1) Should not be needed as its done in /etc/sysctl.d/router.conf

/etc/dhcp/dhcpd.conf
IPv4 automatic host configuration server. IPv6 operates differently

option domain-name "example.com";
 * 1) Set our domain name, if we have one.

option domain-name-servers 203.0.113.100,203.0.113.200;
 * 1) The upstream Domain Nawe Servers

authoritative;

subnet 203.0.113.0/24 netmask 255.255.255.248 { }
 * 1) No service on theses networks
 * 2) IMPORTANT our ISP must be excluded.

subnet 198.51.100.0/24 netmask 255.255.255.0 { }
 * 1) Our DMZ for our servers. Servers need static IPv4


 * 1) Make addresses in the range 192.0.2.0.120 to 192.0.2.139
 * 2) avaiable via DHCP, with a lease time of 3600 to 14400 sec.
 * 3) Thats a good time for WiFi where things come and go.

subnet 192.0.2.0/24 netmask 255.255.255.0 { range 192.0.2.0.120 192.0.2.139; default-lease-time 3600; max-lease-time 14400; option subnet-mask 255.255.255.0; option broadcast-address 192.0.2.255; option routers 192.0.2.252; option interface-mtu 1492; }


 * 1) Tell clients where the router is.
 * 2) Set the Maximum Transmission Unit size to 1492
 * 3) Autodiscovery does not always work.
 * 4) The default is 1500 but PPPoE needs 8 bytes
 * 5) which makes 1492 a good number.


 * 1) Rinse and repeat for other private subnets

Users that have a TFTP server to support PXE booting, for example, for diskless hosts, configure it here.

Shorewall - Overview and Planning - IPv4
Shorewall is big, you just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to the chemist's, but that's just peanuts to Shorewall. Shorewall is a 'short cut' too.

As IPv4 and IPv6 are completely separate network stacks, Shorewall is twice as big as that. It has two separate directories for its configuration files. and

Setting up a firewall is like installing Gentoo. Design decisions are required before you start writing rules.

There are essentially two sorts of firewalls, half open and paranoid.

With half open, everything is allowed out but only responses and things specifically requested are allowed in. That's your typical domestic router. It means that if the bad guys get in, they can phone home.

With a paranoid firewall, nothing is allowed in and nothing is allowed out unless its expressly permitted.

Its not possible to cover the configuration that suits your install, so only a few key points will be covered. The zone and interface naming convention has been borrowed from Smoothwall as that was the first firewall I used.

It helps to draw a diagram to show what is allowed to connect to where.


 * 1) Table below shows firewall setup.  Symbols are
 * 2)       From - To       may not initiate connections
 * 3)       From ? To       connection initiation determined by rules
 * 4)       From / To       its in the same zone - no restrictions


 * 1)       fw IP           |       From    |                   To                  |
 * 2)                                       |  net  | Green |  Blue |  DMZ  |   fw  |
 * 3) 203.0.113.26/29       |  Net          |   /   |   -   |   -   |   ?   |   ?   |
 * 4) 198.51.100.0/28       |  Green        |   ?   |   /   |   ?   |   ?   |   ?   |
 * 5) 198.51.100.128/28     |  Blue         |   ?   |   -   |   /   |   ?   |   -   |
 * 6) 10.10.0.0/16          |  DMZ          |   ?   |   -   |   -   |   /   |   -   |
 * 7) All of the Above      |  fw           |   ?   |   -   |   -   |   ?   |   /   |
 * 1) 198.51.100.0/28       |  Green        |   ?   |   /   |   ?   |   ?   |   ?   |
 * 2) 198.51.100.128/28     |  Blue         |   ?   |   -   |   /   |   ?   |   -   |
 * 3) 10.10.0.0/16          |  DMZ          |   ?   |   -   |   -   |   /   |   -   |
 * 4) All of the Above      |  fw           |   ?   |   -   |   -   |   ?   |   /   |
 * 1) 10.10.0.0/16          |  DMZ          |   ?   |   -   |   -   |   /   |   -   |
 * 2) All of the Above      |  fw           |   ?   |   -   |   -   |   ?   |   /   |
 * 1) All of the Above      |  fw           |   ?   |   -   |   -   |   ?   |   /   |

/etc/shorewall/* - Implementation
Firstly, most of the files can be left untouched. The following is a get-you-going list for IPv4.

zones
Describe your network zones to Shorewall. The firewall itself is its own zone and is required here.




 * 1) ZONE          TYPE            OPTIONS         IN_OPTIONS      OUT_OPTIONS

fw             firewall green          ipv4 dmz            ipv4 blue           ipv4 net            ipv4


 * green the wired fully protected network. No uninvited inbound connections from anywhere are permitted.
 * blue the protected but untrusted network. e.g. WiFi devices, smart TVs. If You don't know what's in it, it goes here.
 * dmz for servers. Inbound connections may be permitted here but only if a service is provided on the requested port.
 * net sometimes known as red, is the big bad internet from the ISP.

The names are case sensitive and will be used in other files.

interfaces
Tell Shorewall how the zones, defined above, map to interfaces.

net    ppp0 dmz    dmz             logmartians=1,nosmurfs blue   blue            dhcp,logmartians=1,nosmurfs green  green           dhcp,logmartians=1,nosmurfs
 * 1) ZONE  INTERFACE       OPTIONS

policy
The policy is applied when there are no more rules left to test. The default for packets from the net is to DROP them on the floor and log them.

Green is allowed connect to the firewall. This matters if the rules are not correct. Its a headless system that will be administered over ssh. As long an the rules permit it. You can lock yourself out.

Everything else is REJECTed, which gives the sender a nice error message.

The first four LOGLEVEL entries for unwanted packets from the net can be removed once correct operation has been tested. The IPv4 address range has been full for several years so the logs will grow very quickly.


 * 1) SOURCE        DEST            POLICY          LOGLEVEL        RATE

net            dmz             DROP            $LOG net            blue            DROP            $LOG net            green           DROP            $LOG net            $FW             DROP            $LOG

green          fw              ACCEPT          $LOG


 * 1) Reject everything else
 * 2) everything that is not explictly allowed is denied
 * 3) locally, we use REJECT as it helps debug

all            all     REJECT          $LOG

It would be more secure to use REJECT for green to the firewall then to open a port for ssh from a single system in the rules file.

params
Map names to static IP addresses and the like, so that names can be used in rules.


 * 1) IP addresses where we run particular services
 * 2) This avoids using name resolution in rules
 * 3) and at the same time, lets us use names for IP addresses
 * 4) Convention is initial capital letters for parameters

Ntp=10.10.10.10
 * 1) Raspberry Pi Timeserver

snat
Source NAT is better known as masquerading as it's the magic that makes Network Address Translation work.
 * 1) ACTION                SOURCE                  DEST            PROTO   PORT    IPSEC   MARK    USER    SWITCH  ORIGDEST        PROBABILITY

MASQUERADE	198.51.100.0/28  ppp0 MASQUERADE	198.51.100.128/28 ppp0 MASQUERADE	10.10.0.0/16 ppp0

rules
The authors rules file is over 20k, so its not shared here. This is the hard bit. The following snippet is for the green zone only. Rinse and repeat for your other zones.


 * 1) ACTION        SOURCE          DEST                    PROTO   DPORT   SPORT           ORIGDEST        RATE            USER    MARK    CONNLIMIT       TIME         HEADERS
 * 1) ACTION        SOURCE          DEST                    PROTO   DPORT   SPORT           ORIGDEST        RATE            USER    MARK    CONNLIMIT       TIME         HEADERS

ACCEPT         green   fw                      tcp     ssh
 * 1) WARNING - Required for remote control of shorewall maybe make this port 222
 * 1) WARNING - Required for remote control of shorewall maybe make this port 222

ACCEPT         green   net                     tcp     www ACCEPT         green   net                     tcp     https ACCEPT         green   net                     udp     domain ACCEPT         green   net                     udp     ntp ACCEPT         green   net                     tcp     ftp ACCEPT         green   net                     tcp     smtp ACCEPT         green   net                     tcp     submission ACCEPT         green   net                     tcp     514 ACCEPT         green   net                     tcp     svn ACCEPT         green   net                     tcp     hkp ACCEPT         green   net                     tcp     pop3s ACCEPT         green   net                     udp     imaps ACCEPT         green   net                     tcp     imaps ACCEPT         green   net                     udp     https ACCEPT         green   net                     tcp     64738 ACCEPT         green   net                     udp     64738 ACCEPT         green   net                     tcp     8880 ACCEPT         green   net                     tcp     4242 ACCEPT         green   net                     udp     9987 ACCEPT         green   net                     tcp     nicname ACCEPT         green   net                     tcp     ircd ACCEPT         green   net                     tcp     urd ACCEPT         green   net                     tcp     git ACCEPT         green   net                     tcp     rsync ACCEPT         green   net                     tcp     git

For every outgoing rule here, by default, shorewall creates a rule to allow the response back in. The service names are taken from.

There is no reason to allow a whole zone out. green:IPaddr only allows tho one host at IPaddr to use the rule.

If you run your own servers, some Destination NAT rules are required DNAT           net             dmz:$Mail               tcp     smtp DNAT           net             dmz:$Shell              tcp     ssh DNAT           net             dmz:$Shell              tcp     https DNAT           net             dmz:$Web                tcp     http

Notice the use of Mail, Shell and Web for IP addresses in the DMZ.

shorewall.conf
TODO

/etc/sysctl.d/router.conf
Turn on some kernel options that are off by default, even when support is built into the kernel.

net.ipv4.ip_forward = 1
 * 1) In order for this file to work properly, you must first
 * 2) enable 'Sysctl support' in the kernel.
 * 3) Look in /proc/sys/ for all the things you can setup.
 * 4) Enable packet forwarding
 * 1) Enable packet forwarding
 * 1) Enable packet forwarding

net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.all.rp_filter = 1
 * 1) Enables source route verification
 * 1) Enable reverse path

net.ipv4.tcp_syncookies = 1
 * 1) Enable SYN cookies (yum!) Helps defend against syn flooding
 * 2) http://cr.yp.to/syncookies.html

Putting the Pieces Together - IPv6
Shorewall6 is completely separate from Shorewall. A long time ago, they were separate packages but the separation idea remains. One day, IPv4 will be switched off but most of the world is not ready for that yet. Feel free to test by blocking IPv4 totally.

/etc/dhcpcd.conf
duid
 * 1) Use the hardware address of the interface for the Client ID.
 * 2) clientid
 * or
 * 1) Use the same DUID + IAID as set in DHCPv6 for DHCPv4 ClientID as per RFC4361.
 * 2) Some non-RFC compliant DHCP servers do not reply with this set.
 * 3) In this case, comment out duid and enable clientid above.

persistent
 * 1) Persist interface configuration when dhcpcd exits.

vendorclassid
 * 1) vendorclassid is set to blank to avoid sending the default of
 * 2) dhcpcd- :: :

option domain_name_servers, domain_name, domain_search option classless_static_routes
 * 1) A list of options to request from the DHCP server.

option interface_mtu
 * 1) Respect the network MTU. This is applied to DHCP routes.

option host_name
 * 1) Request a hostname from the network

option ntp_servers
 * 1) Most distributions have NTP support.

option rapid_commit
 * 1) Rapid commit support.
 * 2) Safe to enable by default because it requires the equivalent option set
 * 3) on the server to actually work.

require dhcp_server_identifier
 * 1) A ServerID is required by RFC2131.

slaac private
 * 1) Generate SLAAC address using the Hardware Address of the interface
 * 2) slaac hwaddr
 * 3) OR generate Stable Private IPv6 Addresses based from the DUID

option interface_mtu
 * 1) Respect the network MTU. This is applied to DHCP routes.

ipv6only

allowinterfaces ppp0 interface ppp0

interface ppp0 IAID 0 iaid 100


 * 1) Request a DHCPv6 Delegated Prefix for iaid.

ia_pd 3 blue green

The author has a /64 for the IPv6 uplink and a delegated /48 prefix. The  entry causes the ISP to assign two /64 subnets from the /48 delegated prefix to the blue and green interfaces. The other 65534 subnets are not used and dropped by the ISP at their boundary router.

As always, your ISP facing interface my not be ppp0. Modify to suit.

/etc/radvd.conf
interface green
 * 1) green - wired

{       ## (Send advertisement messages to other hosts) AdvSendAdvert on; ## (Fragmentation is bad(tm)) AdvLinkMTU 1280; MaxRtrAdvInterval 300; ## (IPv6 subnet prefix we've been assigned by our ISP)


 * 1)       dhcpcd gets our delegated prefix and allocates it
 * 2)       around our interfaces
 * 3)       it also allocates the interface IPv6 address
 * 4)       we just advertise the /64 on the interface

prefix ::/64

{               AdvOnLink on; AdvAutonomous on; }; };

zones

 * 1) a copy of /etc/shorewall/zones with the type changed to ipv6

fw             firewall green          ipv6 dmz            ipv6 blue           ipv6 net            ipv6
 * 1) ZONE          TYPE            OPTIONS         IN_OPTIONS         OUT_OPTIONS
 * 1) ZONE          TYPE            OPTIONS         IN_OPTIONS         OUT_OPTIONS

interfaces
This similar to its IPv4 namesake


 * 1) ZONE          INTERFACE               OPTIONS
 * 1) ZONE          INTERFACE               OPTIONS

net              ppp0                  tcpflags dmz              dmz                   nosmurfs blue             blue                  dhcp,nosmurfs green            green                 tcpflags

policy
This similar to its IPv4 namesake too.


 * 1) SOURCE DEST   POLICY  	LOGLEVEL  RATE    CONNLIMIT
 * 1) SOURCE DEST   POLICY  	LOGLEVEL  RATE    CONNLIMIT

net    dmz     DROP            $LOG net    blue    DROP            $LOG net    green   DROP            $LOG net    $FW     DROP            $LOG all    all     REJECT          $LOG

Again, once everything works all of the $LOG levels except the last can be removed. There is not nearly as much log spam on IPv6 as the address space is almost empty.

params
If you need parameters to help keep your rules file easy to read, define them here.

snat
NAT is not used with IPv6. NAT on IPv4 was designed to delay the exhaustion of the IPv4 address range.

rules
This is almost but not quite a copy of its IPv4 counterpart.


 * 1) ACTION        SOURCE          DEST            PROTO   DPORT   SPORT   ORIGDEST        RATE    USER    MARK    CONNLIMIT       TIME    HEADERS SWITCH  HELPER
 * 1) ACTION        SOURCE          DEST            PROTO   DPORT   SPORT   ORIGDEST        RATE    USER    MARK    CONNLIMIT       TIME    HEADERS SWITCH  HELPER

ACCEPT     	any         	any       ipv6-icmp
 * 1) All ipv6-icmp to/from anywhere

ACCEPT		fw		net		udp	dhcpv6-server ACCEPT		net		fw		udp	dhcpv6-client ACCEPT         fw              net             tcp     dhcpv6-server ACCEPT         net             fw              tcp     dhcpv6-client
 * 1) We get our subnets via dhcpcd over pppoe

When the firewall asks for its IPv6 setup, it uses dhcpv6-server messages. So these must be allowed out. The server responds with dhcpv6-client messages, which are on a different port, so these must be explicitly permitted.

/etc/sysctl.d/router.conf
In addition to the IPv4 entries IPv6 requires net.ipv6.conf.ppp0.accept_ra = 2
 * 1) A very trendy value for a binary flag!

Starting
Set services to start as follows.

dhcpcd |     default dhcpd |     default net.blue |     default net.dmz |     default net.green |     default net.ppp0 |     default net.red |     default radvd |     default shorewall |     default shorewall-init | boot shorewall6 |     default

shorewall-init makes sure that the firewall is closed when the network interfaces start, to avoid being open to the world between the network interfaces starting and Shorewall starting.

Debugging
There are lots of logs in /var/log. dmesg will be flooded with DROPs due to all the logging.

and  will be useful too.