After hours of fuddling with IPSec and windows giving vague useless errors, I decided to give OpenVPN a shot.
I used the pfsense built-in wizard and the addon package openvpn-client-export along with one of the numerous how-to guides.
I setup for a route to be pushed on the server:
I turned on float for the clients:
Finally the VPN connection that connected (the tray icon turned green and the log showed no errors) but connecting to devices within the LAN didn't work (although connecting to the router worked fine). I thought "At least the hard part, authentication, was out of the way".
Fortunately, my neighbor let me use her Wi-Fi to troubleshoot my connection. I already gave up getting another IP from my ISP since they required I convert to a commercial account and using my mobile phone as a Wi-Fi hotspot was out of the question since family is grandfathered into a cheaper unlimited data plan which I could not break out of unless I signed up for my own line at $80 a month. Of course it was fun explaining to the ISP tech support rep how my switch did not have a WAN port, what the difference between a hub and a switch is, and explaining to the mobile phone carrier that 3mbps means megabit not megabyte (and that megabyte is no longer 1024 based, hello mebibit)
You can use Wireshark to verify which machines are receiving which packets.
To start, I started pinging my main machine from my laptop:
Start Wireshark, select the appropriate interface, start capturing, and add a filter 'icmp' to show only the ping packets.
Here we can see that my Mac was not returning replies. I found this curious, if it received the request, where was the reply? You can verify what the router is receiving by capturing its packets.
On pfsense you can use the built-in packet capture tool based on tcpdump. Make sure to select the appropriate interface and filter to ICMP packets.
In my case, pfsense confirmed no replies were being sent. As it turns out, if no route exists to a destination, no packet is sent. This can be further diagnosed using a traceroute:
traceroute -d 10.0.8.2
To view a list of all static routes on a system
We can solve this problem by adding a static route:
sudo route -n add 10.0.8.0/24 10.0.0.1
But this is unsustainable as we would need to add this route to every machine in the LAN. Another option is a DHCP classless static lease.
Static routes over DHCP
You can enter a classless static route into the DHCP options in pfsense under the DHCP server menu using an option number of 121.
The classless static routes have a really screwy format entered as:
In my case of 10.0.8.0 with a subnet mask of 255.255.255.0 and a gateway of 10.0.0.1 I get (in decimal) 24:10:0:8:10:0:0:1 (notice the missing trailing 0 from 10.0.8.0) which in hex is 18:0A:00:08:0A:00:00:01
Be careful setting up a classless static route because Windows will silently refuse to take a DHCP lease if it's not perfectly formatted.
You can debug this situation by trying to manually renew and you will get this output showing invalid data:
To view static routes on MacOS coming from DHCP discovery response:
ipconfig getpacket `en1`
In my case, my Mac has a static lease which doesn't get the special bits from pfsense so I'm sticking with the manually added static route.
Even after all that, my Mac could still not ping my Windows machine connected through the VPN. It turns out Windows Firewall was on! I didn't want to turn off the firewall for public networks where I'm connecting to the VPN from so I needed to get Windows to treat the VPN as a 'private' network. This is not an easy task with the Home version of Windows 10 that came on my laptop. I finally found this exhaustive guide to changing the network location which had the key PowerShell command (make sure to run as Administrator):
(For demonstration purposes only; in this example it shows my home Wi-Fi which is already private)
In the end, I was finally able to connect both ways through the VPN.