Blog

I finally figured out a decent fix for my networking situation. For those unfamiliar with the situation, the Arris BGW210-700 modem/router combo units from AT&T do not support NAT Loopbacking (even though the previous modems did). So I had to do a little network trickery in order to get it to work.

First step was to enable a DNS server on my server. This wasn’t hard as I already have AdGuard up and running for the VMs on my server’s network. It was just a matter of telling the modem to use my server’s DNS resolver instead of the modems. Once I did that, I made AdGuard rewrite all requests to *.heestand.tech to the private IP address of the server, that way packets wouldn’t be routed to the modem.

Afterwards I had to enable NAT loopback in OPNSense since my internal VMs were getting routed to the firewall instead of the server. Once that was over, everything worked fine!

I still wish the Arris BGW210-700 modems supported NAT loopback, this would have saved a ton of time and a large headache on my part. But everything is working as expected now so I can not complain!

A few months ago the AT&T modem my family had for about two years decided to die. It was one of the typical rental modems from AT&T, nothing too fancy (I think it was the Arris BGW210-700 to be exact). It had abysmal WiFi range so we ended up getting a Linksys mesh system and turning of the radio in the modem. Keep in mind, that modem was probably our fifth AT&T modem since they always seemed to die from hardware failure or something else.

Somewhere down the line a few years ago, I made the decision to isolate my server’s traffic from the Linksys system. I wanted to more cyber security stuff like live malware analysis and network traffic analysis from a SSH honeypot I had set up. Everything was working as expected, I could access my server either from the Linksys network or from the wide open internet, it was great. I even set up printing from my server to the printer that was on the Linksys network so I could print papers/other stuff from my Windows Virtual Machine. For a visual representation, the topology of the network looks like this:

Somewhere down the line, Window’s RDP client started to let users “forward” printers, local disks and USB devices from the client machine so you could access them on the server, so my original method of going through the forwarded ports from the mesh system was pretty much obsolete.

With the old method being obsolete, I never considered cleaning up the forwarded ports from the Linksys Mesh system since it never had any impact on anything until back in July.

As usual, the AT&T modem failed so we got a new one. I hooked up and configured all the devices to get the same IP address, just like the old modem. Everything was working fine except the connection to my server. If I navigated to “heestand.tech” from my the Linksys network, it would sometimes time out for no apparent reason, other times it would work for a few minutes.

At first, I thought I messed something up with the port forwarding on the modem end since I checked the access logs on the server and I didn’t see any attempt of my device to connect with my server, but everything seemed to be correct. To make this even more weird, I could access everything just fine outside of my home network. I would go to work and be able to login just fine. So I figured something was up with the modem.

I looked online to see if people had similar issues and I found a few sporadic cases of people having connection issues due to AT&T’s smart firewall feature on the new modem. I disabled the firewall and everything was still the same. Another issue lead me to change the MTU on the modem (since I guess it comes incorrect from the factory?). Still after changing the MTU, the issue still persisted. So I was back at square one.

After some more troubleshooting with ping, traceroute and nslookup, all three commands functioned normally on all of the devices. Additionally RDP and SSH seemed to function normally, it was only HTTP/HTTPS I was having trouble with and it only seemed to be present on the local network. So what is it?

I troubleshooted the DNS again. I statically routed heestand.tech to it’s local IP address (let’s say it is 10.2.1.11) and it seemed to work all the time! Perfect, we now know it is DNS that is causing the issue, but why? Well, I introduce you to a little thing called NAT loopback. If you have a server/device on a local network and you can access it from a public IP address (say 78.8.82.21) most modern networking devices will be able to automatically route the packets to the internal IP address instead of the request having to go out to the public IP address. Well it appears with the new AT&T modems, this is not supported (or it is supported very poorly). Due to this, the modem sees the packet with it’s own public IP address and decides to drop the traffic.

For some unknown reason, this issue was further compounded because of the port forwarding I did on the Linksys system for my printer back a few years ago. One key note is the printer uses port 80 and 443 to host it’s webpage. Once I went through and deleted the old port forwarding for the printer, everything seemed to resolve itself so now I only have to deal with the half working AT&T modem. Which I can live with for now since it is just a little slow.

Hello all, I thought I would hop on and do a mid-year update of the server and some of the services.

Change Notes for Main Server

• All services now use Keycloak for authentication (with the exception of Vaultwarden and Guacamole)
• Mastodon, a federated social media platform has been added to the list of services
• Firefly iii, a manual financial management/budgeting tool has been added to the list of services
• Traefik is now the reverse proxy instead of Nginx
• A Postgres database has been added

So there have been quite a few changes, all of which I am very excited about! One of the most impactful ones in my opinion has been the addition of Keycloak for the sole reason of having one, singular log on for all applications. One surprise that came with it was the addition of “Passwordless” logins with their webauthn integration.

The webauthn integration allows for you to enroll any device as a Security Key. This allows you to use your phone or computer’s login system to log in instead of relying on a username and password. For some devices like a phone or tablet, this will allow you to use your phone’s fingerprint scanner or facial recognition feature to login. For PCs such as Windows 10/11 devices, you can use your TPM PIN code, facial scan or your login password. As a brief note, only newer computers support this feature. Your PC must be up to date. For MacOSX devices, your Keychain can be used to store your login information.

Another feature area I am excited to talk about is the reverse proxy change to Traefik. Previously, Nginx, a very popular webserver was used. However, as Keycloak got integrated into the stack, I found that being able to use Keycloak to authenticate users would be difficult, as such I settled on Traefik since it has really good support for using Keycloak in conjunction with it. It was a big change but much needed. Traefik offers a lot more extensibility than Nginx did. Although I will miss the simplicity of Nginx, Traefik will definitely hold up to the task!

One quick note on Vaultwarden before I close this section off, currently the developers are implementing single sign on that is compatible with Keycloak. This has been taking time due to the complexity of everything, but I am actively monitoring it.

Change Notes for Logging Server

• Logging server has been dockerized

Previously, I manually setup Graylog by manually installing everything. However early in the year, Graylog had a breaking update which seemed to break everything. After cutting my losses and starting fresh, I used a docker-compose file to setup a new instance of Graylog, Mongodb and memcached. Everything works well so far although I still haven’t rebuilt where I was before since the migration to Keycloak and Traefik took up a lot of time.

Change Notes for XOA Server

XOA (Xen Orchestra Appliance) is a web app used to manage Virtual Machines on the server. I used to use the default appliance that got installed with the system however I found that I needed all of the features, so I found a docker-compose file that builds and deploys the full one for me. The only downside is that I am responsible for updating the system, but that is ok with me.

New Servers

The only new server that has been added is the PXE Boot server. While this server is currently not used, I do have plans for it to be used to streamline updates to the virtual machines since I am on limited bandwith. The main idea is this: manually build a Debian based cloud-init image that is automatically set up to run a provided docker-compose file. Once the image is built, the newly built image will automatically be pushed to the Virtual Machines when they reboot next time.

Currently I am still in the implementation phase of this project. I have hit a few snags in the road with using Gitea and Jenkins in regard to how Gitea stores data. So this is on hold until the time being.

Future Plans for the End of the Year

• Setup Graylog fully
• Add OIDC to Guacamole
• Change Nextcloud to Minio

These ones are pretty much self explanatory. I would like to setup all of my rules in Graylog again so I can have a cohesive dashboard for traffic analysis and security purposes. Guacamole has yet to join the party on using Keycloak to authenticate however it is easy enough to do in an evening with a custom dockerfile. I have also been wanting to ditch nextcloud for quite a while and replace it with Minio. I don’t have much against nextcloud but I do think an S3 back storage solution would be better for hosting files than a PHP & Webdav backed solution. S3 storage solutions are pretty much the gold standard nowadays when it comes to storing files.

Closing Thoughts

Quite a lot of work has been done this year and I am very happy to see where everything is going! I might post periodic updates on my mastodon instance so make sure to follow me on mastodon: @kheestand@mastodon.heestand.tech

Thank you all and take care of yourself and others!