A few months ago the AT&T modem my family had for about two years decided to die. It was one of the typical rental modems from AT&T, nothing too fancy (I think it was the Arris BGW210-700 to be exact). It had abysmal WiFi range so we ended up getting a Linksys mesh system and turning of the radio in the modem. Keep in mind, that modem was probably our fifth AT&T modem since they always seemed to die from hardware failure or something else.
Somewhere down the line a few years ago, I made the decision to isolate my server’s traffic from the Linksys system. I wanted to more cyber security stuff like live malware analysis and network traffic analysis from a SSH honeypot I had set up. Everything was working as expected, I could access my server either from the Linksys network or from the wide open internet, it was great. I even set up printing from my server to the printer that was on the Linksys network so I could print papers/other stuff from my Windows Virtual Machine. For a visual representation, the topology of the network looks like this:
Somewhere down the line, Window’s RDP client started to let users “forward” printers, local disks and USB devices from the client machine so you could access them on the server, so my original method of going through the forwarded ports from the mesh system was pretty much obsolete.
With the old method being obsolete, I never considered cleaning up the forwarded ports from the Linksys Mesh system since it never had any impact on anything until back in July.
As usual, the AT&T modem failed so we got a new one. I hooked up and configured all the devices to get the same IP address, just like the old modem. Everything was working fine except the connection to my server. If I navigated to “heestand.tech” from my the Linksys network, it would sometimes time out for no apparent reason, other times it would work for a few minutes.
At first, I thought I messed something up with the port forwarding on the modem end since I checked the access logs on the server and I didn’t see any attempt of my device to connect with my server, but everything seemed to be correct. To make this even more weird, I could access everything just fine outside of my home network. I would go to work and be able to login just fine. So I figured something was up with the modem.
I looked online to see if people had similar issues and I found a few sporadic cases of people having connection issues due to AT&T’s smart firewall feature on the new modem. I disabled the firewall and everything was still the same. Another issue lead me to change the MTU on the modem (since I guess it comes incorrect from the factory?). Still after changing the MTU, the issue still persisted. So I was back at square one.
After some more troubleshooting with ping, traceroute and nslookup, all three commands functioned normally on all of the devices. Additionally RDP and SSH seemed to function normally, it was only HTTP/HTTPS I was having trouble with and it only seemed to be present on the local network. So what is it?
I troubleshooted the DNS again. I statically routed heestand.tech to it’s local IP address (let’s say it is 10.2.1.11) and it seemed to work all the time! Perfect, we now know it is DNS that is causing the issue, but why? Well, I introduce you to a little thing called NAT loopback. If you have a server/device on a local network and you can access it from a public IP address (say 78.8.82.21) most modern networking devices will be able to automatically route the packets to the internal IP address instead of the request having to go out to the public IP address. Well it appears with the new AT&T modems, this is not supported (or it is supported very poorly). Due to this, the modem sees the packet with it’s own public IP address and decides to drop the traffic.
For some unknown reason, this issue was further compounded because of the port forwarding I did on the Linksys system for my printer back a few years ago. One key note is the printer uses port 80 and 443 to host it’s webpage. Once I went through and deleted the old port forwarding for the printer, everything seemed to resolve itself so now I only have to deal with the half working AT&T modem. Which I can live with for now since it is just a little slow.