If you are getting this error message read this article first BEFORE you try to rebooting your firewall.
I ran into this problem recently with a Palo Alto PA-200 series firewall. I tried to login via the WebUI and would get the error “Session timed out”. I could SSH into the firewall and the internet was working. I’ve had this problem before and a quick reboot of my PA-200 solved the problem. Not so this time.
This time when I rebooted the firewall it did not come back up. Well not fully at least. I could PING and SSH it and load the WebUI (still getting the “Session timed out” error when trying to login) but all network traffic stopped flowing. This was bad.
I did some searching online and found that the issue can occur if you run low on disk space on your Palo Alto.
I logged into my PA-200 via SSH and ran the following:
myuser@PA200-MySite> show system disk-space
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 1.9G 1.9G 0M 100% /
/dev/sda5 6.6G 1.3G 5.1G 20% /opt/pancfg
/dev/sda6 1.9G 752M 1.1G 42% /opt/panrepo
tmpfs 1.3G 67M 1.2G 6% /dev/shm
/dev/sda8 2.4G 1.4G 874M 62% /opt/panlogs
You can see the problem. The root (/) volume of the PA-200 was out of disk space.
I reviewed the following Palo Alto KB’s:
Neither of them helped. They didn’t clear up any disk space on the root volume.
To fix the problem I had to call Palo Alto support. They generated a support key which allows them and only them access to the root file system on a Palo Alto device. Form there they cleared out a few logs in /var/log that were eating up all of the disk space. The main problem was /var/log/secure. If you’re familiar with Linux systems you’ll know that log gains entries for all successful and failed login attempts. In the Palo Alto’s case it was all of the successful and failed logins via SSH.
Once Palo Alto cleared out those logs files we gave our PA-200 another reboot and it came back up as per normal.
Then came the part where we wanted to prevent this from happening again. I knew for a fact we had never created a rule that allowed access to the PA-200’s SSH service and there was no way someone internally was hammering the PA-200 trying to break into it. Fortunately I had a really good support rep on the phone that knew exactly where to look. Management Profiles.
If you login to your Palo Alto via the WebUI and go to ‘Network’ and ‘Interfaces’ you’ll see a column labelled ‘Management Profile’. In our case we had a management profile assigned to our public interface that allowed for SSH. This is how the internet in general was accessing our PA-200’s SSH service. That’s not the best part though. The best part is traffic that is allowed via a Management Profile isn’t logged so you can’t even tell this is happening by looking at your traffic logs. Awesome right?
We changed our management profile to only allow ICMP (pings) and called it a day.
I’ve been told Palo Alto is aware this is an issue but it only really affects the PA-200 since it has the smallest hard drive. Palo Alto isn’t making it a priority to fix it by implementing something as simple as logrotate or even truncating the log after 50mb is written to it.
If you have this exact problem I really hope you have you have an active Palo Alto support contact. If you don’t you’re screwed. Palo Alto is the only one who can access the root file system.
I’m hoping they will eventually fix this problem in future PanOS releases.