Palo Alto firewall displays “Session timed out” when you try to login

If you are getting this error message read this article first BEFORE you try to rebooting your firewall.

Screen Shot 2015-01-25 at 09.33.11

I ran into this problem recently with a Palo Alto PA-200 series firewall. I tried to login via the WebUI and would get the error “Session timed out”. I could SSH into the firewall and the internet was working. I’ve had this problem before and a quick reboot of my PA-200 solved the problem. Not so this time.

This time when I rebooted the firewall it did not come back up. Well not fully at least. I could PING and SSH it and load the WebUI (still getting the “Session timed out” error when trying to login) but all network traffic stopped flowing. This was bad.

I did some searching online and found that the issue can occur if you run low on disk space on your Palo Alto.

I logged into my PA-200 via SSH and ran the following:

You can see the problem. The root (/) volume of the PA-200 was out of disk space.

I reviewed the following Palo Alto KB’s:

Neither of them helped. They didn’t clear up any disk space on the root volume.

To fix the problem I had to call Palo Alto support. They generated a support key which allows them and only them access to the root file system on a Palo Alto device. Form there they cleared out a few logs in /var/log that were eating up all of the disk space. The main problem was /var/log/secure. If you’re familiar with Linux systems you’ll know that log gains entries for all successful and failed login attempts. In the Palo Alto’s case it was all of the successful and failed logins via SSH.

Once Palo Alto cleared out those logs files we gave our PA-200 another reboot and it came back up as per normal.

Then came the part where we wanted to prevent this from happening again. I knew for a fact we had never created a rule that allowed access to the PA-200’s SSH service and there was no way someone internally was hammering the PA-200 trying to break into it. Fortunately I had a really good support rep on the phone that knew exactly where to look. Management Profiles.

If you login to your Palo Alto via the WebUI and go to ‘Network’ and ‘Interfaces’ you’ll see a column labelled ‘Management Profile’. In our case we had a management profile assigned to our public interface that allowed for SSH. This is how the internet in general was accessing our PA-200’s SSH service. That’s not the best part though. The best part is traffic that is allowed via a Management Profile isn’t logged so you can’t even tell this is happening by looking at your traffic logs. Awesome right?

We changed our management profile to only allow ICMP (pings) and called it a day.

I’ve been told Palo Alto is aware this is an issue but it only really affects the PA-200 since it has the smallest hard drive. Palo Alto isn’t making it a priority to fix it by implementing something as simple as logrotate or even truncating the log after 50mb is written to it.

If you have this exact problem I really hope you have you have an active Palo Alto support contact. If you don’t you’re screwed. Palo Alto is the only one who can access the root file system.

I’m hoping they will eventually fix this problem in future PanOS releases.

Access Supermicro BMC via SSH tunnels

I’ve got a server at home with a Supermicro motherboard that has a BMC in it. The BMC allows me to access a web interface on a dedicated network interface on the motherboard which will let me control the server in the event the OS has frozen or the hardware has powered down. This is extremely useful if something goes wrong at home while I’m out of town and need to power cycle my server remotely.

The problem I have is that my VPN server is a VM hosted on the server in question. So if the server is down I can’t VPN home and access the BMC. I could forward the ports through my firewall but there is a more secure way of doing things so why not?

I got my hands on a Raspberry Pi2 with the intent of connecting it up so I could remotely access it via SSH in the event my main server was offline at home. From the RPi2 I could then load up a browser and access the BMC interface on my server. One big problem though. While I could access the web interface and power cycle the server I could not access the Java based KVM that comes included with the BMC. The KVM lets you access the server as if you were physically in front of it with a keyboard and mouse AND connect media to the server remotely such as a ISO for some diagnostic software if needed. Unfortunately no matter how much I tried I could not get the Java WebApp to work on my RPi2.

Instead I opted to just use SSH tunnels to connect to the BMC via the RPi2. Again this worked great for the WebUI but failed when using the Java KVM. I did find a work around though and it’s pretty simple. When you’re on the Supermicro BMC page and you start the Java KVM you get a download for a “launch.jnlp” file. Save that to your local computer.

Open the launch.jnlp file in your favourite editor and you’ll see something like this:

You want to edit a few lines:

And you’re done. Save and close the file.

You’ll notice changed the web port on Line 1 from 80 to 5901 because I know 5901 isn’t in use on my local system. Now all I had to do was setup my SSH Tunnels so that local host 5901 forwarded to remote host 80 and local host 5900/623 forwarded to their respective remote host ports via my RPi2.

There is one catch with this method. Every time you go back into your BMC web-interface and click ‘Launch Console’ it appears the BMC generates a new set of security keys. All this means is if you access your BMC via the normal method and then want to use tunnels again you’ll have to get a new .jnlp file and re-apply the above edits.