Fun with Covid-19, lockdown, WFH and a FortiGate firewall

These are some rough notes on the issues encountered with providing remote access for staff when the UK went into lockdown.

Part 1: The Problem

In March 2020 all staff had instructions to work from home with immediate effect (UK Lockdown 1.0).

As with most other organisations, user network traffic flipped from inside-to-outside to outside-to-inside. More users immediately needed remote access.

Over the following days I checked how our FortiGate firewall was coping with the increase in the number of SSL VPN connections. This had only been used before by about 6 concurrent users.

After a day or so it emerged that it wasn’t coping well at all.

When the number of SSL VPN connections reached 90-100 all traffic in and out of the university stopped. As our monitoring system (PRTG) sent unresponsive alerts for the DNS servers I thought I had found the cause.

Some investigation proved this wasn’t the case. The PRTG sensor was testing the DNS servers by initiating a query. As the DNS server couldn’t reach the internet (due to the firewall issue) PRTG reported the DNS server as unresponsive.

Closer inspection of the firewall dashboard revealed high memory usage. This seemed to correlate with the number of SSL VPN connections. The CLI revealed a separate process for each SSL VPN connection.

I was initially confused as the spec sheet for our firewall model (1500D) stated a maximum of 10000 VPN connections.

Reading again it actually said 10000 IPSec VPN connections *not* 10000 SSL VPN connections.

Digging around in the FortiGate support documents revealed that our level of SSL VPNs is not a recommended usage scenario. The firewall will enter “conserve mode” when it detects high memory usage. This had the effect of halting all traffic forwarding (!).

Technical Tip: SSL VPN in web mode use a lot of CPU and memory resources:
https://kb.fortinet.com/kb/documentLink.do?externalID=FD48014

How conserve mode is triggered:
https://kb.fortinet.com/kb/documentLink.do?externalID=FD33103

As SSL VPN connections use far more memory than IPSec VPN connections this was the cause of the problems.

To try and prevent the traffic from stopping completely, I started to kill idle connections from the CLI to keep the connections below 90 and conserve memory.

I noticed the processes using all the memory were named “guacd”. Something I filed away and became significant later …

Meanwhile, a solution was needed (and fast).

Part 2: The Solution

The search for a solution.

Having determined that our solution for users to work from home wasn’t up to it, I started to look for alternatives.

The first thought was to use RDP. As this is not a good idea across the internet, it was quickly discounted.

Next was utilising the FortiGate IPSec VPN client (FortiClient). Licensing was free if not using central management of the VPN clients. This was fine for devices built, secured and controlled by us i.e. university owned devices. Not advisable for user owned devices. The VPN connection would put the devices on to the corporate network.

This has been rolled out for corporate devices but only solved a small part of the issue.

Microsoft RD Gateway was considered as a an RDP solution. Licensing expensive and not a quick solution to build. Possible in the long term.

While browsing during the evenings looking to see what other sysadmins were using to solve the WFH problem, I came across mention of the guacd process that was causing the issues on the FortiGate firewall.

It transpired that this was part of an open source remote desktop gateway product by the Apache Software Foundation; Apache Guacamole.

Apache Guacamole: https://guacamole.apache.org/

Fortinet had customised it to use as their SSL VPN offering on the FortiGate firewalls. It is a standalone product that can be run on a Linux VM.

I thought it was worth a try to take the load away from the firewall and still provide clientless RDP access for our users.

My thought process was that if the VM was getting busy it would be simple to add more memory or CPU from within vCenter.

Crash courses in Docker, Guacamole, MySQL and NGINX (reverse proxy/SSL) produced a test server.

After some testing it was decided to build a production server and get the users on it. What was there to lose?

So far (Dec 2020), it has performed fine with an average of 100 concurrent daily users while we look for a more long term solution.

TLDR: built Guacamole server, solved immediate problem.

Resources used:

Apache Guacamole Manual – Installing Guacamole with Docker: https://guacamole.apache.org/doc/gug/guacamole-docker.html

Alternative combined guide: https://www.linode.com/docs/applications/remote-desktop/remote-desktop-using-apache-guacamole-on-docker/

Installing Docker: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04

phpIPAM pingCheck cron job

After installing phpIPAM on Ubuntu 16.04 from the instructions here I needed to set up automatic host availability checking.

After following the instructions at the phpIPAM site I still could not get it to work.

There appears to be a number of people having the same issue i.e. the syslog shows the pingCheck cron job running but the hosts are showing as unavailable.

We discovered (after some random stumbling around) that adding the path to the php.ini file seemed to get discoveryCheck and pingCheck working properly.

So, we added “-c /etc/php/7.0/cli/php.ini” to the instructions from the phpIPAM site.

Our cron file is below:

# host discovery at 11am every day
0 11 * * * /usr/bin/php -c /etc/php/7.0/cli/php.ini /var/www/html/phpipam/functions/scripts/discoveryCheck.php

# update host status every 30 minutes
*/30 * * * * /usr/bin/php -c /etc/php/7.0/cli/php.ini /var/www/html/phpipam/functions/scripts/pingCheck.php

 

Wifi calling with Three inTouch and O2 TU Go

We recently had a query where a user wanted to use the Three inTouch and O2 TU Go apps for wifi calling as the phone signal was a bit rubbish in the building they were in.

The inital report stated that the connection could be made but there was no sound from either end. The call also terminated itself after a few seconds.

The firewall was apparently preventing these apps from working correctly.

The solution was to allow outbound access on UDP ports 1024-65535 to the 188.30.0.0/15 subnet (for the Three app) and the 91.220.9.0/24 subnet (for the O2 app).

The Three inTouch app by itself didn’t require all of these UDP ports, it actually requires 5060 (SIP), 5061 and 16000-65535.

As we were testing both apps one outbound firewall rule was created to cover both.

Further testing will determine whether the subnets need adding to and if the range of UDP ports can be restricted.

Jessops 360AFD Flash Instruction Manual

Instruction manual for Jessops 360AFD series Flashguns.

These instructions cover the following cameras:

360AFDC – TYPE C – Canon Digital E-TTL, E-TTL II cameras & E-TTL film cameras

360AFDN – TYPE N – Nikon Digital TTL, i-TTL cameras & TTL, i-TTL film cameras

360AFDS – TYPE S – Sony Alpha & Minolta Digital ADI, D Lens cameras & TTL film cameras

Jessops 360AFD Flash Instructions Page 1

PDF download: Jessops 360AFD Page 1

Jessops 360AFD Flash Instructions Page 2

PDF download: Jessops 360AFD Page 2

Syren Alternative Belly Dancers, Cinderford Music Festival 2015

Syren Alternative Belly Dancers, Cinderford Music Festival 2015

Syren Alternative Belly Dancers
Cinderford Music Festival 2015
The Causeway,
Edge Hill Road,
Cinderford,
Gloucestershire
GL14 2QH

Apple TV 2 jailbreak

I had been looking for a way to stream video from my own hard drive to the TV while still using my Apple TV 3. I read through the information on using the Apple TV 3 PlexConnect hack but that meant still having a machine running all the time with the Plex Media Server on it. I already had a Mac Mini running 24/7 so the Apple TV 3 could stream movies through iTunes Home Sharing and I wanted to turn it off and share the movies using some sort of NAS.

As the Apple TV 3 can only stream locally held movies through iTunes Home Sharing this required some creative thinking. Also, as the family are used to using the Apple TV (mostly) I was reluctant to change the streaming box again as the ATV is easy and intuitive to use.

Lots of proprietry and open source NAS OS’s have the open source iTunes protocol running but unfortunately it will only stream audio, so that was not suitable. I briefly tried XBMC on my Raspberry Pi, it worked OK but was (obviously) not as slick as the Apple TV.

However, during my researches I was offered a stock Apple TV 2 through a swap with a colleague at work, who wasn’t really using it, for my Apple TV 3. No money changed hands during this transaction even though they appear to be selling for up to £250.00 on eBay.

First thing was to read up on the jailbreak process and look at installing XBMC (now Kodi) which would allow me to stream video from an SMB (or NFS) share. After doing some research on this it appeared that XBMC could prove a bit unreliable and another colleague recommended looking at Firecore aTV Flash even though it meant paying real money for it. After reading up on the jailbreak/flashing process and looking at reviews and forums I decided to give it a go.

I won’t go through the whole process here, but basically jailbreak the Apple TV 2 using Seas0nPass and then flash with aTV Flash (black) for the ATV2.

Couple of things to note. The Seas0nPass instructions state: Seas0nPass provides an UN-TETHERED jailbreak of the 2nd gen Apple TV running the 5.3 software. However, my ATV2 was running 6.2 and the jailbreak process actually downgrades the software to 5.3 itself. So there are no worries if the ATV2 is running a later software version.

The other thing happened during the jailbreak process itself. The jailbreak instructions state: Step 3: When prompted, connect your Apple TV to your Mac/PC using a micro-USB cable (leave power cable disconnected). When I got to this step I tried pressing the MENU and PLAY/PAUSE buttons on the remote as instructed but nothing was happening. After a quick Google search it appears you DO have to connect the power cable. I did this and all went smoothly.

I connected a USB hard drive to my BT Home Hub 3 router, which will share a FAT32 formatted drive over SMB, connected to the share using InFuse on the jailbroken ATV2 and could stream video to the TV. I haven’t tried a full HD movie file yet, but as the the drive will only take a 4GB file any large HD video files would have to be re-encoded anyway to fit on the disk.

All good so far.

 

Online volunteering with HOT OSM

I have been looking for opportunities to volunteer for a while and, due to family, work and other commitments, I cannot commit to volunteering regularly. This discounts me from volunteering at a local charity as they would want me to be available on a regular schedule so they can plan their workload.

Therefore, I have been searching for online volunteering opportunities. I wanted to contribute online so I could use any spare time I have helping out, whether it was a free 10 minutes or an hour at a time.

I was browsing reddit and came across a post that mentioned the Humanitarian OpenStreetMap Team (HOT OSM) that uses volunteers to trace data from satellite images on to OpenStreetMap for use by aid agencies when responding to situations of political crisis and natural disasters. Microsoft has donated all their Bing satellite images for use by HOT OSM.

I browsed a number of web sites to see if the data was actually being used and came across a few references to the maps being used on the ground. The HOT OSM projects are tied in with the Missing Maps and MapGive projects and do seem to be making a difference. This was one of the criteria for the use of my time.

I first had to learn how to edit OpenStreetMap, so I followed the steps at LearnOSM and practiced on the OpenStreetMap in my local area, including the use of the JOSM application.

I then learned how to use the OSM Tasking Manager from this section at LearnOSM so I could start contributing to the projects.

If you browse the OpenStreetMap wiki there is a lot of help and information if you are unsure about any aspects of the process. For example, I have been working on projects in West Africa and one area I was unsure of was how to identify what type of road I was looking at on the satellite images so I could tag it correctly. After a bit of searching I came across this post which helped enormously.

I wondered, as you probably will, why the aid agencies don’t just use Google Maps. There are several reasons, which can probably be articulated better by others:

• Google decides what is displayed on Google Maps
• anyone can edit OSM and the data stays freely available
• OSM is free for everyone to view, download and print
• in areas of concern Google doesn’t display every road, street, track, path, wall and building (which is what you will be filling in if you volunteer)

An overview of the Humanitarian OSM Team is available at their wiki.

If you are interested in volunteering and, like me, you are a bit techie but are unable to commit regularly, then this is the one of the better uses of your spare time. You will make a difference.

Nexus 4 will not turn on or charge

I gave my daughter the Nexus 4 I was using as I found it a bit too big to have in my pocket all day at work and I started using her Nokia Lumia 620.

After a few weeks she said that the phone had just turned itself off and would not switch back on. I was assured that the battery was not low when this happened.

So I Googled the issue and the first post gave the following solution:

1. Remove charging cable
2. Hold down the power and volume down buttons
3. Insert the USB charging cable into the Nexus 4 while continuing to hold down the power and volume down buttons

This didn’t work so I Googled some more. Basically all I found were variations on the same theme.

Leave charging cable in. Hold volume down (or up) and power buttons for 15 (or 20, or 60) seconds.

Remove charging cable. Hold volume up (or down) and power buttons for 15 (or 20, or 60) seconds. Insert charging cable while still holding buttons down.

Charge for 1 hour. Remove charging cable. Re-insert charging cable within 10 seconds. Hold power button for 15 seconds.

Connect your Nexus device to a computer’s USB port while the computer is on and connected to a power source. After 10–15 minutes, disconnect the USB cable from your Nexus device and reconnect it within 10 seconds. After 1 minute, check to see if a battery icon appears on the screen. Press & hold the Power button for at least 15 seconds to see if your device turns on.

During these operations I should see a red (or orange) led flash (or not).

I tried all of these variations and none of them worked so looked to see if it was still in warranty. Luckily, I discovered there is a two year warranty on the Nexus 4 in the UK.

So I phoned Google Play Support Team and was instructed to go through various combinations of plugging in chargers and pressing buttons, including a new one of holding the volume up AND volume down AND power buttons simultaneously. As before, none of this worked.

After that I went through the warranty replacement process. After Google checked that I had purchased the Nexus 4 through the Google Play Store, and that it was still in warranty (80 days left), I received an email from Google with a link to the purchase of a replacement device. Basically you are authorising Google to charge you £159.00 if the device you return to them is faulty because of something not covered under the warranty.

I spoke to Google on a Wednesday, received the refurbished Nexus 4 on the following Monday and posted the faulty phone back on the Tuesday.

Google did not charge me for the replacement so as far as I am concerned this was a successful warranty claim.

Raspberry Pi wifi configuration

Like most people running a Raspberry Pi I have had loads of problems getting the wifi working on it.

I have a RPi B and use a TP-LINK TL-WN725N USB wifi adapter.

I’ve tried various configurations to bring the wifi interface up. Sometimes it only works when the ethernet cable is connected. I’ve also had the wifi connecting to my router but won’t ping, and I’ve had the wifi interface connecting OK but dropping the connection.

Currently I am using the configuration below which seems to work most of the time. I don’t use the wpa_supplicant.conf file as I have read that I do not need it if my SSID is not hidden. Don’t know if that is correct but it seems to work for me.

### /etc/network/interfaces ### 

auto lo 

iface lo inet loopback 

iface eth0 inet dhcp 

allow-hotplug wlan0 

auto wlan 0 

iface wlan0 inet dhcp 

wpa-ssid “BTHub3-XXXX” 

wpa-psk “xxxxxxxx” 

wireless-power off 

# wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf 

# iface default inet dhcp 

 

I’ve just built my RPi from scratch and this config works but is a bit unreliable. If I don’t do anything on the console for a while the connection drops and I have to ping the IP address for a while to get it to respond.

So, I followed the instructions detailed here and created the file /etc/modprobe.d/8192cu.conf with following configuration:

# Disable power saving

options 8192cu rtw_power_mgnt=0

So far it seems to have fixed it.

Windows Vista KSoD (blacK Screen of Death)

Was given a Dell Inspiron 1520 laptop running Windows Vista to fix. It would boot up but there was no desktop, just a black screen with a working mouse pointer. This appears to be a common problem and there are a number of solutions posted on the web. I tried a few lots of them.

Here are the fixes I tried (spoiler: solution 7 worked):

 

---

Solution 1

 http://www.nuonsoft.com/blog/2009/05/02/possible-ksod-black-screen-of-death-solution-for-windows-vista/comment-page-2/

1. Press F8 and start up computer to enable recovery menu
2. Select Repair your computer
3. In the System Recovery Options dialog box accept the default keyboard layout (US).
4. Select user name from drop down box on next screen and enter password
5. Choose Command Prompt from the list
6. Type in c:
7. Then type in Regedit
8. Clicked “File>Import”
9. Clicked to show “all files” at the bottom
10. Navigated to C:\Windows\System32\Winevt\Logs
11. Deleted all event logs one by one.
    • Alternative solution is to rename the Logs folder to Logs_OLD (ren Logs Logs_OLD) and recreate an empty Logs folder (mkdir Logs).
12. Closed all windows and rebooted.

Didn’t work.
top

---

 Solution 2

https://social.technet.microsoft.com/Forums/windows/en-US/193b7008-ce4b-4d03-acc3-b8d7ffe610d5/vista-black-screen-white-mouse-pointer 

1. Press F8 and start up computer to enable recovery menu
2. Select Repair your computer
3. In the System Recovery Options dialog box accept the default keyboard layout (US).
4. Select user name from drop down box on next screen and enter password
5. Choose Command Prompt from the list
6. At C:>  type  regedit   [Press ENTER]
7. The registry editor will open.
8. In the Registry select  ‘HKEY_LOCAL_MACHINE’ and then go to File/Load Hive.
9. In the ‘Load Hive’ dialog box, navigate to the C:\Windows\System32\Config  folder.
10. In the Config folder, select  SOFTWARE  and click Open.
11. In the Name Hive dialog box, type a unique name, such as EDIT1 and click OK.
12. You should now see the EDIT1 branch, expand this branch and navigate to \Microsoft\Windows NT\CurrentVersion\Winlogon.
13. In the right side of the Winlogon Key you will see a Value called  ‘Shell’, double click this value.
14. In the Value Data box, it should show  ‘Explorer.exe’  and nothing else. If it shows something different, change the Value Data to Explorer.exe.
15. Click OK.
16. Scroll back up and select the EDIT1 branch.
17. Go to File and select Unload Hive.
18. Exit the registry editor and the command prompt.
19. Press the ‘Restart’ button to reboot.

Didn’t work. Value was already set as Explorer.exe.
top

---

Solution 3

http://www.nuonsoft.com/blog/2009/05/02/possible-ksod-black-screen-of-death-solution-for-windows-vista/comment-page-2/

1. Press F8 and start up computer to enable recovery menu
2. Select Repair your computer
3. In the System Recovery Options dialog box accept the default keyboard layout (US).
4. Select user name from drop down box on next screen and enter password
5. Choose Command Prompt from the list
6. Navigate to C:\windows\system32\config\
7. There should be two registry files: “software” and “software_previous”
8. Rename “software” file to “software_corrupt” and “software_previous” to “software”
9. Exit the command prompt.
10. Reboot

Didn’t work.
top

---

Solution 4

http://answers.microsoft.com/en-us/windows/forum/windows_vista-hardware/getting-that-black-screen-of-death-upon-logging/e0312a9a-e84d-452f-9244-5e3ba1a4b8af 

1. Press F8 and start up computer to enable recovery menu
2. Select Repair your computer
3. In the System Recovery Options dialog box accept the default keyboard layout (US).
4. Select user name from drop down box on next screen and enter password
5. Choose Command Prompt from the list
6. At C:>  type  regedit   [Press ENTER]
7. The registry editor will open.
8. In the Registry select  ‘HKEY_LOCAL_MACHINE’ and then go to File/Load Hive.
9. In the ‘Load Hive’ dialog box, navigate to the C:\Windows\System32\Config  folder.
10. Select “SYSTEM”
11. Select Open.
12. In the Load Hive dialog box, type in “MySYSTEM” box for the registry hive that you want to edit.
13. After the hive is loaded, modify the following key value per the instructions below: You will need to know what ControlSet the machine is currently running on, this can be determined by going to HKEY_LOCAL_MACHINE\MySYSTEM\Select and find the “Current” value in the Right hand side. (Example: Current value is 1 then the ControlSet will be ControlSet001)
14. Key: HKEY_LOCAL_MACHINE\SYSTEM\ControlSet00X\Services\RpcSs (X is the Number from the Current Key from above)
15. Value Name: ObjectName
16. Old Value: LocalSystem
17. New Value: NT AUTHORITY\NetworkService
18. Unload the SYSTEM hive by selecting the key “MySYSTEM” and then select File -> Unload Hive… menu item.
19. Exit regedit.exe
20. Reboot the system normally

Didn’t work. Value was already set as NT AUTHORITY\NetworkService.
top

---

 Solution 5

http://www.howtogeek.com/howto/windows-vista/make-user-account-control-uac-stop-blacking-out-the-screen-in-windows-vista/

1. Press F8 and start up computer to enable recovery menu
2. Select Repair your computer
3. In the System Recovery Options dialog box accept the default keyboard layout (US).
4. Select user name from drop down box on next screen and enter password
5. Choose Command Prompt from the list
6. At C:>  type  regedit   [Press ENTER]
7. The registry editor will open.
8. Browse to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System
9. Right-click in the right-hand pane and create a new 32-bit DWORD value called PromptOnSecureDesktop, setting the value to 0.
10. Exit regedit.exe
11. Reboot the system normally

Didn’t work.
top

---

 

Solution 6

1. Created Windows 7 USB boot disk using X17-59183.iso that I saw mentioned on Reddit.
2. Booted faulty laptop from USB drive
3. Accept default language
4. Click Repair your computer in bottom left of Install Windows screen (do not click Install now)
5. Select system hard drive on next screen and click Next
6. Select Command Prompt from the list
7. Run chkdsk c: /f /r (takes a long time)
8. Run sfc /scannow /offbootdir=c:\ /offwindir=c:\windows
9. Reboot

Didn’t work.
top

---

 

Solution 7

Created a Windows Vista installation USB drive by following the instructions here:

http://www.vistax64.com/general-discussion/296167-vista-iso-download.html 

Then re-installed Windows Vista as detailed here:

http://computer.howstuffworks.com/how-to-reinstall-windows-without-losing-data.htm

Hooray!
top