2020-01-02

Ubiquiti UniFi AP AC Pro firmware update

Recently, I needed to expand a UniFi wifi network with a new access point, and the hardware of choice, was the UniFi AP AC Pro.

When I tried "adopting" it, the UniFi controller told me that the AP's firmware (3.x) was incompatible, and that I could click to adopt and upgrade.

That seemed nice, but was a bit different from other devices, which typically would have separate upgrade and adopt actions (or buttons in the UniFi app).

However, this did not work, probably due to a software error in the controller which causes issues with old firmware versions, and trying various ways of updating and bugfixing did not work. The AP was stuck in TFTP mode.

Ubiquiti's support article does not actually explain how to fix it, as vital information is missing from the list of instructions.

There are a bunch of user forum posts about these kinds of issues online, and nobody seems to offer a coherent explanation of how to fix this.

So here is how I fixed it, step by step, from a Mac, some of the text has been copied verbatim, for this educational purpose, from
Ubiquiti's Ubiquiti's support article:


  1. Ensure that you have a computer with wired network access capability, e.g. via a USB/Thunderbolt dongle if you have a laptop.
  2. Download the firmware, e.g. to your Downloads folder.
  3. Same as Ubiquiti's item #3: Go to System Preferences > Network and set your computer's wired network IP address to 192.168.1.25, subnet 255.255.255.0 and gateway 192.168.1.20. You do not need to disable Wifi.
  4. Open a command line window, e.g. Terminal, and assuming you downloaded to "Downloads", type:


    cd Downloads
    tftp
  5. Unplug the ethernet cable from the UniFi AP.
  6. In the UniFi controller, "forget" the UniFi AP. You need to enter the unadopted device's submenu to do so.
  7. Using a paperclip press and hold the UniFi AP's reset button. Do not release the button until step 9.
  8. While keeping the reset button pressed in, plug the ethernet cable back into the AP. Keep the reset button depressed until you see the device's LED flashing in upgrade mode (blue-white-off somewhat rapidly). Ubiquiti say this may take up to 25 seconds, but in my experience, it took around 8-12 seconds.
  9. Release the reset button.
  10. On the TFTP command line, paste these four lines and hit enter:


    connect 192.168.1.20
    binary
    rexmt 1
    timeout 60
  11. Type the command "put" followed by filename of the firmware downloaded. DO NOT enter the full path, as the tftp software does not have permissions to do so in recent MacOS versions.


    put BZ.qca956x.v4.0.69.10871.191109.0533.bin


    If the transfer is successful, you get a confirmation of how much was transferred, and the device will change its status lights to show that a firmware upgrade is in progress. If not, you will most likely get an error like "tftp: sendto: Host is down", and you will need to go back to step 7.
  12. Re-connect the PoE injector's LAN cable into your router. Restore the network IP back to what it was before, or disable the USB/Thunderbolt adapter.
  13. Adopt the UniFi AP.
That should do it. Happy new year!

2017-07-27

MySQL, MariaDB and TLS (SSL)

Do you want to secure your MySQL flavoured database server connections, and have discovered that you cannot do it like you secured your webserver?

Then I have answers for you, at least if you're using a Debian.

This post has multiple parts:


Requirements

  • Debian Wheezy, Jessie, or Stretch
  • GNU certtool or openssl
  • MySQL client software and libraries, either as shipped by Debian, or via Oracle's repositories for MySQL 5.5, 5.6, or 5.7, or MariaDB's repositories for 10.x
  • MySQL server software, similarly

Gotchas

  • Debian's MySQL (Wheezy, Jessie) and MariaDB (Stretch) packages, and MySQL Community Edition precompiled binaries, including WorkBench, all use yaSSL/WolfSSL, which support an extremely limited set of crypto features, compared to OpenSSL.
  • Having OpenSSL installed on the server or client does not help with the above situation.
  • MariaDB ships precompiled binaries that are linked against OpenSSL.
  • All keys and certificates must stay within the minimum supported feature sets (lowest common denominator) of yaSSL and OpenSSL. In essence, this means RSA keys hashed with SHA2-256. Do not attempt using DSA, ECDSA, etc.
  • TLS v1.2 is impossible with older versions of yaSSL, it's only possible with the version bundled in MySQL 5.7.10 or more recent, or with MariaDB binaries that are linked with OpenSSL versions supporting TLS v1.2. Restricting to v1.2 is therefore unwise.

Part A: Certificate Generation

Either follow the MySQL 5.5 instructions religiously, or use certtool. Skip to Part B if you have this under control.

1. Create template for self-signed certificate

Suggested name: ca.cfg


# An organization name
organization = "Blogger Bakkushan TLS Non-Authority"
# Geographical location things
state = "Oslo"
country = "NO"
locality = "Oslo"
# Common name (certificate "name", if you like)
cn = "Bakkushan CA 1"
# Serial number (unimportant, but you need it)
serial = 1
# Expiration - after this long, you need to generate a new CA cert
expiration_days = 3652
# This is a CA certificate, which will be used for signing
ca
cert_signing_key
signing_key
encryption_key

2. Create a self-signed CA certificate


certtool --generate-privkey --rsa --bits 4096 --outfile ca-key.pem
certtool --generate-self-signed --rsa --load-privkey ca-key.pem --template ca.cfg --outfile ca-cert.pem

3. Create template for server certificates

Filename e.g. "server-cert.cfg".

# An organization name
organization = "Blogger Bakkushan MySQL Services"
# Geographical location things
state = "Oslo"
country = "NO"
locality = "Oslo"
# Common name (typically the full DNS name of the server)
cn = "mysql.bakkushan.example"
dns_name = "mysql.bakkushan.example"
# Got a DNS alias? Ok!
dns_name = "alias.copycat.example"
# Serial number (unimportant, but you need it)
serial = 1
# Expiration - after this long, you need to generate a new server
expiration_days = 365
# This is a server certificate, ignore the "www"
tls_www_server
# This is not a CA certificate, which will be used for signing
signing_key
encryption_key

4. Create a server certificate


certtool --generate-privkey --rsa --bits 3072 --outfile server-key.pem
certtool --generate-request --load-privkey server-key.pem --template server-cert.cfg --outfile server-req.pem
certtool --generate-certificate --rsa --load-request server-req.pem --load-ca-privkey ca-key.pem --load-ca-certificate ca-cert.pem --template server-cert.cfg --outfile server-cert.pem

5. Optional: Create template for client certificates

Filename e.g. "client-cert.cfg".

# An organization name
organization = "A Client Organization"
# Geographical location things
state = "Stockholm"
country = "SE"
locality = "Stockholm"
# Common name (something to identify with)
cn = "Medel-Svensson"
# Serial number (unimportant, but you need it)
serial = 1
# Expiration - after this long, you need to generate a new server
expiration_days = 365
# This is a client certificate, ignore the "www"
tls_www_client
# This is not a CA certificate, which will be used for signing
signing_key
encryption_key

6. Optional: Create a client certificate


certtool --generate-privkey --rsa --bits 3072 --outfile client-key.pem
certtool --generate-request --load-privkey client-key.pem --template client-cert.cfg --outfile client-req.pem
certtool --generate-certificate --rsa --load-request client-req.pem --load-ca-privkey ca-key.pem --load-ca-certificate ca-cert.pem --template client-cert.cfg --outfile client-cert.pem

Part B: Server Configuration

Note 1: you need to get the paths precisely correct, or nothing will work.
Note 2: path, directory/folder, and file permissions must all permit the database server instance to read the files mentioned. Directories should need only execution privileges. Whether you do so via ACLs (setfacl) or old-fashioned chown and chmod is up to you.

I like placing these files in /etc/mysql/ssl, a directory that you'll need to create.

ssl-ca = /etc/mysql/ssl/ca-cert.pem
ssl-cert = /etc/mysql/ssl/server-cert.pem
ssl-key = /etc/mysql/ssl/server-key.pem

Part C: Client Configuration

If you created client certificates, you need to distribute the following files:

ca-cert.pem
client-cert.pem
client-key.pem

The client needs to specify the corresponding options for CA certificate, client certificate, and client key. Doing so ensures that you can connect while validating the server certificate, and permitting the server to validate the client certificate. If no validation is required, you can connect, but you must request SSL.

Please note that the latter option leaves your TLS connection open for man-in-the-middle attacks.

Part D: Database Restart

Yeah, you need to restart the database server instance to load the new config.

systemctl restart mysql.service


Part E: Database Grant Statements

I really recommend that you restrict access to TLS, and if you have provided client certificates, require that as well. The following example grants wide privileges to anyone connecting with the correct username and password from anywhere, but limits simultaneous connections for that user:

GRANT ALL PRIVILEGES ON clientdatabase TO 'client'@'%'
REQUIRE SSL
IDENTIFIED BY 'correct horse battery staple'
WITH max_user_connections 10;

This example restricts the client certificate:

GRANT ALL PRIVILEGES ON clientdatabase TO 'client'@'%'
REQUIRE SUBJECT '/C=SE/ST=Stockholm/L=Stockholm/
O=A Client Organization/
CN=Medel-Svensson'
AND ISSUER '/C=NO/ST=Oslo/L=Oslo/
O=Blogger Bakkushan TLS Non-Authority/
CN=Bakkushan CA 1'
IDENTIFIED BY 'correct horse battery staple'
WITH max_user_connections 10;


Conclusion

It is a bit cumbersome to set up properly, as you cannot rely on global CAs, on DANE/TLSA, or other conveniences you may know from HTTPS, POP, IMAP, and SMTP, but if you do not try to be too clever, it is easy enough.


Sources







2015-05-21

Awful slowness with Dell R620 (20 cores), ACPI 4, and Debian Wheezy (Linux 3.2.0)

Summary

This problem has not been adequately described elsewhere, as far as I can tell. I'll try to keep it brief. Some symptoms and a more prosaic description of the problem is included below, but it really just states the same.

The problem

The system is performing horribly, and watchdog and power_saving threads try to use all available CPU time.

Affected systems

EDIT October 25, 2017: Kernel version has no apparent significant effect on the issue. The problem appears to affect all kernel versions for any stable Debian.

  1. Linux kernel older than 3.5.0 (e.g. Debian Squeeze or Wheezy, Centos 6.2, RHEL 6)
  2. EDIT October 25, 2017: Any Debian up to and including Stretch
  3. ACPI 4 compliant hardware (e.g. Dell 12g systems, R620, R720, ...)
  4. Logical processors enabled (hyperthreading, virtual CPUs)
  5. Power-saving enabled
  6. System is not busy, but not completely idle

Mitigation

These are temporary solutions before you can reboot with a more recent kernel.

EDIT October 25, 2017: These are the first steps to mitigate the problem while avoiding a reboot.

  1. rmmod acpi_pad (this takes a while)
  2. echo 0 > /proc/sys/kernel/nmi_watchdog

Solution

EDIT October 25, 2017: Upgrading Debian's kernel has no significant effect, as the problem persists. Only blacklisting the module acpi_pad works.

  1. rmmod acpi_pad (this takes a while)
  2. echo "blacklist acpi_pad > /etc/modprobe.d/acpi_pad.conf"

Upgrade the kernel to something recent, e.g. by compiling the current long term release yourself.

Debian Wheezy users may use the most recent kernel from wheezy-backports, which as of 2015-05-21 is stuck at 3.16. Please note that recent Linux kernels are compressed with xz. If your system is a virtual machine, ensure that the host system supports LZMA compressed kernels!

Symptoms

Maybe you got an alert about 100+ load, or a user report about system slowness. You look at the system, which is indeed responding slowly even to basic commands, and see watchdog and/or power_saving threads eat nearly all your processing capability (100% or more CPU usage).

Here's an example of what top(1) may show you. On some occasions, the power_saving processes may be dominating.

PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND            
   64 root      rt   0     0    0    0 R   331  0.0  52:22.64 watchdog/14        
   72 root      rt   0     0    0    0 R   324  0.0  51:35.85 watchdog/16        
   12 root      rt   0     0    0    0 S   324  0.0  53:32.19 watchdog/1         
   79 root      rt   0     0    0    0 R   322  0.0  50:46.77 watchdog/17        
   92 root      rt   0     0    0    0 R   315  0.0  50:24.58 watchdog/19        
  100 root      rt   0     0    0    0 R   309  0.0  54:09.71 watchdog/21        
  191 root      rt   0     0    0    0 R   301  0.0  62:52.38 watchdog/39        
  183 root      rt   0     0    0    0 R   299  0.0  66:01.97 watchdog/37        
  116 root      rt   0     0    0    0 S   296  0.0  52:55.74 watchdog/25        
  141 root      rt   0     0    0    0 S   295  0.0  54:26.20 watchdog/29        
  121 root      rt   0     0    0    0 S   295  0.0  52:59.55 watchdog/26        
  133 root      rt   0     0    0    0 S   289  0.0  53:37.40 watchdog/27        
  137 root      rt   0     0    0    0 S   286  0.0  53:33.76 watchdog/28        
  145 root      rt   0     0    0    0 S   279  0.0  57:03.61 watchdog/30        
  161 root      rt   0     0    0    0 S   268  0.0  60:25.99 watchdog/34        
   24 root      rt   0     0    0    0 R   258  0.0  51:34.17 watchdog/4         
    7 root      rt   0     0    0    0 R   251  0.0  53:31.10 watchdog/0         
   40 root      rt   0     0    0    0 R   218  0.0  50:17.33 watchdog/8         
   44 root      rt   0     0    0    0 R   214  0.0  40:28.09 watchdog/9         
   88 root      rt   0     0    0    0 R   214  0.0  50:43.48 watchdog/18        
  149 root      rt   0     0    0    0 S   197  0.0  61:30.01 watchdog/31        
   68 root      rt   0     0    0    0 R   195  0.0  50:57.25 watchdog/15        
  157 root      rt   0     0    0    0 S   139  0.0  57:46.94 watchdog/33        
   52 root      rt   0     0    0    0 S   137  0.0  53:56.98 watchdog/11        
44382 root      -2   0     0    0    0 R    99  0.0  16:33.87 power_saving/7     
44396 root      -2   0     0    0    0 R    97  0.0  16:08.18 power_saving/21    
44400 root      -2   0     0    0    0 R    97  0.0  18:03.40 power_saving/25    
44401 root      -2   0     0    0    0 D    94  0.0  17:28.80 power_saving/26    

Likely cause

Linux kernels before 3.5 have a bug in ACPI 4 processor idling (https://bugzilla.kernel.org/show_bug.cgi?id=42981). This can cause horrible slowness, probably some kind of deadlock, when running a somewhat idle (but not totally idle or busy) system, if you have a multi-core setup, with hyperthreading (logical/virtual cores) enabled, where you also try to conserve power, and where the kernel attempts to let cores go idle.

See also: http://en.community.dell.com/support-forums/servers/f/1466/t/19456558

Useless advice

I've also found the following useless/harmful advice. The text has been paraphrased by me. My comments are in parentheses:

  1. "Reboot, that solved the problem for me" (It "solves" it for a very brief period of time, until the problem reappears. Brilliant for not keeping the system up, though.)
  2. "Disable logical processors in BIOS" (That takes us down from 40 to 20 logical CPUs, yay, big help. Not if you want to keep the system up for a while.)
  3. "Disable power-saving, it's a server, this is a laptop feature" (I bought power-saving CPUs and memory for a reason, thank you! See also "keeping the system up".)