I was blown away by a demonstration of the lack of reasoning in ChatGPT, Llama, Claude, and the other LLM systems.

If you ask a LLM to finish a sentence with one word, explicitly stating that that word should have four letters, it cannot do it. You supply the sentence fragment “the cat sat in the ” and it will add a word, but not a four letter one.

I figured this was a funny example, but no way was it something that would still happen. Surely this was fixed and the models upgraded. I was able to instantly reproduce it:

Surely this was just a problem with ChatGPT 3.5, though?

This is a failure endemic to the entire LLM technology. It is possible to add some post-processors to check for obvious mistakes like this, and surely it will not be so simple to prove the flaws for very long, but the technology is incapable of reasoning. It cannot think. If you find yourself using an AI Chat to do thinking or to make decisions, please learn more about how the thing really works before it leads you into a dangerous error.

The video below is a good start.

As I’ve run EasyEngine for some time, the disk space on my cloud instance has been filling up. This seems to be in two general areas, docker volumes and docker overlay. Today I found an answer to the expanding docker volumes, which is configuring logrotate to rotate and compress the ever-growing php and nginx logs.

I used the solution offered in one of the EE GitHub posts. This provides configuration for logrotate for the EasyEngine hosted sites as well as the ingress nginx proxy. Running logrotate --force /etc/logrotate.d/ee a few times reclaimed 9% of my disk space, and this is likely to improve as the giant log files roll off over 30 days.

I’m reading that overlay2 makes for incorrect disk usage readings when the folders are queried with du. I did find that the site backups have aggregated to take a decent chunk of space. I’m currently offloading them to local storage and planning on purging them from the vm. This may get my disk usage down to a reasonable and maintainable level.

I wanted to configure my Unifi home network with a good segregation between between the desktop machines which I use to manage my family photos and finances and the IOT devices that aren’t really created by trustworthy vendors, or reliably patched to prevent compromise. I’ve trashed all my Amazon Alexa devices, since Amazon seems to think it’s a good idea to resell my internet connection with a presumptive opt-in. I try to only use trustworthy devices, but still, the home automation vendors do not necessarily have the budgets to develop high-quality code, and their incentives are not to optimize for my privacy.

Configuring Unify to separate devices into separate networks is pretty easy using their default tools, but there are several concerns that make segrating IOT into a firewalled network more difficult.

IOT device Wifi standard compatibility

Most modern network gear can publish an SSID (network name) that supports both 5ghz and 2.4ghz connections. This allows a device to connect to the best frequency for its current location and to hop between radios as situations change. Unfortunately, many IOT devices do not support 5ghz connections, but their software will attempt to connect to the wrong network during “automatic” configuration. If you are setting up a network for home automation gear, restrict it to only us 2.4ghz connections. Home control gear doesn’t need broader bandwidth than provided by 2.4ghz wifi, and will benefit from the greater penetration through building materials.

mDNS propagation

Most of the Homekit gear I use relies on mDNS (formerly Bonjour) service discovery. Unifi supposedly supports bridging these broadcasts between subnets, but this capability has been broken in their Dream Machine products for years and they have been unresponsive to requests for a fix. Luckily this can be supported by running custom services in a UDMP-hosted Docker container.

Installation Proceedure

Enable SSH

Set up the UDMP to allow connections using SSH.

Create VLANs

Setting VLAN ID and subnet settings for primary and IOT networks.

Create SSIDs

Attach a new SSID to each VLAN.

Create Firewall Rules to block IOT->LAN Traffic

In order to prevent network connections from the IOT network to the private home network, you need to set up firewall rules to drop the traffic.

• New Rule
• LAN IN
• Drop Traffic
• Source “IOT” network
• Destination “LAN” network

When I set up these rules as described in Christian Mohr’s post, I later discovered that the steps described still allowed ipv6 traffic. I need to add the same rules under the ipv6 tab, in addition to the ipv4 rules tab.

Configure mDNS Reflector

Having set up the separate networks and restrictions, we need to set up a broadcast reflector to allow devices on the private home network to discover devices hosted on the IOT network. They can then send control commands to the lower network while being protected in case those devices are compromised.

• Disable mDNS service (Advanced Features -> Advanced Gateway Settings -> Multicast DNS)
• Disable IGMP snooping
• Disable WiFi Multicast Enhancement
• Install Custom mDNS reflector

Install on-boot-script

ssh root@192.168.1.1
curl -L https://github.com/boostchicken/udm-utilities/releases/download/1.0.4/udm-boot_1.0.4_all.deb -o /tmp/udm-boot_1.0.4_all.deb
dpkg -i /tmp/udm-boot_1.0.4_all.deb
exit

Pull and run multicast-relay docker image with the correct bridge numbers for the configured VLANs

podman run -it -d --restart=always --name="multicast-relay" --network=host -e OPTS="" -e INTERFACES="br51 br5" docker.io/scyto/multicast-relay

Create startup script to restart container after reboots

touch /mnt/data/on_boot.d/01-multicast-relay.sh
chmod +x /mnt/data/on_boot.d/01-multicast-relay.sh
vim /mnt/data/on_boot.d/01-multicast-relay.sh

Contents:

#!/bin/sh

### kill all instances of avahi-daemon (UDM spins an instance up even with mDNS services disabled)

killall avahi-daemon

Finally, start the multicast-relay container image (if it’s not currently running).

podman start multicast-relay

Related Links

• https://vninja.net/2019/08/12/unifi-iot-networks/
• https://vninja.net/2019/08/08/creating-isolated-networks-ubiquiti/
• https://www.brandonmartinez.com/2020/09/02/unifi-and-mdns-with-apple-homekit/
• https://github.com/boostchicken/udm-utilities/tree/master/on-boot-script
• https://github.com/scyto/multicast-relay

Easy Engine is supposed to do automatic Let’s Encrypt updates out of the box, but unfortunately this appears to not actually work and forum requests for clarification have gone unanswered. Manually updating certs is easy with the command:

ee site ssl-renew --all

Friendly forum responders have suggested running this continually from the system crontab like so:

0 1 */5 * * bash -lc "/usr/local/bin/ee site ssl-renew --all" >> /opt/easyengine/logs/cron.log 2>&1

Droplet Server Image

First step is to backup the system to make sure that I can restore if something goes sideways. Running poweroff in the console will bring down the system, then using the Digital Ocean console to actually power down the system, create a named drive snapshot, then power up the system gets us a failsafe.

System Updates

apt update
apt upgrade -y
apt autoremove -y

After a reboot all is up and well.

WordPress Site Backups

I found several helpful references on doing Easy Engine wordpress backups and settled on this (modified) script to produce some easy tarball backups of the sites and their dabases before doing an update.

#!/bin/bash

ee="/opt/easyengine/sites"
backup="/home/backup"
sites=$(ls $ee)
year=`date +%Y`
month=`date +%m`
day=`date +%d`
date=`date +%Y-%m-%d`

mkdir -p $backup/$year/$month/$day

for i in $sites
do
    echo "Starting backup for $i"
    ee shell $i --command="wp db export ../$i.sql"
    cd $ee/$i/app && tar --create --gzip --file $i.tar.xz htdocs $i.sql
    rm $ee/$i/app/$i.sql
    mv $ee/$i/app/$i.tar.xz $backup/$year/$month/$day
done

These backup files can be inspected with tar --list --gzip --file <file> and extracted with tar --extract --gzip --file <file>

Update Easy Engine

Update using:

ee cli update

WordPress Application Updates

With backups in place it should be safe to run the automatic updates in the WP sites. I tried running the updates from the wordpress.com/jetpack interface. I verified the updates in the wp admin pages. The jetpack integration seemed to work, but required some page reloads before all the updates would apply. My plugins are set to autoupdate, so no manual intervention was required.

Checkout

A quick view of the sites didn’t show any problems. Disk usage on the droplet is at 50%. The bulk of usage is /var/lib/docker/overlay2. I’ll need to keep an eye on that, since high overlay usage is a sign that changed files are being written in the containers, but not in volumes. It looks like cache plugins are writing temp files in there, which isn’t a big deal, since they shouldn’t grow, but if temp files or log files are getting written, it could be a problem.

Cleanup

Reminder to cleanup old backups in later update runs.

After doing a Ubuntu update, I found my droplet unbootable. Digital Ocean provides a useful utility ISO for restarting a droplet in recovery mode, but by itself I wasn’t able to see or fix the problem. Luckily I found a blog post with the correct set of remounts and chrooting to be able to rerun apt and get everything running.

In order to get things running, you restart the droplet with the Recovery ISO as the startup drive, mount the main disk, choose 6 for interactive shell, then in the shell:

mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys
mount --bind /run /mnt/run"      
chroot /mnt
apt update
apt upgrade

Restart the droplet and boot from main harddisk (you have to power cycle, a soft reboot doesn’t do the trick). After rebooting, I reran the updates just to make sure:

apt update
apt upgrade
apt autoremove

I found that not all my docker containers would come back up, and traced the failure to insufficient peak memory. This lead me to move my swap to a larger swapfile to allow for greater headroom.

fallocate -l 3G /swapfile3g
ls -lh /swapfile
ls -lh /swapfile3g
chmod 600 /swapfile3g
mkswap /swapfile3g
swapon /swapfile3g
swapon -s
vi /etc/fstab
swapoff /swapfile

For a few days after moving my blogs to self-hosted WordPress, I was seeing somewhat frequent restarts of the MariaDB container. I didn’t see any real indications as to the cause, just a few sql formatting warnings.

I added swap to the host vm, thinking that it may have been memory pressure causing issues. Since I have done that, the problem seems to have gone away. EasyEngine recommends adding swap and a few other tuning tweaks if this problem recurs.

I’ve had a number of domains which I had hosted blogs on in the past and that have been lying fallow. I was pretty happy hosting them on wordpress.com, but the number of sites and the monthly cost per site were just too high to justify the cost of WordPress hosting for sites that were simple “for fun” web publishing. I’ve made several attempts at various static-site rendering solutions or Netflify JAMstack implementations, and while I think they are truly awesome, they also require a lot of development and implementation maintenance (tooling versions are constantly evolving and there isn’t much guarantee that content will continue to work over the long haul). I’ve also lost content in previous engine transfers (mostly from exporting from hosted blog platforms that ran out of VC money), so I suppose I’d rather stay with WordPress(.org) if I can. Eventaully someone may create a static rendering pluging for the WordPress engine and hosting this will all be easier with free or cheap static page CDNs and the engine hosted privately, but so fare those solutions do not seem fully baked and well supported.

Essential System Setup

I started with a 1gb Digital Ocean droplet with Ubuntu 20.04 installed and weekly backups configured. I set up a new ssh key and added a ssh config file entry for connect to the host:

Host wordpress
    HostName 167.172.199.82
    User root

The host needed initial package updates:

apt update
apt upgrade
apt install net-tools

I changed the ssh port as well, to reduce amount of useless security probing and port scanning. This isn’t a huge security measure, but I understand that there a lot of unsophisticated nuisance attacks.

vi /etc/ssh/sshd_config
# Uncomment and modify "Port" directive
service sshd restart

This required an update to my ssh config file as well.

EasyEngine Install

The basic install for EasyEngine 4 is a simple bash script download and execute. Before installing, there were 23gb free on my system drive. EasyEngine installs docker and pulls several images.

wget -qO ee rt.cx/ee4 && sudo bash ee

After this install I had 19gb free on my droplet drive.

Configuring a first site and importing from Tumblr

EasyEngine makes setting up a new site a single line command. I want to set this up with LetsEncrypt SSL, so the first step is to get DNS for joshrivers.me and www.joshrivers.me pointed at the IP address of the droplet. I’m disabling Cloudflare proxying here because it currently appears that this conflicts with getting LetsEncrypt setup.

With this configured, setting up the basic site is a single command:

ee site create joshrivers.me --wp --ssl=le

After a few minutes, and entering my email address for certificate renewal notification emails, I had a functional site. I verified that redirect-to-https was configured with curl -v http://www.joshrivers.me and the web server is supplying 301 redirects to the https version of the site.

First step after creating the new site is to login with the generated admin credentials and change the display name of the account (no point in having a randomized username and printing it on every web page).

Outbound email did not work out of the box with the builtin postfix. Trying a test send from the command line gave an error: server message: 451 4.3.0 <my address>: Temporary lookup failure. Installing cli tools into the postfix container shows that name resolution is working from there. It appears there is a configuration error with the dockerized postfix where email fails when the domain matches the hosted site. The errors go away when attempting to deliver to a different domain. Frustratingly, I still don’t see any email delivery at this point. Those emails may be being blocked due to my SPF configuration for my domains. There was a useful fix described on the community forum, but I still have no mail delivery.

Ultimately, I don’t think I need transactional emails from wordpress, so I’m going to leave it alone for now. I can dig at postfix and SPF later if it seems important.

Further settings to update:

• Site Timezone
• Disable Comments
• Day and Name Permalinks

Next I imported my old Tumblr Blog, which required removing the custom domain mapping in Tumblr and using an OAuth login. After importing I verified comments were disabled on all the old posts. And I deleted the “Hello World” post. I used a simpler theme for joshrivers.me than the default WP theme. It could use some work, but it is fine as a start, and at least as decent as my Tumblr theme was. Next I set up Jetpack.

Imports fron WordPress.com

For each of my additional sites, I ran through this outline:

• DNS Reconfiguration
• ee site create subvert.org --wp --ssl=le
• Initial Admin Settings: https://subvert.org/admin
  • Trash sample post and pages
  • Edit default user display names
  • Change site name and tag line
  • Correct site time zone
  • Disable comments
  • Day and Name permalinks
  • Delete default plugins
• Plugins
  • Disable Comments
  • JetPack
• Import
  • Importing with media import seems to work seamlessly. I got an error due to an attachement from my long ago import when I got onto wordpress.com but I don’t think anything was lost there.
  • Deactivate Importer Plugin
• Change default category

Added this to CSS to stop showing the (disabled!) comments link:

.post-comment-link {
  display: none !important;
}

Using the DB shell instructions from the EasyEngine Handbook, I was able to change the admin email address:

SELECT * FROM wp_users;
UPDATE wp_users SET user_email = "<my_address>" WHERE user_email="admin@subvert.org";
SELECT * FROM wp_options WHERE option_name="admin_email";
UPDATE wp_options set option_value = "joshrivers@me.com" WHERE option_name="admin_email";

I next tested this with the outbound email instructions. Still no mail delivery. This is bugging the heck out of me, but I also can’t think of a single reason I need to send emails from my blog server, so I should really just leave it alone, right?

Update: I also needed to do manual re-linking of images. The images exported and imported correctly, but all the image urls in the post records ended up pointing to the WordPress CDN rather than my webserver. The WordPress CDN appeared to not be serving to pages loaded from my server (referrer filtering?)