Nov 3, 2022 - Linux time machine

Linux time machine

A few days ago my nextcloud server broke apart, completely. It had to be re-installed from scratch. I hadn’t realised that my backup wasn’t working, so I didn’t have a mysql dump. Luckily I still could mount the harddisk and rescue the /var/lib/mysql directory, with all the *.frm, *.ibd, etc. database files.

I re-installed the server and tried to simply copy the recovered database files over. Of course that didn’t work, because the database files were created with an older mariadb version. At least the error message said exactly what version they were created with. So I had to get the same version again, then I could spin up the database and do a mysqldump, which I then can re-import into the up-to-date mariadb again.

On a Debian system that’s basically impossible. Unless you’re lucky and a Debian release has exactly the version you need. You can get these older versions from previous releases or a newer version by enabling testing repositories. But even if you could get the version you need, if you’ve ever tried that, you know what a pain this is! Dependency hell!

But there’s Arch to the rescue! So I span up an Arch VM (see Vagrant file at the bottom). Went to the Arch archive: Arch Archive . Found the mariadb package of the specific version and noted the date when it was added. SSH’d into the Arch VM and adjusted the /etc/pacman.d/mirrorlist with the specific date URL e.g. Server=https://archive.archlinux.org/repos/2014/03/30/$repo/os/$arch. Then got the past keyring with pacman -S archlinux-keyring ca-certificates followed by a complete reset of the system to the past with pacman -Syyuu. And I had a linux system with the exact version of mariadb and all the necessary dependencies I wanted. Truely awesome! I don’t know of any other linux distribution which can do this!

Detailed instructions here: How to restore all packages to a specific date

Here’s a Vagrant file if you quickly wanna spin up an Arch VM:

Vagrant.configure("2") do |config|
  config.vm.box = "archlinux/archlinux"
  config.vm.network "private_network", ip: "192.168.56.100", :name => 'vboxnet0', :adapter => 2

  config.vm.provider "virtualbox" do |vb|
    vb.name = "archy"
    vb.memory = "4096"
    vb.cpus = "2"
  end

  config.vm.provision "shell", inline: <<-SHELL
    sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/g' /etc/ssh/sshd_config
    echo 'MaxAuthTries 100' >> /etc/ssh/sshd_config
    systemctl restart sshd.service
  SHELL
end

You can then simply ssh into vagrant@192.168.56.100 (this is using Virtualbox with a ‘vboxnet0’ host network setup); I don’t like the vagrant ssh workaround.

Nov 2, 2022 - Post installation script

Post installation script

In several earlier blog posts I mentioned a few things to do after you set up a new server: Hardening sshd config, enable firewall, etc.

Here’s a compact form of all this as a script, which you can customise and simply run after an installation. It will create a specific user, and only this user is allowed to ssh with public-key authentication (password authentication will be disabled). UFW will be installed and enabled as firewall. Fail2ban installed and enabled as basic brute-force protection. And a classic LAMP stack will be installed. On a debian system you should only have to edit the ‘Options’ section. For other distros you obviously have to adjust the script accordingly.

#!/bin/bash

###### Options

user="SOME_USER"

key="SOME_USERS_SSH_KEY"

aptpackages="vim screen lynx mariadb-server mariadb-client apache2 libapache2-mod-php php" # classic LAMP server

snappackages="certbot" # Have to run 'certbot --apache' manually later

######


# redirect all output to log file
exec >> /root/post_install.log
exec 2>&1


# update base installation
apt-get -y -q update
apt-get -y -q upgrade


# add the main user
adduser --disabled-password --gecos "" $user
mkdir /home/$user/.ssh
chmod 700 /home/$user/.ssh
echo "$key" >> /home/$user/.ssh/authorized_keys
chmod 644 /home/$user/.ssh/authorized_keys
ssh-keygen -t rsa -N '' -f /home/$user/.ssh/id_rsa
chown -R $user:$user /home/$user/.ssh


# harden ssh config
sed -i "s/.*PubkeyAuthentication.*/PubkeyAuthentication yes/g" /etc/ssh/sshd_config
sed -i "s/.*PasswordAuthentication.*/PasswordAuthentication no/g" /etc/ssh/sshd_config
sed -i "s/.*PermitRootLogin.*/PermitRootLogin no/g" /etc/ssh/sshd_config
echo "AllowUsers $user" >> /etc/ssh/sshd_config
echo "$user      ALL=(ALL)       NOPASSWD: ALL" >> /etc/sudoers


# setup firewall
apt-get -y -q install ufw
ufw default deny incoming
ufw allow ssh
ufw allow http
ufw allow https
ufw --force enable


# setup fail2ban
apt-get -y -q install fail2ban
cat <<EOT > /etc/fail2ban/jail.local
[DEFAULT]
bantime = 1h

[sshd]
enabled = true

[apache-auth]
enabled  = true

[apache-badbots]
enabled  = true

[apache-noscript]
enabled  = true

[apache-overflows]
enabled  = true
EOT
systemctl reload fail2ban


# Install additional packages
if [ -n "$aptpackages" ]
then
  apt-get -y -q install $aptpackages
fi


# Install snap packages
if [ -n "$snappackages" ]
then
  apt-get -y -q install snapd
  snap install core
  snap refresh core
  snap install --classic $snappackages
fi

Mar 16, 2022 - Some more find snippets

Some more find snippets

Exec complex commands with find

find * -exec is a great way to run a simple command on multiple files, like shown in Aug 21, 2020 - More Unix tools. But what to do if you want to run more complex commands, where the “file” variable isn’t necessary at the end of the command, or you need more variables?

Lets say you have a similar task, a lot of tar files you want to extract. But now each tar file is in it’s own directory and you want to extract the files inside there. tar has the -C target_directory option which can do that, so for a single file it would be tar xJf SomeDirectory/Something.tar.xz -C SomeDiretory. But how do you do this in combination with find * -exec for multiple files/directories?

You have to do this via -exec sh -c ''! For example:

find * -type f -name "*.xz" -exec sh -c 'dir=$(dirname "$0"); tar xJf "$0" -C "$dir"' {} \;

As usual you pass in the found file as last argument via {} \; to the exec command. But then in that case the exec command launches a shell which takes it in and passes it on to the shell script you specify in -c '...' as first argument $0.

Find + awk to extract information via regex groups

Recently I had to extract the image dimensions from a few thousand of tif images. Again find came to the rescue, in combination with awk. As single and double quotes and there combinations are quite important for both, find * -exec didn’t work directly in that case. I had to iterate over the find output with a for loop.

echo "Filename , Width , Height" >> image_dims.csv"
for i in `find * -type f -name "*.tif"`; do echo -n "$i , "; tiffinfo $i | head -3 | tr '\n' ' ' | awk 'match($0,/.+Width:\ ([0-9]+).+Length:\ ([0-9]+).+/,a) {print a[1],",",a[2]}' >> image_dims.csv; done;

To break it down:

find * -type f -name "*.tif" generates a list of the tif files, for iterates over them and runs the commands between the do and done.

tiffinfo spits out lots of information about an tif image (but only the first 3 lines are needed), e.g.:

TIFF Directory at offset 0x8 (8)
  Subfile Type: multi-page document (2 = 0x2)
  Image Width: 2960 Image Length: 2960

tr '\n' ' ' removes the linebreaks, so everything’s on one line.

awk 'match($0,/.+Width:\ ([0-9]+).+Length:\ ([0-9]+).+/,a) is looking for two groups of decimals, one after “Width: “ and one after “Length: “. These two values will be stored in the array a.

print a[1],",",a[2] simply prints the two values.