Set up a diskless "Alpine Linux" on a "Raspberry PI 4B" with an encrypted data disk

WARNING: I’ve had some major issues running Alpine Linux on my Raspberry PIs. Most incidents happen due to update process of diskless “Alpine Linux” and having only 4GB of memory. switched to Raspberry PI OS aka Debian. I’m done with Alpine Linux as regular operating system.

I purchased a document scanner for my home office some time ago. It allows me to digitize all my business letters and other important paper documents. The conversion of scanned documents into PDF files needs to be both fast and accurate. While I’m satisfied with the accuracy, my current setup is slow. It relies on a single virtual machine and follows a sequential process of working on one document at a time. Scanning and converting documents in this manner is time-consuming. Consequently, I decided to improve my setup and reconfigure the entire architecture.

I considered using “Mayan EDMS” for process automation and document management, but I encountered difficulties in setting up LDAP. As a result, I opted for a different approach. My new server setup will be based on multiple “Raspberry Pis 4B” (Raspberry) running “Alpine Linux” (Alpine) in its diskless mode. To automate the process, I will utilize n8n, a free and open workflow automation tool.

The article you’re currently reading outlines the foundations of my new setup, focusing specifically on the installation and configuration of Alpine Linux. In the future, I plan to delve into more details about the container setup and the overall scanning process. During the server setup, I encountered various challenges related to USB and udev, and I have already prepared a draft for an article discussing these issues.

Requisites for readers

This article is written for people with a basic understanding in “Linux” operating systems. Writing this article, I assume the reader does not have Alpine running on her/his workstation.

I added tags to clarify which commands have to be executed on which of your systems (see Fig. 1):

Workstation: Your local desktop computer or laptop which you use to set up the server
Server: The server you’re going to install. It is controlled directly via monitor and keyboard
Server (SSH): The server you’re going to install. It is controlled over ssh from your workstation

To enhance readability, I have omitted the use of the sudo prefix for commands executed as root. Instead, I’ve utilized the following syntax for the commands in this article. However, for your regular usage, I strongly recommend incorporating the sudo command for proper execution and authorization.

$ command : Running the command as a normal or admin user
# command : Running the command as root

Server setup

Prepare installation of the operating system

Workstation Prepare the microSD card
Nowadays, I utilize a 64 GB SanDisk Extreme A2 U3 microSD card for my setup, which has significantly lower “IOWAIT” times compared to the 32 GB SanDisk Extreme A1 U3 microSD card I previously used.
To create the root partition, you can use the fdisk command. While I won’t delve into detailed instructions here, there are numerous guides available on the internet that provide step-by-step guidance on this process.
```
# fdisk /dev/mmcblk0 --wipe auto
```
After following the steps outlined in the previous section using the fdisk command, the resulting output should resemble the following. It is crucial to note that during the process, ensure you select “EFI” as the partition type and “GPT” as the partition table type to ensure compatibility and proper functioning.
```
# fdisk -l /dev/mmcblk0

Disk /dev/mmcblk0: 59.48 GiB, 63864569856 bytes, 124735488 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt

Device            Start       End   Sectors  Size Type
/dev/mmcblk0p1     2048  20973567  20971520   10G EFI System
```
Workstation Format the SD card with “FAT32”
```
# sudo mkdosfs /dev/mmcblk0p1
```
Workstation Download the Alpine latest “aarch64” archive for the Raspberry PI
Download the latest “aarch64” archive from the Alpine download site.
Workstation Extract the files to SD card
First, mount the newly created partition to /mnt.
```
# mount /dev/mmcblk0p1 /mnt
```
Once the archive has been obtained, proceed to extract its files into the /mnt directory. Keep in mind that the specific path and file name may differ in your particular case, so be sure to adjust them accordingly based on your environment.
To extract the files, use the following command as an example:
```
# sudo tar -xzf ~/tmp/alpine-rpi-3.16.2-aarch64.tar.gz --no-same-owner -C /mnt .
# sync
```
Server Insert the microSD card cardholder of Raspberry
Server Connect the Raspberry to a monitor and keyboard
Server Connect the Raspberry to your local network by cable
Server Connect Raspberry to the power supply and boot it

Configure the base system

Server Run the initial setup for Alpine

Before proceeding with the next step, ensure that you have a functional internet connection.

Once you have confirmed a working network connection, proceed to run the Alpine-installer. This installation process will generate various files, including the repositories file for apk (Alpine Package Keeper), and initiate essential services such as an SSH server.

# setup-alpine

Keyboard layout: us (or whatever suits your preference)
Keyboard variant: us-altgr-intl (or whatever suits your preference)
Hostname: host1 (or whatever suits your preference)
Network interface: eth0
IP address: dhcp
Manual network configuration: no
Password: <Choose safe password>
Again password: <Same password>
Time zone: Europe/Berlin (or whatever suits your preference)
HTTP proxy: none
NTP client: chrony
Mirror: 1 (or whatever suits your preference)
SSH server: openssh
Setup disk: none
Store configs: mmcblk0p1
Apk cache directory: /media/mmcblk0p1/cache

Server Install the bash-shell
WEITER
IMPORTANT: The admin-user meant in this tutorial is NOT root, but a non-privileged user which is allowed to run privileged commands via sudo.
We are going to use the bash-shell for our scripts and as default-shell for the admin-user. To make use of bash, let’s install it first.
```
# apk add bash

(1/1) Installing bash (5.1.4-r0)
Executing bash-5.1.4-r0.post-install
Executing busybox-1.33.1-r2.trigger
OK: 390 MiB in 192 packages
```
Server Add shell history files for root to the overlay-file
All changes done to a diskless setup are recorded in a so-called overlay-file. The lbu-command manages this file. By default, only changes made to files in a few directories are added to the overlay-file — e.g. shell history files are not recorded. I find it very convenient to have access to commands run earlier, so let’s add the path of the history-files to the lbu-configuration file.
```
# lbu add /root/.ash_history
# lbu add /root/.bash_history
```
After you ran some helpful commands, you need to commit the changes to the history file with lbu commit, if you want them to be persisted.
```
# lbu commit
```
If you want to remove old apk overlay files, add the parameter “-d” and run the following command instead.
```
# lbu commit -d
```
You shall see an overlay-file on the partition you created earlier. This file contains all changed files.
```
# ls /media/mmcblk0p1/*.tar.gz

/media/mmcblk0p1/<Your Hostname>.apkovl.tar.gz
```
Server Reduce the message of the day
IMPORTANT: I’m a vim user, so I deliberately chose vi for this tutorial — vi is some kind of an ancestor of vim, has the same set of base functionality, usage patterns and is installed by default. Please do not hesitate to install and use your preferred editor.
Normally, the message of the day contains a bit of information. For daily use, it is not that helpful. So, let’s strip down the file to a minimum of information.
```
# vi /etc/motd
```
```
Welcome to Alpine!
```
Server Commit changes for the base setup
The next steps might break your setup. That is why this is a good time to commit your configuration.
```
# lbu commit -d
```

Server Upgrade base system

apk uses a database to look for installable packages. You need to download it upfront, before we can install packages.

# apk update

fetch http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/v3.14/main/aarch64/APKINDEX.tar.gz
3.14.0 [/media/mmcblk0p1/apks]
v3.14.0-154-gd627ad2cf8 [http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/v3.14/main]
OK: 4817 distinct packages available

Additionally, let’s upgrade all packages of the base installation.

# apk upgrade

Upgrading critical system libraries and apk-tools:
(1/1) Upgrading apk-tools (2.12.5-r1 -> 2.12.6-r0)
Executing busybox-1.33.1-r2.trigger
Continuing the upgrade transaction with new apk-tools:
(1/2) Upgrading busybox (1.33.1-r2 -> 1.33.1-r3)
Executing busybox-1.33.1-r3.post-upgrade
(2/2) Upgrading alpine-baselayout (3.2.0-r15 -> 3.2.0-r16)
Executing alpine-baselayout-3.2.0-r16.pre-upgrade
Executing alpine-baselayout-3.2.0-r16.post-upgrade
Executing busybox-1.33.1-r3.trigger
OK: 390 MiB in 192 packages

Set up the “OpenSSH”- daemon for the use with SSH-keys

In this section, we set up sshd — the “OpenSSH” daemon (ssh server). You are going to need this daemon to control the server remotely. The use of private/public key pairs for authentication is much more convenient than entering your password each time you log in to the server.

Server Fix configuration for the ssh server temporarily
IMPORTANT: Make sure you install your server in a secured network.
First, we need ssh-access as root to the server. The following configuration can be considered as insecure for internet-servers, but is sufficient for a setup of a server in a secured environment. This will activate the password-based login for root via SSH.
```
# vi /etc/ssh/sshd_config
```
```
PermitRootLogin yes
```
Server Restart the ssh-server
To activate the new configuration, please restart the ssh-server.
```
# rc-service sshd restart
```
Server Show the IP address of your server
Before you can proceed, you need to get the IP address of your server. Nowadays, I use ip for this, but ifconfig will work just as fine as well. Make sure you run this command on your server, NOT on your workstation.
```
# ip address

[... ]
2: eth0
inet 192.168.x.x
[... ]
```

Workstation Copy the SSH-public-key from your local system to your new server

IMPORTANT: I assume, you generated a public/private key pair with ssh-keygen upfront.

Please open a new terminal on your local system and copy your local public SSH-key to the server. You can either use scp or the special helper command ssh-copy-id — I recommend using the helper command. This command will add your public-key to /root/.ssh/authorized_keys.

# ssh-copy-id root@<IP address of your server>

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@<IP address of your server>'"
and check to make sure that only the key(s) you wanted were added.

Server (SSH) Add changes to overlay-file for passwordless access for root
Files in /root are not part of the regular Alpine’s “diskless mode” save-list. The following command will add the authorized_keys file of the root-user to the overlay-file.
```
# lbu add /root/.ssh/authorized_keys
```
Server (SSH) Fix configuration for the ssh server permanently
After you added your SSH-key, you can fix the sshd configuration. First, remove the following line.
```
# vi /etc/ssh/sshd_config
```
```
PermitRootLogin yes
```
Then, add one of the following lines instead. Option 1) will allow SSH-access for root via SSH public/private keys, but not with passwords, and option 2) will prevent login with the root user at all. 2) only makes sense if you configure a second non-privileged user — see this chapter for more information about how to add such an admin-user.
```
# vi /etc/ssh/sshd_config
```
```
# 1)
# public / private key only
PermitRootLogin prohibit-password

# or

# 2)
# no root login at all
PermitRootLogin no
```
Server (SSH) Commit changes for remote access permanently
```
# lbu commit -d
```

Server (SSH) Restart sshd

Again, restart sshd after you saved the configuration-file.

# rc-service sshd restart

* Stopping sshd ... [ ok ]
* Starting sshd ... [ ok ]

Workstation Connect to your server using SSH from your local system from a different terminal session
To verify a correctly working server, connect to the server using a second terminal session. Please create a new shell on your workstation and run the following command. It shall output the configured hostname of your server.
```
# ssh root@<IP address of your server> hostname

//<Hostname of your server>
```

Setup software repositories

Alpine supports multiple repositories to install software on your server. It has some predefined ones. Some of those are enabled, some are disabled by default.

Workstation Enable disabled repositories
Please enable the community and the edge-repositories, but “guard” the “edge” repositories with @<tag>. You might need the “edge” repositories for software which is not part of the regular distribution. Run apk add <packagename>@<repository-tag> to install a software package from the given repository. Your file might look a bit different, if you chose another mirror, run another version of Alpine or word your tags differently.
```
# vi /etc/apk/repositories
```
```
/media/mmcblk0p1/apks
http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/v3.14/main
http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/v3.14/community
@edge-main http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/edge/main
@edge-community http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/edge/community
@edge-testing http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/edge/testing
```

Server (SSH) Update repository metadata

apk requires information about packages it can install. To fetch this kind of information, run the following command. There’s a shortcut for this as well: Run apk add -u <package> to update the package information and install the package in one go.

# apk update

fetch http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/v3.14/main/aarch64/APKINDEX.tar.gz
fetch http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/v3.14/community/aarch64/APKINDEX.tar.gz
fetch http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/edge/main/aarch64/APKINDEX.tar.gz
fetch http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/edge/community/aarch64/APKINDEX.tar.gz
fetch http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/edge/testing/aarch64/APKINDEX.tar.gz
3.14.0 [/media/mmcblk0p1/apks]
v3.14.0-126-g36dbfbf5fc [http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/v3.14/main]
v3.14.0-127-g2246237cc9 [http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/v3.14/community]
v3.14.0-2577-gc799bdc805 [http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/edge/main]
v3.14.0-2586-g4ea15ffb4f [http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/edge/community]
v3.14.0-2586-g4ea15ffb4f [http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/edge/testing]
OK: 33102 distinct packages available

Prepare the environment for the encrypted data storage

I prefer to encrypt the disks where I store user data. Before you can create the encrypted disk itself, you need to set up some tools and configurations.

Server (SSH) Install the util-linux package to have some handy tools on your server to gather information about your storage setup.

# apk add util-linux
# # or
# apk add -u util-linux

(1/18) Installing blkid (2.37-r0)
(2/18) Installing libcap-ng (0.8.2-r0)
(3/18) Installing setpriv (2.37-r0)
(4/18) Installing libsmartcols (2.37-r0)
(5/18) Installing findmnt (2.37-r0)
(6/18) Installing mcookie (2.37-r0)
(7/18) Installing hexdump (2.37-r0)
(8/18) Installing lsblk (2.37-r0)
(9/18) Installing libfdisk (2.37-r0)
(10/18) Installing sfdisk (2.37-r0)
(11/18) Installing cfdisk (2.37-r0)
(12/18) Installing partx (2.37-r0)
(13/18) Installing flock (2.37-r0)
(14/18) Installing logger (2.37-r0)
(15/18) Installing uuidgen (2.37-r0)
(16/18) Installing libeconf (0.3.8-r1)
(17/18) Installing util-linux (2.37-r0)
(18/18) Installing util-linux-openrc (2.37-r0)
OK: 384 MiB in 176 packages

Server (SSH) Get information about your SD card

We need to find out the name of the SD card where we already installed Alpine.

# lsblk

NAME          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
loop0           7:0    0 26.9M  1 loop  /.modloop
mmcblk0       179:0    0 59.5G  0 disk
└─mmcblk0p1   179:1    0   10G  0 part  /media/mmcblk0p1

Server (SSH) Create partitions on SD card

Run fdisk to create partitions on the SD card.

# fdisk /dev/mmcblk0

The result of your fdisk steps should be something similar to this.

# fdisk -l /dev/mmcblk0

Disk /dev/mmcblk0: 59.48 GiB, 63864569856 bytes, 124735488 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt

Device            Start       End   Sectors  Size Type
/dev/mmcblk0p1     2048  20973567  20971520   10G EFI System
/dev/mmcblk0p2 21075968 124735454 103659487 49.5G Linux filesystem

Server (SSH) Commit changes permanently
```
# lbu commit -d
```

Create the encrypted disk

Server (SSH) Create directory for the encryption key
First, create the directory where you store the key.
```
# mkdir -p /etc/crypt-config/
```

Server (SSH) Set permissions for the directory

# chown 0700 /etc/crypt-config/

# ls -al /etc/crypt-config/ -d

drwx------ 2 root root 60 Jan  1  1970 /etc/crypt-config/

Server (SSH) Create key for the disk encryption
Next, create the key itself — do not redirect the output of openssl directly to a file, as the command appends a newline at the end of the string — this will cause problems if you enter the key manually without an additional newline at the end.
```
# openssl rand -hex 32

2d2aefb3ae7266f16cb3a39e79ea72f99ecf3e6a15194bed96156b0a02c877d0
```
Copy the key and write it to a file using echo.
```
# echo -n <key> > /etc/crypt-config/mount.key
```
Verify there’s no 0a (newline) at the end of the file.
```
# xxd /etc/crypt-config/mount.key
```

Server (SSH) Set permissions for the key file

# chmod 0600 /etc/crypt-config/mount.key

# ls -al /etc/crypt-config/mount.key

-rw------- 1 root root 64 Jul 15  2021 /etc/crypt-config/mount.key

Server (SSH) Format the disk as LUKS disk

Subsequently, create the encrypted disk using cryptsetup. Please double-check you have chosen the correct partition.

# cryptsetup luksFormat /dev/mmcblk0p2 --key-file /etc/crypt-config/mount.key

WARNING!
========
This will overwrite data on /dev/mmcblk0p2 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES

Server (SSH) Configure dmcrypt to open the disk on boot
Configure dmcrypt to open the encrypted disk. It requires a name as target for cryptsetup, a source-device and the key-file we created before.
```
# vi /etc/conf.d/dmcrypt
```
```
target=cryptdata
source='/dev/mmcblk0p2'
key='/etc/crypt-config/mount.key'
```

Server (SSH) Enable the service to open the encrypted disk during startup

# rc-update add dmcrypt boot

* service dmcrypt added to runlevel boot

# rc-service dmcrypt start

* Setting up dm-crypt mappings ...
*   cryptdata using:   open /dev/mmcblk0p3 cryptdata ...

Server (SSH) Verify the encrypted disk could be opened with the configured key-file
```
# dmsetup.static  ls

cryptdata       (254:0)
```
Server (SSH) Secure the <hostname>.apkovl.tar.gz with a password
We have created the key-file and placed it in the overlay-file unencrypted, but secured by file system permissions. There’s one further action we can take to secure the “data in rest” on the MicroSD-card.
lbu can encrypt the overlay-file with a password. You need to enter this password on every boot of your system. It’s secure, but not a very convenient solution for a server setup. I’m not aware of any other more secure and better automated solution. This command needs to the very last and is presented here only for a better understanding.
```
# lbu commit -e -d

enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.
```

Set up LVM on top of the encrypted disk

The next steps will create a very basic LVM setup. You can add as many logical volumes as your setup requires.

Server (SSH) Create physical volume

# pvcreate /dev/mapper/cryptdata

Physical volume "/dev/mapper/cryptdata" successfully created.

Server (SSH) Create volume group

# vgcreate vg_data /dev/mapper/cryptdata

Volume group "vg_data" successfully created

Server (SSH) Create base for mount points
```
# mkdir -p /storage/
```
Server (SSH) Make mount points persistent across reboots
```
# touch /storage/.keep
```
Server (SSH) Add directory to save list
To make sure your mount points are part of the overlay-file, please run the following commands.
```
# lbu add /storage/.keep
```
Server (SSH) Commit changes to disk
Thereafter, please commit all changes to disk.
```
# lbu commit -d
```

Customize system

Add user for normal administration tasks

Server (SSH) Add user

# adduser -s /bin/bash user

Changing password for user
New password:
Retype password:
passwd: password for user changed by root

Server (SSH) Setup admin group for the use with sudo

# vi /etc/sudoers

%sudo ALL=(ALL) NOPASSWD: ALL

Server (SSH) Add user user to admin group

# apk add shadow

# addgroup sudo

# gpasswd -a user sudo

# /etc/group
sudo:x:1001:user

Server (SSH) Add the shell history to the overlay-file
```
# lbu add /home/user/.bash_history
```

Server (SSH) Setup passwordless access for user user

ssh-copy-id user@host1
# => /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
# => /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
# => user@host1's password:
# =>
# => Number of key(s) added: 1
# =>
# => Now try logging into the machine, with:   "ssh 'user@host1'"
# => and check to make sure that only the key(s) you wanted were added.

Server (SSH) Add changes to overlay-file
Files in /home/user/ are not part of the regular “diskless mode” save list. These commands will save root’s authorized_keys file to the overlay-file.
```
# lbu add /home/user/.ssh/authorized_keys
```

Configure logging

Server (SSH) Install rsyslog

# apk add rsyslog

rlpscan001:/home/user# apk add rsyslog
(1/4) Installing libestr (0.1.11-r1)
(2/4) Installing libfastjson (0.99.9-r0)
(3/4) Installing rsyslog (8.2012.0-r1)
(4/4) Installing rsyslog-openrc (8.2012.0-r1)
Executing busybox-1.33.1-r3.trigger
OK: 390 MiB in 192 packages

Server (SSH) Disable and Stop syslog service

# rc-update del syslog boot

* service syslog removed from runlevel boot

# rc-service syslog stop

* Stopping busybox syslog ... [ ok ]

Server (SSH) Enable and Run rsyslog service

# rc-update add rsyslog boot

* service rsyslog added to runlevel boot

Server (SSH) Run rsyslog service

# rc-service rsyslog start

* Starting rsyslog ... [ ok ]

Server (SSH) Install logrotate

Adding logrotate helps to reduce the size of long-running daemons. If you need to reboot the system, all logs are gone due to the diskless mode.

# apk add logrotate

(1/2) Installing logrotate (3.18.1-r0)
(2/2) Installing logrotate-openrc (3.18.1-r0)
Executing busybox-1.33.1-r3.trigger
OK: 390 MiB in 192 packages

Server (SSH) Configure logrotate

# vi /etc/logrotate.conf

daily
size 10M
rotate 7
missingok

Prepare for kernel updates

Server (SSH) Add mkinitfs to update the kernel and rebuild the “modloop”-image

# apk add mkinitfs

(1/12) Installing lddtree (1.26-r2)
(2/12) Installing xz-libs (5.2.5-r0)
(3/12) Installing kmod (27-r0)
(4/12) Installing kmod-openrc (27-r0)
(5/12) Installing libblkid (2.35.2-r0)
(6/12) Installing argon2-libs (20190702-r1)
(7/12) Installing device-mapper-libs (2.02.186-r1)
(8/12) Installing json-c (0.14-r1)
(9/12) Installing libuuid (2.35.2-r0)
(10/12) Installing cryptsetup-libs (2.3.2-r0)
(11/12) Installing kmod-libs (27-r0)
(12/12) Installing mkinitfs (3.4.5-r3)
Executing mkinitfs-3.4.5-r3.post-install
Executing busybox-1.31.1-r16.trigger
OK: 46 MiB in 47 packages

Server (SSH) Add helper script to update the kernel

# vi /usr/local/bin/update-my-kernel

#!/bin/bash

set -e

if [ $(/usr/bin/id -u) -ne 0 ]; then
  echo "cmd=${0} msg=\"command needs be run as root\"" status=1
  exit 1
fi

source /etc/update-my-kernel.conf

if [ -z "${KERNEL_MODS_ENABLED}" ]; then
  echo "cmd=${0} msg=\"\$KERNEL_MODS_ENABLED is undefined\"" status=1
  exit 1
fi

/bin/mount -o remount,rw "${UPDATE_KERNEL_CONFIG_MNT_POINT}"
/sbin/update-kernel -F "${KERNEL_MODS_ENABLED}" ${UPDATE_KERNEL_ARGS} "${UPDATE_KERNEL_CONFIG_MNT_POINT}/boot/"

Server (SSH) Make the helper script executable

# chmod +x /usr/local/bin/update-my-kernel

# ls -al /usr/local/bin/update-my-kernel

-rwxr-xr-x 1 root root 477 Jun  8 19:10 /usr/local/bin/update-my-kernel

Server (SSH) Add configuration file for helper script
This helps to re-use the script for multiple servers. The configuration file contains the differences between my server. It depends on your setup, what modules etc. you configure in this file.
```
# vi /etc/update-my-kernel.conf
```
```
KERNEL_MODS_ENABLED="ata base cdrom lvm xfs keymap kms scsi usb"
UPDATE_KERNEL_ARGS="-p xfsprogs -p lvm2"
UPDATE_KERNEL_CONFIG_MNT_POINT="/media/mmcblk0p1/"
```
Server (SSH) Add local scripts to overlay-file
```
# lbu add /usr/local/bin/
```

Server (SSH) Enable the lvm service to activate LVM devices during boot

# rc-update add lvm boot

* service lvm added to runlevel boot

Server (SSH) Commit changes to disk
Subsequently, please commit all changes to disk.
```
# lbu commit -d
```
Server (SSH)(optional) Work around low main memory
The script above requires enough free space in your main memory (RAM) to run without errors. You need to modify /sbin/update-kernel to write to local storage, e.g. /storage/tmp if your Raspberry has not enough main memory.
First, create a directory on a persistent storage.
```
# mkdir /storage/tmp
```
Next, look for TMPDIR=$(mktemp -d /tmp/$SCRIPT.XXXXXX) in /sbin/update-kernel. You need to change this line manually with your preferred editor.
```
TMPDIR=$(mktemp -d /storage/tmp/$SCRIPT.XXXXXX)
```

Server (SSH) Run the script

After that change, run the script to check if it’s working correctly.

# /usr/local/bin/update-my-kernel

Parallel mksquashfs: Using 4 processors
Creating 4.0 filesystem on /tmp/update-kernel.5vy2cA/boot/modloop-rpi4, block size 131072.
[========================================================================/] 2268/2268 100%

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 131072
        compressed data, compressed metadata, compressed fragments,
        compressed xattrs, compressed ids
        duplicates are removed
Filesystem size 27589.10 Kbytes (26.94 Mbytes)
        28.07% of uncompressed filesystem size (98292.55 Kbytes)
Inode table size 17948 bytes (17.53 Kbytes)
        24.10% of uncompressed inode table size (74468 bytes)
Directory table size 22158 bytes (21.64 Kbytes)
        43.62% of uncompressed directory table size (50802 bytes)
Number of duplicate files found 17
Number of inodes 2262
Number of files 1883
Number of fragments 386
Number of symbolic links  0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 379
Number of ids (unique uids + gids) 1
Number of uids 1
        root (0)
Number of gids 1
        root (0)

Prepare for package updates

Server (SSH) Create helper script for package updates

# vi /usr/local/bin/update-packages

#!/bin/bash

set -e

if [ $(/usr/bin/id -u) -ne 0 ]; then
  echo "cmd=$0 msg=\"command needs be run as root\"" status=1
  exit 1
fi

/sbin/apk update
/sbin/apk upgrade

Server (SSH) Make the helper script executable

# chmod +x /usr/local/bin/update-packages

# ls -al /usr/local/bin/update-packages

-rwxr-xr-x 1 root root 477 Jun  8 19:30 /usr/local/bin/update-packages

Server (SSH) Commit changes to disk
Following, please commit all changes to disk.
```
# lbu commit -d
```

Handle missing packages on startup

Server (SSH) Add helper script for package installation

On several occasions, Alpine did not install all packages. To work around this problem, add a helper script. This script tries to add all packages and start all services for the “default” boot level.

# vi /etc/local.d/01-packages.start

#!/bin/bash

# redirect logs
exec &> >(tee -a "/var/log/helper-packages.log")

echo msg='install required packages for alpine linux' level=INFO
/sbin/apk add

echo msg='start services for runlevel default' level=info
/sbin/openrc default

# chmod +x /etc/local.d/01-packages.start

# ls -al /etc/local.d/01-packages.start

-rwxr-xr-x 1 root root 985 Feb 22 18:58 /etc/local.d/01-packagesg.start

Server (SSH) Configure logrotate

# vi /etc/logrotate.d/helper-scripts

/var/log/helper-*.log {}

Server (SSH) Activate the local service
The local service runs files in /etc/local.d. Please read the README file in the same directory for more information about how the service works.
```
# rc-update add local

* service local added to runlevel default
```

Setup debug tools

Server (SSH) Create a “debug helper”-installer-script

# vi /usr/local/bin/setup-debug-utils

#!/bin/bash

set -e

if [ $(/usr/bin/id -u) -ne 0 ]; then
  echo $0: run as root
  exit 1
fi

if [ -z "$(/sbin/apk info | grep bind-tools)" ]; then
  echo Install debug packages
  /sbin/apk add bind-tools curl gptfdisk iftop iotop iproute2 iptraf-ng ncdu procps strace usbutils tshark
else
  echo Remove debug packages
  /sbin/apk del bind-tools curl gptfdisk iftop iotop iproute2 iptraf-ng ncdu procps strace usbutils tshark
fi

Server (SSH) Make the debug helper installer script executable
```
# chmod +x /usr/local/bin/setup-debug-utils
```

Add helper for system temperature

Server (SSH) Install the Raspberry support tools

# apk add raspberrypi

(1/2) Installing raspberrypi-libs (0.20200813-r0)
(2/2) Installing raspberrypi (0.20200813-r0)
OK: 390 MiB in 194 packages

Server (SSH) Create a helper script to gather and parse the temperature

# vi /usr/local/bin/measure-temperature

#!/bin/bash

if [ $(id -u) -ne 0 ]; then
  echo "cmd=$0 msg=\"command needs be run as root\"" status=1
  exit 1
fi

/opt/vc/bin/vcgencmd measure_temp | egrep -o '[0-9]*\.[0-9]*'

Server (SSH) Make the “debug helper”-installer-script executable
```
# chmod +x /usr/local/bin/measure-temperature
```

Server (SSH) Run the script

# /usr/local/bin/measure-temperature

42.3

Setup data-disk

Server (SSH) Create logical volume for the data disk

# lvcreate -L 20G -n lv_data vg_data

Logical volume "lv_data" created.

Server (SSH) Create the file system in this disk

I prefered XFS on my servers. Important: Recently, I faced the second totally broken XFS after a kernel update. I’m not sure what caused the broken filesystem. But I never had that much trouble with EXT4.

# mkfs.xfs /dev/mapper/vg_data-lv_data

meta-data=/dev/mapper/vg_data-lv_data isize=512    agcount=4, agsize=1310720 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=5242880, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Server (SSH) Add new disks to /etc/fstab
Hint: Using UUIDs does not work and fails during boot.
```
/dev/mapper/vg_data-lv_data /storage/data xfs defaults 0 2
```
Server (SSH) Create mount points
```
# mkdir -p /storage/data
```
Server (SSH) Make mount points persistent across reboots
```
# touch /storage/data/.keep
```
Server (SSH) Add directory to save list
To make sure your mount points are part of the overlay-file, please run the following commands.
```
# lbu add /storage/data/.keep
```
Server (SSH) Commit changes to disk
Next, please commit all changes to disk.
```
# lbu commit -d
```

Server (SSH) Mount disks to verify the setup

Now you can use the mount command to verify all disks can be mounted.

# mount -a

Your setup should look similar.

# mount | grep storage

/dev/mapper/vg_data-lv_data on /storage/data type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)

Server (SSH) Reboot
At the very end, please reboot your computer to see if your setup works correctly. You should see no errors on mounting the disks in the start-up logs. After the reboot, you should see a login prompt.
```
# reboot
```

Finish configuration

Server (SSH) Commit changes and encrypt the overlay-file with a password

$ lbu commit -e -d

enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
*** WARNING : deprecated key derivation used.
Using -iter or -pbkdf2 would be better.

Conclusion

Now, you’ve got a configured Alpine base-system running. You can further customize your setup — e.g. by installing some container runtime etc. to run your containers.