Birchwood Abbey Networks

The monks of the abbey are masters of the staff (bo) and Emacs.

---
- name: Install Emacs.
  become: yes
  apt: pkg=emacs

The abbey uses several additional email aliases. These are the public mailboxes @birchwood-abbey.net. The institute already funnels the common mailboxes like postmaster and admin into root and root to the machine's privileged account (sysadm). The abbey takes it from there, forwarding sysadm to a real person.

- name: Install abbey email aliases.
  become: yes
  blockinfile:
    block: |
        sysadm:         matt
        keymaster:      root
        codemaster:     matt
        all:            matt, lori, erica
        elders:         matt, lori
        rents:          elders
        puck:           matt
        abbess:         lori
    dest: /etc/aliases
    marker: "# {mark} ABBEY MANAGED BLOCK"
  notify: New aliases.

---
- name: New aliases.
  become: yes
  command: newaliases

The abbey publishes member Git repositories with git-daemon. If Dick (a member of A Small Institute) builds a Foo project Git repository in ~/foo/, he can publish it to the campus by symbolically linking its .git/ into ~/Public/Git/ on Core. If the repository is world readable and contains a git-daemon-export-ok file, it will be served at git://www/~dick/foo.

touch ~/foo/.git/git-daemon-export-ok
ln -s ~/foo/.git ~/Public/Git/foo
chmod -R o+r ~/foo/.git
find ~/foo/.git -type d -print0 | xargs -0 chmod o+rx

User repositories can be made available to the public at a URL like git://small.example.org/~dick/foo by copying it to the same path on Front (~dick/Public/Git/foo/). The following rsync command creates or updates such a copy.

rsync -av ~/foo/.git/ small.example.org:Public/Git/foo/

Note that Dick's Git repository, mirrored to Front (or Core), does not need to be backed up, assuming Dick's home directory (including ~/foo/) is. If updates are git-pushed to a repository on Front, regular backups should be made, but this is Dick's responsibility. There are no regular, system backups on Front.

rsync -av --del small.institute.org:Public/foo/ ~/Public/foo/

With SystemD and the git-daemon-sysvinit package installed, SystemD supervises a git-daemon service unit launched with /etc/init.d/git-daemon. The old SysV init script gets its configuration from the customary /etc/default/git-daemon file. The script then constructs the appropriate git-daemon command. The git-daemon(1) manual page explains the command options in detail. As explained in /usr/share/doc/git-daemon-sysvinit/README.Debian, the service must be enabled by setting GIT_DAEMON_ENABLE to true. The base path is also changed to agree with gitweb.cgi.

User repositories are enabled by adding a user-path option and disabling the default whitelist. To specify an empty whitelist, the default (a list of one directory: /var/lib/git) must be avoided by setting GIT_DAEMON_DIRECTORY to a blank (not empty) string.

The code below is included in both Front and Core configurations, which should be nearly identical for testing purposes. Rather than factor out small roles like abbey-git-server, Emacs Org Mode's Noweb support does the duplication, by multiple references to code blocks like git-tasks and git-handlers.

<<git-tasks>>

- name: Install git daemon.
  become: yes
  apt: pkg=git-daemon-sysvinit

- name: Configure git daemon.
  become: yes
  lineinfile:
    path: /etc/default/git-daemon
    regexp: "{{ item.patt }}"
    line: "{{ item.line }}"
  loop:
  - patt: '^GIT_DAEMON_ENABLE *='
    line: 'GIT_DAEMON_ENABLE=true'
  - patt: '^GIT_DAEMON_OPTIONS *='
    line: 'GIT_DAEMON_OPTIONS="--user-path=Public/Git"'
  - patt: '^GIT_DAEMON_BASE_PATH *='
    line: 'GIT_DAEMON_BASE_PATH="/var/www/git"'
  - patt: '^GIT_DAEMON_DIRECTORY *='
    line: 'GIT_DAEMON_DIRECTORY=" "'
  notify: Restart git daemon.

- name: Create /var/www/git/.
  become: yes
  file:
    path: /var/www/git
    state: directory
    group: staff
    mode: u=rwx,g=srwx,o=rx

<<git-handlers>>

- name: Restart git daemon.
  become: yes
  command: systemctl restart git-daemon

The abbey provides an HTML interface to members' public Git repositories using gitweb.cgi, one of the few CGI scripts allowed on Front. Unlike the Git daemon, the Gitweb interface does not care if the repository contains a git-daemon-export-ok file.

Again Front and Core need to be configured congruently, so the necessary Apache directives are given here and referenced in the Apache configurations.

Like the suggested per-user rewrite rule in the gitweb(1) manual page, the second RewriteRule specifies the root directory of the user's public Git repositories via the GITWEB_PROJECTROOT environment variable. It makes http://www/~dick/git run Gitweb with the project root ~dick/Public/Git/, the same directory the git-daemon makes available. The first RewriteRule directs URLs with no user name to the default. Thus http://www/git lists the repositories found in /var/www/git/.

Alias /gitweb-static/ /usr/share/gitweb/static/
<Directory "/usr/share/gitweb/static/">
    Options MultiViews
</Directory>
RewriteEngine on
RewriteRule ^/git(/.*)?$ \
            /cgi-bin/gitweb.cgi$1 [QSA,L,PT]
RewriteRule ^/\~([^\/]+)/git(/.*)?$ \
            /cgi-bin/gitweb.cgi$2 \
            [QSA,E=GITWEB_PROJECTROOT:/home/$1/Public/Git/,L,PT]

The RewriteRule flags used here are:

QSA | qsappend

Append the request's query string.

E= | env

Set or unset an environment variable.

L | last

Stop with this Last rule.

PT | passthrugh

Treat the result as a URI, not a file path.

The RewriteEngine on directive must be included in the virtual host or no rewriting will take place.

The CGI script and RewriteRule require Apache's cgi and rewrite modules, which are not normally enabled on a small institute's public server. Thus they need to be enabled here. Note that Debian and -Ubuntu install different Apache MPMs (multi-processing modules) -requiring different CGI modules, turning two tasks into three.

The script uses the CGI Perl module, which must be installed.

The rewrite rule maps to the URL /cgi-bin/gitweb.cgi, which is mapped by default to /usr/lib/cgi-bin/gitweb.cgi. The git package installs gitweb.cgi in /usr/share/gitweb/, so it and its related index.cgi script are linked into /usr/lib/cgi-bin/.

The static/ directory, also installed in /usr/share/gitweb/, is made available as http://www/gitweb-static/ via an Alias directive. The global Perl configuration file, /etc/gitweb.conf, overrides the relative URLs Gitweb normally generates, and uses the web site /favicon.ico.

- name: Enable Apache2 rewrite module for Gitweb.
  become: yes
  apache2_module: name=rewrite
  notify: Restart Apache2.

- name: Enable Apache2 cgid module.
  become: yes
  apache2_module: name=cgid
  notify: Restart Apache2.

- name: Install libcgi-pm-perl for Gitweb.
  become: yes
  apt: pkg=libcgi-pm-perl

- name: Link Gitweb into /cgi-bin/.
  become: yes
  file:
    state: link
    path: /usr/lib/cgi-bin/{{ item }}
    src: /usr/share/gitweb/{{ item }}
  loop: [ gitweb.cgi, index.cgi ]

- name: Override Gitweb assets location.
  become: yes
  copy:
    content: |
      $projectroot = $ENV{'GITWEB_PROJECTROOT'} || "/var/www/git";
      @stylesheets = ("/gitweb-static/gitweb.css");
      $logo = "/gitweb-static/git-logo.png";
      $favicon = "/favicon.ico";
      $javascript = "/gitweb-static/gitweb.js";
    dest: /etc/gitweb.conf
    mode: u=rw,g=r,o=r

- name: Restart Apache2.
  become: yes
  systemd:
    service: apache2
    state: restarted

Some of the directives added to the -vhost.conf file are needed by the abbey's documentation, published at https://birchwood-abbey.net/Abbey/. The following template uses a docroot variable for the actual path to the HTML. On Front this variable is set to /home/www. The same template is used on Core, to ensure matching configurations for accurate previews and tests.

The abbey's network documentation currently uses automatic directory indexes, and declares the types of files with several additional filename suffixes.

<Directory {{ docroot }}/Abbey/>
    AllowOverride Indexes FileInfo
    Options +Indexes +FollowSymLinks
</Directory>

The following .htaccess file works with the directives above. It declares most the native source files in the current directory tree to be plain text, so that they are displayed rather than downloaded.

ReadmeName notfound.html
IndexIgnore README.org
AddType text/plain attr campus_vpn cfg cnf conf crt daily_letsencrypt
AddType text/plain domain el htaccess idx j2 key old org pack pem
AddType text/plain private pub public_vpn req rev sample txt yml

Some of the directives added to the -vhost.conf file map the abbey's abstract photo URLs, e.g. /Photos/2022/08/06/, into actual file paths. The following template uses the docroot variable introduced in the previous section. On Front this variable is set to /home/www. The same template is used on Core, to ensure matching configurations for accurate previews and tests.

RedirectMatch /Photos$ /Photos/
RedirectMatch /Photos/(20[0-9][0-9])_([0-9][0-9])_([0-9][0-9])$ \
              /Photos/$1_$2_$3/
AliasMatch /Photos/(20[0-9][0-9])_([0-9][0-9])_([0-9][0-9])/(.+)$ \
           {{ docroot }}/Photos/$1/$2/$3/$4
AliasMatch /Photos/(20[0-9][0-9])_([0-9][0-9])_([0-9][0-9])/$ \
           {{ docroot }}/Photos/$1/$2/$3/index.html
AliasMatch /Photos/$ {{ docroot }}/Photos/index.html

The abbey needs to add some Apache2 configuration directives to the virtual host listening for HTTPS requests to birchwood-abbey.net. Luckily there is support for this in the institutional configuration. The abbey simply creates a birchwood-abbey.net-vhost.conf file in /etc/apache2/sites-available/.

The following task adds the apache-abbey, apache-photos, and apache-gitweb directives described above to the -vhost.conf file, and includes options-ssl-apache.conf from /etc/letsencrypt/. The rest of the Let's Encrypt configuration is discussed in the following Install Let's Encrypt section.

- name: Configure Apache.
  become: yes
  vars:
    docroot: /home/www
  copy:
    content: |
        <<apache-abbey>>
        <<apache-photos>>
        <<apache-gitweb>>

        IncludeOptional /etc/letsencrypt/options-ssl-apache.conf
    dest: /etc/apache2/sites-available/{{ domain_name }}-vhost.conf
  notify: Restart Apache2.

<<apache-gitweb-tasks>>

<<apache-gitweb-handlers>>

These tasks hack Apache's logrotate(8) configuration to rotate weekly, keep the last 12 weeks, and email each week's log to root. The logrotate(8) manual page explains the configuration options.

The Systemd configuration drop tells logrotate to use a special script for its mail program. Postfix's mail work-alike did not take the subject as a command line argument as provided by logrotate. The replacement logrotate-mailer does, and includes it in a Subject header prepended to logrotate's message.

- name: Configure Apache log archival.
  become: yes
  lineinfile:
    path: /etc/logrotate.d/apache2
    regexp: "{{ item.regexp }}"
    line: "{{ item.line }}"
  loop:
  - { regexp: '^ *daily', line: "\tweekly" }
  - { regexp: '^ *rotate', line: "\trotate 12" }

- name: Configure Apache log email.
  become: yes
  lineinfile:
    path: /etc/logrotate.d/apache2
    regexp: "{{ item.regexp }}"
    line: "{{ item.line }}"
    insertbefore: " *}"
    firstmatch: yes
  loop:
  - { regexp: "^\tmail ", line: "\tmail webmaster" }
  - { regexp: "^\tmailfirst", line: "\tmailfirst" }

- name: Configure logrotate.
  become: yes
  copy:
    src: logrotate-mailer.conf
    dest: /etc/systemd/system/logrotate.service.d/mailer.conf
  notify: Reload systemd.

- name: Install logrotate mailer.
  become: yes
  copy:
    src: logrotate-mailer
    dest: /usr/local/sbin/logrotate-mailer
    mode: u=rwx,g=rx,o=rx

- name: Reload systemd.
  become: yes
  systemd:
    daemon_reload: yes

Note that the first setting for ExecStart is intended to clear the system's ExecStart in /lib/systemd/system/logrotate.service. (A oneshot service like this can have multiple ExecStart settings. See the description of ExecStart in the systemd.service(5) manual page.)

[Service]
ExecStart=
ExecStart=/usr/sbin/logrotate \
                --mail /usr/local/sbin/logrotate-mailer \
                /etc/logrotate.conf

The /usr/local/sbin/logrotate-mailer script (below) was originally needed because Postfix does not provide an emulation of mail(1) and some translation to sendmail(1) was required. Since then the script has learned to compute the date-dependent file name, compress the log, convert it to base64, and encapsulate it in MIME format, before sending it on to sendmail. Note that there is no encryption (yet). This is a low priority because much of the data is available to Droplet's ISP's Mom, the NSA/CIA/NWO.

#!/bin/bash -e

if [ "$#" != 3 -o "$1" != "-s" ]; then
    echo "usage: $0 -s subject recipient" 1>&2
    exit 1
fi

D=`date -d yesterday "+%Y%m%d"`
if [[ "$2" == *error.log* ]]; then
    F="$D-error.log.gz"
else
    F="$D.log.gz"
fi

( echo "Subject: $2"
  echo "Content-Type: multipart/mixed; boundary=\"boundary\""
  echo "MIME-Version: 1.0"
  echo ""
  echo "--boundary"
  echo "Content-Type: text/plain"
  echo "Content-Transfer-Encoding: 8bit"
  echo ""
  echo "$F"
  echo "--boundary"
  echo "Content-Type: application/gzip; name=\"$F\""
  echo "Content-Disposition: attachment; filename=\"$F\""
  echo "Content-Transfer-Encoding: base64"
  echo ""
  gzip | base64
  echo ""
  echo "--boundary--" ) | sendmail "$3"

The abbey uses a Let's Encrypt certificate to authenticate its public web site and email services. Initial installation of a Let's Encrypt certificate is a terminal session affair (with prompts and lines entered as shown below).

$ sudo apt install python3-certbot-apache
$ sudo certbot --apache -d birchwood-abbey.net
...
Enter email address (...) (Enter 'c' to cancel): webmaster@birchwood-
bbey.net
...
Please read the Terms of Service at
...
(A)gree/(C)ancel: A
...
Would you be willing to share your email address...
...
(Y)es/(N)o: Y
...
Deploying Certificate to VirtualHost /etc/apache2/sites-enabled/birch
ood-abbey.net.conf

Please choose whether or not to redirect HTTP traffic to HTTPS, remov
ng HTTP access.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: No redirect - Make no further changes to the webserver configurati
n.
...
Select the appropriate number [1-2] then [enter] (press 'c' to cancel
: 1

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations! You have successfully enabled https://birchwood-abbe
.net

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=birchwood-abbey.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

IMPORTANT NOTES:
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory wil
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
...
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/birchwood-abbey.net/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/birchwood-abbey.net/privkey.pem
   Your cert will expire on 2019-01-13. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot agai
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"

When the /etc/letsencrypt/ directory is restored from a backup copy, and the following tasks performed, the web server will be prepared to do ACME (the certificate protocol) when next Let's Encrypt calls (quarterly). The following tasks ensure the python3-cerbot-apache package is installed and its live/ subdirectory is world readable.

- name: Install Certbot for Apache.
  become: yes
  apt: pkg=python3-certbot-apache

- name: Ensure Let's Encrypt certificate is readable.
  become: yes
  file:
    mode: u=rwx,g=rx,o=rx
    path: /etc/letsencrypt/live

Front's Dovecot (and Postfix) certificate and key are in separate files despite their warning about a race condition (when updating the pair of files) mainly because that is how they are provided (and updated) by Let's Encrypt, but also because Let's Encrypt's symbolic links keep the window for a mismatch extremely small.

With the institutional configuration, Postfix, Dovecot and Apache servers get their certificate&key from /etc/server.crt&.key. The institutional roles check that they exist, but will not create them. In this abbey specific role, /etc/server.crt&key are ours to frob. The following tasks ensure they are symbolic links to /etc/letsencrypt/live/birchwood-abbey.net/fullchain&privkey.pem. If /etc/letsencrypt/ was restored from a backup, the servers should be restarted manually.

- name: Use Let's Encrypt certificate&key.
  file:
    state: link
    src: "{{ item.target }}"
    path: "{{ item.link }}"
    force: yes
  loop:
  - target: /etc/letsencrypt/live/birchwood-abbey.net/fullchain.pem
    link: /etc/server.crt
  - target: /etc/letsencrypt/live/birchwood-abbey.net/privkey.pem
    link: /etc/server.key

The following task arranges to rotate Certbot's logs files.

- name: Install Certbot logrotate configuration.
  become: yes
  copy:
    src: certbot_logrotate
    dest: /etc/logrotate.d/certbot
    mode: u=rw,g=r,o=r

/var/log/letsencrypt/*.log {
    rotate 12
    weekly
    compress
    missingok
}

A backup copy of Let's Encrypt's data (/etc/letsencrypt/) is sent to root@core in S/MIME encrypted email every time it changes. Changes are detected by keeping a copy in /etc/letsencrypt~/ for comparison.

- name: Install Let's Encrypt archive script.
  become: yes
  copy:
    src: cron.daily_letsencrypt
    dest: /etc/cron.daily/letsencrypt
    mode: u=rwx,g=rx,o=rx

#!/bin/bash -e

cd /etc/

[ -d letsencrypt~ ] \
&& diff -rq letsencrypt/ letsencrypt~/ \
&& exit 0

( echo "Subject: New /etc/letsencrypt/ on Droplet."
  echo ""
  tar czf - letsencrypt/ \
  | gpg --encrypt --armor \
        --trust-model always --recipient root@core ) \
| sendmail root \
|| exit $?

rm -rf letsencrypt~
cp -a letsencrypt letsencrypt~

The message is encrypted with root@core's public key, which is imported into root@front's GnuPG key file.

- name: Copy root@core's public key.
  become: yes
  copy:
    src: ../Secret/root-pub.pem
    dest: /root/.gnupg-root-pub.pem
    mode: u=r,g=r,o=r
  notify: Import root@core's public key.

- name: Import root@core's public key.
  become: yes
  command: gpg --import ~/.gnupg-root-pub.pem

In this abbey specific document, most abbey particulars are not replaced with variables, but specified in-line. Some, however, are private (e.g. database passwords), not to be published in this document, and so replaced with variables set in private/vars-abbey.yml. The file path is relative to the playbook's directory, playbooks/.

---
- name: Include private abbey variables.
  include_vars: ../private/vars-abbey.yml

The scripts that maintain the abbey's web site and run the Weather project use a number of additional software packages. The /WWW/live/Private/make-top-index script uses HTML::TreeBuilder in the libhtml-tree-perl package. The house task list uses JQuery. Weather scripts use mit-scheme and gnuplot (in pseudonymous packages).

- name: Install additional packages.
  apt:
    pkg: [ libhtml-tree-perl, libjs-jquery, mit-scheme, gnuplot ]

The abbey uses several additional email aliases. These are the campus mailboxes @*.birchwood.private. The institute already includes some standard system aliases, as well as mailboxes for accounts running services: www-data and monkey. The institute funnels these to root and forwards root to sysadm (as on Front). The abbey takes it from there, forwarding sysadm to a real person and including mailboxes for all accounts running services on any campus machine. (They should all be relaying to smtp.birchwood.private which delivers any .birchwood.private email, e.g. mythtv@mythtv.birchwood.private, locally.)

- name: Install abbey email aliases.
  become: yes
  blockinfile:
    block: |
        sysadm:         matt
        house:          sysadm
        mythtv:         sysadm
        scanner:        sysadm
    dest: /etc/aliases
    marker: "# {mark} ABBEY MANAGED BLOCK"
  notify: New aliases.

---
- name: New aliases.
  become: yes
  command: newaliases

These tasks are identical to those executed on Front, for similar Git services on Front and Core. See 3.3 and Configure Gitweb on Front for more information.

<<git-tasks>>

<<git-handlers>>

The Apache2 configuration on Core specifies three web sites (live, test, and campus). The live and test sites must operate just like the site on Front. Their configurations include the same apache-abbey, apache-photos, and apache-gitweb used on Front.

- name: Configure live website.
  become: yes
  vars:
    docroot: /WWW/live
  copy:
    content: |
        <<apache-abbey>>
        <<apache-photos>>
        <<apache-gitweb>>
    dest: /etc/apache2/sites-available/live-vhost.conf
    mode: u=rw,g=r,o=r
  notify: Restart Apache2.

- name: Configure test website.
  become: yes
  vars:
    docroot: /WWW/test
  copy:
    content: |
        <<apache-abbey>>
        <<apache-photos>>
        <<apache-gitweb>>
    dest: /etc/apache2/sites-available/test-vhost.conf
    mode: u=rw,g=r,o=r
  notify: Restart Apache2.

<<apache-gitweb-tasks>>

<<apache-gitweb-handlers>>

The institute serves its /usr/share/doc/ on the house (campus) web site. This is a debugging convenience, making some HTML documentation more accessible, especially the documentation of software installed on Core and not on typical desktop clients. Also included: the Apache2 directives that enable user Git publishing with Gitweb (defined here).

- name: Configure house website.
  become: yes
  copy:
    content: |
      Alias /doc /usr/share/doc
      <Directory /usr/share/doc/>
          Options Indexes
      </Directory>
      <<apache-gitweb>>
    dest: /etc/apache2/sites-available/www-vhost.conf
    mode: u=rw,g=r,o=r
  notify: Restart Apache2.

The abbey uses the Apt-Cacher:TNG package cache on Core. The apt-cacher domain name is defined in private/db.domain.

- name: Install Apt-Cacher:TNG.
  become: yes
  apt: pkg=apt-cacher-ng

Core itself will benefit from using the package cache.

- name: Use the local Apt package cache.
  become: yes
  copy:
    content: >
      Acquire::http::Proxy
      "http://apt-cacher.{{ domain_priv }}.:3142";
    dest: /etc/apt/apt.conf.d/01proxy
    mode: u=rw,g=r,o=r

A small institute uses nagios4 to monitor the health of its network, with an initial smattering of monitors adopted from the Debian monitoring-plugins package. Thus a NAGIOS4 server on the abbey's Core monitors core network services, and uses nagios-nrpe-server to monitor Gate. The abbey adds several more monitors, installing additional configuration files in /etc/nagios4/conf.d/, and another customized check_sensors plugin (abbey_pisensors) in /usr/local/sbin/ on the Raspberry Pis.

The abbey adds monitoring of the space remaining on the volume at /home/ on Core. (The small institute only monitors the space remaining on roots.)

- name: Configure NAGIOS monitoring for Core /home/.
  become: yes
  copy:
    content: |
      define service {
          use                     local-service
          host_name               core
          service_description     Home Partition
          check_command           check_local_disk!20%!10%!/home
      }
    dest: /etc/nagios4/conf.d/abbey.cfg
  notify: Reload NAGIOS4.

- name: Reload NAGIOS4.
  become: yes
  systemd:
    service: nagios4
    state: reloaded

The check_sensors plugin is included in the package monitoring-plugins-basic, but it does not report any readings. The small institute substitutes a Custom NAGIOS Monitor inst_sensors that reports core CPU temperatures, but the sensors command on a Raspberry Pi does not reveal core CPU temperatures, so the abbey includes yet another version, abbey_pisensors, that reports any recognizable temperature in the sensors output.

#!/bin/sh

PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
export PATH
PROGNAME=`basename $0`
REVISION="2.3.1"

. /usr/lib/nagios/plugins/utils.sh

print_usage() {
        echo "Usage: $PROGNAME" [--ignore-fault]
}

print_help() {
        print_revision $PROGNAME $REVISION
        echo ""
        print_usage
        echo ""
        echo "This plugin checks hardware status using" \
                "the lm_sensors package."
        echo ""
        support
        exit $STATE_OK
}

brief_data() {
    echo "$1" | sed -n -E -e '
  /^temp[0-9]+: +[-+][0-9.]+.?C/ {
    s/^temp[0-9]+: +([-+][0-9.]+).?C.*/ \1/; H }
  $ { x; s/\n//g; p }'
}

case "$1" in
--help)
        print_help
        exit $STATE_OK
        ;;
-h)
        print_help
        exit $STATE_OK
        ;;
--version)
        print_revision $PROGNAME $REVISION
        exit $STATE_OK
        ;;
-V)
        print_revision $PROGNAME $REVISION
        exit $STATE_OK
        ;;
*)
        sensordata=`sensors 2>&1`
        status=$?
        if test ${status} -eq 127; then
                text="SENSORS UNKNOWN - command not found"
                text="$text (did you install lmsensors?)"
                exit=$STATE_UNKNOWN
        elif test ${status} -ne 0; then
                text="WARNING - sensors returned state $status"
                exit=$STATE_WARNING
        elif echo ${sensordata} | egrep ALARM > /dev/null; then
                text="SENSOR CRITICAL -`brief_data "${sensordata}"`"
                exit=$STATE_CRITICAL
        elif echo ${sensordata} | egrep FAULT > /dev/null \
            && test "$1" != "-i" -a "$1" != "--ignore-fault"; then
                text="SENSOR UNKNOWN - Sensor reported fault"
                exit=$STATE_UNKNOWN
        else
                text="SENSORS OK -`brief_data "${sensordata}"`"
                exit=$STATE_OK
        fi

        echo "$text"
        if test "$1" = "-v" -o "$1" = "--verbose"; then
                echo ${sensordata}
        fi
        exit $exit
        ;;
esac

The abbey adds monitoring for more servers: Kamino, Kessel and Devaron. They are abbey-cloister servers, so they are configured as small institute campus servers, like Gate, with an NRPE (a NAGIOS Remote Plugin Executor) server and an inst_sensors command.

The configurations for the servers are very similar to Gate's, but are idiosyncratically in flux. In particular, Kamino does not irritate check_total_procs, yet Kessel does. Both are Pop!_OS 22.04, but Kessel is a wireless host while Kamino is wired. Devaron, the Raspberry Pi OS (ARM64) machine, uses the abbey_pisensors monitor.

The abbey's public web site's access and error logs are emailed regularly to webmaster, who saves them in /Logs/apache2-public/ and runs analog to generate /WWW/campus/analog.html, available to the campus as http://www/analog.html.

- name: Install Analog.
  become: yes
  apt: pkg=analog

- name: Configure Analog (removing old /var/log/apache/ LOGFILEs).
  become: yes
  lineinfile:
    path: /etc/analog.cfg
    regexp: '^LOGFILE /var/log/apache/'
    state: absent

- name: Configure Analog (adding new configuration lines).
  become: yes
  lineinfile:
    path: /etc/analog.cfg
    line: "{{ item }}"
    insertafter: EOF
  loop:
  - "LOGFILE /Logs/apache2-public/*-access.log.gz"
  - "ALLCHART OFF"
  - "DNS WRITE"
  - "HOSTNAME \"{{ full_name }}\""
  - "OUTFILE /WWW/campus/analog.html"

- name: Create /Logs/.
  become: yes
  file:
    path: /Logs
    state: directory
    mode: u=rwx,g=rx,o=rx

- name: Create /Logs/apache2-public/.
  become: yes
  file:
    path: /Logs/apache2-public
    state: directory
    owner: monkey
    group: staff
    mode: u=rwx,g=srwx,o=rx

Monkey needs to be in www-data so that it can run /WWW/live/Photos/Private/cronjob to publish photos from multiple user cloud accounts, found in files owned by www-data, files like InstantUpload/Camera/2021/01/IMG_20210115_092838.jpg in /var/www/nextcloud/data/$USER/files/.

- name: Add Monkey to Nextcloud group.
  become: yes
  user:
    name: monkey
    append: yes
    groups: www-data

Monkey's photo processing scripts use netpbm commands like jpegtopnm.

- name: Install netpbm.
  become: yes
  apt: pkg=netpbm

Monkey on Core runs /WWW/campus/Weather/Private/cronjob every 5 minutes and cronjob-midnight at midnight.

- name: Create Monkey's weather job.
  become: yes
  cron:
    name: weather
    hour: "*"
    minute: "*/5"
    job: "[ -d /WWW/house ] && /WWW/house/Weather/Private/cronjob"
    user: monkey

The abbey gate's lan interface is the PC's built-in Ethernet interface, connected to the cloister Ethernet, a Gigabit Ethernet switch. Its wifi interface is a USB3.0 Ethernet adapter connected with a cross-over cable to the WAN interface of a Think Penguin TPE-R1300 (and at one time a Linksys WRT1900AC). The isp interface is another USB3.0 Ethernet adapter connected with a cross-over cable to the Ethernet interface of a "cable modem" (a Starlink terminal).

The MAC address of each interface is set in private/vars.yml (see Institute/private/vars.yml) as the values of the gate_lan_mac, gate_wifi_mac and gate_isp_mac variables.

The abbey connects to Starlink via Ethernet, and disables Starlink's Wi-Fi access point. An Ethernet adapter add-on (ordered separately) was installed on the Starlink cable, and a second USB-Ethernet dongle on Gate. The adapters were then connected with a cross-over cable.

The abbey could have avoided buying a separate cloister Wi-Fi access point, and used Starlink's Wi-Fi instead, with or without its add-on Ethernet interface. Instead, the abbey invested in a 2.4GHz-only Think Penguin access point, and connected it to a third Ethernet interface on Gate. This was preferred for a number of reasons.

The abbey uses ISPs other than Starlink, tethering to a cellphone when under trees, or even limping along on campground Wi-Fi where the land of woven trees has cut off even cell service.

The abbey uses long and complex passwords, especially on public facing services like Wi-Fi. Such a password has been laboriously entered into several household IoT devices. Connecting them to a dedicated, ISP-independent cloister Wi-Fi access point ensures a reliable IoT with zero re-configuration.

Using Starlink's add-on Ethernet interface allowed its Wi-Fi to be disabled, reducing the Wi-Fi clutter in the campground ether.

The Think Penguin access point is transparent, trustworthy hardware that has earned a Respects Your Freedom certification (see https://ryf.fsf.org/).

And most importantly, a dedicated and trustworthy cloister Wi-Fi keeps at least our local network traffic out of view of our ISPs.

The abbey used to use a cell phone on a USB tether to get Internet service. At that time, Gate's /etc/netplan/60-isp.yaml file was the following.

network:
  ethernets:
    tether:
      match:
        name: usb0
      set-name: isp
      dhcp4: true
      dhcp4-overrides:
        use-dns: false

The abbey has occasionally used a campground Wi-Fi for Internet service, using a 60-isp.yaml file similar to the lines below.

network:
  wifis:
    tether:
      match:
        name: wlan0
      set-name: isp
      dhcp4: true
      dhcp4-overrides:
        use-dns: false
      access-points:
        "AP with password":
          password: "password"
        "AP with no password": {}

The Apt-Cacher:TNG program does not work well on the frontier, so is not a common part of a small institute. But it is helpful even for a cloister with less than a dozen hosts (especially to a homogeneous cloister using many of the same packages), so it is tolerable to the abbey's monks. Monks are patient enough to re-run failed scans repeatedly until few or no incomplete or damaged files are found. Depending on the quality of the Internet connection, this may take a while.

---
- name: Use the local Apt package cache.
  become: yes
  copy:
    content: >
      Acquire::http::Proxy
      "http://apt-cacher.{{ domain_priv }}.:3142";
    dest: /etc/apt/apt.conf.d/01proxy
    mode: u=rw,g=r,o=r

Each cloistered host is a small institute campus host and thus is already running an NRPE server (a NAGIOS Remote Plugin Executor server) with a custom inst_sensors monitor (described in Configure NRPE of A Small Institute). The abbey adds one complication: yet another check_sensors variant, abbey_pisensors, installed on Raspberry Pis (architecture aarch64) only.

- name: Install abbey_pisensors NAGIOS plugin.
  become: yes
  copy:
    src: ../abbey-core/files/abbey_pisensors
    dest: /usr/local/sbin/abbey_pisensors
    mode: u=rwx,g=rx,o=rx
  when: ansible_architecture == 'aarch64'

- name: Configure NAGIOS command.
  become: yes
  copy:
    content: |
      command[abbey_pisensors]=/usr/local/sbin/abbey_pisensors
    dest: /etc/nagios/nrpe.d/abbey.cfg
  when: ansible_architecture == 'aarch64'
  notify: Reload NRPE server.

- name: Reload NRPE server.
  become: yes
  systemd:
    service: nagios-nrpe-server
    state: reloaded

The monks of the abbey are masters of the staff and Emacs.

- name: Install monastic software.
  become: yes
  apt: pkg=emacs

The abbey currently has one weather host, Gate, and a couple 1-Wire sensor modules. The modules measure inside and outside temperature and humidity. Their desired locations are 7-8m from the core servers so they are plugged into a custom Y cable, with the inside sensor cable spliced into the middle of the outside/main cable. The proximal end's RJ11 plugs into a 1-Wire USB adapter (a DS9490R) plugged into Gate. The outside end goes out the window with the Starlink cable.

The Ansible code in the abbey-weather role assumes it is working with a cloistered host (as described in Cloistering) and proceeds in two phases. The first installs the ow-server package and configures it to use a DS9490 (USB adapter) rather than a debugging fake. After the first ./abbey config new, the new weather host seems to need a reboot before the 1-Wire bus becomes visible via owdir.

After a reboot owdir should list one or more type 26 device IDs. Listing them (e.g. running owdir /26.nnnnnnnn or owdir /26.nnnnnnnn/HIH) should reveal "files" named temperature and HIH/humidity. These pseudo-file paths are used in the daemon script below. A test session is shown below.

monkey@new$ owdir
...
    /26.2153B6000000/
...
monkey@new$ owdir /26.2153B6000000
...
    /26.2153B6000000/temperature
...
monkey@new$ owread /26.2153B6000000/temperature; echo
26.125
monkey@new$ 

The second phase of weather host configuration waits for the host- specific weather daemon script to appear in the role's files/.

Different weather hosts, with different 1-Wire devices, need different daemon scripts, to call owread with different paths (containing the IDs of each host's devices). At the moment there is just the one weather host, anoat.

#!/usr/bin/perl -w
# -*- CPerl -*-
#
# Weather/daemon
#
# Fetches data from the local owserver once per minute.  Appends to
# Log/{In,Out}side/YEAR/MONTH/DAY.txt.

use strict;
use IO::File;
use Date::Format;

my $ILOG;
my $OLOG;
my $ymd = "";
sub mymkdir ($);
sub reopen_logs ()
{
  my $time = time;
  my $datime = time2str ("%Y-%m-%d %H:%M:%S", $time, "UTC");
  my ($year, $month, $day) = $datime =~ /^(\d{4})-(\d\d)-(\d\d) /;
  my $new_ymd = "$year/$month/$day";
  return if $new_ymd eq $ymd;
  close $ILOG if defined $ILOG;
  close $OLOG if defined $OLOG;
  umask 07;
  mymkdir "Inside/$year/$month";
  mymkdir "Outside/$year/$month";
  umask 027;
  my $filename = "Inside/$new_ymd.txt";
  $ILOG = new IO::File;
  open $ILOG, ">>$filename" or die "Could not open $filename: $!\n";
  $filename = "Outside/$new_ymd.txt";
  $OLOG = new IO::File;
  open $OLOG, ">>$filename" or die "Could not open $filename: $!\n";
  $ymd = $new_ymd;
}

sub logit ($$$);
sub main () {
  die "usage: $0\n" if @ARGV != 0;
  $0 = "weatherd";
  chdir "/home/monkey/Weather/Log" or die;
  umask 027;
  my $start = time;
  {
    my $secs = 60 - $start % 60;
    $start += $secs;
    sleep ($secs);
  }
  while (1) {
    reopen_logs;
    logit $OLOG, "T", "/26.2153B6000000/temperature";
    logit $OLOG, "H", "/26.2153B6000000/HIH4000/humidity";
    logit $ILOG, "T", "/26.8859B6000000/temperature";
    logit $ILOG, "H", "/26.8859B6000000/HIH4000/humidity";
    $start += 60;
    my $now = time;
    while ($start < $now) { $start += 60; }
    my $secs = $start - $now;
    sleep  ($secs);
  }
}

sub logit ($$$)
{
  my ($log, $name, $query) = @_;

  my $tries = 0;
  while ($tries < 3) {
    my $time = time;
    my $datime = time2str ("%Y-%m-%d %H:%M:%S", $time, "UTC");
    $tries += 1;
    my @lines = `/usr/bin/owread $query`;
    chomp @lines;
    my $status = $?;
    my $sig = $status & 127;
    $status >>= 8;
    if ($status != 0) {
      my $L = join "\\n", @lines;
      print $log "$datime\t$name\terror: status $status: $L\n";
      $log->flush;
    } elsif (@lines != 1) {
      my $L = join "\\n", @lines;
      print $log "$datime\t$name\terror: multiple lines: $L\n";
      $log->flush;
    } elsif ($lines[0] !~ /^ *(-?\d+(\.\d+)?)$/) {
      my $L = $lines[0];
      print $log "$datime\t$name\terror: bogus line: $L\n";
      $log->flush;
    } else {
      my $datum = $1;
      print $log "$datime\t$name\t$datum\n";
      $log->flush;
      return;
    }
  }
}

sub mymkdir ($)
{
  my ($dirpath) = @_;

  my @path_names = split /\//, $dirpath;
  my $path;
  if (!$path_names[0]) {
    $path = "/";
    shift @path_names;
  } else {
    $path = ".";
  }
  my @created;
  while (@path_names) {
    $path .= "/" . shift @path_names;
    if (! -d $path) {
      if (-e $path) {
        die "mkdir $dirpath: already exists; not a directory!\n";
      }
      if (! mkdir $path) {
        die "mkdir $path: $!\n";
      } else {
        chmod 02775, $path;
        push @created, $path;
      }
    }
  }
  return @created;
}

main;

The above Perl script uses the Date::Format module, which is installed by the following task.

---
- name: Install weather daemon packages.
  become: yes
  apt: pkg=libtimedate-perl

This next task installs the 1-Wire server and shell commands. The abbey uses the Dallas Semiconductor DS9490R, a USB to 1-Wire adapter, on all its weather hosts, so it also configures the server to use the USB adapter (rather than a test "fake" adapter).

- name: Install 1-Wire server.
  become: yes
  apt:
    pkg: [ owserver, ow-shell ]

- name: Configure 1-Wire server.
  become: yes
  lineinfile:
    path: /etc/owfs.conf
    regexp: "{{ item.regexp }}"
    line: "{{ item.line }}"
    backrefs: yes
  loop:
  - { regexp: '^[# ]*server: *FAKE(.*)$', line: '#server: FAKE\1' }
  - { regexp: '^[# ]*server: *usb(.*)$', line: 'server: usb\1' }

Monkey on Core will want to download log records (files) using rsync(1).

- name: Install Rsync.
  become: yes
  apt: pkg=rsync

The weather daemon is run by an unprivileged monkey account (not sysadm) which allows monkey on Core shell access. This is also executed during the initial phase of configuration, allowing the administrator to login on the new weather host as monkey and thus to test access to the 1-Wire adapter and devices. To facilitate debugging, the sysadm account is included in the monkey group.

- name: Create monkey.
  become: yes
  user:
    name: monkey
    system: yes

- name: Authorize monkey@core.
  become: yes
  vars:
    pubkeyfile: ../Secret/ssh_monkey/id_rsa.pub
  authorized_key:
    user: monkey
    key: "{{ lookup('file', pubkeyfile) }}"
    manage_dir: yes

- name: Add {{ ansible_user }} to monkey group.
  become: yes
  user:
    name: "{{ ansible_user }}"
    append: yes
    groups: monkey

The weather daemon is kept alive as a Systemd service unit. This task creates and starts that service after the host-specific files/daemon-HOST file becomes available.

The ExecStartPre=/bin/sleep 30 is intended to avoid recent hangs in owread.

- name: Install weather directory.
  become: yes
  file:
    path: /home/monkey/Weather/Log
    state: directory
    owner: monkey
    group: monkey
    mode: u=rwx,g=rx,o=rx

- name: Test for weather daemon script.
  vars:
    dir: ../roles/abbey-weather/files
    file: "{{ dir }}/daemon-{{ inventory_hostname }}"
  stat: path="{{ file }}"
  delegate_to: localhost
  register: weather

- name: Note missing weather daemon script.
  vars:
    dir: ../roles/abbey-weather/files
    script: "{{ dir }}/daemon-{{ inventory_hostname }}"
  debug:
    msg: "{{ script }}: not found"
  when: not weather.stat.exists

- name: Install weather daemon.
  become: yes
  vars:
    dir: ../roles/abbey-weather/files
    script: "{{ dir }}/daemon-{{ inventory_hostname }}"
  copy:
    src: "{{ script }}"
    dest: /home/monkey/Weather/daemon
    owner: monkey
    group: monkey
    mode: u=rwx,g=rx,o=
  when: weather.stat.exists

- name: Install weatherd service.
  become: yes
  copy:
    content: |
      [Unit]
      Description=Weather Logger
      After=owserver.service

      [Service]
      User=monkey
      ExecStartPre=/bin/sleep 30
      ExecStart=/home/monkey/Weather/daemon
      Restart=always

      [Install]
      WantedBy=multi-user.target
    dest: /etc/systemd/system/weatherd.service
  when: weather.stat.exists
  notify:
  - Reload Systemd.
  - Restart weather daemon.

- name: Enable/Start weather daemon.
  become: yes
  systemd:
    service: weatherd
    enabled: yes
    state: started
  when: weather.stat.exists

---
- name: Reload Systemd.
  become: yes
  command: systemctl daemon-reload

- name: Restart weather daemon.
  become: yes
  systemd:
    service: weatherd
    state: restarted

The installation process involves some manual intervention. The first time a host is enrolled, Ansible will install the necessary packages, but it cannot create the database, nor the database user (yet, in the first pass). After adding the new machine to the dvrs group in 10.2, run Ansible to get the Zoneminder software installed.

./abbey config HOST

Several configuration steps will be skipped because /Zoneminder/ has not been created yet. To proceed, first create the database and database user manually, as described in section Manually Create Zoneminder DB and User.

/Zoneminder/ should be a separate, large volume lest Zoneminder fill the root file system. For acceptable performance, /Zoneminder/ should also be the mount point of a solid-state disk (SSD). A symbolic link at /var/cache/zoneminder/events targets /Zoneminder to make it Zoneminder's "default" storage area. (The PurgeWhenFull filter only works with the default storage area in v1.34.)

Once the zm database (and zmuser database user) are created, and a large volume mounted at /Zoneminder/, Ansible can continue with the Zoneminder configuration.

./abbey configure HOST

Configuring Zoneminder's cameras is still a manual process as described in the final section, Configure Cameras, below.

Private variables in private/vars-abbey.yml are needed, and included here, as in the abbey-core role. The file path is relative to the playbook's directory, playbooks/.

---
- name: Include private abbey variables.
  include_vars: ../private/vars-abbey.yml

The latest version of Zoneminder (1.36) was manually downloaded, built and installed, but it immediately had problems, randomly producing short events, dropping "problem" cameras entirely, etc. Version 1.34 did not have those problems, but could still melt down (thrash?) when /Zoneminder/ was a Seagate Barracuda in a USB3.1gen2 external drive enclosure. A Western Digital Passport Ultra seemed to work much better, for a short while. Ultimately a solid-state drive (a 2TB USB3.2 Gen2 Samsung T7 Shield) mounted at /Zoneminder/ got Zoneminder 1.34 to work reliably.

After uninstalling 1.36, the Debian 11 package (1.34) was installed and configured per the instructions in sections "Web server set-up" and "Time Zone" in /usr/share/doc/zoneminder/README.Debian.gz.

- name: Install Zoneminder.
  become: yes
  apt: pkg=zoneminder

- name: Enable Apache modules for Zoneminder.
  become: yes
  apache2_module:
    name: "{{ item }}"
  loop: [ cgid, rewrite, expires, headers ]
  notify: Restart Apache2.

- name: Enable Zoneminder Apache configuration.
  become: yes
  command:
    cmd: a2enconf zoneminder
    creates: /etc/apache2/conf-enabled/zoneminder.conf
  notify: Restart Apache2.

- name: Configure MySQL for Zoneminder.
  become: yes
  copy:
    content: |
      [mysqld]
      sql_mode = NO_ENGINE_SUBSTITUTION
    dest: /etc/mysql/conf.d/zoneminder.cnf
  notify: Restart MySQL.

- name: Configure PHP date.timezone.
  become: yes
  lineinfile:
    regexp: date.timezone ?=
    line: date.timezone = {{ lookup('file', '/etc/timezone') }}
    path: "{{ item }}"
  loop:
  - /etc/php/8.2/cli/php.ini
  - /etc/php/8.2/apache2/php.ini
  notify: Restart Apache2.

- name: Enable/Start Apache2.
  become: yes
  systemd:
    service: apache2
    enabled: yes
    state: started

---
- name: Restart MySQL.
  become: yes
  systemd:
    service: mysql
    state: restarted

- name: Restart Apache2.
  become: yes
  systemd:
    service: apache2
    state: restarted

The following Rsyslog configuration drop-in gets Zoneminder's natter out of /var/log/syslog.

- name: Use /var/log/zoneminder.log
  become: yes
  copy:
    content: |
      :programname,startswith,"zm" -/var/log/zoneminder.log
      & stop
    dest: /etc/rsyslog.d/40-zoneminder.conf

Zoneminder's MariaDB database is created by the following task, when the mysql_db Ansible module supports check_implicit_admin.

- name: Create Zoneminder DB.
  become: yes
  mysql_db:
    check_implicit_admin: yes
    name: zm
    collation: utf8mb4_general_ci
    encoding: utf8mb4

Unfortunately it does not currently, yet the institute prefers the more secure Unix socket authentication method. Rather than create a privileged DB user, the zm database is created manually (below).

The following task would create the DB user (mysql_user supports check_implicit_admin) but the zm database was not created above.

The DB user's password is taken from the zoneminder_dbpass variable, kept in private/vars-abbey.yml, and generated e.g. with the apg -n 1 -x 12 -m 12 command.

zoneminder_dbpass:           gakJopbikJadsEdd

- name: Create Zoneminder DB user.
  become: yes
  mysql_user:
    check_implicit_admin: yes
    name: zmuser
    password: "{{ zoneminder_dbpass }}"
    priv: >-
      zm.*:
      lock tables,alter,create,index,select,insert,update,delete

The Zoneminder database and database user are created manually with the following SQL (with the zoneminder_dbpass spliced in). The SQL commands are entered at the SQL prompt of the sudo mysql command, or perhaps piped into the command.

create database zm
    character set utf8mb4
    collate utf8mb4_general_ci;
grant lock tables,alter,create,index,select,insert,update,delete
    on zm.*
    to 'zmuser'@'localhost'
    identified by '{{ zoneminder_dbpass }}';
flush privileges;
exit;

Finally, zm's tables are created, completing the database setup,

sudo mysql < /usr/share/zoneminder/db/zm_create.sql

The following tasks start with a test for the existence of /Zoneminder. Configuration tasks that require /Zoneminder/ or the zm database are executed only when zoneminder.stat.exists. The last "Link…" task below "forces" the link, whether the target exists or not (yet).

- name: Test for /Zoneminder/.
  stat:
    path: /Zoneminder
  register: zoneminder
- debug:
    msg: "/Zoneminder/ does not yet exist"
  when: not zoneminder.stat.exists

- name: Check /Zoneminder/.
  become: yes
  file:
    state: directory
    path: /Zoneminder
    owner: www-data
    group: www-data
    mode: u=rwx,g=rx,o=rx
  when: zoneminder.stat.exists

- name: Link to /Zoneminder/.
  become: yes
  file:
    state: link
    src: /Zoneminder
    path: /var/cache/zoneminder/events
    force: yes
    follow: no

The remaining tasks ensure that the /etc/zm/zm.conf file has the proper permissions and contains the correct password.

- name: Set /etc/zm/zm.conf permissions.
  become: yes
  file:
    path: /etc/zm/zm.conf
    owner: root
    group: www-data
    mode: u=rw,g=r,o=

- name: Set Zoneminder passphrase.
  become: yes
  lineinfile:
    regexp: '^ *ZM_DB_PASS *='
    line: ZM_DB_PASS={{ zoneminder_dbpass }}
    path: /etc/zm/zm.conf

Finally, Zoneminder's service unit can be enabled (and started) if /Zoneminder/ exists. It is assumed that, if /Zoneminder/ exists, the zm database has also been created, and the service is ready to run.

- name: Enable/Start Zoneminder.
  become: yes
  systemd:
    service: zoneminder
    enabled: yes
    state: started
  when: zoneminder.stat.exists

A new security camera is setup as described in Cloistering, after which the camera should be accessible by name on the abbey networks. Assuming ping -c1 new works, the camera's web interface will be accessible at http://new/.

The abbey's administrator logs into http://new/ and turns off any OSD (on-screen display). Zoneminder will add its own timestamp, for the best accuracy and reliability. The administrator also turns down the frame rate to 5fps. The abbey prefers HD resolution (e.g. 1080p) and long duration logs, thus fewer frames per second. The administrator also creates an unprivileged user with a short password e.g. user:gobbledygook.

After Ansible has configured and started Zoneminder, a camera can be created by clicking on "Add" in the Zoneminder console. (If the Zoneminder host was named "security", the Zoneminder console can be found at http://security/zm/.) In the Add dialog, the following settings should be changed. (The parenthesized settings are default settings that should be checked but are probably already correctly set.)

• In the "General" tab, specify:
  • Name: Front
  • (Server: None)
  • (Source type: Ffmpeg)
  • Function: Record
  • Enabled: yes
  • (Analysis FPS: <blank>)
  • (Maximum FPS: <blank>)
  • (Alarm Maximum FPS: <blank>)
• In the "Source" tab, specify:
  • Src path: rtsp://user:gobbledygook@new.small.private.:554/11
  • (Method: TCP)
  • (Target colorspace: 32 bit colour)
  • Capture Resolution: 1920x1080 1080p
• In the "Timestamp" tab, specify:
  • Timestamp Label X: 10
  • Timestamp Label Y: 10
  • Font Size: Large
• In the "Buffers" tab, specify:
  • Image Buffer Size (frames): 40

Neither Debian nor the MythTV project provide binary packages of MythTV and MythWeb. The project recommends building from source according to their Build from Source wiki page. To do this, the target host will need several dozen "developer" packages installed. Thus the abbey's TVR role proceeds in two phases.

In the first phase, the MythTV project's Ansible code, in mythtv-ansible/, is used to assemble a list of packages needed during the build. The packages are installed and the rest of the role's tasks are skipped. This allows the administrator to manually build and install MythTV, creating /usr/local/bin/mythtv-setup. The administrator will also download and install MythWeb before running the TVR role again for its second phase. The administrator will not be able to run mythtv-setup before completing the second phase.

In the second phase, the role finds mythtv-setup has been installed on the target host and so proceeds with the "Post-installation tasks" section of the wiki page. This still leaves a number of manual steps to be performed with the mythtv-setup program, e.g. configuring a video source and capture card, after which the backend can be started.

A new TVR machine needs only Cloistering to prepare it for Ansible. As part of that process, it should be added to the tvrs group in the hosts file. An existing server can become a TVR machine simply by adding it to the tvrs group.

Private variables in private/vars-abbey.yml are needed, as in the abbey-core role. The file path is relative to the playbook's directory, playbooks/.

---
- name: Include private abbey variables.
  include_vars: ../private/vars-abbey.yml

A number of developer packages are needed to build MythTV. The wiki page recommends Ansible playbooks to assemble the appropriate list of package names (several dozen count) depending on the target OS version. The playbooks are in https://github.com/MythTV/ansible which contains a README.md.

The instructions in the README.md are to clone the repository and run sudo ansible-playbook -i hosts qt5.yml on the build machine. However the abbey prefers to keep the Ansible code on an administrator's machine with the rest of the abbey's roles. The following commands were used to create a mythtv-ansible/ subdirectory. (A git pull origin command in this subdirectory might be appropriate to download updates.)

git clone https://github.com/MythTV/ansible mythtv-ansible
cd mythtv-ansible
git checkout fixes/32

The abbey-tvr role uses a couple tasks files in mythtv-ansible/ directly, bypassing the inventories, playbooks and roles, after "fixing" the final apt tasks by adding become: yes. After making these edits, the git diff command should produce something like the following.

diff --git a/roles/mythtv-deb/tasks/main.yml b/roles/mythtv-deb/tasks
index 868c9b7..3dcf115 100644
--- a/roles/mythtv-deb/tasks/main.yml
+++ b/roles/mythtv-deb/tasks/main.yml
@@ -366,6 +366,7 @@
       '{{ lookup("flattened", deb_pkg_lst) }}'

 - name: install packages
+  become: yes
   apt:
     name:
       '{{ lookup("flattened", deb_pkg_lst ) }}'
diff --git a/roles/qt5/tasks/qt5-deb.yml b/roles/qt5/tasks/qt5-deb.ym
index 7a1a0bc..26ba782 100644
--- a/roles/qt5/tasks/qt5-deb.yml
+++ b/roles/qt5/tasks/qt5-deb.yml
@@ -25,6 +25,7 @@
       '{{ lookup("flattened", deb_pkg_lst) }}'

 - name: install deb qt5 packages
+  become: yes
   apt:
     name:
       '{{ lookup("flattened", deb_pkg_lst ) }}'

- name: Install MythTV runtime requisites.
  become: yes
  apt:
    pkg: [ mariadb-server, xmltv ]

- name: Install MythTV build requisites.
  include_tasks: "{{ item }}"
  loop:
  - ../mythtv-ansible/roles/mythtv-deb/tasks/main.yml
  - ../mythtv-ansible/roles/qt5/tasks/qt5-deb.yml

The tasks above install runtime and compile-time requisites during the "first" run of e.g. ./abbey config new. The "first" run can be repeated until successful. The remaining tasks are skipped until MythTV is built and installed.

After a successful "first" run of e.g. ./abbey config new, the target machine is prepared to build (and install) MythTV. The following commands are used.

cd /usr/local/src/
git clone https://github.com/MythTV/mythtv
cd mythtv/
git checkout fixes/32
cd mythtv/
./configure
make
sudo make install

The make install command does not need to be run as root if bin/, lib/, include/, share/ in /usr/local/ and dist-packages/ in /usr/local/lib/python3.9/ on the target machine are writable by the builder.

The following task probes for the mythtv-setup program, installed in /usr/local/bin/, to detect that the build/install process has completed. It registers the results in the mythtv variable. Several of the remaining installation steps are skipped unless mythtv.stat.exists.

- name: Test for MythTV binary packages.
  stat:
    path: /usr/local/bin/mythtv-setup
  register: mythtv
- debug:
    msg: "/usr/local/bin/mythtv-setup does not yet exist"
  when: not mythtv.stat.exists

MythTV Backend needs to run as its own user: mythtv.

- name: Create mythtv.
  become: yes
  user:
    name: mythtv
    system: yes

MythTV's MariaDB database is created by the following task, when the mysql_db Ansible module supports check_implicit_admin.

- name: Create MythTV DB.
  become: yes
  mysql_db:
    check_implicit_admin: yes
    name: mythconverg
    collation: utf8mb4_general_ci
    encoding: utf8mb4

Unfortunately it does not currently, yet the institute prefers the more secure Unix socket authentication method. Rather than create a privileged DB user, the mythconverg database is created manually (below).

The DB user's password is taken from the mythtv_dbpass variable, kept in private/vars-abbey.yml, and generated e.g. with the apg -n 1 -x 12 -m 12 command.

mythtv_dbpass:           daJkibpoJkag

The following task would create the DB user (mysql_user supports check_implicit_admin) but the mythconverg database was not created above.

- name: Create MythTV DB user.
  become: yes
  mysql_user:
    check_implicit_admin: yes
    name: mythtv
    password: "{{ mythtv_dbpass }}"
    priv: "mythconverg.*:all"

The MythTV database and database user are created manually with the following SQL (with the mythtv_dbpass spliced in). The SQL commands are entered at the SQL prompt of the sudo mysql command, or perhaps piped into the command.

create database mythconverg
    character set utf8mb4
    collate utf8mb4_general_ci;
create user 'mythtv'@'%' identified by '{{ mythtv_dbpass }}';
create user 'mythtv'@'localhost' identified by '{{ mythtv_dbpass }}';
grant all privileges on mythconverg.*
    to 'mythtv'@'%' with grant option;
grant all privileges on mythconverg.*
    to 'mythtv'@'localhost' with grant option;
flush privileges;
exit;

Starting with MythTV version 0.26, the time zone tables must be loaded into MySQL. The MariaDB installed by Debian 11 seems to need this too. The test SQL produced NULL.

SELECT CONVERT_TZ(NOW(), 'SYSTEM', 'Etc/UTC');

After running the following command line, the test SQL produced e.g. 2022-09-13 20:15:41.

mysql_tzinfo_to_sql /usr/share/zoneinfo | sudo mysql mysql

This task installs the mythtv-backend.service file.

- name: Create mythtv-backend service.
  become: yes
  copy:
    content: |
      [Unit]
      Description=MythTV Backend
      Documentation=https://www.mythtv.org/wiki/Mythbackend
      After=mysql.service network.target

      [Service]
      User=mythtv
      ExecStartPre=/bin/sleep 30
      #TimeoutStartSec=infinity
      ExecStart=/usr/local/bin/mythbackend --quiet --syslog local7
      StartLimitBurst=10
      StartLimitInterval=10m
      Restart=on-failure
      RestartSec=1

      [Install]
      WantedBy=multi-user.target
    dest: /etc/systemd/system/mythtv-backend.service
  when: mythtv.stat.exists
  notify: Reload Systemd.

---
- name: Reload Systemd.
  become: yes
  command: systemctl daemon-reload

This task checks PHP's timezone. If unset, MythTV's backend logs bitter complaints.

- name: Configure PHP date.timezone.
  become: yes
  lineinfile:
    regexp: date.timezone ?=
    line: date.timezone = {{ lookup('file', '/etc/timezone') }}
    path: "{{ item }}"
  loop:
  - /etc/php/8.2/cli/php.ini
  - /etc/php/8.2/apache2/php.ini
  when: mythtv.stat.exists
  notify: Restart Apache2.

- name: Restart Apache2.
  become: yes
  systemd:
    service: apache2
    state: restarted

The backend does not have a default storage area for its recordings. A path to an appropriate directory must be set with the mythtv-setup program (as described below). The abbey uses /home/mythtv/Recordings/ for MythTV's default storage. This task creates that directory and ensures it has appropriate permissions.

- name: Create MythTV storage area.
  become: yes
  file:
    state: directory
    dest: /home/mythtv/Recordings
    owner: mythtv
    group: mythtv
    mode: u=rwx,g+rwx,o=rx

With MythTV built and installed, and the post-installation tasks addressed, MythTV Setup (the mythtv-setup program) can be run. It must be run by the mythtv user, whose home directory will contain the MythTV (and XMLTV) configuration files. The program is best run remotely (unless there is a graphical desktop on the server) by a command like ssh -X mythtv@new mythtv-setup.

Patience is required. The mythtv-setup program was not written for X11 and the X11 adapter has a difficult job. It is often hard to determine what button is selected or how to proceed (sometimes simply with ESC!). Sticking to the arrow, enter and escape keys best emulates a TV remote (for which the interface was designed).

In MythTV Setup:

• In the initial MythTV Startup Status ("Unable to connect to Database."), use the "Setup" button to get to "Database Configuration". Leave the default hostname (localhost), port (3306), database name (mythconverg) and user (mythtv). Enter the value of mythtv_dbpass (in private/vars-abbey.yml) for the password. Leave the rest of the settings at their default values. Leave "Database Configuration" by pressing Escape and confirming "Save and Exit".
• Once in MythTV Setup proper, you will see the main menu. Scroll down and choose "Storage Directories". In the Local Storage Groups dialog, add to the "Local 'Default' Storage Group Directories" a new directory: /home/mythtv/Recordings.

The abbey has a Silicon Dust Homerun HDTV Duo (with two tuners). It is setup as described in Cloistering, after which the tuner is accessible by name (e.g. new) on the cloister network. Assuming ping -c1 new works, the tuner should be accessible via the hdhomerun_config_gui command, a graphical interface contributed to Debian by Silicon Dust and found in the hdhomerun-config-gui package. The program, run with the command hdhomerun_config_gui, will broadcast on the localnet to find any Homeruns there, but the new tuner's domain name or IP address can also be entered.

In MythTV Setup:

• Choose "Capture cards".
  • Choose "(Add Capture Card)", then the "New Capture Card".
  • Choose Card Type and select "HDHomeRun Networked Tuner".
  • Press the right arrow key to see card type parameters. Choose the tuner's address, which should be listed assuming the tuner and TVR are on the same subnet (e.g. the private Ethernet).
  • Save and Exit (via Escape key).
• Choose "Video sources".
  • Choose "(New Video Source)", then the "New Video Source".
  • Enter video source name "Mr.Antenna".
  • Choose listings grabber "Schedules Direct JSON API (xmltv)".
  • Save and Exit.
• Choose "Input Connections".
  • Choose the HDHomeRun.
  • Choose video source "Mr.Antenna".
  • Save and Exit.
• Choose "Capture cards".
  • Add a second HDHomeRun as above.
  • Save and Exit.
• Choose "Input connections".
  • Connect the second HDHomeRun to Mr.Antenna as above.
  • Save and Exit.
• Exit MythTV Setup or continue directly to Scan for New Channels. In any case, do not run mythfilldatabase.

In MythTV Setup:

• Choose "Channel Editor".
  • Navigate to the "Delete" button, leaving Video Source All (right and down and down, or left six times, or sump'n). Confirm deletion of all channels.
  • Choose video source Mr.Antenna, then Channel Scan. Scroll down to the "scan" button and choose it (select and Enter).
  • Choose "Insert All" when the scan is complete and the count of channels is presented. Delete All unused transports.
  • Save and Exit from the scan. Exit from the channel editor.
• Exit MythTV Setup. Do not run mythfilldatabase.

The xmltv package, specifically its tv_grab_zz_sdjson program, is used to download broadcast listings from Schedules Direct. The program is run by the mythtv user (like mythtv-setup) and is initially configured (the first time) using its --configure option.

tv_grab_zz_sdjson --configure
cp ~/.xmltv/tv_grab_zz_sdjson.conf ~/.mythtv/Mr.Antenna.xmltv

The --configure command above prompts with many questions and creates ~/.xmltv/tv_grab_zz_sdjson.conf, which is copied to ~/.mythtv/Mr.Antenna.xmltv where mythfilldatabase will find it. Afterwards any re-configuration should use the following command.

tv_grab_zz_sdjson --configure \
        --config-file ~/.mythtv/Mr.Antenna.xmltv

Here is a transcript of a session with tv_grab_zz_sdjson. Note that the list of "inputs" available in a postal code typically ends with the OTA (over the air) broadcasts.

$ tv_grab_zz_sdjson --configure --config-file .mythtv/Mr.Antenna.xml
Cache file for lineups, schedules and programs.
Cache file: [/home/mythtv/.xmltv/tv_grab_zz_sdjson.cache]
If you are migrating from a different grabber selecting an alternate
 channel ID format can make the migration easier.
Select channel ID format:
0: Default Format (eg: I12345.json.schedulesdirect.org)
1: tv_grab_na_dd Format (eg: I12345.labs.zap2it.com)
2: MythTV Internal DD Grabber Format (eg: 12345)
Select one: [0,1,2 (default=0)] 
As the JSON data only includes the previously shown date normally th
 XML output should only have the date. However some programs such as
 older versions of MythTV also need a time.
Select previously shown format:
0: Date Only
1: Date And Time
Select one: [0,1 (default=0)] 
Schedules Direct username.
Username: USERNAME
Schedules Direct password.
Password: PASSWORD
** POST https://json.schedulesdirect.org/20141201/token ==> 200 OK
** GET https://json.schedulesdirect.org/20141201/status ==> 200 OK (
** GET https://json.schedulesdirect.org/20141201/lineups ==> 200 OK
This step configures the lineups enabled for your Schedules Direct
 account. It impacts all other configurations and programs using the
 JSON API with your account. A maximum of 4 lineups can by added to
 your account. In a later step you will choose which lineups or
 channels to actually use for this configuration.
Current lineups enabled for your Schedules Direct account:
#. Lineup ID | Name | Location | Transport
1. USA-OTA-57719 | Local Over the Air Broadcast | 57719 | Antenna
Edit account lineups: [continue,add,delete (default=continue)] 
Choose whether you want to include complete lineups or individual
 channels for this configuration.
Select mode: [lineups,channels (default=lineups)] 
** GET https://json.schedulesdirect.org/20141201/lineups ==> 200 OK
Choose lineups to use for this configuration.
USA-OTA-57719 [yes,no,all,none (default=no)] all

Once configured, the mythfilldatabase program should be able to use tv_grab_zz_sdjson to connect to Schedules Direct and download the chosen line-up. However mythfilldatabase is happiest when the backend is running, so it is not run until then.

If the mythfilldatabase command fails or expected listings do not appear, more information is available by adding the --verbose option. The --help option also reveals much, including a --manual option for "interactive configuration".

sudo -H -u mythtv mythfilldatabase --verbose

The command might, for example, show that it is failing to run a tv_grab_zz_sdjson command like the following.

nice tv_grab_zz_sdjson \
        --config-file '/home/mythtv/.mythtv/Mr.Antenna.xmltv' \
        --output /tmp/myths5Sq35 --quiet

Running a similar command (without --quiet) might be more revealing.

sudo -H -u mythtv \
    tv_grab_zz_sdjson \
        --config-file '/home/mythtv/.mythtv/Mr.Antenna.xmltv' \
        --output /tmp/mythFUBAR

The abbey directs MythTV log messages to /var/log/mythtv.log (and away from /var/log/syslog) and rotates the log file.

- name: Install =/etc/rsyslog.d/40-mythtv.conf.
  become: yes
  copy:
    content: |
      :msg,startswith," myth" -/var/log/mythtv.log
      & stop
    dest: /etc/rsyslog.d/40-mythtv.conf

- name: Install =/etc/logrotate.d/mythtv=.
  become: yes
  copy:
    content: |
      /var/log/mythtv.log {
          daily
          size=10M
          rotate 7
          notifempty
          copytruncate
          missingok
          postrotate
              reload rsyslog >/dev/null 2>&1 || true
          endscript
      }
    dest: /etc/logrotate.d/mythtv

After configuring with mythtv-setup as discussed above, start and enable (at boot time) the mythtv-backend service.

sudo systemctl enable mythtv-backend
sudo systemctl start mythtv-backend
systemctl status -l mythtv-backend
sudo -u mythtv mythfilldatabase

MythWeb, like MythTV, is installed from a Git repository. The following commands create /usr/local/share/mythtv/mythweb/ by cloning the MythWeb repository in /usr/local/src/mythweb/, checking out the appropriate branch, and copying the appropriate portion.

cd /usr/local/src/
git clone https://github.com/MythTV/mythweb
( cd mythweb/; git checkout fixes/32 )
rsync -C mythweb /usr/local/share/mythtv/

The following tasks take care of the rest of the installation.

- name: Install MythWeb requisites.
  become: yes
  apt:
    pkg: [ apache2, php, php-mysql ]

- name: Install MythWeb in web server DocumentRoot.
  file:
    state: link
    src: /usr/local/share/mythtv/mythweb
    dest: /var/www/html/mythweb

- name: Configure MythWeb data directory.
  file:
    state: directory
    dest: /var/www/html/mythweb/data
    group: www-data
    mode: u=rwx,g+rwx,o=rx

- name: Install MythWeb configuration.
  become: yes
  template:
    src: mythweb.conf.j2
    dest: /etc/apache2/sites-available/mythweb.conf
  notify: Restart Apache2.

- name: Enable MythWeb configuration.
  become: yes
  command:
    cmd: a2ensite -q mythweb
    creates: /etc/apache2/sites-enabled/mythweb.conf
  notify: Restart Apache2.

#
# Apache configuration directives for MythWeb.
#
# Note that this file is maintained by the network administration.
<Directory "/var/www/html/mythweb/data">
    # For Apache 2.2
    #Options -All +FollowSymLinks +IncludesNoExec
    # For Apache 2.4+
    Options +FollowSymLinks +IncludesNoExec
</Directory>
<Directory "/var/www/html/mythweb" >
    <Files mythweb.*>
    setenv db_server "127.0.0.1"
    setenv db_name "mythconverg"
    setenv db_login "mythtv"
    setenv db_password "{{ mythtv_dbpass }}"
    </Files>
    <Files *.php>
        php_value file_uploads                  0
        php_value allow_url_fopen               On
        php_value zlib.output_handler           Off
        php_value memory_limit                  64M
        php_value max_execution_time 30
        php_value display_startup_errors        On
        php_value display_errors                On
    </Files>
    RewriteEngine  on
    RewriteRule \
^(css|data|images|js|themes|skins|README|INSTALL|[a-z_]+\.(php|pl))(/|$)\
        - [L]
    RewriteRule ^(pl(/.*)?)$            mythweb.pl/$1  [QSA,L]
    RewriteRule ^(.+)$                  mythweb.php/$1 [QSA,L]
    RewriteRule ^(.*)$                  mythweb.php    [QSA,L]
    AllowOverride All
    Options         FollowSymLinks
    AddType video/nuppelvideo   .nuv
    AddType image/x-icon        .ico
    <IfModule deflate_module>
        BrowserMatch ^Mozilla/4 gzip-only-text/html
        BrowserMatch ^Mozilla/4\.0[678] no-gzip
        BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
        AddOutputFilterByType DEFLATE text/html
        AddOutputFilterByType DEFLATE text/css
        AddOutputFilterByType DEFLATE application/x-javascript
    </IfModule>
    <IfModule headers_module>
        Header append Vary User-Agent env=!dont-vary
    </IfModule>
    <Files *.pl>
        SetHandler cgi-script
        Options +ExecCGI
    </Files>

</Directory>

The abbey changes location almost weekly, so its HDTV broadcast area changes frequently. At the start of a long stay the administrator uses the MythTV Setup program to scan for the new area's channels, as described in Scan for New Channels.

To change MythTV's "listings", the administrator needs the new area's postal code and the username and password of the abbey's Schedules Direct account. The administrator then runs the tv_grab_zz_sdjson program as user mythtv.

tv_grab_zz_sdjson --configure \
        --config-file ~/.mythtv/Mr.Antenna.xmltv

The program will prompt for the zip code and offer a list of "inputs" available in that area, as described in Configure XMLTV.

This is much like the example (test) institutional configuration file, except the roles are found in Institute/roles/ as well as roles/.

[defaults]
interpreter_python=/usr/bin/python3
vault_password_file=Secret/vault-password
inventory=hosts
roles_path=roles:Institute/roles

This playbook provisions the entire network by applying first the institutional roles, then the liturgical roles.

---
- name: Configure All
  hosts: all
  roles: [ all ]

- name: Configure Front
  hosts: front
  roles: [ front, abbey-front ]

- name: Configure Gate
  hosts: gate
  roles: [ gate ]

- name: Configure Core
  hosts: core
  roles: [ core, abbey-core ]

- name: Configure Cloister
  hosts: campus
  roles: [ campus, abbey-cloister ]

- name: Configure Weather
  hosts: weather
  roles: [ abbey-weather ]

- name: Configure DVRs
  hosts: dvrs
  roles: [ abbey-dvr ]

- name: Configure TVRs
  hosts: tvrs
  roles: [ abbey-tvr ]

Institutional sub-commands:

config

Check/Set the configuration of one or all hosts.

new

Create system accounts for a new member.

old

Disable system accounts for a former member.

pass

Set the password of a current member.

client

Produce an OpenVPN configuration (.ovpn) file for a member's device.

Liturgical sub-commands:

tz

Run timedatectl set-timezone on cloister servers.

upgrade

Run apt update; apt full-upgrade --autoremove on all hosts.

reboots

Look for /run/reboot* on all hosts.

versions

Report ansible_distribution, _distribution_version, and _architecture for all hosts.

The script begins with the following prefix and trampolines.

#!/usr/bin/perl -w
#
# DO NOT EDIT.  This file was tangled from README.org.

use strict;

if (grep { $_ eq $ARGV[0] } qw(CA config new old pass client)) {
  exec "./Institute/inst", @ARGV;
}

The small institute's ./inst command expects to be running in Institute/, not ./, but it only references public/, private/, Secret/ and playbooks/check-inst-vars.yml, and will find the abbey specific versions of these. The roles_path setting in ansible.cfg effectively merges the institutional roles into the distinctly named abbey specific roles. The roles likewise reference files with relative names, and will find the abbey specific private/ directory (named ../private/ relative to playbooks/).

Ansible does not implement a playbooks_path key, so the following code block "duplicates" the action of the institute's check-inst-vars.yml.

- import_playbook: ../Institute/playbooks/check-inst-vars.yml

The script implements an upgrade sub-command that runs apt update and apt full-upgrade --autoremove on all abbey managed machines. It recognizes an optional -n flag indicating that the upgrade tasks should only be checked. Any other (single, optional) argument must be a limit pattern. For example:

./abbey upgrade
./abbey upgrade -n
./abbey upgrade core
./abbey upgrade -n core
./abbey upgrade '!front'

if ($ARGV[0] eq "upgrade") {
  shift;
  my @args = ( "-e", "\@Secret/become.yml" );
  if (defined $ARGV[0] && $ARGV[0] eq "-n") {
    shift;
    push @args, "--check", "--diff";
  }
  if (defined $ARGV[0]) {
    my $limit = $ARGV[0];
    shift;
    die "illegal characters: $limit"
      if $limit !~ /^!?[a-z][-a-z0-9,!]+$/;
    push @args, "-l", $limit;
  }
  exec ("ansible-playbook", @args, "playbooks/upgrade.yml");
}

- hosts: all
  tasks:

  - name: Upgrade packages.
    become: yes
    apt:
      update_cache: yes
      upgrade: full
      autoremove: yes
      purge: yes
      autoclean: yes

  - name: Check for /run/reboot-required.
    stat:
      path: /run/reboot-required
    no_log: true
    register: st

  - debug:
      msg: Reboot required.
    when: st.stat.exists

The script implements a reboots sub-command that looks for /run/reboot-required on all abbey managed machines.

if ($ARGV[0] eq "reboots") {
  exec ("ansible-playbook", "-e", "\@Secret/become.yml",
        "playbooks/reboots.yml");
}

---
- hosts: all
  tasks:

  - stat:
      path: /run/reboot-required
    register: st

  - debug:
      msg: Reboot required.
    when: st.stat.exists

The script implements a versions sub-command that reports the operating system version of all abbey managed machines.

if ($ARGV[0] eq "versions") {
  exec ("ansible-playbook", "-e", "\@Secret/become.yml",
        "playbooks/versarch.yml");
}

- hosts: all
  tasks:
  - debug:
      msg: >-
        {{ ansible_distribution }}
        {{ ansible_distribution_version }}
        {{ ansible_architecture }}

The abbey changes location almost weekly, so its timezone changes occasionally. Droplet does not move. Gate and other simple servers (the weather monitors) are kept in UTC. Core, the DVRs, TVRs, and the desktops all want updating to the current local timezone. The desktops are managed maually, but the rest can all be updated using Ansible.

The tz sub-command runs the timezone.yml playbook, which uses the current timezone/city on the administrator's notebook and updates Core, the DVRs and TVRs. Each runs timedatectl set-timezone and restarts the affected services.

This is an experimental playbook until it is used/tested with separate machines hosting the DVR and TVR services. It assumes each host sees the new_tz result registered by it in a previous play and not by the last host in the previous play.

if ($ARGV[0] eq "tz") {
  my $city = `cat /etc/timezone`; chomp $city;
  my $zone = `date +%Z`; chomp $zone;
  print "Setting timezones to $city.\n";
  exec ("ansible-playbook", "-e", "\@Secret/become.yml",
        "-e", "zone=$zone", "-e", "city=$city",
        "playbooks/timezone.yml");
}

---
- hosts: core, dvrs, tvrs, webtvs
  tasks:
  - name: Update timezone.
    become: yes
    command: timedatectl set-timezone {{ city }}
    when: ansible_date_time.tz != zone
    register: new_tz

- hosts: dvrs
  tasks:
  - name: Restart Zoneminder.
    become: yes
    systemd:
      service: "{{ item }}"
      state: restarted
    loop: [ mysql, zoneminder ]
    when: new_tz.changed

- hosts: tvrs
  tasks:
  - name: Restart MythTV.
    become: yes
    systemd:
      service: "{{ item }}"
      state: restarted
    loop: [ mysql, mythtv-backend ]
    when: new_tz.changed

- hosts: core
  tasks:
  - name: Update PHP date.timezone.
    become: yes
    lineinfile:
      regexp: date.timezone ?=
      line: date.timezone = {{ city }}
      path: "{{ item }}"
    loop:
    - /etc/php/8.2/cli/php.ini
    - /etc/php/8.2/apache2/php.ini
    notify: Restart Apache2.
  handlers:
  - name: Restart Apache2.
    become: yes
    systemd:
      service: apache2
      state: restarted

A wireless IoT device (smart TV, Blu-ray deck, etc.) cannot install Debian nor even an OpenVPN app from F-Droid. And it shouldn't. As an untrustworthy bit of kit, it should have no access to the cloister, merely the Internet. It need not appear in the Ansible inventory.

IoT devices trusted enough to be patched to the cloister Ethernet (IP cameras, TV Tuners, etc.) are added to /etc/dhcp/dhcpd.conf and given a private domain name as described in the following steps.

• Add to Core DHCP
• Create Wired Domain Name

Wireless IoT devices are manually configured with the cloister Wi-Fi password and may be given a private domain name as described in the last step:

• Create Wireless Domain Name

The abbey's Raspberry Pi runs the Raspberry Pi OS desktop off an external, USB3.0 SSD. A fresh install should go something like this:

• Write the disk image, 2023-12-05-raspios-bookworm-arm64.img.xz, to the SSD and plug it into the Pi. Leave the µSD card socket empty.
• Attach an HDMI monitor, a USB keyboard/mouse, and the cloister Ethernet, and power up.
• Answer first-boot installation questions:
  • Language: English (USA)
  • Keyboard: English (USA)
  • new username: sysadm
  • new password: fubar
• Add to Core DHCP
• Create Wired Domain Name
• Log in as sysadm on the console.
• Run sudo raspi-config and use the following menu items.
  • S4 Hostname (Set name for this computer on a network): new
  • I1 SSH (Enable/disable remote command line access using SSH): enable
  • A1 Expand Filesystem (Ensures that all of the SD card is available)
• Update From Cloister Apt Cache
• Authorize Remote Administration
• Configure with Ansible

If the Pi is going to operate wirelessly, the following additional steps are taken.

• Connect to Cloister Wi-Fi
• Connect to Cloister VPN
• Create Wireless Domain Name

Most of the abbey's machines, like Core and Gate, are general-purpose PCs running Debian. The process of cloistering these machines follows.

• Write the disk image, e.g. debian-12.2.0-amd64-netinst.iso, to a USB drive and insert it in the PC.
• Attach an HDMI monitor, a USB keyboard/mouse, and the cloister Ethernet, and power up. Choose to boot from the USB drive.
• Answer first-boot installation questions:
  • Language: English (USA)
  • Keyboard: English (USA)
  • new username: sysadm
  • new password: fubar
• Add to Core DHCP
• Create Wired Domain Name
• Log in as sysadm on the console.
• Update From Cloister Apt Cache

• Install OpenSSH. Plain Debian does not come with OpenSSH installed.

  sudo apt install openssh-server
  

• Authorize Remote Administration
• Configure with Ansible

If the PC is going to operate wirelessly, the following additional steps are taken.

• Connect to Cloister Wi-Fi
• Connect to Cloister VPN
• Create Wireless Domain Name

When a new machine is connected to the cloister Ethernet, its MAC address must be added to Core's DHCP configuration. Core does not provide network addresses to new devices automatically.

IoT devices (IP cameras, HDTV tuners, etc.) often have their MAC address printed on their case or mentioned in a configuration page. The MAC address must also appear in the device's DHCP Discover broadcasts, which are logged to /var/log/daemon.log on Core. As a last (or first!) resort, the following command line should reveal the new device's MAC.

tail -100 /var/log/daemon.log | grep DISCOVER

With the new device's Ethernet MAC in hand, a stanza like the following is added to the bottom of private/core-dhcpd.conf. The IP address must be unique. Typically the next host number after the last entry is chosen.

host new {
  hardware ethernet 08:00:27:f3:41:66; fixed-address 192.168.56.4; }

The DHCP service is then restarted (not reloaded).

sudo systemctl restart isc-dhcp-server

Soon after this the device should be offered a lease for its IP address, 192.168.56.4. It might be power cycled to speed up the process.

When successful, the following command shows the device is accessible, reporting 1 packets transmitted, 1 received, 0% packet loss....

ping -c1 192.168.56.4

A wired device is assigned an IP address when it is added to Core's DHCP configuration (as in Add to Core DHCP). A private domain name is then associated with this address. If the device is intended to operate wirelessly, the name for its address is modified with a -w suffix. Thus new-w.small.private would be the name of the new device while it is temporarily connected to the cloister Ethernet, and new.small.private would be its "normal" name used when it is on the cloister Wi-Fi.

The private domain name is created by adding a line like the following to private/db.domain and incrementing the serial number at the top of the file.

new-w   IN      A       192.168.56.4

The reverse mapping is also created by adding a line like the following to private/db.private and incrementing the serial number at the top of that file.

4       IN      PTR     new-w.small.private.

After ./abbey config core updates Core, resolution of the new-w name can be tested.

resolvectl query new-w.small.private.
resolvectl query 192.168.56.4

• Log in as sysadm on the console.

• Create /etc/apt/apt.conf.d/01proxy.

  D=apt-cacher.small.private.
  echo "Acquire::http::Proxy \"http://$D:3142\";" \
  | sudo tee /etc/apt/apt.conf.d/01proxy
  

• Update the system and reboot.

  sudo apt update
  sudo apt full-upgrade --autoremove
  sudo reboot
  

To remotely administer new-w, Ansible must be authorized to login as sysadm@new-w without a login password, using an SSH key pair. This is accomplished by copying Ansible's SSH public key to new-w.

scp Secret/ssh_admin/id_rsa.pub sysadm@new-w:admin_key

Then on new-w (logged in as sysadm) the public key is installed in ~sysadm/.ssh/authorized_keys.

( cd; umask 077; mkdir .ssh; cp admin_key .ssh/authorized_keys )

Now the administrator can test access to new-w using Ansible's SSH key.

ssh -i Secret/ssh_admin/id_rsa sysadm@new-w

With remote administration authorized and tested (as in Authorize Remote Administration), and the machine connected to the cloister Ethernet, the configuration of new-w can be completed by Ansible. Note that if the machine is staying on the cloister Ethernet, its domain name will be new (having had no -w suffix added).

First new-w is added to Ansible's inventory in hosts. A new-w section is added to the list of all hosts, and an empty section of the same name is added to the list of campus hosts. If the machine uses the usual privileged account name, sysadm, the ansible_user key in not needed.

hosts:
  ...
  new-w:
    ansible_user: pi
    ansible_become_password: "{{ become_new }}"
  ...
children:
  ...
  campus:
    hosts:
      ...
      new-w:

If the sudo command on new-w never prompts sysadm for a password, then the ansible_become_password setting is also not needed. Otherwise, the password is added to Secret/become.yml as shown below.

echo -n "become_new: " >>Secret/become.yml
ansible-vault encrypt_string PASSWORD >>Secret/become.yml

Finally the ./abbey config new-w command is run. It will install several additional software packages and change several more configuration files.

./abbey config new-w

On an IoT device, or a Debian or Android "desktop", the cloister Wi-Fi name and password are entered manually. Once the device is connected, its Wi-Fi IP address may be discovered in its network settings, and perhaps via the access point's local domain, e.g. as new.lan on a desktop connected to the cloister Wi-Fi.

Wireless Debian machines use ifupdown configured with a short /etc/network/interfaces.d/wifi drop-in. In this example, the Wi-Fi interface on new is named wlan0.

auto wlan0
iface wlan0 inet dhcp
    wpa-ssid "Birchwood Abbey"
    wpa-psk "PASSWORD"

Once the sudo ifup wlan0 command is successful, the machine will get an IP address on the access point's local network (revealed by the command ip addr show dev wlan0).

The new Wi-Fi IP address, e.g. 192.168.10.225, should be tested on a desktop connected to the Wi-Fi using the following ping command.

ping -c1 192.168.10.225

Wireless devices (with the cloister Wi-Fi password) can get an IP address and a default route to the Internet with no special configuration. Neither said devices nor the access point require special configuration. Any Wi-Fi access point, e.g. as found in a cable modem, will work with zero configuration. The abbey's networks, however, are not accessible except via the cloister VPN.

Connections to the cloister VPN are authorized by OpenVPN configuration (.ovpn) files generated by the ./abbey client... command (aka The Client Command). These are secret files, kept readable only by their owners and are deleted after use. They are copied to new OpenVPN clients using secure (ssh) connections.

A wireless machine is assigned a Wi-Fi address when it connects to the cloister Wi-Fi, and a "VPN address" when it connects to Gate's OpenVPN server. The VPN address can be discovered by running ip addr show dev ovpn on the machine, or inspecting /etc/openvpn/ipp.txt on Gate. Once discovered, a private domain name, e.g. new.small.private, can be associated with the VPN address, e.g 10.84.138.7. The administrator adds a line like the following to private/db.domain and increments the serial number at the top of the file.

new     IN      A       10.84.138.7

The administrator also creates the reverse mapping by adding a line like the following to private/db.campus_vpn and incrementing the serial number at the top of that file.

7       IN      PTR     new.small.private.

After ./abbey config core updates Core, the administrator can test resolution of the new name.

resolvectl query new.small.private.
resolvectl query 10.84.138.7

A wireless device with no Ethernet interface and unable to run OpenVPN gets just a Wi-Fi address. It can be given a private domain name (e.g. new.small.private) associated with the Wi-Fi address (e.g. 192.168.10.225), but a reverse lookup on a machine connected to the Wi-Fi may yield a name like new.lan (provided by the access point) while elsewhere (e.g. on the cloister Ethernet) the IP address will not resolve at all. (There is no "reverse mapping" to be added to private/db.campus_vpn.)