A Small Institute

The institute has a public domain, e.g. small.example.org, and a private domain, e.g. small.private. The public has access only to the former and, as currently configured, to only one address (A record): Front's public IP address. Members connected to the campus, via wire or VPN, use the campus name server which can resolve institute private domain names like core.small.private. If small.private is also used as a search domain, members can use short names like core.

Front provides the public SMTP (Simple Mail Transfer Protocol) service that accepts email from the Internet, delivering messages addressed to the institute's domain name, e.g. to postmaster@small.example.org. Its Postfix server accepts email for member accounts and any public aliases (e.g. postmaster). Messages are delivered to member ~/Maildir/ directories via Dovecot.

If the campus is connected to the Internet, the new messages are quickly picked up by Core and stored in member ~/Maildir/ directories there. Securely stored on Core, members can decrypt and sort their email using common, IMAP-based tools. (Most mail apps can use IMAP, the Internet Message Access Protocol.)

Core transfers messages from Front using Fetchmail's --idle option, which instructs Fetchmail to maintain a connection to Front so that it can (with good campus connectivity) get notifications to pick up new email. Members of the institute typically employ email apps that work similarly, alerting them to new email on Core. Thus members enjoy email messages that arrive as fast as text messages (but with the option of real, end-to-end encryption).

If the campus loses connectivity to the Internet, new email accumulates in ~/Maildir/ directories on Front. If a member is abroad, with Internet access, their new emails can be accessed via Front's IMAPS (IMAP Secured [with SSL/TLS]) service, available at the institute domain name. When the campus regains Internet connectivity, Core will collect the new email.

Core is the campus mail hub, securely storing members' incoming emails, and relaying their outgoing emails. It is the "smarthost" for the campus. Campus machines send all outgoing email to Core, and Core's Postfix server accepts messages from any of the institute's networks.

Core delivers messages addressed to internal host names locally. For example webmaster@test.small.private is delivered to webmaster on Core. Core relays other messages to its smarthost, Front, which is declared by the institute's SPF (Sender Policy Framework) DNS record to be the only legitimate sender of institute emails. Thus the Internet sees the institute's outgoing email coming from a server at an address matching the domain's SPF record. The institute does not sign outgoing emails per DKIM (Domain Keys Identified Mail) yet.

TXT    v=spf1 ip4:159.65.75.60 -all

There are a number of configuration settings that, for interoperability, should be in agreement on the Postfix servers and the campus clients. Policy also requires certain settings on both Postfix or both Dovecot servers. To ensure that the same settings are applied on both, the shared settings are defined here and included via noweb reference in the server configurations. For example the Postfix setting for the maximum message size is given in a code block labeled postfix-message-size below and then included in both Postfix configurations wherever <<postfix-message-size>> appears.

Front provides the public HTTP service that serves institute web pages at e.g. https://small.example.org/. The small institute initially runs with a self-signed, "snake oil" server certificate, causing browsers to warn of possible fraud, but this certificate is easily replaced by one signed by a recognized authority, as discussed in The Front Role.

The Apache2 server finds its web pages in the /home/www/ directory tree. Pages can also come from member home directories. For example the HTML for https://small.example.org/~member would come from the /home/member/Public/HTML/index.html file.

The server does not run CGI scripts. This keeps Front's CPU requirements cheap. CGI scripts can be used on Core. Indeed Nextcloud on Core uses PHP and the whole LAMP (Linux, Apache, MySQL, PHP) stack.

Core provides a campus HTTP service with several virtual hosts. These web sites can only be accessed via the campus Ethernet or an institute VPN. In either situation Core's many private domain names become available, e.g. www.small.private. In many cases these domain names can be shortened e.g. to www. Thus the campus home page is accessible in a dozen keystrokes: http://www/ (plus Enter).

Core's web sites:

http://www/

is the small institute's campus web site. It serves files from the staff-writable /WWW/campus/ directory tree.

http://live/

is a local copy of the institute's public web site. It serves the files in the /WWW/live/ directory tree, which is mirrored to Front.

http://test/

is a test copy of the institute's public web site. It tests new web designs in the /WWW/test/ directory tree. Changes here are merged into the live tree, /WWW/live/, once they are complete and tested.

http://core/

is the Debian default site. The institute does not munge this site, to avoid conflicts with Debian-packaged web services (e.g. Nextcloud, AgentDVR, MythTV's MythWeb).

Core runs a cron job under a system account named monkey that mirrors /WWW/live/ to Front's /home/www/ every 15 minutes. Vandalism on Front should not be possible, but if it happens Monkey will automatically wipe it within 15 minutes.

Core runs Nextcloud to provide a private institute cloud at https://core.small.private/nextcloud/. It is managed manually per The Nextcloud Server Administration Guide. The code and data, including especially database dumps, are stored in /Nextcloud/ which is included in Core's backup procedure as described in Backups. The default Apache2 configuration expects to find the web scripts in /var/www/nextcloud/, so the institute symbolically links this to /Nextcloud/nextcloud/.

Note that authenticating to a non-HTTPS URL like http://core.small.private/ is often called out as insecure, but the domain name is private and the service is on a directly connected private network.

A small institute has just a handful of members. For simplicity (and thus security) static configuration files are preferred over complex account management systems, LDAP, Active Directory, and the like. The Ansible scripts configure the same set of user accounts on Core and Front. The Institute Commands (e.g. ./inst new dick) capture the processes of enrolling, modifying and retiring members of the institute. They update the administrator's membership roll, and run Ansible to create (and disable) accounts on Core, Front, Nextcloud, etc.

The small institute does not use disk quotas nor access control lists. It relies on Unix group membership and permissions. It is Debian based and thus uses "user groups" by default. Sharing is typically accomplished via the campus cloud and the resulting desktop files can all be private (readable and writable only by the owner) by default.

The institute keeps its "master secrets" in an encrypted volume on an off-line hard drive, e.g. a LUKS (Linux Unified Key Setup) format partition on a USB pen/stick. The Secret/ sub-directory is actually a symbolic link to this partition's automatic mount point, e.g. /media/sysadm/ADE7-F866/. Unless this volume is mounted (unlocked) at Secret/, none of the ./inst commands will work.

Chief among the institute's master secrets is the SSH key authorized to access privileged accounts on all of the institute servers. It is stored in Secret/ssh_admin/id_rsa. The complete list of the institute's SSH keys:

Secret/ssh_admin/

The SSH key pair for A Small Institute Administrator.

Secret/ssh_monkey/

The key pair used by Monkey to update the website on Front (and other unprivileged tasks).

Secret/ssh_front/

The host key pair used by Front to authenticate itself. The automatically generated key pair is not used. (Thus Core's configuration does not depend on Front's.)

The institute uses a couple X.509 certificates to authenticate servers. They are created by the EasyRSA Certificate Authority stored in Secret/CA/.

Secret/CA/pki/ca.crt

The institute CA certificate, used to sign the other certificates.

Secret/CA/pki/issued/small.example.org.crt

The public Postfix, Dovecot and Apache servers on Front.

Secret/CA/pki/issued/core.small.private.crt

The campus Postfix, Dovecot and Apache (thus Nextcloud) servers on Core.

The ./inst client command updates the institute membership roll, which lists members and their clients' public keys, and is stored in private/members.yml.

Finally, the institute uses an OpenPGP key to secure sensitive emails (containing passwords or private keys) to Core.

Secret/root.gnupg/

The "home directory" used to create the public/secret key pair.

Secret/root-pub.pem

The ASCII armored OpenPGP public key for e.g. root@core.small.private.

Secret/root-sec.pem

The ASCII armored OpenPGP secret key.

The institute administrator updates a couple encrypted copies of this drive after enrolling new members, changing a password, (de)authorizing a VPN client, etc.

rsync -a Secret/ Secret2/
rsync -a Secret/ Secret3/

This is out of consideration for the fragility of USB drives, and the importance of a certain SSH private key, without which the administrator will have to login with a password, hopefully stored in the administrator's password keep, to install a new SSH key.

The small institute backs up its data, but not so much so that nothing can be deleted. It actually mirrors user directories (/home/), the web sites (/WWW/), Nextcloud (/Nextcloud/), and any capitalized root directory entry, to a large off-line disk. Where incremental backups are desired, a CMS like git is used.

Off-site backups are not a priority due to cost and trust issues, and the low return on the investment given the minuscule risk of a catastrophe big enough to obliterate all local copies. And the institute's public contributions are typically replicated in public code repositories like GitHub and GNU Savannah.

The following example /usr/local/sbin/backup script pauses Nextcloud, dumps its database, rsyncs /home/, /WWW/ and /Nextcloud/ to a /backup/ volume (mounting and unmounting /backup/ if necessary), then continues Nextcloud. The script assumes the backup volume is labeled Backup and formatted per LUKS version 2.

Given the -n flag, the script does a "pre-sync" which does not pause Nextcloud nor dump its DB. A pre-sync gets the big file (video) copies done while Nextcloud continues to run. A follow-up sudo backup, without -n, produces the complete copy (with all the files mentioned in the Nextcloud database dump).

#!/bin/bash -e
#
# DO NOT EDIT.
#
# Maintained (will be replaced) by Ansible.
#
# sudo backup [-n]

if [ `id -u` != "0" ]
then
    echo "This script must be run as root."
    exit 1
fi

if [ "$1" = "-n" ]
then
    presync=yes
    shift
fi

if [ "$#" != "0" ]
then
    echo "usage: $0 [-n]"
    exit 2
fi

function cleanup () {
    sleep 2
    finish
}

trap cleanup SIGHUP SIGINT SIGQUIT SIGPIPE SIGTERM

function start () {

    if ! mountpoint -q /backup/
    then
        echo "Mounting /backup/."
        cryptsetup luksOpen /dev/disk/by-partlabel/Backup backup
        mount /dev/mapper/backup /backup
    else
        echo "Found /backup/ already mounted."
    fi

    if [ ! -d /backup/home ]
    then
        echo "The backup device should be mounted at /backup/"
        echo "yet there is no /backup/home/ directory."
        exit 2
    fi

    if [ ! $presync ]
    then
        echo "Putting Nextcloud into maintenance mode."
        ( cd /Nextcloud/nextcloud/
          sudo -u www-data php occ maintenance:mode --on &>/dev/null )

        echo "Dumping Nextcloud database."
        ( cd /Nextcloud/
          umask 07
          BAK=`date +"%Y%m%d%H%M"`-dbbackup.bak.gz
          CNF=/Nextcloud/dbbackup.cnf
          mysqldump --defaults-file=$CNF nextcloud | gzip > $BAK
          chmod 440 $BAK
          ls -t1 *-dbbackup.bak.gz | tail -n +4 \
          | while read; do rm "$REPLY"; done
        )
    fi

}

function finish () {

    if [ ! $presync ]
    then
        echo "Putting Nextcloud back into service."
        ( cd /Nextcloud/nextcloud/
          sudo -u www-data php occ maintenance:mode --off &>/dev/null )
    fi

    if mountpoint -q /backup/
    then
        echo "Unmounting /backup/."
        umount /backup
        cryptsetup luksClose backup
        echo "Done."
        echo "The backup device can be safely disconnected."
    fi
}

start

for D in /home /[A-Z]*; do
    echo "Updating /backup$D/."
    ionice --class Idle --ignore \
        rsync -av --delete --exclude=.NoBackups $D/ /backup$D/
done

finish

The small institute's domain name is used quite frequently in the Ansible code. The example used here is small.example.org. The following line sets domain_name to that value. (Ansible will then replace {{ domain_name }} in the code with small.example.org.)

---
domain_name: small.example.org

The institute's private domain is treated as sensitive information, and so is "tangled" into the example file private/vars.yml rather than public/vars.yml. The example file is used for testing, and serves as the template for an actual, private, private/var.yml file that customizes this Ansible code for an actual, private, small institute.

The institute's private domain name should end with one of the top-level domains set aside for this purpose: .intranet, .internal, .private, .corp, .home or .lan.¹ It is hoped that doing so will increase the chances that some abomination like DNS-over-HTTPS will pass us by.

---
domain_priv: small.private

The small institute uses a private Ethernet, two VPNs, and a "wild", untrusted Ethernet for the campus Wi-Fi access point(s) and wired IoT appliances. Each must have a unique private network address. Hosts using the VPNs are also using foreign private networks, e.g. a notebook on a hotel Wi-Fi. To better the chances that all of these networks get unique addresses, the small institute uses addresses in the IANA's (Internet Assigned Numbers Authority's) private network address ranges except the 192.168 address range already in widespread use. This still leaves 69,632 8 bit networks (each addressing up to 254 hosts) from which to choose. The following table lists their CIDRs (subnet numbers in Classless Inter-Domain Routing notation) in abbreviated form (eliding 69,624 rows).

The following Emacs Lisp randomly chooses one of these 8 bit subnets. The small institute used it to pick its four private subnets. An example result follows the code.

(let ((bytes
         (let ((i (random (+ 256 16))))
           (if (< i 256)
               (list 10        i         (1+ (random 254)))
             (list  172 (+ 16 (- i 256)) (1+ (random 254)))))))
  (format "%d.%d.%d.0/24" (car bytes) (cadr bytes) (caddr bytes)))

The four private networks are named and given example CIDRs in the code block below. The small institute treats these addresses as sensitive information so again the code block below "tangles" into private/vars.yml rather than public/vars.yml. Two of the addresses are in 192.168 subnets because they are part of a test configuration using mostly-default VirtualBoxes (described here).

private_net_cidr:           192.168.56.0/24
wild_net_cidr:              192.168.57.0/24
public_wg_net_cidr:         10.177.87.0/24
campus_wg_net_cidr:         10.84.139.0/24

The network addresses are needed in several additional formats, e.g. network address and subnet mask (10.84.139.0 255.255.255.0). The following boilerplate uses Ansible's ipaddr filter to set several corresponding variables, each with an appropriate suffix, e.g. _net_and_mask rather than _net_cidr.

private_net:
           "{{ private_net_cidr | ansible.utils.ipaddr('network') }}"
private_net_mask:
           "{{ private_net_cidr | ansible.utils.ipaddr('netmask') }}"
private_net_and_mask:      "{{ private_net }} {{ private_net_mask }}"
wild_net:     "{{ wild_net_cidr | ansible.utils.ipaddr('network') }}"
wild_net_mask:
              "{{ wild_net_cidr | ansible.utils.ipaddr('netmask') }}"
wild_net_and_mask:               "{{ wild_net }} {{ wild_net_mask }}"
wild_net_broadcast:
            "{{ wild_net_cidr | ansible.utils.ipaddr('broadcast') }}"
public_wg_net:
         "{{ public_wg_net_cidr | ansible.utils.ipaddr('network') }}"
public_wg_net_mask:
         "{{ public_wg_net_cidr | ansible.utils.ipaddr('netmask') }}"
public_wg_net_and_mask:
                       "{{ public_wg_net }} {{ public_wg_net_mask }}"
campus_wg_net:
         "{{ campus_wg_net_cidr | ansible.utils.ipaddr('network') }}"
campus_wg_net_mask:
         "{{ campus_wg_net_cidr | ansible.utils.ipaddr('netmask') }}"
campus_wg_net_and_mask:
                       "{{ campus_wg_net }} {{ campus_wg_net_mask }}"

This is obvious, site-independent, non-private boilerplate and so goes in a defaults/main.yml file in each role. The variables can then be overridden by adding them to the site-specific private/vars.yml. The block is referenced with <<network-vars>> and tangled into each role's defaults/main.yml file.

The institute prefers to configure its services with IP addresses rather than domain names, and one of the most important for secure and reliable operation is Front's public IP address known to the world by the institute's Internet domain name.

front_addr: 192.168.15.4

The example address is a private network address because the example configuration is intended to run in a test jig made up of VirtualBox virtual machines and networks.

Finally, five host addresses are needed frequently in the Ansible code. Each is made available in both CIDR and IPv4 address formats. Again this is site-independent, non-private boilerplate referenced with address-vars in the default/main.yml files.

core_addr_cidr:  "{{ private_net_cidr | ansible.utils.ipaddr('1') }}"
gate_addr_cidr:  "{{ private_net_cidr | ansible.utils.ipaddr('2') }}"
gate_wild_addr_cidr:
                    "{{ wild_net_cidr | ansible.utils.ipaddr('1') }}"
front_wg_addr_cidr:
               "{{ public_wg_net_cidr | ansible.utils.ipaddr('1') }}"
core_wg_addr_cidr:
               "{{ public_wg_net_cidr | ansible.utils.ipaddr('2') }}"

core_addr:   "{{ core_addr_cidr | ansible.utils.ipaddr('address') }}"
gate_addr:   "{{ gate_addr_cidr | ansible.utils.ipaddr('address') }}"
gate_wild_addr:
        "{{ gate_wild_addr_cidr | ansible.utils.ipaddr('address') }}"
front_wg_addr:
         "{{ front_wg_addr_cidr | ansible.utils.ipaddr('address') }}"
core_wg_addr:
          "{{ core_wg_addr_cidr | ansible.utils.ipaddr('address') }}"

Front is the small institute's public facing server, a virtual machine on the Internets. It needs only as much disk as required by the institute's public web site. Often the cheapest offering (4GB RAM, 1 core, 20GB disk) is sufficient. The provider should make it easy and fast to (re)initialize the machine to a factory fresh Debian Server, and install additional Debian software packages. Indeed it should be possible to quickly re-provision a new Front machine from a frontier Internet café using just the administrator's notebook.

Core is the small institute's private file, email, cloud and whatnot server. It should have some serious horsepower (RAM, cores, GHz) and storage (hundreds of gigabytes). An old desktop system might be sufficient and if later it proves it is not, moving Core to new hardware is "easy" and good practice. It is also straightforward to move the heaviest workloads (storage, cloud, internal web sites) to additional machines.

Core need not have a desktop, and will probably be more reliable if it is not also playing games. It will run continuously 24/7 and will benefit from a UPS (uninterruptible power supply). It's file system and services are critical.

The following example prepared a new core on a PC with Debian 11 freshly installed. During installation, the machine was named core, no desktop or server software was installed, no root password was set, and a privileged account named sysadm was created (per the policy in The Administration Accounts).

New password: oingstramextedil
Retype new password: oingstramextedil
...
        Full Name []: System Administrator
...
Is the information correct? [Y/n] 

The password was generated by gpw, saved in the administrator's password keep, and later added to Secret/become.yml as shown below. (Producing a working Ansible configuration with Secret/become.yml file is described in The Ansible Configuration.)

notebook$ gpw 1 16
oingstramextedil
notebook$ echo -n "become_core: " >>Secret/become.yml
notebook$ ansible-vault encrypt_string oingstramextedil \
notebook_     >>Secret/become.yml

With Debian freshly installed, Core needed several additional software packages. The administrator temporarily plugged Core into a cable modem and installed them as shown below.

$ sudo apt install wireguard systemd-resolved unattended-upgrades \
_                  chrony isc-dhcp-server bind9 apache2 postfix \
_                  dovecot-imapd fetchmail rsync gnupg

Manual installation of Postfix prompted for configuration type and mail name. The answers given are listed here.

• General type of mail configuration: Internet Site
• System mail name: core.small.private

The host then needed to be rebooted to get its name service working again after systemd-resolved was installed. (Any help with this will be welcome!) After rebooting and re-logging in, yet more software packages were installed.

The Nextcloud configuration required Apache2, MariaDB and a number of PHP modules. Installing them while Core was on a cable modem sped up final configuration "in position" (on a frontier).

$ sudo apt install mariadb-server php php-{apcu,bcmath,curl,gd,gmp}\
_                  php-{json,mysql,mbstring,intl,imagick,xml,zip} \
_                  imagemagick libapache2-mod-php

Similarly, the NAGIOS configuration required a handful of packages that were pre-loaded via cable modem (to test a frontier deployment).

$ sudo apt install nagios4 monitoring-plugins-basic lm-sensors \
_                  nagios-nrpe-plugin

Next, the administrator concatenated a personal public ssh key and the key found in Secret/ssh_admin/ (created by The CA Command) into an admin_keys file, copied it to Core, and installed it as the authorized_keys for sysadm.

notebook$ cat ~/.ssh/id_rsa.pub Secret/ssh_admin/id_rsa.pub \
notebook_     > admin_keys
notebook$ scp admin_keys sysadm@core.lan:
The authenticity of host 'core.lan' can't be established.
....
Are you sure you want to continue connecting (...)? yes
...
sysadm@core.lan's password: oingstramextedil
notebook$ ssh sysadm@core.lan
sysadm@core.lan's password: oingstramextedil
sysadm@core$ ( umask 077; mkdir .ssh; \
sysadm@core_   cp admin_keys .ssh/authorized_keys )
sysadm@core$ rm admin_keys
sysadm@core$ logout
notebook$ rm admin_keys
notebook$

Note that the name core.lan should be known to the cable modem's DNS service. An IP address might be used instead, discovered with an ip -4 a command on Core.

Now Core no longer needed the Internets so it was disconnected from the cable modem and connected to the campus Ethernet switch. Its primary Ethernet interface was temporarily (manually) configured with a new, private IP address and a default route.

In the example command lines below, the address 10.227.248.1 was generated by the random subnet address picking procedure described in Subnets, and is named core_addr in the Ansible code. The second address, 10.227.248.2, is the corresponding address for Gate's Ethernet interface, and is named gate_addr in the Ansible code.

sysadm@core$ sudo ip address add 10.227.248.1 dev enp82s0
sysadm@core$ sudo ip route add default via 10.227.248.2 dev enp82s0

At this point Core was ready for provisioning with Ansible.

Gate is the small institute's route to the Internet, and the campus Wi-Fi's route to the private Ethernet. It has three network interfaces.

1. lan is its main Ethernet interface, connected to the campus's private Ethernet switch.
2. wild is its second Ethernet interface, connected to the untrusted network of campus IoT appliances and Wi-Fi access point(s).
3. isp is its third network interface, connected to the campus ISP. This could be an Ethernet device connected to a cable modem, a USB port tethered to a phone, a wireless adapter connected to a campground Wi-Fi access point, etc.

=============== | ==================================================
                |                                           Premises
          (Campus ISP)                                              
                |            +----Member's notebook on campus       
                |            |                                      
                | +----(Campus Wi-Fi)                               
                | |                                                 
============== Gate ================================================
                |                                            Private
                +----Ethernet switch                                

The all role's task contains a reference to a common institute particular, the institute's domain_name, a variable found in the public/vars.yml file. Thus the first task of the all role is to include the variables defined in this file (described in The Particulars). The code block below is the first to tangle into roles/all/tasks/main.yml.

---
- name: Include public variables.
  include_vars: ../public/vars.yml

The systemd-networkd and systemd-resolved service units are not enabled by default in Debian, but are the default in Ubuntu. The institute attempts to make use of their link-local name resolution, so they are enabled on all institute hosts.

The /usr/share/doc/systemd/README.Debian.gz file recommends both services be enabled and /etc/resolv.conf be replaced with a symbolic link to /run/systemd/resolve/resolv.conf. The institute follows these recommendations (and not the suggestion to enable "persistent logging", yet). In Debian 12 there is a systemd-resolved package that symbolically links /etc/resolv.conf (and provides /lib/systemd/systemd-resolved, formerly part of the systemd package).

- name: Install systemd-resolved.
  become: yes
  apt: pkg=systemd-resolved
  when:
  - ansible_distribution == 'Debian'
  - 11 < ansible_distribution_major_version|int

- name: Start systemd-networkd.
  become: yes
  systemd:
    service: systemd-networkd
    state: started
  tags: actualizer

- name: Enable systemd-networkd.
  become: yes
  systemd:
    service: systemd-networkd
    enabled: yes

- name: Start systemd-resolved.
  become: yes
  systemd:
    service: systemd-resolved
    state: started
  tags: actualizer

- name: Enable systemd-resolved.
  become: yes
  systemd:
    service: systemd-resolved
    enabled: yes

- name: Link /etc/resolv.conf.
  become: yes
  file:
    path: /etc/resolv.conf
    src: /run/systemd/resolve/resolv.conf
    state: link
    force: yes
  when:
  - ansible_distribution == 'Debian'
  - 12 > ansible_distribution_major_version|int

All servers should recognize the institute's Certificate Authority as trustworthy, so its certificate is added to the set of trusted CAs on each host. More information about how the small institute manages its X.509 certificates is available in Keys.

- name: Trust the institute CA.
  become: yes
  copy:
    src: ../Secret/CA/pki/ca.crt
    dest: /usr/local/share/ca-certificates/{{ domain_name }}.crt
    mode: u=r,g=r,o=r
    owner: root
    group: root
  notify: Update CAs.

---
- name: Update CAs.
  become: yes
  command: update-ca-certificates

The front role sets a number of variables to default values in its defaults/main.yml file.

---
<<network-vars>>
<<address-vars>>
<<membership-rolls>>

The membership-rolls reference defines membership_rolls which is used to select an empty membership roll if one has not been written yet. (See section 12.7.)

The first task, as in The All Role, is to include the institute particulars. The front role refers to private variables and the membership roll, so these are included was well.

---
- name: Include public variables.
  include_vars: ../public/vars.yml

- name: Include private variables.
  include_vars: ../private/vars.yml

- name: Include members.
  include_vars: "{{ lookup('first_found', membership_rolls) }}"
  tags: accounts

This task ensures that Front's /etc/hostname and /etc/mailname are correct. The correct /etc/mailname is essential to proper email delivery.

- name: Configure hostname.
  become: yes
  copy:
    content: "{{ domain_name }}\n"
    dest: "{{ item }}"
  loop:
  - /etc/hostname
  - /etc/mailname

- name: Update hostname.
  become: yes
  command: hostname -F /etc/hostname
  when: domain_name != ansible_fqdn
  tags: actualizer

The administrator often needs to read (directories of) log files owned by groups root and adm. Adding the administrator's account to these groups speeds up debugging.

- name: Add {{ ansible_user }} to system groups.
  become: yes
  user:
    name: "{{ ansible_user }}"
    append: yes
    groups: root,adm

The SSH service on Front needs to be known to Monkey. The following tasks ensure this by replacing the automatically generated keys with those stored in Secret/ssh_front/etc/ssh/ and restarting the server.

- name: Install SSH host keys.
  become: yes
  copy:
    src: ../Secret/ssh_front/etc/ssh/{{ item.name }}
    dest: /etc/ssh/{{ item.name }}
    mode: "{{ item.mode }}"
  loop:
  - { name: ssh_host_ecdsa_key,       mode: "u=rw,g=,o=" }
  - { name: ssh_host_ecdsa_key.pub,   mode: "u=rw,g=r,o=r" }
  - { name: ssh_host_ed25519_key,     mode: "u=rw,g=,o=" }
  - { name: ssh_host_ed25519_key.pub, mode: "u=rw,g=r,o=r" }
  - { name: ssh_host_rsa_key,         mode: "u=rw,g=,o=" }
  - { name: ssh_host_rsa_key.pub,     mode: "u=rw,g=r,o=r" }
  notify: Reload SSH server.

---
- name: Reload SSH server.
  become: yes
  systemd:
    service: ssh
    state: reloaded
  tags: actualizer

The small institute runs cron jobs and web scripts that generate reports and perform checks. The un-privileged jobs are run by a system account named monkey. One of Monkey's more important jobs on Core is to run rsync to update the public web site on Front. Monkey on Core will login as monkey on Front to synchronize the files (as described in *Configure Apache2). To do that without needing a password, the monkey account on Front should authorize Monkey's SSH key on Core.

- name: Create monkey.
  become: yes
  user:
    name: monkey
    password: "!"

- name: Authorize monkey@core.
  become: yes
  vars:
    pubkeyfile: ../Secret/ssh_monkey/id_rsa.pub
  authorized_key:
    user: monkey
    key: "{{ lookup('file', pubkeyfile) }}"
    manage_dir: yes

- name: Add {{ ansible_user }} to monkey group.
  become: yes
  user:
    name: "{{ ansible_user }}"
    append: yes
    groups: monkey

Monkey uses Rsync to keep the institute's public web site up-to-date.

- name: Install rsync.
  become: yes
  apt: pkg=rsync

The institute prefers to install security updates as soon as possible.

- name: Install basic software.
  become: yes
  apt: pkg=unattended-upgrades

User accounts are created immediately so that Postfix and Dovecot can start delivering email immediately, without returning "no such recipient" replies. The Account Management chapter describes the members and usernames variables used below.

- name: Create user accounts.
  become: yes
  user:
    name: "{{ item }}"
    password: "{{ members[item].password_front }}"
    update_password: always
    home: /home/{{ item }}
  loop: "{{ usernames }}"
  when: members[item].status == 'current'
  tags: accounts

- name: Disable former users.
  become: yes
  user:
    name: "{{ item }}"
    password: "!"
  loop: "{{ usernames }}"
  when: members[item].status != 'current'
  tags: accounts

- name: Revoke former user authorized_keys.
  become: yes
  file:
    path: /home/{{ item }}/.ssh/authorized_keys
    state: absent
  loop: "{{ usernames }}"
  when: members[item].status != 'current'
  tags: accounts

The servers on Front use the same certificate (and key) to authenticate themselves to institute clients. They share the /etc/server.crt and /etc/server.key files, the latter only readable by root.

- name: Install server certificate/key.
  become: yes
  copy:
    src: ../Secret/CA/pki/{{ item.path }}.{{ item.typ }}
    dest: /etc/server.{{ item.typ }}
    mode: "{{ item.mode }}"
    force: no
  loop:
  - { path: "issued/{{ domain_name }}", typ: crt,
      mode: "u=r,g=r,o=r" }
  - { path: "private/{{ domain_name }}", typ: key,
      mode: "u=r,g=,o=" }
  notify:
  - Restart Postfix.
  - Restart Dovecot.

Front uses Postfix to provide the institute's public SMTP service, and uses the institute's domain name for its host name. The default Debian configuration (for an "Internet Site") is nearly sufficient. Manual installation may prompt for configuration type and mail name. The appropriate answers are listed here but will be checked (corrected) by Ansible tasks below.

• General type of mail configuration: Internet Site
• System mail name: small.example.org

As discussed in The Email Service above, Front's Postfix configuration includes site-wide support for larger message sizes, shorter queue times, the relaying configuration, and the common path to incoming emails. These and a few Front-specific Postfix configurations settings make up the complete configuration (below).

Front relays messages from the institute's public WireGuard™ subnet via which Core relays messages from the campus.

- p: mynetworks
  v: >-
     {{ public_wg_net_cidr }}
     127.0.0.0/8
     [::ffff:127.0.0.0]/104
     [::1]/128

Front uses one recipient restriction to make things difficult for spammers, with permit_mynetworks at the start to not make things difficult for internal hosts, who do not have (public) domain names.

- p: smtpd_recipient_restrictions
  v: >-
     permit_mynetworks
     reject_unauth_pipelining
     reject_unauth_destination
     reject_unknown_sender_domain

Front uses Postfix header checks to strip Received headers from outgoing messages. These headers contain campus host and network names and addresses in the clear (un-encrypted). Stripping them improves network privacy and security. Front also strips User-Agent headers just to make it harder to target the program(s) members use to open their email. These headers should be stripped only from outgoing messages; incoming messages are delivered locally, without smtp_header_checks.

- p: smtp_header_checks
  v: regexp:/etc/postfix/header_checks.cf

/^Received:/    IGNORE
/^User-Agent:/  IGNORE

The complete Postfix configuration for Front follows. In addition to the options already discussed, it must override the loopback-only Debian default for inet_interfaces.

- { p: smtpd_tls_cert_file, v: /etc/server.crt }
- { p: smtpd_tls_key_file, v: /etc/server.key }
<<postfix-front-networks>>
<<postfix-front-restrictions>>
<<postfix-relaying>>
<<postfix-message-size>>
<<postfix-queue-times>>
<<postfix-maildir>>
<<postfix-header-checks>>

The following Ansible tasks install Postfix, modify /etc/postfix/main.cf according to the settings given above, and start and enable the service.

- name: Install Postfix.
  become: yes
  apt: pkg=postfix

- name: Configure Postfix.
  become: yes
  lineinfile:
    path: /etc/postfix/main.cf
    regexp: "^ *{{ item.p }} *="
    line: "{{ item.p }} = {{ item.v }}"
  loop:
  <<postfix-front>>
  notify: Restart Postfix.

- name: Install Postfix header_checks.
  become: yes
  copy:
    content: |
      <<postfix-header-checks-content>>
    dest: /etc/postfix/header_checks.cf
  notify: Postmap header checks.

- name: Start Postfix.
  become: yes
  systemd:
    service: postfix
    state: started
  tags: actualizer

- name: Enable Postfix.
  become: yes
  systemd:
    service: postfix
    enabled: yes

- name: Restart Postfix.
  become: yes
  systemd:
    service: postfix
    state: restarted
  tags: actualizer

- name: Postmap header checks.
  become: yes
  command:
    chdir: /etc/postfix/
    cmd: postmap header_checks.cf
  notify: Restart Postfix.

The institute's Front needs to deliver email addressed to a number of common aliases as well as those advertised on the web site. System daemons like cron(8) may also send email to system accounts like monkey. The following aliases make these customary mailboxes available. The aliases are installed in /etc/aliases in a block with a special marker so that additional blocks can be installed by other Ansible roles. Note that the postmaster alias forwards to root in the default Debian configuration, and the following aliases do not include the crucial root alias that forwards to the administrator. It could be included here or in a separate block created by a more specialized role.

- name: Install institute email aliases.
  become: yes
  blockinfile:
    block: |
        abuse:          root
        webmaster:      root
        admin:          root
        monkey:         monkey@{{ front_wg_addr }}
        root:           {{ ansible_user }}
    path: /etc/aliases
    marker: "# {mark} INSTITUTE MANAGED BLOCK"
  notify: New aliases.

- name: New aliases.
  become: yes
  command: newaliases
  tags: actualizer

Front uses Dovecot's IMAPd to allow user Fetchmail jobs on Core to pick up messages. Front's Dovecot configuration is largely the Debian default with POP and IMAP (without TLS) support disabled. This is a bit "over the top" given that Core accesses Front via VPN, but helps to ensure privacy even when members must, in extremis, access recent email directly from their accounts on Front. For more information about Front's role in the institute's email services, see The Email Service.

The institute follows the recommendation in the package README.Debian (in /usr/share/dovecot-core/). Note that the default "snake oil" certificate can be replaced with one signed by a recognized authority (e.g. Let's Encrypt) so that email apps will not ask about trusting the self-signed certificate.

The following Ansible tasks install Dovecot's IMAP daemon and its /etc/dovecot/local.conf configuration file, then starts the service and enables it to start at every reboot.

- name: Install Dovecot IMAPd.
  become: yes
  apt: pkg=dovecot-imapd

- name: Configure Dovecot IMAPd.
  become: yes
  copy:
    content: |
      <<dovecot-tls>>
      ssl_cert = </etc/server.crt
      ssl_key = </etc/server.key
      <<dovecot-ports>>
      <<dovecot-maildir>>
    dest: /etc/dovecot/local.conf
  notify: Restart Dovecot.

- name: Start Dovecot.
  become: yes
  systemd:
    service: dovecot
    state: started
  tags: actualizer

- name: Enable Dovecot.
  become: yes
  systemd:
    service: dovecot
    enabled: yes

- name: Restart Dovecot.
  become: yes
  systemd:
    service: dovecot
    state: restarted
  tags: actualizer

This is the small institute's public web site. It is simple, static, and thus (hopefully) difficult to subvert. There are no server-side scripts to run. The standard Debian install runs the server under the www-data account, which does not need any permissions. It will serve only world-readable files.

The server's document root, /home/www/, is separate from the Debian default /var/www/html/ and (presumably) on the largest disk partition. The directory tree, from the document root to the leaf HTML files, should be owned by monkey, and only writable by its owner. It should not be writable by the Apache2 server (running as www-data).

The institute uses several SSL directives to trim protocol and cipher suite compatibility down, eliminating old and insecure methods and providing for forward secrecy. Along with an up-to-date Let's Encrypt certificate, these settings win the institute's web site an A rating from Qualys SSL Labs (https://www.ssllabs.com/).

The apache-ciphers block below is included last in the Apache2 configuration, so that its SSLCipherSuite directive can override (narrow) any list of ciphers set earlier (e.g. by Let's Encrypt!²). The protocols and cipher suites specified here were taken from https://www.ssllabs.com/projects/best-practices in 2022.

SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLHonorCipherOrder on
SSLCipherSuite {{ [ 'ECDHE-ECDSA-AES128-GCM-SHA256',
                    'ECDHE-ECDSA-AES256-GCM-SHA384',
                    'ECDHE-ECDSA-AES128-SHA',
                    'ECDHE-ECDSA-AES256-SHA',
                    'ECDHE-ECDSA-AES128-SHA256',
                    'ECDHE-ECDSA-AES256-SHA384',
                    'ECDHE-RSA-AES128-GCM-SHA256',
                    'ECDHE-RSA-AES256-GCM-SHA384',
                    'ECDHE-RSA-AES128-SHA',
                    'ECDHE-RSA-AES256-SHA',
                    'ECDHE-RSA-AES128-SHA256',
                    'ECDHE-RSA-AES256-SHA384',
                    'DHE-RSA-AES128-GCM-SHA256',
                    'DHE-RSA-AES256-GCM-SHA384',
                    'DHE-RSA-AES128-SHA',
                    'DHE-RSA-AES256-SHA',
                    'DHE-RSA-AES128-SHA256',
                    'DHE-RSA-AES256-SHA256',
                    '!aNULL',
                    '!eNULL',
                    '!LOW',
                    '!3DES',
                    '!MD5',
                    '!EXP',
                    '!PSK',
                    '!SRP',
                    '!DSS',
                    '!RC4' ] |join(":") }}

The institute supports public member (static) web pages. A member can put an index.html file in their ~/Public/HTML/ directory on Front and it will be served as https://small.example.org/~member/ (if the member's account name is member and the file is world readable).

On Front, a member's web pages are available only when they appear in /home/www-users/ (via a symbolic link), giving the administration more control over what appears on the public web site. The tasks below create or remove the symbolic links.

The following are the necessary Apache2 directives: a UserDir directive naming /home/www-users/ and matching Directory block that includes the standard Require and AllowOverride directives used on all of the institute's web sites.

UserDir /home/www-users
<Directory /home/www-users/>
        Require all granted
        AllowOverride None
</Directory>

The institute requires the use of HTTPS on Front, so its default HTTP virtual host permanently redirects requests to their corresponding HTTPS URLs.

<VirtualHost *:80>
        Redirect permanent / https://{{ domain_name }}/
</VirtualHost>

The complete Apache2 configuration for Front is given below. It is installed in /etc/apache2/sites-available/{{ domain_name }}.conf (as expected by Let's Encrypt's Certbot). It includes the fragments described above and adds a VirtualHost block for the HTTPS service (also as expected by Certbot). The VirtualHost optionally includes an additional configuration file to allow other Ansible roles to specialize this configuration without disturbing the institute file.

The DocumentRoot directive is accompanied by a Directory block that authorizes access to the tree, and ensures .htaccess files within the tree are disabled for speed and security. This and most of Front's Apache2 directives (below) are intended for the top level, not the inside of a VirtualHost block. They should apply globally.

ServerName {{ domain_name }}
ServerAdmin webmaster@{{ domain_name }}

DocumentRoot /home/www
<Directory /home/www/>
        Require all granted
        AllowOverride None
</Directory>

<<apache-userdir-front>>

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

<<apache-redirect-front>>

<VirtualHost *:443>
        SSLEngine on
        SSLCertificateFile /etc/server.crt
        SSLCertificateKeyFile /etc/server.key
        IncludeOptional \
            /etc/apache2/sites-available/{{ domain_name }}-vhost.conf
</VirtualHost>

<<apache-ciphers>>

Ansible installs the configuration above in e.g. /etc/apache2/sites-available/small.example.org.conf and runs a2ensite -q small.example.org to enable it.

- name: Install Apache2.
  become: yes
  apt: pkg=apache2

- name: Enable Apache2 modules.
  become: yes
  apache2_module:
    name: "{{ item }}"
  loop: [ ssl, userdir ]
  notify: Restart Apache2.

- name: Create DocumentRoot.
  become: yes
  file:
    path: /home/www
    state: directory
    owner: monkey
    group: monkey

- name: Configure web site.
  become: yes
  copy:
    content: |
      <<apache-front>>
    dest: /etc/apache2/sites-available/{{ domain_name }}.conf
  notify: Restart Apache2.

- name: Enable web site.
  become: yes
  command:
    cmd: a2ensite -q {{ domain_name }}
    creates: /etc/apache2/sites-enabled/{{ domain_name }}.conf
  notify: Restart Apache2.

- name: Start Apache2.
  become: yes
  systemd:
    service: apache2
    state: started
  tags: actualizer

- name: Enable Apache2.
  become: yes
  systemd:
    service: apache2
    enabled: yes

- name: Restart Apache2.
  become: yes
  systemd:
    service: apache2
    state: restarted
  tags: actualizer

Furthermore, the default web site and its HTTPS version is disabled so that it does not interfere with its replacement.

- name: Disable default vhosts.
  become: yes
  file:
    path: /etc/apache2/sites-enabled/{{ item }}
    state: absent
  loop: [ 000-default.conf, default-ssl.conf ]
  notify: Restart Apache2.

The redundant default other-vhosts-access-log configuration option is also disabled. There are no other virtual hosts, and it stores the same records as access.log.

- name: Disable other-vhosts-access-log option.
  become: yes
  file:
    path: /etc/apache2/conf-enabled/other-vhosts-access-log.conf
    state: absent
  notify: Restart Apache2.

Finally, the UserDir is created and populated with symbolic links to the users' ~/Public/HTML/ directories.

- name: Create UserDir.
  become: yes
  file:
    path: /home/www-users/
    state: directory

- name: Create UserDir links.
  become: yes
  file:
    path: /home/www-users/{{ item }}
    src: /home/{{ item }}/Public/HTML
    state: link
    force: yes
    follow: false
  loop: "{{ usernames }}"
  when: members[item].status == 'current'
  tags: accounts

- name: Disable former UserDir links.
  become: yes
  file:
    path: /home/www-users/{{ item }}
    state: absent
  loop: "{{ usernames }}"
  when: members[item].status != 'current'
  tags: accounts

Front uses WireGuard™ to provide a public (Internet accessible) VPN service. Core has an interface on this VPN and is expected to forward packets between it and the institute's other private networks.

The following tasks install WireGuard™, configure it with private/front-wg0.conf (or private/front-wg0-empty.conf if it does not exist), and enable the service.

- name: Enable IP forwarding.
  become: yes
  sysctl:
    name: net.ipv4.ip_forward
    value: "1"
    state: present

- name: Install WireGuard™.
  become: yes
  apt: pkg=wireguard

- name: Configure WireGuard™.
  become: yes
  vars:
    srcs:
      - ../private/front-wg0.conf
      - ../private/front-wg0-empty.conf
  copy:
    src: "{{ lookup('first_found', srcs) }}"
    dest: /etc/wireguard/wg0.conf
    mode: u=r,g=,o=
    owner: root
    group: root
  notify: Restart WireGuard™.
  tags: accounts

- name: Start WireGuard™.
  become: yes
  systemd:
    service: wg-quick@wg0
    state: started
  tags: actualizer

- name: Enable WireGuard™.
  become: yes
  systemd:
    service: wg-quick@wg0
    enabled: yes

- name: Restart WireGuard™.
  become: yes
  systemd:
    service: wg-quick@wg0
    state: restarted
  tags: actualizer

The "empty" WireGuard™ configuration file (below) is used until the ./inst client command adds the first client, and generates an actual private/front-wg0.conf.

[Interface]
Address = 10.177.87.1/24
ListenPort = 39608
PostUp = wg set %i private-key /etc/wireguard/private-key
PostUp = resolvectl dns %i 192.168.56.1
PostUp = resolvectl domain %i small.private

Front uses Kamailio to provide a SIP service on the public VPN so that members abroad can chat privately. This is a connection-less UDP service that can be used with or without encryption. The VPN's encryption can be relied upon or an extra layer can be used when necessary. (Apps cannot tell if a network is secure and often assume the luser is an idiot, so they insist on doing some encryption.)

Kamailio listens on all network interfaces by default, but the institute expects its SIP traffic to be aggregated and encrypted via the public VPN. To enforce this expectation, Kamailio is instructed to listen only on Front's public VPN. The private name sip.small.private resolves to this address for the convenience of members configuring SIP clients. The server configuration specifies the actual IP, known here as front_wg_addr.

listen=udp:{{ front_wg_addr }}:5060

The Ansible tasks that install and configure Kamailio follow, but before Kamailio is configured (thus started), the service is tweaked by a configuration drop (which must notify Systemd before the service starts).

The first step is to install Kamailio.

- name: Install Kamailio.
  become: yes
  apt: pkg=kamailio

Now the configuration drop concerns the network device on which Kamailio will be listening, the wg0 device created by WireGuard™. The added configuration settings inform Systemd that Kamailio should not be started before the wg0 device has appeared.

- name: Create Kamailio/Systemd configuration drop.
  become: yes
  file:
    path: /etc/systemd/system/kamailio.service.d
    state: directory

- name: Create Kamailio dependence on WireGuard™ interface.
  become: yes
  copy:
    content: |
      [Unit]
      After=wg-quick@wg0.service
      Requires=sys-devices-virtual-net-wg0.device
    dest: /etc/systemd/system/kamailio.service.d/depend.conf
  notify: Reload Systemd.

- name: Reload Systemd.
  become: yes
  systemd:
    daemon-reload: yes
  tags: actualizer

Finally, Kamailio can be configured and started.

- name: Configure Kamailio.
  become: yes
  copy:
    content: |
      <<kamailio>>
    dest: /etc/kamailio/kamailio-local.cfg
  notify: Restart Kamailio.

- name: Start Kamailio.
  become: yes
  systemd:
    service: kamailio
    state: started
  tags: actualizer

- name: Enable Kamailio.
  become: yes
  systemd:
    service: kamailio
    enabled: yes

- name: Restart Kamailio.
  become: yes
  systemd:
    service: kamailio
    state: restarted
  tags: actualizer

As in The Front Role, the core role sets a number of variables to default values in its defaults/main.yml file.

---
<<network-vars>>
<<address-vars>>
<<membership-rolls>>

The first task, as in The Front Role, is to include the institute particulars and membership roll.

---
- name: Include public variables.
  include_vars: ../public/vars.yml
  tags: accounts

- name: Include private variables.
  include_vars: ../private/vars.yml
  tags: accounts

- name: Include members.
  include_vars: "{{ lookup('first_found', membership_rolls) }}"
  tags: accounts

This task ensures that Core's /etc/hostname and /etc/mailname are correct. Core accepts email addressed to the institute's public or private domain names, e.g. to dick@small.example.org as well as dick@small.private. The correct /etc/mailname is essential to proper email delivery.

- name: Configure hostname.
  become: yes
  copy:
    content: "{{ item.name }}\n"
    dest: "{{ item.file }}"
  loop:
  - { name: "core.{{ domain_priv }}", file: /etc/mailname }
  - { name: "{{ inventory_hostname }}", file: /etc/hostname }

- name: Update hostname.
  become: yes
  command: hostname -F /etc/hostname
  when: inventory_hostname != ansible_hostname
  tags: actualizer

Core runs the campus name server, so Resolved is configured to use it (or dns.google), to include the institute's domain in its search list, and to disable its cache and stub listener.

- name: Configure resolved.
  become: yes
  lineinfile:
    path: /etc/systemd/resolved.conf
    regexp: "{{ item.regexp }}"
    line: "{{ item.line }}"
  loop:
  - { regexp: '^ *DNS *=', line: "DNS=127.0.0.1" }
  - { regexp: '^ *FallbackDNS *=', line: "FallbackDNS=8.8.8.8" }
  - { regexp: '^ *Domains *=', line: "Domains={{ domain_priv }}" }
  - { regexp: '^ *Cache *=', line: "Cache=no" }
  - { regexp: '^ *DNSStubListener *=', line: "DNSStubListener=no" }
  notify:
  - Reload Systemd.
  - Restart Systemd resolved.

---
- name: Reload Systemd.
  become: yes
  systemd:
    daemon-reload: yes
  tags: actualizer

- name: Restart Systemd resolved.
  become: yes
  systemd:
    service: systemd-resolved
    state: restarted
  tags: actualizer

Core's network interface is statically configured using the systemd-networkd configuration files 10-lan.link and 10-lan.network installed in /etc/systemd/network/. Those files statically assign Core's IP address (as well as the campus name server and search domain), and its default route through Gate. A second route, through Core itself to Front, is advertised to other hosts, and is routed through a WireGuard™ interface connected to Front's public WireGuard™ VPN.

Note that the [Match] sections of the .network files should specify only a MACAddress. Getting systemd-udevd to rename interfaces has thusfar been futile (short of a reboot), so specifying a Name means the interface does not match, leaving it un-configured (until the next reboot).

The configuration needs the MAC address of the primary (only) NIC, an example of which is given here. (A clever way to extract that name from ansible_facts would be appreciated. The ansible_default_ipv4 fact was an empty hash at first boot on a simulated campus Ethernet.)

core_lan_mac:               08:00:27:b3:e5:5f

- name: Install 10-lan.link.
  become: yes
  copy:
    content: |
      [Match]
      MACAddress={{ core_lan_mac }}

      [Link]
      Name=lan
    dest: /etc/systemd/network/10-lan.link

- name: Install 10-lan.network.
  become: yes
  copy:
    content: |
      [Match]
      MACAddress={{ core_lan_mac }}

      [Network]
      Address={{ core_addr_cidr }}
      Gateway={{ gate_addr }}
      DNS={{ core_addr }}
      Domains={{ domain_priv }}
    dest: /etc/systemd/network/10-lan.network
  notify: Reload networkd.

- name: Reload networkd.
  become: yes
  command: networkctl reload
  tags: actualizer

Core speaks DHCP (Dynamic Host Configuration Protocol) using the Internet Software Consortium's DHCP server. The server assigns unique network addresses to hosts plugged into the private Ethernet as well as advertising local net services, especially the local Domain Name Service.

The example configuration file, private/core-dhcpd.conf, uses RFC3442's extension to encode a second (non-default) static route. The default route is through the campus ISP at Gate. A second route directs campus traffic to the Front VPN through Core. This is just an example file, with MAC addresses chosen to match VirtualBox test machines. In actual use private/core-dhcpd.conf refers to a replacement file.

option domain-name "small.private";
option domain-name-servers 192.168.56.1;

default-lease-time 3600;
max-lease-time 7200;

ddns-update-style none;

authoritative;

log-facility daemon;

option rfc3442-routes code 121 = array of integer 8;

subnet 192.168.56.0 netmask 255.255.255.0 {
  option subnet-mask 255.255.255.0;
  option broadcast-address 192.168.56.255;
  option routers 192.168.56.2;
  option ntp-servers 192.168.56.1;
  option rfc3442-routes 24, 10,177,87, 192,168,56,1,
                        0,             192,168,56,2;
}

host dick {
  hardware ethernet 08:00:27:dc:54:b5; fixed-address 192.168.56.4; }

The following tasks install ISC's DHCP server and configure it with the real private/core-dhcpd.conf (not the example above).

- name: Install DHCP server.
  become: yes
  apt: pkg=isc-dhcp-server

- name: Configure DHCP interface.
  become: yes
  lineinfile:
    path: /etc/default/isc-dhcp-server
    regexp: "^INTERFACESv4="
    line: "INTERFACESv4=\"lan\""
  notify: Restart DHCP server.

- name: Configure DHCP subnet.
  become: yes
  copy:
    src: ../private/core-dhcpd.conf
    dest: /etc/dhcp/dhcpd.conf
  notify: Restart DHCP server.

- name: Start DHCP server.
  become: yes
  systemd:
    service: isc-dhcp-server
    state: started
  tags: actualizer

- name: Enable DHCP server.
  become: yes
  systemd:
    service: isc-dhcp-server
    enabled: yes

- name: Restart DHCP server.
  become: yes
  systemd:
    service: isc-dhcp-server
    state: restarted
  tags: actualizer

Core uses BIND9 to provide name service for the institute as described in The Name Service. The configuration supports reverse name lookups, resolving many private network addresses to private domain names.

The following tasks install and configure BIND9 on Core.

- name: Install BIND9.
  become: yes
  apt: pkg=bind9

- name: Configure BIND9 with named.conf.options.
  become: yes
  copy:
    content: |
      <<bind-options>>
    dest: /etc/bind/named.conf.options
  notify: Reload BIND9.

- name: Configure BIND9 with named.conf.local.
  become: yes
  copy:
    content: |
      <<bind-local>>
    dest: /etc/bind/named.conf.local
  notify: Reload BIND9.

- name: Install BIND9 zonefiles.
  become: yes
  copy:
    src: ../private/db.{{ item }}
    dest: /etc/bind/db.{{ item }}
  loop: [ domain, private, public_vpn, campus_vpn ]
  notify: Reload BIND9.

- name: Start BIND9.
  become: yes
  systemd:
    service: bind9
    state: started
  tags: actualizer

- name: Enable BIND9.
  become: yes
  systemd:
    service: bind9
    enabled: yes

- name: Reload BIND9.
  become: yes
  systemd:
    service: bind9
    state: reloaded
  tags: actualizer

Examples of the necessary zone files, for the "Install BIND9 zonefiles." task above, are given below. If the campus ISP provided one or more IP addresses for stable name servers, those should probably be used as forwarders rather than Google.

acl "trusted" {
        {{ private_net_cidr }};
        {{ wild_net_cidr }};
        {{ public_wg_net_cidr }};
        {{ campus_wg_net_cidr }};
        localhost;
};

options {
        directory "/var/cache/bind";

        forwarders {
                8.8.4.4;
                8.8.8.8;
        };

        allow-query { any; };
        allow-recursion { trusted; };
        allow-query-cache { trusted; };

        dnssec-validation yes;

        listen-on {
                {{ core_addr }};
                localhost;
        };
};

include "/etc/bind/zones.rfc1918";

zone "{{ domain_priv }}." {
        type master;
        file "/etc/bind/db.domain";
};

zone "{{ private_net_cidr | ansible.utils.ipaddr('revdns')
         | regex_replace('^0\.','') }}" {
        type master;
        file "/etc/bind/db.private";
};

zone "{{ public_wg_net_cidr | ansible.utils.ipaddr('revdns')
         | regex_replace('^0\.','') }}" {
        type master;
        file "/etc/bind/db.public_vpn";
};

zone "{{ campus_wg_net_cidr | ansible.utils.ipaddr('revdns')
         | regex_replace('^0\.','') }}" {
        type master;
        file "/etc/bind/db.campus_vpn";
};

;
; BIND data file for a small institute's PRIVATE domain names.
;
$TTL    604800
@       IN      SOA     small.private. root.small.private. (
                              1         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      core.small.private.
$TTL    7200
mail    IN      CNAME   core.small.private.
smtp    IN      CNAME   core.small.private.
ns      IN      CNAME   core.small.private.
www     IN      CNAME   core.small.private.
test    IN      CNAME   core.small.private.
live    IN      CNAME   core.small.private.
ntp     IN      CNAME   core.small.private.
sip     IN      A       10.177.87.1
;
core    IN      A       192.168.56.1
gate    IN      A       192.168.56.2

;
; BIND reverse data file for a small institute's private Ethernet.
;
$TTL    604800
@       IN      SOA     small.private. root.small.private. (
                              1         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      core.small.private.
$TTL    7200
1       IN      PTR     core.small.private.
2       IN      PTR     gate.small.private.

;
; BIND reverse data file for a small institute's public VPN.
;
$TTL    604800
@       IN      SOA     small.private. root.small.private. (
                              1         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      core.small.private.
$TTL    7200
1       IN      PTR     front-p.small.private.
2       IN      PTR     core-p.small.private.

;
; BIND reverse data file for a small institute's campus VPN.
;
$TTL    604800
@       IN      SOA     small.private. root.small.private. (
                              1         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      core.small.private.
$TTL    7200
1       IN      PTR     gate-c.small.private.

The administrator often needs to read (directories of) log files owned by groups root and adm. Adding the administrator's account to these groups speeds up debugging.

- name: Add {{ ansible_user }} to system groups.
  become: yes
  user:
    name: "{{ ansible_user }}"
    append: yes
    groups: root,adm

The small institute runs cron jobs and web scripts that generate reports and perform checks. The un-privileged jobs are run by a system account named monkey. One of Monkey's more important jobs on Core is to run rsync to update the public web site on Front (as described in *Configure Apache2).

- name: Create monkey.
  become: yes
  user:
    name: monkey
    password: "!"
    append: yes
    groups: staff

- name: Add {{ ansible_user }} to staff groups.
  become: yes
  user:
    name: "{{ ansible_user }}"
    append: yes
    groups: monkey,staff

- name: Create /home/monkey/.ssh/.
  become: yes
  file:
    path: /home/monkey/.ssh
    state: directory
    mode: u=rwx,g=,o=
    owner: monkey
    group: monkey

- name: Configure monkey@core.
  become: yes
  copy:
    src: ../Secret/ssh_monkey/{{ item.name }}
    dest: /home/monkey/.ssh/{{ item.name }}
    mode: "{{ item.mode }}"
    owner: monkey
    group: monkey
  loop:
  - { name: config,      mode: "u=rw,g=r,o=" }
  - { name: id_rsa.pub,  mode: "u=rw,g=r,o=r" }
  - { name: id_rsa,      mode: "u=rw,g=,o=" }

- name: Configure Monkey SSH known hosts.
  become: yes
  vars:
    pubkeypath: ../Secret/ssh_front/etc/ssh
    pubkeyfile: "{{ pubkeypath }}/ssh_host_ecdsa_key.pub"
    pubkey: "{{ lookup('file', pubkeyfile) }}"
  lineinfile:
    regexp: "^{{ domain_name }},{{ front_addr }} ecdsa-sha2-nistp256 "
    line: "{{ domain_name }},{{ front_addr }} {{ pubkey }}"
    path: /home/monkey/.ssh/known_hosts
    create: yes
    owner: monkey
    group: monkey
    mode: "u=rw,g=,o="

The institute prefers to install security updates as soon as possible.

- name: Install basic software.
  become: yes
  apt: pkg=unattended-upgrades

User accounts are created immediately so that backups can begin restoring as soon as possible. The Account Management chapter describes the members and usernames variables.

- name: Create user accounts.
  become: yes
  user:
    name: "{{ item }}"
    password: "{{ members[item].password_core }}"
    update_password: always
    home: /home/{{ item }}
  loop: "{{ usernames }}"
  when: members[item].status == 'current'
  tags: accounts

- name: Disable former users.
  become: yes
  user:
    name: "{{ item }}"
    password: "!"
  loop: "{{ usernames }}"
  when: members[item].status != 'current'
  tags: accounts

- name: Revoke former user authorized_keys.
  become: yes
  file:
    path: /home/{{ item }}/.ssh/authorized_keys
    state: absent
  loop: "{{ usernames }}"
  when: members[item].status != 'current'
  tags: accounts

The servers on Core use the same certificate (and key) to authenticate themselves to institute clients. They share the /etc/server.crt and /etc/server.key files, the latter only readable by root.

- name: Install server certificate/key.
  become: yes
  copy:
    src: ../Secret/CA/pki/{{ item.path }}.{{ item.typ }}
    dest: /etc/server.{{ item.typ }}
    mode: "{{ item.mode }}"
  loop:
  - { path: "issued/core.{{ domain_priv }}", typ: crt,
      mode: "u=r,g=r,o=r" }
  - { path: "private/core.{{ domain_priv }}", typ: key,
      mode: "u=r,g=,o=" }
  notify:
  - Restart Postfix.
  - Restart Dovecot.

Core uses Chrony to provide a time synchronization service to the campus. The default daemon's default configuration is fine.

- name: Install Chrony.
  become: yes
  apt: pkg=chrony

- name: Configure NTP service.
  become: yes
  copy:
    content: |
      allow {{ private_net_cidr }}
      allow {{ public_wg_net_cidr }}
      allow {{ campus_wg_net_cidr }}
    dest: /etc/chrony/conf.d/institute.conf
  notify: Restart Chrony.

- name: Restart Chrony.
  become: yes
  systemd:
    service: chrony
    state: restarted

Core uses Postfix to provide SMTP service to the campus. The default Debian configuration (for an "Internet Site") is nearly sufficient. Manual installation may prompt for configuration type and mail name. The appropriate answers are listed here but will be checked (corrected) by Ansible tasks below.

• General type of mail configuration: Internet Site
• System mail name: core.small.private

As discussed in The Email Service above, Core delivers email addressed to any internal domain name locally, and uses its smarthost Front to relay the rest. Core is reachable only on institute networks, so there is little benefit in enabling TLS, but it does need to handle larger messages and respect the institute's expectation of shortened queue times.

Core relays messages from any institute network.

- p: mynetworks
  v: >-
     {{ private_net_cidr }}
     {{ public_wg_net_cidr }}
     {{ campus_wg_net_cidr }}
     127.0.0.0/8
     [::ffff:127.0.0.0]/104
     [::1]/128

Core uses Front to relay messages to the Internet.

- { p: relayhost, v: "[{{ front_wg_addr }}]" }

Core uses a Postfix transport file, /etc/postfix/transport, to specify local delivery for email addressed to any internal domain name. Note the leading dot at the beginning of the sole line in the file.

.{{ domain_name }}      local:$myhostname
.{{ domain_priv }}      local:$myhostname

The complete list of Core's Postfix settings for /etc/postfix/main.cf follow.

<<postfix-relaying>>
- { p: smtpd_tls_security_level, v: none }
- { p: smtp_tls_security_level, v: none }
<<postfix-message-size>>
<<postfix-queue-times>>
<<postfix-maildir>>
<<postfix-core-networks>>
<<postfix-core-relayhost>>
- { p: inet_interfaces, v: "127.0.0.1 {{ core_addr }}" }

The following Ansible tasks install Postfix, modify /etc/postfix/main.cf, create /etc/postfix/transport, and start and enable the service. Whenever /etc/postfix/transport is changed, the postmap transport command must also be run.

- name: Install Postfix.
  become: yes
  apt: pkg=postfix

- name: Configure Postfix.
  become: yes
  lineinfile:
    path: /etc/postfix/main.cf
    regexp: "^ *{{ item.p }} *="
    line: "{{ item.p }} = {{ item.v }}"
  loop:
  <<postfix-core>>
  - { p: transport_maps, v: "hash:/etc/postfix/transport" }
  notify: Restart Postfix.

- name: Configure Postfix transport.
  become: yes
  copy:
    content: |
      <<postfix-transport>>
    dest: /etc/postfix/transport
  notify: Postmap transport.

- name: Start Postfix.
  become: yes
  systemd:
    service: postfix
    state: started
  tags: actualizer

- name: Enable Postfix.
  become: yes
  systemd:
    service: postfix
    enabled: yes

- name: Restart Postfix.
  become: yes
  systemd:
    service: postfix
    state: restarted
  tags: actualizer

- name: Postmap transport.
  become: yes
  command:
    chdir: /etc/postfix/
    cmd: postmap transport
  notify: Restart Postfix.

The institute's Core needs to deliver email addressed to institute aliases including those advertised on the campus web site, in X.509 certificates, etc. System daemons like cron(8) may also send email to e.g. monkey. The following aliases are installed in /etc/aliases with a special marker so that additional blocks can be installed by more specialized roles.

- name: Install institute email aliases.
  become: yes
  blockinfile:
    block: |
        admin:          root
        www-data:       root
        monkey:         root
        root:           {{ ansible_user }}
    path: /etc/aliases
    marker: "# {mark} INSTITUTE MANAGED BLOCK"
  notify: New aliases.

- name: New aliases.
  become: yes
  command: newaliases
  tags: actualizer

Core uses Dovecot's IMAPd to store and serve member emails. As on Front, Core's Dovecot configuration is largely the Debian default with POP and IMAP (without TLS) support disabled. This is a bit "over the top" given that Core is only accessed from private (encrypted) networks, but helps to ensure privacy even when members accidentally attempt connections from outside the private networks. For more information about Core's role in the institute's email services, see The Email Service.

The institute follows the recommendation in the package README.Debian (in /usr/share/dovecot-core/) but replaces the default "snake oil" certificate with another, signed by the institute. (For more information about the institute's X.509 certificates, see Keys.)

The following Ansible tasks install Dovecot's IMAP daemon and its /etc/dovecot/local.conf configuration file, then starts the service and enables it to start at every reboot.

- name: Install Dovecot IMAPd.
  become: yes
  apt: pkg=dovecot-imapd

- name: Configure Dovecot IMAPd.
  become: yes
  copy:
    content: |
      <<dovecot-tls>>
      ssl_cert = </etc/server.crt
      ssl_key = </etc/server.key
      <<dovecot-maildir>>
    dest: /etc/dovecot/local.conf
  notify: Restart Dovecot.

- name: Start Dovecot.
  become: yes
  systemd:
    service: dovecot
    state: started
  tags: actualizer

- name: Enable Dovecot.
  become: yes
  systemd:
    service: dovecot
    enabled: yes

- name: Restart Dovecot.
  become: yes
  systemd:
    service: dovecot
    state: restarted
  tags: actualizer

Core runs a fetchmail for each member of the institute. Individual fetchmail jobs can run with the --idle option and thus can download new messages instantly. The jobs run as Systemd services and so are monitored and started at boot.

In the ~/.fetchmailrc template below, the item variable is a username, and members[item] is the membership record associated with the username. The template is only used when the record has a password_fetchmail key providing the member's plain-text password.

# Permissions on this file may be no greater than 0600.

set no bouncemail
set no spambounce
set no syslog
#set logfile /home/{{ item }}/.fetchmail.log

poll {{ front_wg_addr }} protocol imap timeout 15
    username {{ item }}
    password "{{ members[item].password_fetchmail }}" fetchall
    ssl sslproto tls1.2+ sslcertck sslcommonname {{ domain_name }}

The Systemd service description.

[Unit]
Description=Fetchmail --idle task for {{ item }}.
AssertPathExists=/home/{{ item }}/.fetchmailrc
After=wg-quick@wg0.service
Wants=sys-devices-virtual-net-wg0.device

[Service]
User={{ item }}
ExecStart=/usr/bin/fetchmail --idle
Restart=always
RestartSec=1m
NoNewPrivileges=true

[Install]
WantedBy=default.target

The following tasks install fetchmail, a ~/.fetchmailrc and Systemd .service file for each current member, start the services, and enable them to start on boot. To accommodate any member of the institute who may wish to run their own fetchmail job on their notebook, only members with a fetchmail_password key will be provided the Core service.

- name: Install fetchmail.
  become: yes
  apt: pkg=fetchmail

- name: Configure user fetchmails.
  become: yes
  copy:
    content: |
      <<fetchmail-config>>
    dest: /home/{{ item }}/.fetchmailrc
    owner: "{{ item }}"
    group: "{{ item }}"
    mode: u=rw,g=,o=
  loop: "{{ usernames }}"
  when:
  - members[item].status == 'current'
  - members[item].password_fetchmail is defined
  tags: accounts

- name: Create user fetchmail services.
  become: yes
  copy:
    content: |
      <<fetchmail-service>>
    dest: /etc/systemd/system/fetchmail-{{ item }}.service
  loop: "{{ usernames }}"
  when:
  - members[item].status == 'current'
  - members[item].password_fetchmail is defined
  tags: accounts

- name: Enable/Start user fetchmail services.
  become: yes
  systemd:
    service: fetchmail-{{ item }}.service
    enabled: yes
    state: started
  loop: "{{ usernames }}"
  when:
  - members[item].status == 'current'
  - members[item].password_fetchmail is defined
  tags: accounts, actualizer

Finally, any former member's Fetchmail service on Core should be stopped and disabled from restarting at boot, deleted even.

- name: Stop former user fetchmail services.
  become: yes
  systemd:
    service: fetchmail-{{ item }}
    state: stopped
    enabled: no
  loop: "{{ usernames }}"
  when:
  - members[item].status != 'current'
  - members[item].password_fetchmail is defined
  tags: accounts

If the .service file is deleted, then Ansible cannot use the systemd module to stop it, nor check that it is still stopped. Otherwise the following task might be appropriate.

- name: Delete former user fetchmail services.
  become: yes
  file:
    path: /etc/systemd/system/fetchmail-{{ item }}.service
    state: absent
  loop: "{{ usernames }}"
  when:
  - members[item].status != 'current'
  - members[item].password_fetchmail is defined
  tags: accounts

This is the small institute's campus web server. It hosts several web sites as described in The Web Services.

The live (and test) web site content (eventually) is intended to be copied to Front, so the live and test sites are configured as identically to Front's as possible. The directories and files are owned by monkey but are world readable, thus readable by www-data, the account running Apache2.

The campus web site is much more permissive. Its directories are owned by root but writable by the staff group. It runs CGI scripts found in any of its directories, any executable with a .cgi file name. It runs them as www-data so CGI scripts that need access to private data must Set-UID to the appropriate account.

The UserDir directives for all of Core's web sites are the same, and punt the indirection through a /home/www-users/ directory, simply naming a sub-directory in the member's home directory on Core. The <Directory> block is the same as the one used on Front.

UserDir Public/HTML
<Directory /home/*/Public/HTML/>
        Require all granted
        AllowOverride None
</Directory>

The virtual host for the live web site is given below. It should look like Front's top-level web configuration without the permanent redirect, the encryption ciphers and certificates.

<VirtualHost *:80>
        ServerName live
        ServerAlias live.{{ domain_priv }}
        ServerAdmin webmaster@core.{{ domain_priv }}

        DocumentRoot /WWW/live
        <Directory /WWW/live/>
                Require all granted
                AllowOverride None
        </Directory>

        <<apache-userdir-core>>

        ErrorLog ${APACHE_LOG_DIR}/live-error.log
        CustomLog ${APACHE_LOG_DIR}/live-access.log combined

        IncludeOptional /etc/apache2/sites-available/live-vhost.conf
</VirtualHost>

The virtual host for the test web site is given below. It should look familiar.

<VirtualHost *:80>
        ServerName test
        ServerAlias test.{{ domain_priv }}
        ServerAdmin webmaster@core.{{ domain_priv }}

        DocumentRoot /WWW/test
        <Directory /WWW/test/>
                Require all granted
                AllowOverride None
        </Directory>

        <<apache-userdir-core>>

        ErrorLog ${APACHE_LOG_DIR}/test-error.log
        CustomLog ${APACHE_LOG_DIR}/test-access.log combined

        IncludeOptional /etc/apache2/sites-available/test-vhost.conf
</VirtualHost>

The virtual host for the campus web site is given below. It too should look familiar, but with a notably loose Directory directive. It assumes /WWW/campus/ is secure, writable only by properly trained staffers, monitored by a revision control system, etc.

<VirtualHost *:80>
        ServerName www
        ServerAlias www.{{ domain_priv }}
        ServerAdmin webmaster@core.{{ domain_priv }}

        DocumentRoot /WWW/campus
        <Directory /WWW/campus/>
                Options Indexes FollowSymLinks MultiViews ExecCGI
                AddHandler cgi-script .cgi
                Require all granted
                AllowOverride None
        </Directory>

        <<apache-userdir-core>>

        ErrorLog ${APACHE_LOG_DIR}/campus-error.log
        CustomLog ${APACHE_LOG_DIR}/campus-access.log combined

        IncludeOptional /etc/apache2/sites-available/www-vhost.conf
</VirtualHost>

The tasks below install Apache2 and edit its default configuration.

- name: Install Apache2.
  become: yes
  apt: pkg=apache2

- name: Enable Apache2 modules.
  become: yes
  apache2_module:
    name: "{{ item }}"
  loop: [ userdir, cgid, ssl ]
  notify: Restart Apache2.

- name: Configure Apache2 SSL certificate.
  become: yes
  lineinfile:
    path: /etc/apache2/sites-available/default-ssl.conf
    regexp: "^([\t ]*){{ item.p }}"
    line: "\\1{{ item.p }}\t{{ item.v }}"
    backrefs: yes
  loop:
    - { p: SSLCertificateFile, v: "/etc/server.crt" }
    - { p: SSLCertificateKeyFile, v: "/etc/server.key" }
  notify: Restart Apache2.

With Apache installed there is a /etc/apache/sites-available/ directory into which the above site configurations can be installed. The a2ensite command enables them.

- name: Install live web site.
  become: yes
  copy:
    content: |
      <<apache-live>>
    dest: /etc/apache2/sites-available/live.conf
    mode: u=rw,g=r,o=r
  notify: Restart Apache2.

- name: Install test web site.
  become: yes
  copy:
    content: |
      <<apache-test>>
    dest: /etc/apache2/sites-available/test.conf
    mode: u=rw,g=r,o=r
  notify: Restart Apache2.

- name: Install campus web site.
  become: yes
  copy:
    content: |
      <<apache-campus>>
    dest: /etc/apache2/sites-available/www.conf
    mode: u=rw,g=r,o=r
  notify: Restart Apache2.

- name: Enable web sites.
  become: yes
  command:
    cmd: a2ensite -q {{ item }}
    creates: /etc/apache2/sites-enabled/{{ item }}.conf
  loop: [ live, test, www, default-ssl ]
  notify: Restart Apache2.

- name: Start Apache2.
  become: yes
  systemd:
    service: apache2
    state: started
  tags: actualizer

- name: Enable Apache2.
  become: yes
  systemd:
    service: apache2
    enabled: yes

- name: Restart Apache2.
  become: yes
  systemd:
    service: apache2
    state: restarted
  tags: actualizer

Monkey on Core runs /usr/local/sbin/webupdate every 15 minutes via a cron job. The example script mirrors /WWW/live/ on Core to /home/www/ on Front.

#!/bin/bash -e
#
# DO NOT EDIT.
#
# This file was tangled from a small institute's README.org.

cd /WWW/live/

rsync -avz --delete --chmod=g-w         \
        --filter='exclude *~'           \
        --filter='exclude .git*'        \
        ./ 192.168.15.4:/home/www/

The following tasks install the webupdate script from private/, and create Monkey's cron job. An example webupdate script is provided here.

- name: "Install Monkey's webupdate script."
  become: yes
  copy:
    src: ../private/webupdate
    dest: /usr/local/sbin/webupdate
    mode: u=rx,g=rx,o=
    owner: monkey
    group: staff

- name: "Create Monkey's webupdate job."
  become: yes
  cron:
    minute: "*/15"
    job: "[ -d /WWW/live ] && /usr/local/sbin/webupdate"
    name: webupdate
    user: monkey

Core connects to Front's WireGuard™ service to provide members abroad with a route to the campus networks. As described in Configure Public WireGuard™ Subnet for Front, Core is expected to forward packets from/to the private networks.

The following tasks install WireGuard™, configure it and enable the service.

- name: Enable IP forwarding.
  become: yes
  sysctl:
    name: net.ipv4.ip_forward
    value: "1"
    state: present

- name: Install WireGuard™.
  become: yes
  apt: pkg=wireguard

- name: Configure WireGuard™.
  become: yes
  copy:
    content: |
      [Interface]
      Address = {{ core_wg_addr }}
      PostUp = wg set %i private-key /etc/wireguard/private-key

      # Front
      [Peer]
      EndPoint = {{ front_addr }}:{{ public_wg_port }}
      PublicKey = {{ front_wg_pubkey }}
      AllowedIPs = {{ front_wg_addr }}
      AllowedIPs = {{ public_wg_net_cidr }}
    dest: /etc/wireguard/wg0.conf
    mode: u=r,g=,o=
    owner: root
    group: root
  notify: Restart WireGuard™.

- name: Start WireGuard™.
  become: yes
  systemd:
    service: wg-quick@wg0
    state: started
  tags: actualizer

- name: Enable WireGuard™.
  become: yes
  systemd:
    service: wg-quick@wg0
    enabled: yes

- name: Restart WireGuard™.
  become: yes
  systemd:
    service: wg-quick@wg0
    state: restarted
  tags: actualizer

Core runs a nagios4 server to monitor "services" on institute hosts. The following tasks install the necessary packages and configure the server via edits to /etc/nagios4/nagios.cfg. The monitors are installed in /etc/nagios4/conf.d/institute.cfg which is tangled from code blocks described in the following subsections.

The institute NAGIOS configuration includes a customized version of the check_sensors plugin named inst_sensors. Both versions rely on the sensors command (from the lm-sensors package). The custom version (below) is installed in /usr/local/sbin/inst_sensors on both Core and Campus (and thus Gate) machines.

- name: Install NAGIOS4.
  become: yes
  apt:
    pkg: [ nagios4, monitoring-plugins-basic, nagios-nrpe-plugin,
           lm-sensors ]

- name: Install inst_sensors NAGIOS plugin.
  become: yes
  copy:
    src: inst_sensors
    dest: /usr/local/sbin/inst_sensors
    mode: u=rwx,g=rx,o=rx

- name: Configure NAGIOS4.
  become: yes
  lineinfile:
    path: /etc/nagios4/nagios.cfg
    regexp: "{{ item.regexp }}"
    line: "{{ item.line }}"
    backrefs: yes
  loop:
  - regexp: "^( *cfg_file *=.*/localhost.cfg)"
    line: "#\\1"
  - regexp: "^( *admin_email *= *)"
    line: "\\1{{ ansible_user }}@localhost"
  notify: Reload NAGIOS4.

- name: Configure NAGIOS4 contacts.
  become: yes
  lineinfile:
    path: /etc/nagios4/objects/contacts.cfg
    regexp: "^( *email +)"
    line: "\\1sysadm@localhost"
    backrefs: yes
  notify: Reload NAGIOS4.

- name: Configure NAGIOS4 monitors.
  become: yes
  template:
    src: nagios.cfg
    dest: /etc/nagios4/conf.d/institute.cfg
  notify: Reload NAGIOS4.

- name: Start NAGIOS4.
  become: yes
  systemd:
    service: nagios4
    state: started
  tags: actualizer

- name: Enable NAGIOS4.
  become: yes
  systemd:
    service: nagios4
    enabled: yes

- name: Reload NAGIOS4.
  become: yes
  systemd:
    service: nagios4
    state: reloaded
  tags: actualizer

The following task installs the backup script from private/. An example script is provided in here.

- name: Install backup script.
  become: yes
  copy:
    src: ../private/backup
    dest: /usr/local/sbin/backup
    mode: u=rx,g=r,o=

Core runs Nextcloud to provide a private institute cloud, as described in The Cloud Service. Installing, restoring (from backup), and upgrading Nextcloud are manual processes documented in The Nextcloud Admin Manual, Maintenance. However Ansible can help prepare Core before an install or restore, and perform basic security checks afterwards.

As in The Core Role, the gate role sets a number of variables to default values in its defaults/main.yml file.

---
<<network-vars>>
<<address-vars>>

The following should be familiar boilerplate by now.

---
- name: Include public variables.
  include_vars: ../public/vars.yml

- name: Include private variables.
  include_vars: ../private/vars.yml

Gate's network interfaces are configured using SystemD NetworkD configuration files that specify their MAC addresses. (One or more might be plug-and-play USB dongles.) These addresses are provided by the private/vars.yml file as in the example code here.

gate_lan_mac:               08:00:27:f3:16:79
gate_wild_mac:              08:00:27:4a:de:d2
gate_isp_mac:               08:00:27:3d:42:e5

The tasks in the following sections install the necessary configuration files.

Gate provides name service on the wild Ethernet by having its "stub listener" listen there. That stub should not read /etc/hosts lest gate resolve to 127.0.1.1, nonsense to the wild.

- name: Configure resolved.
  become: yes
  lineinfile:
    path: /etc/systemd/resolved.conf
    regexp: "{{ item.regexp }}"
    line: "{{ item.line }}"
  loop:
  - regexp: '^ *DNSStubListenerExtra *='
    line: "DNSStubListenerExtra={{ gate_wild_addr }}"
  - regexp: '^ *ReadEtcHosts *='
    line: "ReadEtcHosts=no"
  notify:
  - Reload Systemd.
  - Restart Systemd resolved.

- name: Reload Systemd.
  become: yes
  systemd:
    daemon-reload: yes
  tags: actualizer

- name: Restart Systemd resolved.
  become: yes
  systemd:
    service: systemd-resolved
    state: restarted
  tags: actualizer

Gate uses the Uncomplicated FireWall (UFW) to install its packet filters at boot-time. The institute does not use a firewall except to configure Network Address Translation (NAT) and forwarding. Members expect to be able to exercise experimental services on random ports. The default policy settings in /etc/default/ufw are ACCEPT and ACCEPT for input and output, and DROP for forwarded packets. Forwarding was enabled in the kernel previously (when configuring WireGuard™) using Ansible's sysctl module. It does not need to be set in /etc/ufw/sysctl.conf.

NAT is enabled per the ufw-framework(8) manual page, by introducing nat table rules in a block at the end of /etc/ufw/before.rules. They translate packets going to the ISP. These can come from the private Ethernet or the untrusted Ethernet (campus IoT, including Wi-Fi APs). Hosts on the other institute networks (the two VPNs) should not be routing their Internet traffic through their VPN.

-A POSTROUTING -s {{ private_net_cidr }} -o isp -j MASQUERADE
-A POSTROUTING -s {{    wild_net_cidr }} -o isp -j MASQUERADE

Forwarding rules are also needed. The nat table is a post routing rule set, so the default routing policy (DENY) will drop packets before NAT can translate them. The following rules are added to allow packets to be forwarded from the campus Ethernet or its wild subnet to an ISP on the isp interface. A generic routing rule in UFW accepts any related or established packet (according to the kernel's connection tracking).

-A ufw-before-forward -i lan  -o isp -j ACCEPT
-A ufw-before-forward -i wild -o isp -j ACCEPT

Forwarding rules are also needed to route packets from the campus VPN (the wg0 WireGuard™ tunnel device) to the institute's LAN and back. The public VPN on Front will also be included since its packets arrive at Gate's lan interface, coming from Core. Thus forwarding between public and campus VPNs is also allowed.

-A ufw-before-forward -i lan  -o wg0 -j ACCEPT
-A ufw-before-forward -i wg0  -o lan -j ACCEPT
-A ufw-before-forward -i wg0  -o wg0 -j ACCEPT

The third rule above may seem curious; it is. It short circuits filters in subsequent chains (e.g. ufw-reject-forward) that, by default, log and reject packets, even those from subnet to the same subnet (if it is a WireGuard™ subnet?).

Note that there are no forwarding rules to allow packets to pass from the wild device to the lan device, just the wg0 device.

The following tasks install the Uncomplicated Firewall (UFW), set its policy in /etc/default/ufw, and install the institute's rules in /etc/ufw/before.rules.

- name: Install UFW.
  become: yes
  apt: pkg=ufw

- name: Configure UFW policy.
  become: yes
  lineinfile:
    path: /etc/default/ufw
    line: "{{ item.line }}"
    regexp: "{{ item.regexp }}"
  loop:
  - line: "DEFAULT_INPUT_POLICY=\"ACCEPT\""
    regexp: "^DEFAULT_INPUT_POLICY="
  - line: "DEFAULT_OUTPUT_POLICY=\"ACCEPT\""
    regexp: "^DEFAULT_OUTPUT_POLICY="
  - line: "DEFAULT_FORWARD_POLICY=\"DROP\""
    regexp: "^DEFAULT_FORWARD_POLICY="

- name: Configure UFW rules.
  become: yes
  blockinfile:
    block: |
      *nat
      :POSTROUTING ACCEPT [0:0]
      <<ufw-nat>>
      COMMIT
      *filter
      <<ufw-forward-nat>>
      <<ufw-forward-private>>
      COMMIT
    dest: /etc/ufw/before.rules
    insertafter: EOF
    prepend_newline: yes

- name: Enable UFW.
  become: yes
  ufw: state=enabled
  tags: actualizer

Gate uses WireGuard™ to provide a campus VPN service. Gate's routes and firewall rules allow packets to be forwarded to/from the institute's private networks: the private Ethernet and the public VPN. (It should not forward packets to/from the wild Ethernet.) The only additional route Gate needs is to the public VPN via Core. The rest (private Ethernet and campus VPN) are directly connected.

The following tasks install WireGuard™, configure it with private/gate-wg0.conf (or private/gate-wg0-empty.conf if it does not exist), and enable the service.

- name: Enable IP forwarding.
  become: yes
  sysctl:
    name: net.ipv4.ip_forward
    value: "1"
    state: present

- name: Install WireGuard™.
  become: yes
  apt: pkg=wireguard

- name: Configure WireGuard™.
  become: yes
  vars:
    srcs:
      - ../private/gate-wg0.conf
      - ../private/gate-wg0-empty.conf
  copy:
    src: "{{ lookup('first_found', srcs) }}"
    dest: /etc/wireguard/wg0.conf
    mode: u=r,g=,o=
    owner: root
    group: root
  notify: Restart WireGuard™.
  tags: accounts

- name: Start WireGuard™.
  become: yes
  systemd:
    service: wg-quick@wg0
    state: started
  tags: actualizer

- name: Enable WireGuard™.
  become: yes
  systemd:
    service: wg-quick@wg0
    enabled: yes

- name: Restart WireGuard™.
  become: yes
  systemd:
    service: wg-quick@wg0
    state: restarted
  tags: actualizer

The "empty" WireGuard™ configuration file (below) is used until the ./inst client command adds the first client, and generates an actual private/gate-wg0.conf.

[Interface]
Address = 10.84.139.1/24
ListenPort = 51820
PostUp = wg set %i private-key /etc/wireguard/private-key

As in The Gate Role, the campus role sets a number of variables to default values in its defaults/main.yml file.

---
<<network-vars>>
<<address-vars>>

The following should be familiar boilerplate by now.

---
- name: Include public variables.
  include_vars: ../public/vars.yml

- name: Include private variables.
  include_vars: ../private/vars.yml

Clients should be using the expected host name.

- name: Configure hostname.
  become: yes
  copy:
    content: "{{ item.content }}"
    dest: "{{ item.file }}"
  loop:
  - { file: /etc/hostname,
      content: "{{ inventory_hostname }}\n" }
  - { file: /etc/mailname,
      content: "{{ inventory_hostname }}.{{ domain_priv }}\n" }

- name: Update hostname.
  become: yes
  command: hostname -F /etc/hostname
  when: inventory_hostname != ansible_hostname
  tags: actualizer

The institute uses a common time reference throughout the campus. This is essential to campus security, improving the accuracy of log and file timestamps.

- name: Configure timesyncd.
  become: yes
  lineinfile:
    path: /etc/systemd/timesyncd.conf
    line: NTP=ntp.{{ domain_priv }}
  notify: Restart systemd-timesyncd.

---
- name: Restart systemd-timesyncd.
  become: yes
  systemd:
    service: systemd-timesyncd
    state: restarted
  tags: actualizer

The administrator often needs to read (directories of) log files owned by groups root and adm. Adding the administrator's account to these groups speeds up debugging.

- name: Add {{ ansible_user }} to system groups.
  become: yes
  user:
    name: "{{ ansible_user }}"
    append: yes
    groups: root,adm

The institute prefers to install security updates as soon as possible.

- name: Install basic software.
  become: yes
  apt: pkg=unattended-upgrades

The Postfix settings used by the campus include message size, queue times, and the relayhost Core. The default Debian configuration (for an "Internet Site") is otherwise sufficient. Manual installation may prompt for configuration type and mail name. The appropriate answers are listed here but will be checked (corrected) by Ansible tasks below.

• General type of mail configuration: Internet Site
• System mail name: new.small.private

- name: Install Postfix.
  become: yes
  apt: pkg=postfix

- name: Configure Postfix.
  become: yes
  lineinfile:
    path: /etc/postfix/main.cf
    regexp: "^ *{{ item.p }} *="
    line: "{{ item.p }} = {{ item.v }}"
  loop:
  <<postfix-relaying>>
  <<postfix-message-size>>
  <<postfix-queue-times>>
  <<postfix-maildir>>
  - { p: myhostname,
      v: "{{ inventory_hostname }}.{{ domain_priv }}" }
  - { p: mydestination,
      v: "{{ postfix_mydestination | default('') }}" }
  - { p: relayhost, v: "[smtp.{{ domain_priv }}]" }
  - { p: inet_interfaces, v: loopback-only }
  notify: Restart Postfix.

- name: Start Postfix.
  become: yes
  systemd:
    service: postfix
    state: started
  tags: actualizer

- name: Enable Postfix.
  become: yes
  systemd:
    service: postfix
    enabled: yes

- name: Restart Postfix.
  become: yes
  systemd:
    service: postfix
    state: restarted
  tags: actualizer

The host's fully qualified (private) domain name (FQDN) is set by an alias in its /etc/hosts file, as is customary on Debian. (See "The "recommended method of setting the FQDN" in the hostname(1) manpage.)

- name: Set domain name.
  become: yes
  vars:
    name: "{{ inventory_hostname }}"
  lineinfile:
    path: /etc/hosts
    regexp: "^127.0.1.1[        ].*"
    line: "127.0.1.1    {{ name }}.{{ domain_priv }} {{ name }}"

Each campus host runs an NRPE (a NAGIOS Remote Plugin Executor) server so that the NAGIOS4 server on Core can collect statistics. The NAGIOS service is discussed in the Configure NRPE section of The Core Role.

- name: Install NRPE.
  become: yes
  apt:
    pkg: [ nagios-nrpe-server, lm-sensors ]

- name: Install inst_sensors NAGIOS plugin.
  become: yes
  copy:
    src: ../core/files/inst_sensors
    dest: /usr/local/sbin/inst_sensors
    mode: u=rwx,g=rx,o=rx

- name: Configure NRPE server.
  become: yes
  copy:
    content: |
      allowed_hosts=127.0.0.1,::1,{{ core_addr }}
    dest: /etc/nagios/nrpe_local.cfg
  notify: Reload NRPE server.

- name: Configure NRPE commands.
  become: yes
  copy:
    src: nrpe.cfg
    dest: /etc/nagios/nrpe.d/institute.cfg
  notify: Reload NRPE server.

- name: Start NRPE server.
  become: yes
  systemd:
    service: nagios-nrpe-server
    state: started
  tags: actualizer

- name: Enable NRPE server.
  become: yes
  systemd:
    service: nagios-nrpe-server
    enabled: yes

- name: Reload NRPE server.
  become: yes
  systemd:
    service: nagios-nrpe-server
    state: reloaded
  tags: actualizer

The Ansible configuration file ansible.cfg contains just a handful of settings, some included just to create a test jig as described in Testing.

• interpreter_python is set to suppress a warning from Ansible's "automatic interpreter discovery" (described here). It declares that Python 3 can be expected on all institute hosts.
• vault_password_file is set to suppress prompts for the vault password. The institute keeps its vault password in Secret/ (as described in Keys) and thus sets this parameter to Secret/vault-password.
• inventory is set to avoid specifying it on the command line.
• roles_path is set to the recently tangled roles files in roles_t/ which are preferred in the test configuration.

[defaults]
interpreter_python=/usr/bin/python3
vault_password_file=Secret/vault-password
inventory=hosts
roles_path=roles_t

The Ansible inventory file hosts describes all of the institute's machines starting with the main servers Front, Core and Gate. It provides the IP addresses, administrator account names and passwords for each machine. The IP addresses are all private, campus network addresses except Front's public IP. The following example host file describes three test servers named front, core and gate.

all:
  vars:
    ansible_user: sysadm
    ansible_ssh_extra_args: -i Secret/ssh_admin/id_rsa
  hosts:
    front:
      ansible_host: 192.168.58.3
      ansible_become_password: "{{ become_front }}"
    core:
      ansible_host: 192.168.56.1
      ansible_become_password: "{{ become_core }}"
    gate:
      ansible_host: 192.168.56.2
      ansible_become_password: "{{ become_gate }}"
  children:
    campus:
      hosts:
        gate:

The values of the ansible_become_password key are references to variables defined in Secret/become.yml, which is loaded as "extra" variables by a -e option on the ansible-playbook command line.

become_front: !vault |
        $ANSIBLE_VAULT;1.1;AES256
        3563626131333733666466393166323135383838666338666131336335326
        3656437663032653333623461633866653462636664623938356563306264
        3438660a35396630353065383430643039383239623730623861363961373
        3376663366566326137386566623164313635303532393335363063333632
        363163316436380a336562323739306231653561613837313435383230313
        1653565653431356362
become_core: !vault |
        $ANSIBLE_VAULT;1.1;AES256
        3464643665363937393937633432323039653530326465346238656530303
        8633066663935316365376438353439333034666366363739616130643261
        3232380a66356462303034636332356330373465623337393938616161386
        4653864653934373766656265613636343334356361396537343135393663
        313562613133380a373334393963623635653264663538656163613433383
        5353439633234666134
become_gate: !vault |
        $ANSIBLE_VAULT;1.1;AES256
        3138306434313739626461303736666236336666316535356561343566643
        6613733353434333962393034613863353330623761623664333632303839
        3838350a37396462343738303331356134373634306238633030303831623
        0636537633139366333373933396637633034383132373064393939363231
        636264323132370a393135666335303361326330623438613630333638393
        1303632663738306634

The passwords are individually encrypted just to make it difficult to acquire a list of all institute privileged account passwords in one glance. The multi-line values are generated by the ansible-vault encrypt_string command, which uses the ansible.cfg file and thus the Secret/vault-password file.

The example playbooks/site.yml playbook (below) applies the appropriate institutional role(s) to the hosts and groups defined in the example inventory: hosts.

---
- name: Configure All
  hosts: all
  roles: [ all ]

- name: Configure Front
  hosts: front
  roles: [ front ]

- name: Configure Gate
  hosts: gate
  roles: [ gate ]

- name: Configure Core
  hosts: core
  roles: [ core ]

- name: Configure Campus
  hosts: campus
  roles: [ campus ]

As already mentioned, the small institute keeps its Ansible vault password, a "master secret", on the encrypted partition mounted at Secret/ in a file named vault-password. The administrator generated a 16 character pronounceable password with gpw 1 16 and saved it like so: gpw 1 16 >Secret/vault-password. The following example password matches the example encryptions above.

alitysortstagess

A working Ansible configuration can be "tangled" from this document to produce the test configuration described in the Testing chapter. The tangling is done by Emacs's org-babel-tangle function and has already been performed with the resulting tangle included in the distribution with this document.

An institution using the Ansible configuration herein can include this document and its tangle as a Git submodule, e.g. in institute/, and thus safely merge updates while keeping public and private particulars separate, in sibling subdirectories public/ and private/. The following example commands create a new Git repo in ~/network/ and add an institute/ submodule.

cd
mkdir network
cd network
git init
git submodule add git://birchwood-abbey.net/~puck/institute
git add institute

An institute administrator would then need to add several more files.

• A top-level Ansible configuration file, ansible.cfg, would be created by copying institute/ansible.cfg and changing the roles_path to roles:institute/roles.
• A host inventory, hosts, would be created, perhaps by copying institute/hosts and changing its IP addresses.
• A site playbook, site.yml, would be created in a new playbooks/ subdirectory by copying institute/playbooks/site.yml with appropriate changes.
• All of the files in institute/public/ and institute/private/ would be copied, with appropriate changes, into new subdirectories public/ and private/.
• ~/network/Secret would be a symbolic link to the (auto-mounted?) location of the administrator's encrypted USB drive, as described in section Keys.

The files in institute/roles_t/ were "tangled" from this document and must be copied to institute/roles/ for reasons discussed in the next section. This document does not "tangle" directly into roles/ to avoid clobbering changes to a working (debugged!) configuration.

The playbooks/ directory must include the institutional playbooks, which find their settings and templates relative to this directory, e.g. in ../private/vars.yml. Running institutional playbooks from ~/network/playbooks/ means they will use ~/network/private/ rather than the example ~/network/institute/private/.

cp -r institute/roles_t institute/roles
( cd playbooks; ln -s ../institute/playbooks/* . )

Given these preparations, the inst script should work in the super-project's directory.

./institute/inst config -n

The Ansible roles currently tangle into the roles_t/ directory to ensure that debugged Ansible code in roles/ is not clobbered by code tangled from this document. Comparing roles_t/ with roles/ will reveal any changes made to roles/ during debugging that need to be reconciled with this document as well as any policy changes in this document that require changes to the current roles/.

When debugging literate programs becomes A Thing, then this document can tangle directly into roles/, and literate debuggers can find their way back to the code block in this document.

The code blocks in this chapter tangle into the inst script. Each block examines the script's command line arguments to determine whether its sub-command was intended to run, and exits with an appropriate code when it is done.

The first code block is the header of the ./inst script.

#!/usr/bin/perl -w
#
# DO NOT EDIT.
#
# This file was tangled from a small institute's README.org.

use strict;
use IO::File;

The next code block does not implement a sub-command; it implements part of all ./inst sub-commands. It performs a "sanity check" on the current directory, warning of missing files or directories, and especially checking that all files in private/ have appropriate permissions. It probes past the Secret/ mount point (probing for Secret/become.yml) to ensure the volume is mounted.

sub note_missing_file_p ($);
sub note_missing_directory_p ($);

{
  my $missing = 0;
  if (note_missing_file_p "ansible.cfg") { $missing += 1; }
  if (note_missing_file_p "hosts") { $missing += 1; }
  if (note_missing_directory_p "Secret") { $missing += 1; }
  if (note_missing_file_p "Secret/become.yml") { $missing += 1; }
  if (note_missing_directory_p "playbooks") { $missing += 1; }
  if (note_missing_file_p "playbooks/site.yml") { $missing += 1; }
  if (note_missing_directory_p "roles") { $missing += 1; }
  if (note_missing_directory_p "public") { $missing += 1; }
  if (note_missing_directory_p "private") { $missing += 1; }

  for my $filename (glob "private/*") {
    my $perm = (stat $filename)[2];
    if ($perm & 077) {
      print "$filename: not private\n";
    }
  }
  die "$missing missing files\n" if $missing != 0;
}

sub note_missing_file_p ($) {
  my ($filename) = @_;
  if (! -f $filename) {
    print "$filename: missing\n";
    return 1;
  } else {
    return 0;
  }
}

sub note_missing_directory_p ($) {
  my ($dirname) = @_;
  if (! -d $dirname) {
    print "$dirname: missing\n";
    return 1;
  } else {
    return 0;
  }
}

To ensure that Ansible and ./inst are sympatico vis-a-vi certain variable values (esp. private values like network addresses), a check-inst-vars.yml playbook is used to update the Perl syntax file private/vars.pl before ./inst loads it. The Perl code in inst declares the necessary global variables and private/vars.pl sets them.

sub mysystem (@) {
  my $line = join (" ", @_);
  print "$line\n";
  my $status = system $line;
  die "status: $status\nCould not run $line: $!\n" if $status != 0;
}

mysystem "ansible-playbook playbooks/check-inst-vars.yml >/dev/null";

our ($domain_name, $domain_priv,
     $front_addr, $front_wg_pubkey,
     $public_wg_net_cidr, $public_wg_port,
     $private_net_cidr, $wild_net_cidr,
     $gate_wild_addr, $gate_wg_pubkey,
     $campus_wg_net_cidr, $campus_wg_port,
     $core_addr, $core_wg_pubkey);
do "./private/vars.pl";

The playbook that updates private/vars.pl:

- hosts: localhost
  gather_facts: no
  roles: [ check-inst-vars ]

This role is executed by playbooks/check-inst-vars.yml and is not just a playbook because it needs a copy of the role defaults.

---
<<network-vars>>
<<address-vars>>

---
- include_vars: ../public/vars.yml
- include_vars: ../private/vars.yml
- copy:
    content: |
      $domain_name = "{{ domain_name }}";
      $domain_priv = "{{ domain_priv }}";

      $front_addr = "{{ front_addr }}";
      $front_wg_pubkey = "{{ front_wg_pubkey }}";

      $public_wg_net_cidr = "{{ public_wg_net_cidr }}";
      $public_wg_port = "{{ public_wg_port }}";

      $private_net_cidr = "{{ private_net_cidr }}";
      $wild_net_cidr = "{{ wild_net_cidr }}";

      $gate_wild_addr = "{{ gate_wild_addr }}";
      $gate_wg_pubkey = "{{ gate_wg_pubkey }}";

      $campus_wg_net_cidr = "{{ campus_wg_net_cidr }}";
      $campus_wg_port = "{{ campus_wg_port }}";

      $core_addr = "{{ core_addr }}";
      $core_wg_pubkey = "{{ core_wg_pubkey }}";
    dest: ../private/vars.pl
    mode: u=rw,g=,o=

Most of these settings are already in private/vars.yml. The following few provide the servers' public keys and ports.

front_wg_pubkey: S+6HaTnOwwhWgUGXjSBcPAvifKw+j8BDTRfq534gNW4=
public_wg_port:  39608

gate_wg_pubkey:  y3cjFnvQbylmH4lGTujpqc8rusIElmJ4Gu9hh6iR7QI=
campus_wg_port:  51820

core_wg_pubkey:  lGhC51IBgZtlq4H2bsYFuKvPtV0VAEwUvVIn5fW7D0c=

The next code block implements the CA sub-command, which creates a new CA (certificate authority) in Secret/CA/ as well as SSH and PGP keys for the administrator, Monkey, Front and root, also in sub-directories of Secret/. The CA is created with the "common name" provided by the full_name variable. An example is given here.

full_name: Small Institute LLC

The Secret/ directory is on an off-line, encrypted volume plugged in just for the duration of ./inst commands, so Secret/ is actually a symbolic link to a volume's automount location.

ln -s /media/sysadm/ADE7-F866/ Secret

The Secret/CA/ directory is prepared using Easy RSA's make-cadir command. The Secret/CA/vars file thus created is edited to contain the appropriate names (or just to set EASYRSA_DN to cn_only).

sudo apt install easy-rsa
( cd Secret/; make-cadir CA )
./inst CA

Running ./inst CA creates the new CA and keys. The command prompts for the Common Name (or several levels of Organizational names) of the certificate authority. The full_name is given: Small Institute LLC. The CA is used to issue certificates for front, gate and core, which are installed on the servers during the next ./inst config.

if (defined $ARGV[0] && $ARGV[0] eq "CA") {
  die "usage: $0 CA" if @ARGV != 1;
  die "Secret/CA/easyrsa: not an executable\n"
    if ! -x "Secret/CA/easyrsa";
  die "Secret/CA/pki/: already exists\n" if -e "Secret/CA/pki";

  umask 077;
  mysystem "cd Secret/CA; ./easyrsa init-pki";
  mysystem "cd Secret/CA; ./easyrsa build-ca nopass";
  # Common Name: small.example.org

  my $dom = $domain_name;
  my $pvt = $domain_priv;
  mysystem "cd Secret/CA; ./easyrsa build-server-full $dom nopass";
  mysystem "cd Secret/CA; ./easyrsa build-server-full core.$pvt nopass";
  umask 077;

  mysystem "mkdir --mode=700 Secret/root.gnupg";
  mysystem ("gpg --homedir Secret/root.gnupg",
            "--batch --quick-generate-key --passphrase ''",
            "root\@core.$pvt");
  mysystem ("gpg --homedir Secret/root.gnupg",
            "--export --armor --output Secret/root-pub.pem",
            "root\@core.$pvt");
  chmod 0440, "root-pub.pem";
  mysystem ("gpg --homedir Secret/root.gnupg",
            "--export-secret-key --armor --output Secret/root-sec.pem",
            "root\@core.$pvt");
  chmod 0400, "root-sec.pem";

  mysystem "mkdir Secret/ssh_admin";
  chmod 0700, "Secret/ssh_admin";
  mysystem ("ssh-keygen -q -t rsa",
            "-C A\\ Small\\ Institute\\ Administrator",
            "-N '' -f Secret/ssh_admin/id_rsa");

  mysystem "mkdir Secret/ssh_monkey";
  chmod 0700, "Secret/ssh_monkey";
  mysystem "echo 'HashKnownHosts  no' >Secret/ssh_monkey/config";
  mysystem ("ssh-keygen -q -t rsa -C monkey\@core",
            "-N '' -f Secret/ssh_monkey/id_rsa");

  mysystem "mkdir Secret/ssh_front";
  chmod 0700, "Secret/ssh_front";
  mysystem "ssh-keygen -A -f Secret/ssh_front -C $dom";
  exit;
}

The next code block implements the config sub-command, which provisions network services by running the site.yml playbook described in playbooks/site.yml. It recognizes an optional -n flag indicating that the service configurations should just be checked. Given an optional host name, it provisions (or checks) just the named host.

Example command lines:

./inst config
./inst config -n
./inst config HOST
./inst config -n HOST

if (defined $ARGV[0] && $ARGV[0] eq "config") {
  die "Secret/CA/easyrsa: not executable\n"
    if ! -x "Secret/CA/easyrsa";
  shift;
  my $cmd = "ansible-playbook -e \@Secret/become.yml";
  if (defined $ARGV[0] && $ARGV[0] eq "-n") {
    shift;
    $cmd .= " --check --diff"
  }
  if (@ARGV == 0) {
    ;
  } elsif (defined $ARGV[0]) {
    my $hosts = lc $ARGV[0];
    die "$hosts: contains illegal characters"
      if $hosts !~ /^!?[a-z][-a-z0-9,!]+$/;
    $cmd .= " -l $hosts";
  } else {
    die "usage: $0 config [-n] [HOSTS]\n";
  }
  $cmd .= " playbooks/site.yml";
  mysystem $cmd;
  exit;
}

For general information about members and their Unix accounts, see Accounts. The account management sub-commands maintain a mapping associating member "usernames" (Unix account names) with their records. The mapping is stored among other things in private/members.yml as the value associated with the key members.

A new member's record in the members mapping will have the status key value current. That key gets value former when the member leaves.³ Access by former members is revoked by invalidating the Unix account passwords, removing any authorized SSH keys from Front and Core, and removing their public keys from the WireGuard™ configurations.

The example file (below) contains a membership roll with one membership record, for an account named dick, which registered the public keys of devices named dick, dicks-phone and dicks-razr. dicks-razr is presumably a replacement for dicks-phone, which was lost and its key invalidated. Lastly, Dick's membership record includes a vault-encrypted password (for Fetchmail) and the two password hashes installed on Front and Core. (The example hashes are truncated versions.)

---
members:
  dick:
    status: current
    clients:
    - dick 4 4qd4xdRztZBKhFrX9jI/b4fnMzpKQ5qhg691hwYSsX8=
    - dicks-phone 5 --WFbTSff17QiYObXoU+7mjaEUCqKjgvLqA49pAxqVeWg=
    - dicks-razr 6 zho0qMxoLclJSQu4GeJEcMkk0hx4Q047OcNc8vOejVw=
    password_front:
      $6$17h49U76$c7TsH6eMVmoKElNANJU1F1LrRrqzYVDreNu.QarpCoSt9u0gTHgiQ
    password_core:
      $6$E9se3BoSilq$T.W8IUb/uSlhrVEWUQsAVBweiWB4xb3ebQ0tguVxJaeUkqzVmZ
    password_fetchmail: !vault |
      $ANSIBLE_VAULT;1.1;AES256
      38323138396431323564366136343431346562633965323864633938613363336
      4333334333966363136613264636365383031376466393432623039653230390a
      39366232633563646361616632346238333863376335633639383162356661326
      4363936393530633631616630653032343465383032623734653461323331310a
      6535633263656434393030333032343533626235653332626330666166613833
usernames:
- dick
clients:
- thing 3 LdsCsgfjKCfd5+VKS+Q/dQhWO8NRNygByDO2VxbXlSQ=

The members.yml file will be modified during testing, and should not be overwritten by a re-tangle during testing, so it not tangled from this file. Thus in the fresh built (e.g. test) system private/members.yml does not exist, not until a ./inst new command creates the first member. Until then, Ansible includes the private/members-empty.yml file. It does that using the first_found lookup plugin and a list of the two files with members.yml first and members-empty.yml last. That list is the value of membership_rolls.

membership_rolls:
- "../private/members.yml"
- "../private/members-empty.yml"

---
members: {}
usernames: []
clients: []

Using the standard Perl library YAML::XS, the subroutine for reading the membership roll is simple, returning the top-level hash read from the file. The dump subroutine is another story (below).

use YAML::XS qw(LoadFile DumpFile);

sub read_members_yaml () {
  my $path;
  $path = "private/members.yml";
  if (-e $path) { return LoadFile ($path); }
  $path = "private/members-empty.yml";
  if (-e $path) { return LoadFile ($path); }
  die "private/members.yml: not found\n";
}

sub write_members_yaml ($) {
  my ($yaml) = @_;
  my $old_umask = umask 077;
  my $path = "private/members.yml";
  print "$path: "; STDOUT->flush;
  eval { #DumpFile ("$path.tmp", $yaml);
         dump_members_yaml ("$path.tmp", $yaml);
         rename ("$path.tmp", $path)
           or die "Could not rename $path.tmp: $!\n"; };
  my $err = $@;
  umask $old_umask;
  if ($err) {
    print "ERROR\n";
  } else {
    print "updated\n";
  }
  die $err if $err;
}

sub dump_members_yaml ($$) {
  my ($pathname, $yaml) = @_;
  my $O = new IO::File;
  open ($O, ">$pathname") or die "Could not open $pathname: $!\n";
  print $O "---\n";
  if (keys %{$yaml->{"members"}}) {
    print $O "members:\n";
    for my $user (sort keys %{$yaml->{"members"}}) {
      print_member ($O, $user, $yaml->{"members"}->{$user});
    }
    print $O "usernames:\n";
    for my $user (sort keys %{$yaml->{"members"}}) {
      print $O "- $user\n";
    }
  } else {
    print $O "members: {}\n";
    print $O "usernames: []\n";
  }
  if (@{$yaml->{"clients"}}) {
    print $O "clients:\n";
    for my $name (@{$yaml->{"clients"}}) {
      print $O "- $name\n";
    }
  } else {
    print $O "clients: []\n";
  }
  close $O or die "Could not close $pathname: $!\n";
}

The first implementation using YAML::Tiny balked at the !vault data type. The current version using YAML::XS (Simonov's libyaml) does not support local data types neither, but does not abort. It just produces a multi-line string. Luckily the structure of members.yml is relatively simple and fixed, so a purpose-built printer can add back the !vault data types at appropriate points. YAML::XS thus provides only a borked parser. Also luckily, the YAML produced by the for-the-purpose printer makes the resulting membership roll easier to read, with the username and status at the top of each record.

sub print_member ($$$) {
  my ($out, $username, $member) = @_;
  print $out "  ", $username, ":\n";
  print $out "    status: ", $member->{"status"}, "\n";
  if (@{$member->{"clients"} || []}) {
    print $out "    clients:\n";
    for my $name (@{$member->{"clients"} || []}) {
      print $out "    - ", $name, "\n";
    }
  } else {
    print $out "    clients: []\n";
  }
  print $out "    password_front: ", $member->{"password_front"}, "\n";
  print $out "    password_core: ", $member->{"password_core"}, "\n";
  if (defined $member->{"password_fetchmail"}) {
    print $out "    password_fetchmail: !vault |\n";
    for my $line (split /\n/, $member->{"password_fetchmail"}) {
      print $out "      $line\n";
    }
  }
  my @standard_keys = ( "status", "clients",
                        "password_front", "password_core",
                        "password_fetchmail" );
  my @other_keys = (sort
                    grep { my $k = $_;
                           ! grep { $_ eq $k } @standard_keys }
                    keys %$member);
  for my $key (@other_keys) {
    print $out "    $key: ", $member->{$key}, "\n";
  }
}

The next code block implements the new sub-command. It adds a new member to the institute's membership roll. It runs an Ansible playbook to create the member's Nextcloud user, updates private/members.yml, and runs the site.yml playbook. The site playbook (re)creates the member's accounts on Core and Front, (re)installs the member's personal homepage on Front, and the member's Fetchmail service on Core. All services are configured with an initial, generated password.

sub valid_username (@);
sub shell_escape ($);
sub strip_vault ($);

if (defined $ARGV[0] && $ARGV[0] eq "new") {
  my $user = valid_username (@ARGV);
  my $yaml = read_members_yaml ();
  my $members = $yaml->{"members"};
  die "$user: already exists\n" if defined $members->{$user};

  my $pass = `apg -n 1 -x 12 -m 12`; chomp $pass;
  print "Initial password: $pass\n";
  my $epass = shell_escape $pass;
  my $front = `mkpasswd -m sha-512 "$epass"`; chomp $front;
  my $core = `mkpasswd -m sha-512 "$epass"`; chomp $core;
  my $vault = strip_vault `ansible-vault encrypt_string "$epass"`;
  mysystem ("ansible-playbook -e \@Secret/become.yml",
            "playbooks/nextcloud-new.yml",
            "-e user=$user", "-e pass=\"$epass\"",
            ">/dev/null");
  $members->{$user} = { "status" => "current",
                        "password_front" => $front,
                        "password_core" => $core,
                        "password_fetchmail" => $vault };
  write_members_yaml $yaml;
  mysystem ("ansible-playbook -e \@Secret/become.yml",
            "-t accounts -l core,front playbooks/site.yml",
            ">/dev/null");
  exit;
}

sub valid_username (@) {
  my $sub = $_[0];
  die "usage: $0 $sub USER\n"
    if @_ != 2;
  my $username = lc $_[1];
  die "$username: does not begin with an alphabetic character\n"
    if $username !~ /^[a-z]/;
  die "$username: contains non-alphanumeric character(s)\n"
    if $username !~ /^[a-z0-9]+$/;
  return $username;
}

sub shell_escape ($) {
  my ($string) = @_;
  my $result = "$string";
  $result =~ s/([\$`"\\ ])/\\$1/g;
  return ($result);
}

sub strip_vault ($) {
  my ($string) = @_;
  die "Unexpected result from ansible-vault: $string\n"
    if $string !~ /^ *!vault [|]/;
  my @lines = split /^ */m, $string;
  return (join "", @lines[1..$#lines]);
}

- hosts: core
  tasks:
  - name: Run occ user:add.
    become: yes
    shell:
      chdir: /var/www/nextcloud/
      cmd: >
        sudo -u www-data sh -c
        "OC_PASS={{ pass }}
        php occ user:add {{ user }} --password-from-env"

The institute's passwd command on Core securely emails root with a member's desired password (hashed). The command may update the servers immediately or let the administrator do that using the ./inst pass command. In either case, the administrator needs to update the membership roll, and so receives an encrypted email, which gets piped into ./inst pass. This command decrypts the message, parses the (YAML) content, updates private/members.yml, and runs the full Ansible site.yml playbook to update the servers. If all goes well a message is sent to member@core.

The old command disables a member's account (and thus their clients).

if (defined $ARGV[0] && $ARGV[0] eq "old") {
  my $user = valid_username (@ARGV);
  my $yaml = read_members_yaml ();
  my $members = $yaml->{"members"};
  my $member = $members->{$user};
  die "$user: does not exist\n" if ! defined $member;

  mysystem ("ansible-playbook -e \@Secret/become.yml",
            "playbooks/nextcloud-old.yml -e user=$user",
            ">/dev/null");
  $member->{"status"} = "former";
  umask 077;
  write_members_yaml $yaml;
  write_wireguard $yaml;
  mysystem ("ansible-playbook -e \@Secret/become.yml",
            "-t accounts playbooks/site.yml",
            ">/dev/null");
  exit;
}

- hosts: core
  tasks:
  - name: Run occ user:disable.
    become: yes
    shell:
      chdir: /var/www/nextcloud/
      cmd: >
        sudo -u www-data sh -c
        "php occ user:disable {{ user }}"

The client command registers the public key of a client wishing to connect to the institute's WireGuard™ subnets. The command allocates a host number, associates it with the provided public key, and updates the configuration files front-wg0.conf and gate-wg0.conf. These are distributed to the servers, which are then reset. Thereafter the servers recognize the new peer (and drop packets from any "peer" that is no longer authorized).

The client command also generates template WireGuard™ configuration files for the client. They contain the necessary parameters except the client's PrivateKey, which in most cases should be found in the local /etc/wireguard/private-key, not in the configuration files. Private keys (and corresponding public keys) should be generated on the client (i.e. by the WireGuard for Android™ app) and never revealed (i.e. sent in email, copied to a network drive, etc.).

The generated configurations vary depending on the type of client, which must be given as the first argument to the command. For most types, two configuration files are generated. campus.conf contains the client's campus VPN configuration, and public.conf the client's public VPN configuration.

• ./inst client android NAME USER PUBKEY
  An android client runs WireGuard for Android™ or work-alike.
• ./inst client debian NAME USER PUBKEY
  A debian client runs a Debian/Linux desktop with NetworkManager (though wg-quick is currently used).
• ./inst client campus NAME PUBKEY
  A campus client is an institute machine (with or without desktop) that is used by the institute generally, is not the property of a member, never roams off campus, and so is remotely administered with Ansible. Just one configuration file is generated: campus.conf.

The administrator emails the template .conf files to new members. (They contain no secrets.) The members will have already installed the wireguard package in order to run the wg genkey and wg pubkey commands. After receiving the .conf templates, they paste in their private keys and install the resulting files in e.g. /etc/wireguard/wg0.conf and wg1.conf. To connect, members run a command like systemctl start wg-quick@wg0. (There may be better support in NetworkManager soon.)

sub write_wg_server ($$$$$);
sub write_wg_client ($$$$$$);
sub hostnum_to_ipaddr ($$);
sub hostnum_to_ipaddr_cidr ($$);

if (defined $ARGV[0] && $ARGV[0] eq "client") {
  my $type = $ARGV[1]||"";
  my $name = $ARGV[2]||"";
  my $user = $ARGV[3]||"";
  my $pubkey = $ARGV[4]||"";
  if ($type eq "android" || $type eq "debian") {
    die "usage: $0 client $type NAME USER PUBKEY\n" if @ARGV != 5;
    die "$name: invalid host name\n" if $name !~ /^[a-z][-a-z0-9]+$/;
  } elsif ($type eq "campus") {
    die "usage: $0 client campus NAME PUBKEY\n" if @ARGV != 4;
    die "$name: invalid host name\n" if $name !~ /^[a-z][-a-z0-9]+$/;
    $pubkey = $user;
    $user = "";
  } else {
    die "usage: $0 client [debian|android|campus]\n";
  }
  my $yaml = read_members_yaml;
  my $members = $yaml->{"members"};
  my $member = $members->{$user};
  die "$user: does not exist\n"
    if !defined $member && $type ne "campus";
  die "$user: no longer current\n"
    if defined $member && $member->{"status"} ne "current";

  my @campus_peers # [ name, hostnum, type, pubkey, user|"" ]
     = map { [ (split / /), "" ] } @{$yaml->{"clients"}};

  my @member_peers = ();
  for my $u (sort keys %$members) {
    push @member_peers,
         map { [ (split / /), $u ] } @{$members->{$u}->{"clients"}};
  }

  my @all_peers = sort { $a->[1] <=> $b->[1] }
                       (@campus_peers, @member_peers);

  for my $p (@all_peers) {
    my ($n, $h, $t, $k, $u) = @$p;
    die "$n: name already in use by $u\n"
        if $name eq $n && $u ne "";
    die "$n: name already in use on campus\n"
        if $name eq $n && $u eq "";
  }

  my $hostnum = (@all_peers
                 ? 1 + $all_peers[$#all_peers][1]
                 : 3);

  push @{$type eq "campus"
         ? $yaml->{"clients"}
         : $member->{"clients"}},
       "$name $hostnum $type $pubkey";

  umask 077;
  write_members_yaml $yaml;
  write_wireguard $yaml;

  umask 033;
  write_wg_client ("public.conf",
                   hostnum_to_ipaddr ($hostnum, $public_wg_net_cidr),
                   $type,
                   $front_wg_pubkey,
                   "$front_addr:$public_wg_port",
                   hostnum_to_ipaddr (1, $public_wg_net_cidr))
    if $type ne "campus";
  write_wg_client ("campus.conf",
                   hostnum_to_ipaddr ($hostnum, $campus_wg_net_cidr),
                   $type,
                   $gate_wg_pubkey,
                   "$gate_wild_addr:$campus_wg_port",
                   hostnum_to_ipaddr (1, $campus_wg_net_cidr));

  mysystem ("ansible-playbook -e \@Secret/become.yml",
            "-l gate,front",
            "-t accounts playbooks/site.yml",
            ">/dev/null");
  exit;
}

sub write_wireguard ($) {
  my ($yaml) = @_;

  my @campus_peers # [ name, hostnum, type, pubkey, user|"" ]
     = map { [ (split / /), "" ] } @{$yaml->{"clients"}};

  my $members = $yaml->{"members"};
  my @member_peers = ();
  for my $u (sort keys %$members) {
    next if $members->{$u}->{"status"} ne "current";
    push @member_peers,
         map { [ (split / /), $u ] } @{$members->{$u}->{"clients"}};
  }

  my @all_peers = sort { $a->[1] <=> $b->[1] }
                       (@campus_peers, @member_peers);

  my $core_wg_addr = hostnum_to_ipaddr (2, $public_wg_net_cidr);
  my $extra_front_config = "
PostUp = resolvectl dns %i $core_addr
PostUp = resolvectl domain %i $domain_priv

# Core
[Peer]
PublicKey = $core_wg_pubkey
AllowedIPs = $core_wg_addr
AllowedIPs = $private_net_cidr
AllowedIPs = $wild_net_cidr
AllowedIPs = $campus_wg_net_cidr\n";

  write_wg_server ("private/front-wg0.conf", \@member_peers,
                   hostnum_to_ipaddr_cidr (1, $public_wg_net_cidr),
                   $public_wg_port, $extra_front_config);
  write_wg_server ("private/gate-wg0.conf", \@all_peers,
                   hostnum_to_ipaddr_cidr (1, $campus_wg_net_cidr),
                   $campus_wg_port, "\n");
}

sub write_wg_server ($$$$$) {
  my ($file, $peers, $addr_cidr, $port, $extra) = @_;
  my $O = new IO::File;
  open ($O, ">$file.tmp") or die "Could not open $file.tmp: $!\n";
  print $O "[Interface]
Address = $addr_cidr
ListenPort = $port
PostUp = wg set %i private-key /etc/wireguard/private-key$extra";
  for my $p (@$peers) {
    my ($n, $h, $t, $k, $u) = @$p;
    next if $k =~ /^-/;
    my $ip = hostnum_to_ipaddr ($h, $addr_cidr);
    print $O "
# $n
[Peer]
PublicKey = $k
AllowedIPs = $ip\n";
  }
  close $O or die "Could not close $file.tmp: $!\n";
  rename ("$file.tmp", $file)
    or die "Could not rename $file.tmp: $!\n";
}

sub write_wg_client ($$$$$$) {
  my ($file, $addr, $type, $pubkey, $endpt, $server_addr) = @_;

  my $O = new IO::File;
  open ($O, ">$file.tmp") or die "Could not open $file.tmp: $!\n";

  my $DNS = ($type eq "android"
             ? "
DNS = $core_addr
Domain = $domain_priv"
             : "
PostUp = wg set %i private-key /etc/wireguard/private-key
PostUp = resolvectl dns %i $core_addr
PostUp = resolvectl domain %i $domain_priv");

  my $WILD = ($file eq "public.conf"
              ? "
AllowedIPs = $wild_net_cidr"
              : "");

  print $O "[Interface]
Address = $addr$DNS

[Peer]
PublicKey = $pubkey
EndPoint = $endpt
AllowedIPs = $server_addr
AllowedIPs = $private_net_cidr$WILD
AllowedIPs = $public_wg_net_cidr
AllowedIPs = $campus_wg_net_cidr\n";
  close $O or die "Could not close $file.tmp: $!\n";
  rename ("$file.tmp", $file)
    or die "Could not rename $file.tmp: $!\n";
}

sub hostnum_to_ipaddr ($$)
{
  my ($hostnum, $net_cidr) = @_;

  # Assume 24bit subnet, 8bit hostnum.
  # Find a Perl library for more generality?
  die "$hostnum: hostnum too large\n" if $hostnum > 255;
  my ($prefix) = $net_cidr =~ m"^(\d+\.\d+\.\d+)\.\d+/24$";
  die if !$prefix;
  return "$prefix.$hostnum";
}

sub hostnum_to_ipaddr_cidr ($$)
{
  my ($hostnum, $net_cidr) = @_;

  # Assume 24bit subnet, 8bit hostnum.
  # Find a Perl library for more generality?
  die "$hostnum: hostnum too large\n" if $hostnum > 255;
  my ($prefix) = $net_cidr =~ m"^(\d+\.\d+\.\d+)\.\d+/24$";
  die if !$prefix;
  return "$prefix.$hostnum/24";
}

This should be the last block tangled into the inst script. It catches any command lines that were not handled by a sub-command above.

die "usage: $0 [CA|config|new|pass|old|client] ...\n";

The networks used in the test:

public

A NAT Network, simulating the cloud provider's and campus ISP's networks. This is the only network with DHCP and DNS services provided by the hypervisor. It is not the default NAT network because gate and front need to communicate.

vboxnet0

A Host-only network, simulating the institute's private Ethernet switch. It has no services, no DHCP, just the host machine at 192.168.56.10 pretending to be the administrator's notebook.

vboxnet1

Another Host-only network, simulating the wild Ethernet between Gate and the campus IoT (and Wi-Fi APs). It has no services, no DHCP, just the host at 192.168.57.2.

vboxnet2

A third Host-only network, used only to directly connect the host to front.

In this simulation the IP address for front is not a public address but a private address on the NAT network public. Thus front is not accessible by the host, by Ansible on the administrator's notebook. To work around this restriction, front gets a second network interface connected to the vboxnet2 network. The address of this second interface is used by Ansible to access front.⁴

The networks described above are created and "started" with the following VBoxManage commands.

VBoxManage natnetwork add --netname public \
                          --network 192.168.15.0/24 \
                          --enable --dhcp on --ipv6 off
VBoxManage natnetwork start --netname public
VBoxManage dhcpserver modify --network=public --lower-ip=192.168.15.5
VBoxManage hostonlyif create # vboxnet0
VBoxManage hostonlyif ipconfig vboxnet0 --ip=192.168.56.10
VBoxManage hostonlyif create # vboxnet1
VBoxManage hostonlyif ipconfig vboxnet1 --ip=192.168.57.2
VBoxManage hostonlyif create # vboxnet2
VBoxManage hostonlyif ipconfig vboxnet2 --ip=192.168.58.1

Note that only the NAT network public should have a DHCP server enabled (to simulate an ISP and cloud for gate and front respectively). Yet front is statically assigned an IP address outside the DHCP server's pool. This ensures it gets front_addr without more server configuration.

Note also that actual ISPs and clouds will provide Gate and Front with public network addresses. In this simulation "they" provide addresses in 192.168.15.0/24, on the NAT network public.

The virtual machines are created by VBoxManage command lines in the following sub-sections. They each start with a recent Debian release (e.g. debian-12.5.0-amd64-netinst.iso) in their simulated DVD drives. Preparation of The Hardware installed additional software packages and keys while the machines had Internet access. They were then moved to the new campus network where Ansible completed the configuration without Internet access.

Preparation of the test machines is automated by "preparatory scripts" that install the same "additional software packages" and the same test keys given in the examples. The scripts are run on each VM while they are still attached to the host's NAT network and have Internet access. They prepare the machine to reboot on the simulated campus network without Internet access, ready for final configuration by Ansible and the first launch of services. The "move to campus" is simulated by shutting each VM down, executing a VBoxManage command line or two, and restarting.

At this point the three test machines core, gate, and front are running fresh Debian systems with select additional packages, on their final networks, with a privileged account named sysadm that authorizes password-less access from the administrator's notebook, ready to be configured by Ansible. However the administrator's notebook may not recognize the test VMs or, worse yet, remember different public keys for them (from previous test machines). For this reason, the administrator executes the following commands before the initial ./inst config.

ssh sysadm@192.168.56.1 date
ssh sysadm@192.168.56.2 date
ssh sysadm@192.168.58.3 date
./inst config

Note that this initial run should exercise all of the handlers, and that subsequent runs probably do not.

Presumably the ./inst config command completed successfully, but before testing begins, gate is restarted. Basic networking tests will fail unless the interfaces on gate are renamed, and nothing less than a restart will get systemd-udevd to rename the isp and wifi interfaces.

At this point the test institute is just core, gate and front, no other campus servers, no members nor their VPN client devices. On each machine, Systemd should assess the system's state as running with 0 failed units.

systemctl status

gate and thus core should be able to reach the Internet and front. If core can reach the Internet and front, then gate is forwarding (and NATing). On core (and gate):

ping -c 1 8.8.4.4      # dns.google
ping -c 1 192.168.15.4 # front_addr

gate and thus core should be able to resolve internal and public domain names. (Front does not use the institute's internal domain names yet.) On core (and gate):

host dns.google
host core.small.private
host www

The last resort email address, root, should deliver to the administrator's account. On core, gate and front:

/sbin/sendmail root
Testing email to root.
.

Two messages, from core and gate, should appear in /home/sysadm/Maildir/new/ on core in just a couple seconds. The message from front should be delivered to the same directory but on front. While members' emails are automatically fetched (with fetchmail(1)) to core, the system administrator is expected to fetch system emails directly to their desktop (and to give them instant attention).

Further tests involve Nextcloud account management. Nextcloud is installed on core as described in Install Nextcloud. Once /Nextcloud/ is created, ./inst config core will validate or update its configuration files.

The administrator will need a desktop system in the test campus networks (using the campus name server). The test Nextcloud configuration requires that it be accessed with the domain name core.small.private. The following sections describe how a client desktop is simulated and connected to the test VPNs (and test campus name server). Its browser can then connect to core.small.private to exercise the test Nextcloud.

The process starts with enrolling the first member of the institute using the ./inst new command and registering a client's public key with the ./inst client command.

A member must be enrolled so that a member's client machine can be authorized and then test the VPNs, Nextcloud, and the web sites. The first member enrolled in the simulated institute is New Hampshire innkeeper Dick Loudon. Mr. Loudon's accounts on institute servers are named dick, as is his notebook.

./inst new dick

Take note of Dick's initial password.

A test member's notebook is created next, much like the servers, except with memory and disk space doubled to 2GiB and 8GiB, and a desktop. This machine is not configured by Ansible. Rather, its WireGuard™ tunnels and web browser test the VPN configurations on gate and front, and the Nextcloud installation on core.

NAME=dick
RAM=2048
DISK=8192
create_vm
VBoxManage modifyvm $NAME --macaddress1 080027dc54b5
VBoxManage modifyvm $NAME --nic1 hostonly --hostonlyadapter1 vboxnet1

Dick's notebook, dick, is initially connected to the host-only network vboxnet1 as though it were the campus wireless access point. It simulates a member's notebook on campus, connected to (NATed behind) the access point.

Debian is installed much as detailed in A Test Machine except that the SSH server option is not needed and the GNOME desktop option is. When the machine reboots, the administrator logs into the desktop and installs a couple additional software packages (which require several more).

sudo apt install systemd-resolved \
                 wireguard nextcloud-desktop evolution

The ./inst client command is used to register the public key of a client wishing to connect to the institute's VPNs. In this test, new member Dick wants to connect his notebook, dick, to the institute VPNs. First he generates a pair of WireGuard™ keys by running the following commands on his notebook.

( umask 077; wg genkey \
  | sudo tee /etc/wireguard/private-key ) | wg pubkey

Dick then sends the resulting public key to the administrator, who runs the following command.

./inst client debian dick dick \
  4qd4xdRztZBKhFrX9jI/b4fnMzpKQ5qhg691hwYSsX8=

The command generates campus.conf and public.conf configuration files, which the administrator sends, openly (e.g. in email) to Dick. Dick then installs the configuration files in /etc/wireguard/ and creates the campus interface.

sudo cp {campus,public}.conf /etc/wireguard/
sudo wg-quick up campus
sudo systemctl enable wg-quick@campus

A few basic tests are then performed in a terminal.

systemctl status
ping -c 1 8.8.8.8      # dns.google
ping -c 1 192.168.56.1 # core
host dns.google
host core.small.private
host www

Next, the administrator copies Backup/WWW/ (included in the distribution) to /WWW/ on core and sets the file permissions appropriately.

sudo chown -R monkey:staff /WWW/campus /WWW/live /WWW/test
sudo chmod 02775 /WWW/*
sudo chmod 664 /WWW/*/index.html
sudo -u monkey /usr/local/sbin/webupdate

then uses Firefox on dick to fetch the following URLs. They should all succeed and the content should be a simple sentence identifying the source file.

• http://www/
• http://www.small.private/
• http://live/
• http://live.small.private/
• http://test/
• http://test.small.private/

The first will probably be flagged as unverifiable, signed by an unknown issuer, etc. Otherwise, each should be accessible, displaying a short description of the website that was being simulated.

The simulated public web site at http://192.168.15.4/ is also tested. It should redirect to https://small.example.org/, which does not exist. However, the web site at https://192.168.15.4/ (with httpS) should exist and produce a legible page (after the usual warnings).

Next the administrator modifies /WWW/live/index.html on core and waits 15 minutes for the edit to appear in the web page at https://192.168.15.4/ (and in the file /home/www/index.html on front). The same is done to /home/www/index.html on front and the edit observed immediately, and its correction within 15 minutes.

Using the browser on the simulated member notebook, the Nextcloud installation on core can be completed. The following steps are performed on dick's desktop.

• Get http://core/nextcloud/. The attempt produces a warning about using Nextcloud via an untrusted name.
• Get https://core.small.private/nextcloud/. Receive a login page.
• Login as sysadm with password fubar.
• Examine the security & setup warnings in the Settings > Administration > Overview web page. A few minor warnings are expected.
• Download and enable Calendar and Contacts in the Apps > Featured web page.
• Logout and login as dick with Dick's initial password (noted above).
• Use the Nextcloud app to sync ~/Nextcloud/ with the cloud. In the Nextcloud Desktop app's Connection Wizard (the initial dialog), login with the URL https://core.small.private/nextcloud. The web browser should pop up with a new tab: "Connect to your account". Press "Log in" and "Grant access". The Nextcloud Connection Wizard then prompts for sync parameters. The defaults are fine. Presumably the Local Folder is /home/sysadm/Nextcloud/.
• Drop a file in ~/Nextcloud/, then find it in the Nextcloud Files web page.

• Create a Mail account in Evolution. This step does not involve Nextcloud, but placates Evolution's Welcome Wizard, and follows in the steps of the newly institutionalized luser. CardDAV and CalDAV accounts can be created in Evolution later.

  The account's full name is Dick Loudon and its email address is dick@small.example.org. The Receiving Email Server Type is IMAP, its name is mail.small.private and it uses the IMAPS port (993). The Username on the server is dick. The encryption method is TLS on a dedicated port. Authentication is by password. The Receiving Option defaults are fine. The Sending Email Server Type is SMTP with the name smtp.small.private using the default SMTP port (25). It requires neither authentication nor encryption.

  At some point Evolution will find that the server certificate is self-signed and unknown. It must be accepted (permanently).

• Create a CardDAV account in Evolution. Choose Edit, Accounts, Add, Address Book, Type CardDAV, name Small Institute, and user dick. The URL starts with https://core.small.private/nextcloud/ and ends with remote.php/dav/addressbooks/users/dick/contacts/ (yeah, 88 characters!). Create a contact in the new address book and see it in the Contacts web page. At some point Evolution will need Dick's password to access the address book.
• Create a CalDAV account in Evolution just like the CardDAV account except add a Calendar account of Type CalDAV with a URL that ends remote.php/dav/calendars/dick/personal/ (only 79 characters). Create an event in the new calendar and see it in the Calendar web page. At some point Evolution will need Dick's password to access the calendar.

With Evolution running on the member notebook dick, one second email delivery can be demonstrated. The administrator runs the following commands on front

/sbin/sendmail dick
Subject: Hello, Dick.

How are you?
.

and sees a notification on dick's desktop in a second or less.

Outgoing email is also tested. A message to sysadm@small.example.org should be delivered to /home/sysadm/Maildir/new/ on front just as fast.

At this point, dick can move abroad, from the campus Wi-Fi (host-only network vboxnet1) to the broader Internet (the NAT network public). The following command makes the change. The machine does not need to be shut down if the GUI is used to change its NIC.

VBoxManage modifyvm dick --nic1 natnetwork --natnetwork1 public

Then the campus VPN is disconnected and the public VPN connected.

systemctl stop wg-quick@wg0
systemctl start wg-quick@wg1

Again, some basics are tested in a terminal.

ping -c 1 8.8.8.8      # dns.google
ping -c 1 192.168.56.1 # core
host dns.google
host core.small.private
host www

And, again, these web pages are fetched with a browser.

• http://www/
• http://www.small.private/
• http://live/
• http://live.small.private/
• http://test/
• http://test.small.private/
• http://192.168.15.4/
• https://192.168.15.4/
• http://core.small.private/nextcloud/

The Nextcloud web pages too should still be refresh-able, editable, and Evolution should still be able to edit messages, contacts and calendar events.

To test the ./inst pass command, the administrator logs in to core as dick and runs passwd. A random password is entered, more obscure than fubar (else Nextcloud will reject it!).

The administrator then finds the password change request message in the most recent file in /home/sysadm/Maildir/new/ and pipes it to the ./inst pass command. The administrator might do that by copying the message to a more conveniently named temporary file on core, e.g. ~/msg, copying that to the current directory on the administrator's notebook, and feeding it to ./inst pass on standard input.

On core, logged in as sysadm:

sudo -u dick passwd
( cd ~/Maildir/new/
  cp `ls -1t | head -1` ~/msg )
grep Subject: ~/msg

To ensure that the most recent message is indeed the password change request, the last command should find the line Subject: New password.. Then on the administrator's notebook:

scp sysadm@192.168.56.1:msg ./
./inst pass < msg

The last command should complete without error.

Finally, the administrator verifies that dick can login on core, front and Nextcloud with the new password.

One more institute command is left to exercise. The administrator retires dick and his main device dick.

./inst old dick

The administrator tests Dick's access to core, front and Nextcloud, and attempts to access the campus VPN. All of these should fail.

The current network monitoring is rudimentary. It could use some love, like intrusion detection via Snort or similar. Services on Front are not monitored except that the webupdate script should be emailing sysadm whenever it cannot update Front (every 15 minutes!).

Pro-active monitoring might include notifying root of any vandalism corrected by Monkey's quarter-hourly web update. This is a non-trivial task that must ignore intentional changes.

Monkey's cron jobs on Core should be systemd.timer and .service units.

The institute's reverse domains (e.g. 86.177.10.in-addr.arpa) are not available on Front, yet.

The testing process described in the previous chapter is far from complete. Additional tests are needed.

The strategy pursued in The Hardware is two phase: prepare the servers on the Internet where additional packages are accessible, then connect them to the campus facilities (the private Ethernet switch, Wi-Fi AP, ISP), manually configure IP addresses (while the DHCP client silently fails), and avoid names until BIND9 is configured.

The strategy of Starting With Gate concentrates on configuring Gate's connection to the campus ISP in hope of allowing all to download additional packages. This seems to require manual configuration of Core or a standard rendezvous.

• Connect Gate to ISP, e.g. apartment WAN via Wi-Fi or Ethernet.

• Connect Gate to private Ethernet switch.

  sudo ip address add GATE dev ISPDEV
  

• Configure Gate to NAT from private Ethernet.

• Configure Gate to serve DHCP on Ethernet, temporarily!

  • Push default route through Gate, DNS from 8.8.8.8.

  Or statically configure Core with address, route, and name server.

  sudo ip address add CORE dev PRIVETH
  sudo ip route add default via GATE
  sudo sh -c 'echo "nameserver 8.8.8.8" >/etc/resolve.conf'
  

• Configure admin's notebook similarly?
• Test remote access from administrator's notebook.

• Finally, configure Gate and Core.

  ansible-playbook -l gate site.yml
  ansible-playbook -l core site.yml
  

A refinement of the current strategy might avoid the need to maintain (and test!) lists of "additional" packages. With Core and Gate and the admin's notebook all together on a café Wi-Fi, Ansible might be configured (e.g. tasks tagged) to just install the necessary packages. The administrator would put Core's and Gate's localnet IP addresses in Ansible's inventory file, then run just the Ansible tasks tagged base-install, leaving the new services in a decent (secure, innocuous, disabled) default state.

ansible-playbook -l core -t base-install site.yml
ansible-playbook -l gate -t base-install site.yml