Network setup of the test farm:

LAB network -> fd_farm USB ethernet -> fd-farm NUC -> fd-farm internal ethernet -> switch -> servo USBC ethernet

fd-farm (10.42.0.1 on internal network) runs:
- gitlab-runner for dragonboards
- dnsmasq run through network-manager for internal network DHCP
- nginx for caching fd.o accesses
- nfs server for db rootfses

servo (10.42.0.10 on internal network) runs:
- gitlab-runner for cheza
- servods
- dnsmasq as separate service for cheza tftpboot
- nfs server for cheza rootfs

Sysadmin notes:

- fd-farm is 100.107.2.169 on ipv4, and
  2401:fa00:481:4c03:96c6:91ff:feaf:453a/64 on ipv6.  Note that ipv6
  is not usable from cloudtopm so the ipv4 is usually more useful.
  Also, gitlab admin interface's ip addr for the runners is not this
  address, it's some google proxy.

- In the past we've had issues with running out of disk space on servo
  due to gitlab not cleaning its docker images.  fd.o's cleaning
  script doesn't work because our pyhthon is too old.  We should have
  docker-gc cleaning in a cron job, but maybe that's busted somehow?
  docker system prune -a is a big hammer to free space.

- servo runs kernel 4.9 because later ones have a USB bug with servos
  connected where USB will die occasionally.

- servo ID mapping from ~anholt/.servodrc:
cheza1,CMO653-00166-040491U02703,9981,cheza
cheza2,CMO653-00166-040491U01290,9982,cheza
cheza3,CMO653-00166-040491U02099,9983,cheza
cheza4,CMO653-00166-040491U00849,9984,cheza
cheza5,CMO653-00166-040489J03930,9985,cheza
cheza6,CMO653-00166-040489J03435,9986,cheza
cheza7,CMO653-00166-040491U03294,9987,cheza
cheza8,CMO653-00166-040491U02092,9988,cheza
cheza9,CMO653-00166-040491U01239,9989,cheza


Journal:

2021-08-30:

Post move from wells fargo, nginx failed to come up due to not binding
to the address.  chadv fixed it hopefully, with:

anholt@fd-farm:/home/anholt% cat /etc/systemd/system/nginx.service.d/10-custom.conf 
# See <https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/>.
# See man:systemd.special(7), entries network.target and network-online.target.

[Unit]
After=network-online.target
Wants=network-online.target

Additionally, post move Fritz found that servo's internal ethernet had
gone away, and added a USBC ethernet to replace it.  anholt adjusted
it to use servo's static IP.

As of today, offline boards are:
cheza1: boot loops looking for enough power to boot
cheza5: servo not probed
cheza7: servo silent
db410c-4 (relay power doesn't reach, no usb micro cable)
db410c on top of stack (9th, no usb micro cable, no relay power)
db820c-4

plugged usb cable back in for cheza5 and servo probed, but still
silent.  board power light was off.  swapping power cables into usbc
dongle didn't help, but swapping dongles did.  dongle presumed broken.
replaced with a HooToo and updated the MAC in fd-farm's dnsmasq, and
it seems to work.

cheza1's wasn't getting good power.  swapping cables occasionally
helped, but never fully stable.  swapped to an external power brick,
and it seems stable now.  probably flaky usbc ethernet, but we're
short on those.

cheza7 was also not getting power.  power port on USBC dongle full of
some horribly sticky goop.

conclusion is that we need two more ethernet/power dongles, for cheza7 plus a spare.