07:57bentiss: It's Friday morning and we have a gitlab security update pending... So how about I do it now?
08:13bentiss: FWIW, the release notes mention a minor bump of postgresql, so I'll also have to do it once the current gitlab migration ends
08:30daniels: bentiss: go for it - it’s Easter so should be quiet
08:31bentiss: hopefully :)
08:32bentiss: right now the gitlab migration is still running... hopefully a restart/upgrade of the db will help for the next one
09:49emersion: IPv6 is broken again
12:26__tim: have sen multiple issues now with merge request merges failing but the commits actually successfully having been added to the main branch fwiw
12:27__tim: manually pressing the merge button sorts it after it spins for a long time
12:28__tim: (example: https://gitlab.freedesktop.org/gstreamer/cerbero/-/merge_requests/1433 just in case you need something to look for in the logs)
12:59bentiss: __tim: the migration of the db didn't completely went through, so this is not overly surprising
12:59bentiss: re-lauching it ATM
13:01bentiss: sigh... main: == 20240212031520 SyncIndexForPCiBuildsPart3: migrated (12752.2315s) ========== that one took a little bit longer than the 1h timeout
16:23alanc: Hopefully we don't have sshd installed in too many CI containers, because I know we have xz-utils installled in many of them (for "make distcheck" or "meson dist") and at least some are built using Debian testing or unstable: https://www.openwall.com/lists/oss-security/2024/03/29/4
16:25mingdao: Does your sshd link to lzma?
16:25mingdao: Ours doesn't.
16:26mingdao: xz-utils needs to be downgraded, probably.
16:31alanc: apparently that depends on whether patches have been applied for systemd support in sshd
16:32alanc: I don't have access to scan the CI containers to see which do or don't - that's a challenge for the gitlab admins
16:52bentiss: alanc: our runners are pretty much trashable anytime, and are gate-kept by runner_gating.sh. So if we get attacked through this way, that mean we put too much trust in one person and can probably talk to them
16:53bentiss: also you don't need such backdoor to gain access to them, it's fairly easy to escape the container jail given that they ar privileged
16:53bentiss: but thanks for the heads up!
16:53bentiss: side note: the migration still didn't complete, so that's a tad worrying
16:54bentiss:now restarts the db
16:56bentiss:now re-re-restart the db migration for gitrlab 16.10.1
16:57alanc: I wasn't worried so much about our users doing bad things as much as someone external being able to break in that way, but I suppose they'd also have to be lucky enough to hit the small windows when each container is running, since for most projects they only run for a few minutes (I know we have some massive outliers like Mesa though)(
16:58bentiss: alanc: but that bug shouldn't affect the host, only the container, and you can not directly ssh into a container AFAICT
16:58bentiss: (because we already use port 22 on the host, so the container can not bind to it)
16:59alanc: oh right, hadn't thought about the port already being bound
16:59bentiss: \o/ the db migration now seems to be going further after the restart
17:01bentiss: \o/ \o/ migration almost over
17:01bentiss: \o/ \o/ \o/ migration done, now we need for webservice to kick in before we enter the era of gitlab 16.10
17:02bentiss: alanc: also the containers are using a NAT behind the host, I don't think we use the host network
17:03bentiss: and there we go. Gitlab 16.10.1 deployed (finally)
17:07alanc: and I know all the CI containers I'd worked on don't have sshd, but I thought I'd seen some way to set it up so you could login to debug stuff, but perhaps I misremembered that
17:08bentiss: well, there was a remote service which was doing reverse ssh tunneling that I disabled by redirecting it to 127.0.0.1, but otherwise you probably need some host configuration we don't have
17:20__tim: bentiss, nice, seems snappier too, but maybe that's just the public holiday
17:20bentiss: __tim: could also be the db that was stuck in limbos, because the migration was hitting some locks that couldn't be released