04:15mupuf: mirai: you need to get verified before you can fork anything
04:15mupuf: https://gitlab.freedesktop.org/freedesktop/freedesktop/-/issues/new <-- select the user verification template and answer the questions
05:17mupuf: ha ha, gitlab's clock is 53 years in the future :D
05:18mupuf: oh, no, it's my laptop's clock that is :D
05:19mupuf: System clock synchronized: yes --> I doubt it :D
06:05mupuf: bentiss: may the force be with you!
06:05mupuf: Or whatever one can say to wish luck
06:16bentiss: mupuf: thanks!
06:17bentiss: and sigh... while I woke up: "but why don't I also tdo the full gitlab db migration while I said that there will be a downtime???"
06:36mupuf: Lol
06:37mupuf: Well, if the registry goes well, then maybe you'll have time for gitlab too
07:40bentiss: Shutting sown gitlab and gitalb-registry access now, and starting the migrations
07:40bentiss: *down
07:47daniels: bentiss: bonne chance!
07:48daniels: I’ll be around for most of the day if you need but only sporadically as I’m on trains
07:48bentiss: daniels: hopefully by tonight we will have gitlab 16.3 :)
07:48bentiss: daniels: no worries, let's hope I can cope with everything by myself
08:12bentiss: migrating the registry was too easy.... it's done now, though non functional given that gitlab.fd.o is deactivated
08:42mupuf: bentiss: crossing fingers
08:44bentiss: what worries me is that sometimes I lose connectivity between the 2 clusters on some nodes only, and this is going to be a problem for the hot migration of the git data
10:18karolherbst: "504 Gateway Time-out" :')
10:19mupuf: karolherbst: yep, gitlab is being relocated to another datacenter
10:19karolherbst: ahh
10:19karolherbst: but it's also Sunday and it's my fault for checking emails :D
10:23bentiss: TBH, I think I'll also migrate the gitaly data offline, this will be a slightly longer downtime, but it'll save me 2 weeks of monitoring the gitlay transfers.
10:29mupuf: bentiss: sounds fair
10:29mupuf: hope everything is going smoothly for you
10:30bentiss: just to be clear "slightly" in that a backup of gitaly-0 (which I just started and which is the longest) is roughly 8h, so probably same anmount of time to restore
10:31bentiss: I could reuse the backups from this morning, but that would drop the changes between 4 to 8 AM UTC. Not sure if there were a lot
10:34mupuf: bentiss: so... 16 more hours of work... doesn't seem workable
10:36bentiss: mupuf: it's fire and forget, so it should be fine
10:36mupuf: but you can;t start gitlab until it is done, right?
10:36bentiss: which is way better than me babysitting the connections between the two cluster very 2h
10:37bentiss: I can probably restore most of the data and then start gitlab
10:37bentiss: some repos will return 404 but that shouldn't impact the rest
10:38mupuf: Ack
10:46mupuf: bentiss: restarting Mesa CI will be fun
10:51MrCooper: bentiss: getting only 504s with git via https
10:52MrCooper: tried xorg/proto/xorgproto and xorg/xserver so far
11:30bentiss: MrCooper: yes, see above, In the end I ĺl do the full gitlab migration today I think
12:39bentiss: turns out that trying to restore the main db was a good idea, I should have deleted the postgres disk on the new server where I did my tests first
13:05kisak: morning, fwiw, I'm seeing 504 gateway time-outs over here when trying to get to gitlab.fd.o.
13:05daniels: kisak: yeah, see all the messages above
13:07kisak: Oh right, parrot brain forgot the upgrade notice I saw yesterday. Sorry for the noise.
13:07alatiera:sends warm coffee over to admins
13:26bentiss: oops... I actually tried to restore the db on the old cluster... which means this is completely failing... nothing to worry about FWIW
13:28alatiera: I am sure we all wouldn't mind if you accidentally wiped all the open issues 😆
14:47mupuf: 3 people so far reporting the 504 after 8h of outage. Sundays are good days for upgrades
14:55lazka: my CI is failing, but I learned to read the channel topic last time :)
14:56bentiss: damn... second time the db backup fails (with postgres just hanging)
14:56bentiss: I'll try to restore both the db on the new and old cluster this time
15:22just_a_user: hello, folks. gitlab.freedesktop.org is down?
15:23just_a_user: Who knows of any mirrors?
15:23kisak: See topic, scheduled upgrade in progress.
15:24just_a_user: thanks a lot
15:26bentiss: I think I found out why the db was not restoring properly: if someone accesses it while it's restoring, postgres just hangs... and this makes the restore fail
15:28bentiss: (and yes, I had one rogue gitlab-exporter pod around)
15:29bentiss: but that also means I can not restore gitaly while I restore the db, so I might not do the cold gitaly restoration and have to the the manual hot one :(
15:31Inky1003: What happened to freedesktop's gitlab?
15:31Inky1003: I needed to see wireplumber docs, but It seems It's down :(
15:32Inky1003: Oh sorry, I haven't saw the description
15:32Inky1003: nvm
15:40mupuf: bentiss: oh no, 2 weeks of pain then
15:41bentiss: mupuf: yeah... or I can try to do something otherwise, but right now it's that way :(
15:42bentiss: anyway, so far the restoration is working, and I'm past the couple of previous failures, so that's a good sign
15:43bentiss: mupuf: also TBH, gitaly-0 failed its last backup when I messed up the old db, so I don't have the latest shiniest data on those in a backup
15:50mupuf: bentiss: hopefully not much of value will be lost
15:51mupuf: And it's git: just re-push
18:58bentiss: the db restoration on the new cluster is still crashing. Rebooting one node that is showing issues. Meanwhile, the restoration on the old cluster is litterally taking ages
19:31mupuf: Crossing fingers! I'm off to sleep.
20:59DavidHeidelberg[m]: 🤞
21:06meghand: is there a web page explaining what the gitlab maintenance is doing?
21:13bentiss: meghand: no, sorry. long story short: things did not went as well as expected and it's still not over
21:28bentiss: alright status update: both db backup (new/old clusters) are still going on. The new db just passed ~50% in 2h20 min, when the old cluster one is ~60% done in 6h of time
21:29bentiss: I definitely be asleep when this finishes, so that means no gitlab on Monday morning for everyone (especially the .au guys)
21:41DavidHeidelberg[m]: bentiss: good night and sleep well :)
22:06karolherbst: thanks for all the great work and dealing with all of this. Hopefully things will go smoothly tomorrow
22:06emersion: ♥
22:07alyssa: ++
22:17daniels: bentiss: thankyou & sleep well!
23:16zondino: What is meant by “while”? It’s already been a while. I cannot get working desktop without Libinput, where I can download v-1.24.0?
23:19zondino: V-1.24.0?????
23:19whot: zondino: you can grab the fedora src.rpm which includes the source tarball https://kojipkgs.fedoraproject.org//packages/libinput/1.24.0/1.fc40/x86_64/ 23:21zondino: Ok going after
23:24meghand: also cgit is still up https://cgit.freedesktop.org/ 23:26zondino: Thanks @whot