04:15 mupuf: mirai: you need to get verified before you can fork anything
04:15 mupuf: https://gitlab.freedesktop.org/freedesktop/freedesktop/-/issues/new <-- select the user verification template and answer the questions
05:17 mupuf: ha ha, gitlab's clock is 53 years in the future :D
05:18 mupuf: oh, no, it's my laptop's clock that is :D
05:19 mupuf: System clock synchronized: yes --> I doubt it :D
06:05 mupuf: bentiss: may the force be with you!
06:05 mupuf: Or whatever one can say to wish luck
06:16 bentiss: mupuf: thanks!
06:17 bentiss: and sigh... while I woke up: "but why don't I also tdo the full gitlab db migration while I said that there will be a downtime???"
06:36 mupuf: Lol
06:37 mupuf: Well, if the registry goes well, then maybe you'll have time for gitlab too
07:40 bentiss: Shutting sown gitlab and gitalb-registry access now, and starting the migrations
07:40 bentiss: *down
07:47 daniels: bentiss: bonne chance!
07:48 daniels: I’ll be around for most of the day if you need but only sporadically as I’m on trains
07:48 bentiss: daniels: hopefully by tonight we will have gitlab 16.3 :)
07:48 bentiss: daniels: no worries, let's hope I can cope with everything by myself
08:12 bentiss: migrating the registry was too easy.... it's done now, though non functional given that gitlab.fd.o is deactivated
08:42 mupuf: bentiss: crossing fingers
08:44 bentiss: what worries me is that sometimes I lose connectivity between the 2 clusters on some nodes only, and this is going to be a problem for the hot migration of the git data
10:18 karolherbst: "504 Gateway Time-out" :')
10:19 mupuf: karolherbst: yep, gitlab is being relocated to another datacenter
10:19 karolherbst: ahh
10:19 karolherbst: but it's also Sunday and it's my fault for checking emails :D
10:23 bentiss: TBH, I think I'll also migrate the gitaly data offline, this will be a slightly longer downtime, but it'll save me 2 weeks of monitoring the gitlay transfers.
10:29 mupuf: bentiss: sounds fair
10:29 mupuf: hope everything is going smoothly for you
10:30 bentiss: just to be clear "slightly" in that a backup of gitaly-0 (which I just started and which is the longest) is roughly 8h, so probably same anmount of time to restore
10:31 bentiss: I could reuse the backups from this morning, but that would drop the changes between 4 to 8 AM UTC. Not sure if there were a lot
10:34 mupuf: bentiss: so... 16 more hours of work... doesn't seem workable
10:36 bentiss: mupuf: it's fire and forget, so it should be fine
10:36 mupuf: but you can;t start gitlab until it is done, right?
10:36 bentiss: which is way better than me babysitting the connections between the two cluster very 2h
10:37 bentiss: I can probably restore most of the data and then start gitlab
10:37 bentiss: some repos will return 404 but that shouldn't impact the rest
10:38 mupuf: Ack
10:46 mupuf: bentiss: restarting Mesa CI will be fun
10:51 MrCooper: bentiss: getting only 504s with git via https
10:52 MrCooper: tried xorg/proto/xorgproto and xorg/xserver so far
11:30 bentiss: MrCooper: yes, see above, In the end I ĺl do the full gitlab migration today I think
12:39 bentiss: turns out that trying to restore the main db was a good idea, I should have deleted the postgres disk on the new server where I did my tests first
13:05 kisak: morning, fwiw, I'm seeing 504 gateway time-outs over here when trying to get to gitlab.fd.o.
13:05 daniels: kisak: yeah, see all the messages above
13:07 kisak: Oh right, parrot brain forgot the upgrade notice I saw yesterday. Sorry for the noise.
13:07 alatiera:sends warm coffee over to admins
13:26 bentiss: oops... I actually tried to restore the db on the old cluster... which means this is completely failing... nothing to worry about FWIW
13:28 alatiera: I am sure we all wouldn't mind if you accidentally wiped all the open issues 😆
14:47 mupuf: 3 people so far reporting the 504 after 8h of outage. Sundays are good days for upgrades
14:55 lazka: my CI is failing, but I learned to read the channel topic last time :)
14:56 bentiss: damn... second time the db backup fails (with postgres just hanging)
14:56 bentiss: I'll try to restore both the db on the new and old cluster this time
15:22 just_a_user: hello, folks. gitlab.freedesktop.org is down?
15:23 just_a_user: Who knows of any mirrors?
15:23 kisak: See topic, scheduled upgrade in progress.
15:24 just_a_user: thanks a lot
15:26 bentiss: I think I found out why the db was not restoring properly: if someone accesses it while it's restoring, postgres just hangs... and this makes the restore fail
15:28 bentiss: (and yes, I had one rogue gitlab-exporter pod around)
15:29 bentiss: but that also means I can not restore gitaly while I restore the db, so I might not do the cold gitaly restoration and have to the the manual hot one :(
15:31 Inky1003: What happened to freedesktop's gitlab?
15:31 Inky1003: I needed to see wireplumber docs, but It seems It's down :(
15:32 Inky1003: Oh sorry, I haven't saw the description
15:32 Inky1003: nvm
15:40 mupuf: bentiss: oh no, 2 weeks of pain then
15:41 bentiss: mupuf: yeah... or I can try to do something otherwise, but right now it's that way :(
15:42 bentiss: anyway, so far the restoration is working, and I'm past the couple of previous failures, so that's a good sign
15:43 bentiss: mupuf: also TBH, gitaly-0 failed its last backup when I messed up the old db, so I don't have the latest shiniest data on those in a backup
15:50 mupuf: bentiss: hopefully not much of value will be lost
15:51 mupuf: And it's git: just re-push
18:58 bentiss: the db restoration on the new cluster is still crashing. Rebooting one node that is showing issues. Meanwhile, the restoration on the old cluster is litterally taking ages
19:31 mupuf: Crossing fingers! I'm off to sleep.
20:59 DavidHeidelberg[m]: 🤞
21:06 meghand: is there a web page explaining what the gitlab maintenance is doing?
21:13 bentiss: meghand: no, sorry. long story short: things did not went as well as expected and it's still not over
21:28 bentiss: alright status update: both db backup (new/old clusters) are still going on. The new db just passed ~50% in 2h20 min, when the old cluster one is ~60% done in 6h of time
21:29 bentiss: I definitely be asleep when this finishes, so that means no gitlab on Monday morning for everyone (especially the .au guys)
21:41 DavidHeidelberg[m]: bentiss: good night and sleep well :)
22:06 karolherbst: thanks for all the great work and dealing with all of this. Hopefully things will go smoothly tomorrow
22:06 emersion: ♥
22:07 alyssa: ++
22:17 daniels: bentiss: thankyou & sleep well!
23:16 zondino: What is meant by “while”? It’s already been a while. I cannot get working desktop without Libinput, where I can download v-1.24.0?
23:19 zondino: V-1.24.0?????
23:19 whot: zondino: you can grab the fedora src.rpm which includes the source tarball https://kojipkgs.fedoraproject.org//packages/libinput/1.24.0/1.fc40/x86_64/
23:21 zondino: Ok going after
23:24 meghand: also cgit is still up https://cgit.freedesktop.org/
23:26 zondino: Thanks @whot