03:48 uajain: Where do I contact gitlab.freedesktop.org admin staff?
03:53 airlied: here
05:18 uajain: airlied: Thanks, I want to request account deletion since I have lost all means of reviving it again
05:19 airlied: maybe just put the account name here and see if bentiss can look
05:20 uajain: Yes, here it is : https://gitlab.freedesktop.org/uajain
06:44 bentiss: uajain: happy to help (though it's an unusual request I must say). Would you mind changing your profile Skype (or LinkedIn or Twitter) name to "delete_me" so I'm sure the uajain here is the same uajain on gitlab.fd.o?
07:00 bentiss: FTR, (and the other admins) I fixed uajain account, so no deletion is required
07:00 uajain: Thank you for the quick help!
09:02 bentiss: When I'm going to put the cluster down, progress can be seen at https://gitlab.com/bentiss/fdo/-/issues/1
10:01 bentiss: putting the cluster down now
10:17 DavidHeidelberg: oh poor cluster. At least he had nice life.
10:18 bentiss: :)
10:34 bentiss: alright, that should be it!
10:40 bentiss: looks like pages is having issues
10:40 mvlad: oh, oauth is also having some issues. certificate verify failed (hostname mismatch
10:41 vsyrjala: i get a 404 from https://gitlab.freedesktop.org/drm/intel/uploads/abb6c4e2dd13b39a3cba3124b63bec96/logs.zip, linked from https://gitlab.freedesktop.org/drm/intel/-/issues/9347#note_2136931
10:42 bentiss: vsyrjala: yeah, I think all os S3 storage is not correctly configured
10:42 bentiss: mvlad: will look into it in a bit
10:43 emersion: pushing fails with:
10:43 emersion: remote: ERROR: A repository for this project does not exist yet.
10:45 bentiss: emersion: damn
10:45 bentiss: emersion: which project?
10:45 emersion: emersion/libicc
10:45 bentiss: k, thanks
10:45 emersion: was created a few days ago
10:52 bentiss: WTF, we are missing all commits from the past couple of days
10:57 bentiss: emersion: if you just push it again, will this work?
10:57 emersion: nope
10:58 emersion: it seems like the repo isn't even in the DB
10:58 bentiss: yeah, which is utterly wrong
10:59 emersion: so repo data + DB seems to be a snapshot from a few days ago
10:59 bentiss: because I can see changes from today in the db
10:59 dj-death: what's the expectations for the gitlab instance atm?
10:59 dj-death: you can't do anything with it?
10:59 emersion: dj-death: i wouldn't recommend writing to it
10:59 dj-death: alright, thanks
10:59 bentiss: dj-death: working mostly, thought there are some data from the past couple of days missing
11:00 dj-death: I just tried to post a comment but apparently it's broken
11:00 dj-death: it won't let me comment, something aboutt line code being blank
11:01 bentiss: if it's mesa, yes, I'm trying to restore back the git history ATM
11:03 emersion: maybe the config points to the wrong DB or something?
11:03 emersion: how was the DB migrated to the new cluster?
11:04 bentiss: emersion: with a standby hot replication
11:06 bentiss: could it be redis cache?
11:06 bentiss: trying this
11:14 bentiss: so I'm actually missing some data on the gitaly pods themselves
11:14 bentiss: and now I've messed up https://cgit.freedesktop.org/mesa/mesa too
11:29 ishitatsuyuki: I can't sign in. Says 'Could not authenticate you from GoogleOauth2 because "Ssl connect returned=1 errno=0 peeraddr=18.154.227.36:443 state=error: certificate verify failed (hostname mismatch)". '
11:30 emersion: yeah, known issue
11:30 bentiss: ishitatsuyuki: yeah, known issue, see above (but I have a couple of more urgent ones first on my plate)
11:30 ishitatsuyuki: oh, missed that message. thanks
11:55 shadeslayer: OOI have you tried using cloudflare to lower gitlab egress costs?
12:10 pq: I suppose not being able to log in makes the Gitlab frontend effectively read-only for everyone, so no further work can be lost in the mean time?
12:12 emersion: login works, external login doesn't
12:12 emersion: it would be a good idea to put everything in readonly, though
12:12 emersion: https://docs.gitlab.com/ee/administration/read_only_gitlab.html
12:13 mupuf: bentiss: thoughts on the readonly proposal?
12:15 emersion: (if the issue can't be figured out, it would be nice to roll back to the old cluster, and continue the investigation on the new one)
12:15 emersion: (let me know if there's anything i can do to help -- but this is all dark magic to me…)
12:17 bentiss: emersion: the problem of rolling back is that we are going to lose git data since today's backup, which is why I wanted to do it as close as possible from the last backup fetch
12:18 emersion: you mean git data since we switched to the new cluster?
12:18 emersion: or more than that?
12:18 bentiss: git data since this morning (backup was on the old cluster), and it also requires 24h of downtime
12:19 bentiss: I've restored mesa
12:19 bentiss: FWIW
12:19 emersion: but 1 week worth of issues, comments, MRs, git, are lost
12:19 bentiss: 1 week???
12:19 emersion: maybe a bit less
12:20 __tim: gstreamer has the same issue (and my private forks)
12:20 bentiss: emersion: you are talking about your libicc project?
12:20 emersion: i'm talking about all projects
12:20 karolherbst: I'm seeing comments from 3 hours ago
12:20 bentiss: from what I can see the db is fine, the git data not so much
12:20 karolherbst: so it's not all projects
12:20 emersion: karolherbst: but are you seeing comments from yesterday?
12:21 karolherbst: yes
12:21 karolherbst: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25837?commit_id=5af81d42a8000aa2514db0c22bcd727502070c01#note_2136751
12:21 bentiss: yeah, comments, issues, are good, unless you are on a "recent" project (less than 48h)
12:21 emersion: 3h would be after migration
12:21 karolherbst: that MR just doesn't have any commits apparently ...
12:21 emersion: hm, indeed, it seems like issues/comments/etc from yesterday were not lost on another project
12:22 bentiss: emersion: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25847 -> created 11h ago, updated 6h ago
12:22 karolherbst: heh.. but changes are listed
12:22 karolherbst: funky
12:22 karolherbst: just not the content
12:22 mupuf: inconsistencies are the worst
12:22 karolherbst: well.. in that case I can just push again later I guess
12:23 karolherbst: git content lost is like.. yeah whatever, but comments and updates on MRs/issues are kinida important
12:23 emersion: okay, my bad, i thought the issue was much worse
12:23 bentiss: karolherbst: feel free to push it once again
12:23 karolherbst: could also be that just some random time slices are gone
12:23 mupuf: I have a lot of examples of commits missing, if you want data
12:23 karolherbst: remote: error: cannot lock ref 'refs/heads/rusticl/zink/fixes/mr_1': 'refs/heads/rusticl/zink' exists; cannot create 'refs/heads/rusticl/zink/fixes/mr_1'
12:24 karolherbst: To https://gitlab.freedesktop.org/karolherbst/mesa.git
12:24 karolherbst: ! [remote rejected] rusticl/zink/fixes/mr_1 -> rusticl/zink/fixes/mr_1 (failed to update ref)
12:24 karolherbst: error: failed to push some refs to 'https://gitlab.freedesktop.org/karolherbst/mesa.git'
12:24 bentiss: AFAICT, ceph told me the 2 images were mirrored, but maybe it wasn't real-time, and we got the snapshot from yesterday
12:25 karolherbst: bentiss: wanna look into that, or should I try removing the ref?
12:25 bentiss: karolherbst: thinking very hard ATM
12:25 karolherbst: btw I was able to push to another branch an hour ago
12:26 karolherbst: just that one ref doesn't seem to exist anymore
12:26 bentiss: karolherbst: k, then please do not touch it, I'll see what I can do
12:26 karolherbst: okay
12:26 karolherbst: I've created that MR and branch like 2-3 days ago
12:29 karolherbst: branch from that MR is affected the same way (didn't try to push, but the branch is gone) https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25844
12:30 karolherbst: the most recently updated branch before the migration was pushed "Oct 18, 2023 7:10pm GMT+0200"
12:31 bentiss: that would match when I did the copy of the image
12:31 bentiss: the initial one
12:31 kxkamil: igt-gpu-tools is back to Oct 19th
12:31 bentiss: yeah, Thu
12:34 daniels: karolherbst: that's telling you that you have a branch called rusticl/zink, so you can't create one called rusticl/zink/fixes/mr_1
12:34 karolherbst: mhhhhhhhhhh
12:34 daniels: nothing to do with gitlab, that's just git itself
12:34 karolherbst: right.. but I've deleted that one
12:34 daniels: seemingly not :P
12:34 karolherbst: hey, rewriting history is against the rules
12:34 karolherbst: *the past
12:35 karolherbst: but that would explain why those two branches are kinda lost
12:35 karolherbst: mhh well.. not really. just that git was reset to some older state I guess
12:36 karolherbst: bentiss: anyway yeah, I think this is just my fork having an old state and I have to delete some old refs
12:36 karolherbst: if you are okay with it, I remove the old ref, push the two new ones and see if that fixes my MRs, unless you want to repair more forks or something
12:38 bentiss: karolherbst: if you can, then yes please
12:38 bentiss: karolherbst: I can also fetch the backup data for you if you want
12:39 karolherbst: nah, that's all I have been working on anyway
12:39 karolherbst: okay, so that seems to have worked
12:39 karolherbst: and the MRs are fine as well
12:39 karolherbst: there was one odd error from git though:
12:39 karolherbst: - [deleted] rusticl/zink
12:39 karolherbst: error: cannot lock ref 'refs/remotes/karolherbst/rusticl/zink': 'refs/remotes/karolherbst/rusticl/zink/fixes/mr_1' exists; cannot create 'refs/remotes/karolherbst/rusticl/zink'
12:40 karolherbst: `Karol Herbst restored source branch rusticl/zink/fixes/mr_1 1 minute ago` funky
12:41 bentiss: karolherbst: all you did was a git push -f?
12:42 karolherbst: can somebody answer here that "you removed the branch" is not correct and that the contributor should repush the branch? https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25848#note_2137617 thanks
12:42 karolherbst: bentiss: yeah
12:42 karolherbst: pushing rusticl/zink/fixes/mr_1 didn't report any errors after that though
12:43 mupuf: bentiss: if you want examples of broken trees: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25793/commits or https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25843/commits
12:43 karolherbst: maybe that error happened because there is an MR referencing that? no idea :)
12:43 kxkamil: should I wait for some time for restore?
12:43 mupuf: pushed 16h ago, and 2 days ago
12:43 karolherbst: yeah.. so it looks like git pushes in the past ~3 days are kinda lost
12:44 mupuf: *some* pushes
12:44 karolherbst: yeah, fair
12:44 mupuf: bentiss: is it possible to re-synchronize from the old server? I guess not...
12:45 kxkamil: or rather - when it will be safe to push?
12:45 bentiss: mupuf: yeah, I managed to restore mesa/main thanks to marge I think, we can consider anything that was pushed in the past couple of days "lost" as in everybody should re-push
12:45 karolherbst: slap an admin message onto gitlab and tell devs to repush their branches if they find broken MRs 🙃
12:45 mupuf: +1 to that
12:45 karolherbst: if some git branches is the only thing lost I think that's fine
12:46 bentiss: mupuf: that's the problem, once I switched, I couldn't manage to make both copies up at the same time, I had to disable the one on the old server
12:46 karolherbst: I highlyt doubt there is anybody who wouldn't have the work locally anymore
12:46 bentiss: and disabling deletes the data :(
12:46 mupuf: bentiss: Right, I saw your sad smiley
12:46 mupuf: well, then so be it
12:46 mupuf: #goodenough
12:47 mupuf: and next time, we'll know that we'll need to do a cold migration
12:47 bentiss: I can also cherry pick in the backups, and I'm going to do that for emersion and other repos created in the past few days
12:47 bentiss: they are a little bit more of an issue because the repo doesn't exists on gitaly, and so it fails, really hard
12:47 emersion: bentiss: don't bother, i can re-create
12:48 karolherbst: any ETA on enabling external auth? Becuase that's the only way I can login.. otherwise somebody really should comment on MR 25848 so the contributor doesn't do something silly
12:48 bentiss: mupuf: this was a couple of hours of migration compared to a coupleof week or a month of monitoring and hand picking repos... I'll take the "too bad, it failed for the past 2 days
12:49 bentiss: emersion: k, I'll try to work on the others
12:49 emersion: ty for your work bentiss
12:49 mupuf: bentiss: right, fair enough!
12:49 emersion: next time, i should maybe try to watch over your shoulder so that i can learn
12:49 mupuf: thanks a lot, datacenter migrations suck!
12:49 emersion: if you'd be open to that
12:49 bentiss: mupuf: but next time I'll 1. put down gitlab, 2. do the snapshot migration, 3. change the main
12:49 bentiss: (without counting on mirroring actually happening)
12:50 bentiss: emersion: sure (hopefully not soon)
12:50 mupuf: I guess all we can do is pre-cache data in the hope of speeding up migration, but indeed, we probably should just have gitlab down while this is happening
12:50 mupuf: https://docs.mesa3d.org/drivers/radv.html <-- seems like there are certificate issues here
12:51 bentiss: mupuf: yeah, pages is not in a good shape
12:51 bentiss: on both cluster
12:51 bentiss: I was trying to debug that before I got notified about the missing git history
12:53 kxkamil: I got Friday changes and few moments ago git tells I am ahead of remote ...
12:53 karolherbst: yeah, assume all changes to your fork from the last 3-4 days are gone
12:53 kxkamil: any hints? should I wait for a day?
12:53 karolherbst: just repush
12:53 bentiss: kxkamil: gitp ush -f
12:55 mupuf: bentiss: I'm writing a message to say the migration is done but that some branches/projects were lost in the process
12:55 mupuf: I;ll ask people to re-push, and if they can't contacting us to look for it in our backups
12:56 mupuf: sounds good? I'll send the message here when I am ready
12:57 bentiss: mupuf: sounds good
12:57 bentiss: thanks
12:57 vyivel: my avatar seems to be gone
12:57 kxkamil: ok, is it safe to push -f now?
12:57 kxkamil: better ti be safe ...
12:58 karolherbst: I already done it on my repo and it fixed the MRs I have opened against mesa. So maybe you should try one and see if it works and report back if something breaks
12:59 ndufresne: We have some broken MR, were gitlab thinks there is no changes, https://gitlab.freedesktop.org/gstreamer/gstreamer/-/merge_requests/4549/diffs#note_2137223
12:59 Wallbraker: Yeah I'm also seeing broken MRs
12:59 karolherbst: repush the branches
12:59 ndufresne: easy to say when you have handful changes, but having to do that on behavl of few hundreds submitters, that a lot of work
13:00 karolherbst: tldr: data migration didn't went well, git history lost for 3-4 days
13:00 karolherbst: yeah.. if it's a lot of work, then ask bentiss once the message is out
13:00 karolherbst: but it only seem to affect pushes from the last 3-4 days
13:00 vyivel: https://gitlab.freedesktop.org/vyivel/glhf (which is a fork) is empty, trying to fork the original project again leads to the same link; is it safe to delete it and re-fork?
13:00 bentiss: karolherbst: what's teh omniauth error BTW
13:01 karolherbst: https cert stuff
13:01 karolherbst: Could not authenticate you from GitHub because "Ssl connect returned=1 errno=0 peeraddr=18.154.227.36:443 state=error: certificate verify failed (hostname mismatch)".
13:03 bentiss: huh. don't know which cert is an issue here :)
13:04 emersion: ah, the "no repository" page is actually only displayed when the Git data is missing, but the repo exists in the DB
13:04 bentiss: emersion: yeah
13:05 bentiss: so only for the past couple of days of new repos, I should be able to do that by hand
13:05 karolherbst: bentiss: maybe ours, because pages also causes https issues
13:05 karolherbst: *runs into
13:05 bentiss: karolherbst: pages is slightly different
13:05 karolherbst: oh noes
13:05 bentiss: some pages are pointing at the new cluster, some still to the old
13:05 karolherbst: ahh
13:06 karolherbst: tried to open the "callback" url for the oauth stuff?
13:06 bentiss: can't remember where to find that :)
13:06 karolherbst: :)
13:07 mupuf: "The migration is now over... but some git branches and new projects were lost in the process! Feel free to force push your local version to fix the merge requests. If your entire repo disapeared, please contact us on OTFC's #freedesktop channel."
13:07 kxkamil: pipelines cannot work, err 503
13:07 bentiss: mupuf: sounds good, thanks!
13:07 mupuf: ok, it's live!
13:07 mupuf: let's refine it as time goes
13:08 vyivel: typo in "disapeared"
13:09 mupuf: Thanks, will fix
13:09 vyivel: also vyivel/glhf is one of the repos that is gone
13:09 mupuf: vyivel: fixed
13:11 bentiss: ooh... *maybe* omniauth is related to IPv6
13:11 bentiss: actually no, shouldn't impact it
13:13 Wallbraker: Our CI is borked as well https://gitlab.freedesktop.org/monado/monado/-/jobs/50632970
13:20 ndufresne: Same here, "Error: authenticating creds for "registry.freedesktop.org": pinging container registry registry.freedesktop.org: invalid status code from registry 503 (Service Unavailable)"
13:21 vyivel: "This project was scheduled for deletion, but failed with the following message: Failed to open TCP connection to gitlab-prod-registry.gitlab.svc:5000 (Connection refused - connect(2) for "gitlab-prod-registry.gitlab.svc" port 5000) "
13:22 vyivel: which i presume is the same error as the message above
13:22 bentiss: yeah, saw that registry is broken. Give me a minute
13:26 ndufresne: you can have more if you need ;-P (specially if it means a longer holding fix)
13:31 alexander: Hello, is there a ETA on when third-party logins will work again?
13:31 bentiss: registry should be fixed
13:31 bentiss: https://gitlab.freedesktop.org/monado/monado/-/jobs/50633763 worked at least
13:32 bentiss: alexander: no, not yet, not sure what's the issue
13:32 bentiss: and I need to grab something to eat, bbiab
13:33 alexander: Ok, thanks! Its just that my mesa MR is now properly in need of a rebase, and I cant do it atm since my SSH key for my laptop is not registered :meme:
13:33 Venemo: hey guys
13:33 Venemo: it looks like gitlab absolutely destroyed my MR
13:34 ndufresne: thanks bentiss, worked for me too
13:34 Venemo: it seems like all (?) branches in my private repo were lost
13:35 slomo: bentiss: do you know who has access to the equinix runners? maybe they have a commit in their cache that apparently nobody else has :)
13:36 slomo: i mean, i have the change... but if i inject that myself the commit id changes and that might make someone unhappy
13:36 Venemo: are the contents of the branches actually lost or is it just the MR UI that is confused?
13:37 slomo: Venemo: lost, yes. if it's your MR, just push it again
13:37 Venemo: next time can we get some advance warning about possibly losing our work? I imagine some people might have had some work in branches in a private repo which are not backed up elsewhere
13:38 kxkamil: few pipes failed, for example: https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/jobs/50634078
13:38 Venemo: in fact I might have had some of those
13:38 Venemo: fortunately nothing irreplaceble, but still feels bad to lose things without warning
13:38 i509vcb: Seems that the GitHub SSO login is erroring out?
13:39 DragoonAethis: Venemo: Look at the bright side, now you can go back in time and make those MRs perfect on the first try ;D
13:39 Venemo: sorry but I'm not laughing
13:40 i509vcb: GitHub SSO error is: Ssl connect returned=1 errno=0 peeraddr=18.154.227.126:443 state=error: certificate verify failed (hostname mismatch)
13:40 nirbheek_: Data loss is uh yeah
13:41 mupuf: Venemo: there are backups, just not for the past 2 days
13:42 alexander: Also, somewhat unrelated, but when is it acceptable to bump a MR? Im not sure how long I should wait before mentioning it, especially since its quite simple...
13:42 Venemo: mupuf: I guess it is fortunate that I haven't done too much in the past 2 days then, but still...
13:42 mupuf: Atomic updates with TB of data is hard. It was supposed to be transparent but it failed
13:43 slomo: bentiss: nevermind, we found it. ironically on a github mirror of the repo
13:43 mupuf: And knowing it failed required switching to the new data required removing the old one
13:44 mupuf: It is a shitty situation, but migrating TBs of data between multiple datacenters is slooooooow
13:44 vsyrjala: Venemo: you don't have that private repo as a remote in your local git repo?
13:44 mupuf: Took a month for the first sync
13:44 Venemo: vsyrjala: yes I have it, which is why I'm saying I'm fortunate
13:45 emersion: if you think you lost data, it can probably be recovered from the backup
13:45 Venemo: fortunately, it seems I didn't lose anything important, just had one MR messed up I think
13:46 Venemo: or maybe more than one, I would have to check
13:46 Venemo: yeah thus far 3 borked MRs
13:47 Venemo: also gitlab seems to be super slow now
13:48 Venemo: actually, 5 borked MRs
13:49 Venemo: I'm honestly not sure what to say. I don't understand why somebody feels that it's OK to mess up other people's work in this manner
13:50 bentiss: Venemo: I can't let you say that there were no warning. I put an un-dismissable banner saying that a migration will happen today. And I would have expected people to realize that it might come at a cost
13:50 emersion: Venemo: do you think people are infailible?
13:51 emersion: complaining is not constructive
13:51 bentiss: and now the situation is shit: only a couple of days is lost, which means that restoring everything from the last backup will take ~8-12 hours, for a little gain
13:52 MrCooper: Venemo: what makes you think anybody considers it OK?
13:52 bentiss: Venemo: and regarding "I don't understand why somebody feels that it's OK to mess up other people's work in this manner", the admin club is not closed. If you think you can do better, please join
13:52 Venemo: well, sorry but should I apologize to you for you deleting my branches?
13:53 MrCooper: no, just for your inappropriate attitude
13:53 Venemo: why is it inappropriate to ask questions here?
13:53 Consolatis: Venemo: this is not the right time for this. A proper post mortem of the whole situation would be interesting though (once things settle down)
13:54 bentiss: Consolatis: thanks. And yeah, if I spend time to speak here, I'm not restoring the service
13:54 MrCooper: this isn't how one "just asks questions"
13:54 emersion: it is inappropriate to make personal attacks like this
13:54 Venemo: I don't think I spoke to anyone here personally, I just came here to understand what happened
13:55 Venemo: I do remember seeing a notice about a migration, but it didn't mention that it will delete the contents from my MRs, hence I didn't expect that to happen, is all
13:55 emersion: nobody knew that this would happen
13:55 Venemo: now I come here to ask what's up and everyone's at my throat for that
13:55 Venemo: weird
13:55 MrCooper: you didn't even bother reading scrollback before you started complaining
13:56 emersion: you're aggressive, and you're wondering why people don't like your attitude?
13:56 Venemo: I'm not sure how to respond to that, so maybe better if I just don't
13:59 Consolatis: Venemo: Suggest just accepting things as they are for now. Complaining doesn't make it better, it just steals time which could be spend to fix things instead. In a few days when everything is settled these kind of questions can be discussed way more relaxed from all sides.
13:59 Venemo: understood
13:59 yann-kaelig: Hello
14:00 yann-kaelig: Trying to reset my password on https://gitlab.freedesktop.org I get a "500 We're sorry. Something went wrong on our end."
14:04 gawin: hi, nouveau wiki seems offline, not sure if due migration or something else earlier.
14:04 gawin: For example https://nouveau.freedesktop.org/CodeNames.html
14:04 bentiss: gawin: yeah, gitlab pages is broken, looking into it right now
14:05 bentiss: yann-kaelig: yeah, external authentication is broken
14:05 gawin: Thanks, nothing important on my side, just noticed.
14:05 yann-kaelig: ok, thanks
14:07 yann-kaelig: Only for my curiosity. The migration will fix the spam gitlab issue ?
14:08 emersion: no
14:08 yann-kaelig: good
14:08 yann-kaelig: ^^
14:08 yann-kaelig: Is it an issue that can be fixed in gitlab ?
14:17 mupuf: yann-kaelig: maybe, but never fully.
14:20 mupuf: bentiss: after discussing with Venemo, I'm proposing to update the banner to: "The migration is now over... but some git branches and new projects were lost in the process! If at all possible, force-push your branches to restore broken merge requests, or branches. If your entire repo disappeared, or you can't find the HEAD of a branch, please notify us on OTFC's #freedesktop channel and we'll try to address it as soon as possible. Sorry for the
14:20 mupuf: inconvenience..."
14:21 bentiss: mupuf: thanks!
14:21 mupuf: updated
14:22 mupuf: I'll leave it on for a week
14:23 alexander: I can't currently check my branch, since im not at home and can not log in here, but im assuming if the page for a MRs commits says "There are no commits yet." (https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25818/commits) then I will need to force push it?
14:24 sima: that seems to be the solution for missing git data
14:24 sima: you might need to delete some older branches if you have a name conflict
14:24 alexander: Its my first MR and branch so I dont think that will be a problem
14:24 alexander: Thanks!
14:25 pobrn: hi, I cannot log in with github: >>Could not authenticate you from GitHub because "Ssl connect returned=1 errno=0 peeraddr=18.154.227.36:443 state=error: certificate verify failed (hostname mismatch)".<< is this issue known?
14:25 emersion: yes
14:28 __tim: I found up to date commits in /srv/anongit.freedesktop.org/projects/ on annarchy btw, in case it helps anyone
14:30 pobrn: ok sorry, I didn't see it mentioned on https://gitlab.freedesktop.org/freedesktop/freedesktop/-/issues/966
14:31 yann-kaelig: I was using github to sign-in to gitlab.freedesktop.org. I
14:33 yann-kaelig: I'm not sure I created an gitlab.freedesktop.org and I deleted my github account. Can I recover my user used in gitlab.freedesktop.org ?
14:34 yann-kaelig: The thing is that I can not register if I want with the same username.
14:35 pobrn: yann-kaelig: have you tried "forgotten password"?
14:35 yann-kaelig: yes and I get a "500 We're sorry. Something went wrong on our end."
14:36 pobrn: yann-kaelig: well, me too when I want to log in, so I'm guessing there are some issues with gitlab atm still
14:37 yann-kaelig: ok, wil lsee but I'm curious right now, if now username has been registered and I can not use the same username, it's look like something could be broken with external authentification.
14:37 yann-kaelig: now/no
14:38 daniels: yeah, external authentication is broken, and will be fixed at some point, but right now there is no ETA
14:39 pobrn: daniels: I can't even log in using non-external auth
14:40 yann-kaelig: my bad, maybe I was not clear. I used github to authenticate on gitlab, but my github account has been deleted. Can I recover the username ?
14:40 yann-kaelig: on gitlab
14:41 yann-kaelig: or it's lost and I will have to create a new one
14:43 yann-kaelig: This is a deletion on github on my own initiative since it requires double authentication asking for my phone number
14:44 yann-kaelig: Not going to give my phon number for an account only used to report issues.
14:54 bentiss: yann-kaelig: I can quickly disable teh external auth for you (in theory)
14:55 bentiss: yann-kaelig: the link to github is deleted, maybe try re-asking a password
14:57 Consolatis: yann-kaelig: although kind of off-topic: GH doesn't require a phone number. You can use any random TOTP application (on your desktop or phone) for the 2nd factor.
14:58 bentiss: FWIW, gitlab-pages should be fixed except for www.networkmanager.dev. Not sure why the HTTP01 challenges fails
14:59 dmorin: I'm also experimenting authentication failure using GoogleOauth2. Error reported is certificate verify failed (hostname mismatch)
15:00 bentiss: dmorin: yeah same issue with GH or gitlab.com, still unsure what is happening
15:00 bentiss: our certs are correct in theory ;/
15:03 pobrn: bentiss: any idea why I wouldn't be able to log in using local credentials? my account was created through github, but I set a password on fdo gitlab proper a couple days ago
15:03 yann-kaelig: Consolatis: Well, that was not explained or maybe not clear enough and hidden somewhere, but I remembe, however, that the news announced a mandatory double authentication linked to a phone number
15:04 bentiss: pobrn: I don't know, no, sorry
15:06 yann-kaelig: bentiss: still the same issue.
15:11 bentiss: I think I found out why we were having 500 errors -> I had too many webservice pods on the old cluster
15:13 bentiss: could also be something wrong with ml-29
15:40 bentiss: now the registry is fixed...
15:46 Mario_: Hi; I've noticed that I can't authenticate using Github anymore to gitlab.freedesktop.org.
15:46 Mario_: Could not authenticate you from GitHub because "Ssl connect returned=1 errno=0 peeraddr=18.154.227.54:443 state=error: certificate verify failed (hostname mismatch)".
15:47 emersion: known issue
15:47 Mario_: OK thanks
15:49 MrCooper: not using external auth, so I can log in, keep getting logged out though FWIW
15:54 bentiss: MrCooper: are you using a consistent IP or is it flickering?
15:55 bentiss: MrCooper: if the IPv4 points at the old cluster when IPv6 is available on the new, chances are that the redis cache is not happy and logs you out
16:03 MrCooper: you mean the IP used by Firefox? Not sure
16:11 bentiss: MrCooper: works here. When you say "keep getting logged out", is like you can not login or is it after a couple of minutes?
16:12 bentiss: emersion: to be able to restore libicc, did you just git push -f, or have you done something on the UI too?
16:15 emersion: bentiss: "init empty repo" in the UI, then push
16:16 bentiss: emersion: k, thanks, much easier than me initializing the repo in the gitaly pod ;) thanks!
16:26 MrCooper: bentiss: after some time
16:44 nanokatze: hey, when trying to log into my fdo gitlab account with my gitlab.com account, I get "Ssl connect returned=1 errno=0 peeraddr=18.154.227.54:443 state=error: sslv3 alert handshake failure", is this known?
16:45 emersion: known issue
16:46 nanokatze: thanks
16:48 Consolatis: might be worth adding to the banner (or a new banner) that there are currently issues with 3rd party logins and that its being worked on
16:50 bentiss: mupuf: ^
16:50 bentiss: Consolatis: good point!
17:07 bentiss: alright. Manually restored all recently created repository to a working state (to their upstream counterpart), So now, we should only ask people to re-push their changes
17:10 bentiss: OK... so if I use the old cluster IP address, oauth works.
17:11 bentiss: now I "just" need to compare the 2 kubernetes states :/
17:21 anholt: is "some git pushes appear to have been lost in the git trees but the web ui knows they happened" known?
17:22 bentiss: anholt: yes, the db is fine, but the git data no. So just re-pushing the git tree is sufficient to restore teh MRs and whatnot
17:37 koike: is "yaml invalid Undefined error (01HDERKVBND9T3VHHMY3492SJQ) " known ? I pushed the same commit hash to two branches, one of them was valid and the other invalid
18:01 paulosv: Hi, after the gitlab migration, I am having some issues logging in and registering... registering gives 500 after submiting the form, and logging in via Gitlab.com throws an error "Could not authenticate you from GitLab because 'Ssl connect returned=1 errno=0 peeraddr=18.154.227.126:443 state=error: sslv3 alert handshake failure".' Are you aware of this / where can I report this?
18:02 emersion: yeah, that's a known issue
18:02 paulosv: ok, thx
18:04 paulosv: and do you know whether at least Gmail login works now? :D
18:05 bentiss: hmm... "Error during Akismet spam check, flagging as not spam: SSL_connect returned=1" that seems to be the same
18:09 bentiss: Haha! running on a webservice pod "curl https://google.com" returns an SSL error
18:09 emersion: bad CA store?
18:10 bentiss: and the same works on the old cluster
18:10 bentiss: so yes, bad CA store.... but why???
18:14 bentiss: emersion: do you know off hand how to fix this? it seems /etc/ssl/certs is correctly populated on the pod
18:15 bentiss: on the pod: dpkg -l ca-certificates -> ii ca-certificates 20210119 all Common CA certificates
18:16 bentiss: so it's installed
18:20 bentiss: Ahhh, on the old cluster /etc/pki/ca-trust/extracted/pem is filled, on the new, not
18:30 daniels: bentiss: iirc it’s update-ca-certificates you want
18:30 daniels: at the very least, try tab-completing from update-
19:10 bentiss: daniels: thanks... but /etc/pki is mounted as ro, and this is populated in an init container
19:15 bentiss: even weirder. I killed on pod on the old cluster, and the ca store is fine but /etc/pki/ca-trust/extracted/pem/ is empty too
19:17 bentiss: oops. I thought I was in a pod when I looked at /etc/pki... but I was on my own computer. At least that explains why ;)
19:34 bentiss: so... the problem is I'm getting redirected to cloudfront instead of contacting google.com in my curl experience
19:35 bentiss: probably a coredns config issue then
19:51 bentiss: maybe a quick fix for the night is to restore the old IP address
19:57 bentiss: alright, this is done. And it seems to be working
19:57 bentiss: hanging up for today
20:00 Consolatis: bentiss: thanks for your work the whole day
21:32 ngcortes: anybody else having an issue with gitlab where it can't find a source branch for an MR?
21:33 ngcortes: ie. https://gitlab.freedesktop.org/Mesa_CI/mesa_jenkins/-/merge_requests/2455
21:33 ngcortes: the branch exists but gitlab says it doesn't
21:33 slomo: yes, i just pushed to https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/merge_requests/1367 2 hours ago and now it can't find the branch
21:36 ngcortes: ack