01:33 DavidHeidelberg[m]: what most likely happens when I'm s3cp to same file (except caches runner proxy cache would keep it)
08:46 ishitatsuyuki: I'm seeing a bunch of ERROR: Job failed (system failure): Error response from daemon: container create: allocating lock for new container: allocation failed; exceeded num_locks (2048) (docker.go:534:0s)
08:46 ishitatsuyuki: on https://gitlab.freedesktop.org/mesa/mesa/-/pipelines/804146
09:01 MrCooper: seems to be the fdo-equinix-m3l-11 runner
09:22 MrCooper: bentiss: any idea why ccache from F37 hangs in the futex syscall in CI?
09:37 bentiss: MrCooper: ouch, that's way too many words for me to parse in the morning :)
09:37 MrCooper: sorry :)
09:37 bentiss: heh, no worries
09:38 MrCooper: https://gitlab.freedesktop.org/daenzer/mesa/-/jobs/36120295 was hanging for minutes before I cancelled it
09:41 bentiss: FWIW, fdo-equinix-m3l-11 has a lot of crosvm jobs running
09:42 bentiss: fdo-equinix-m3l-12 is pretty much unused now but has a load average of 15, meaning that it was quite busy not so long ago
09:47 bentiss: let me reboot/upgrade them. Though I am afraid virglrenderer is messing with the runner
09:51 MrCooper: to be clear, ccache hanging seems unrelated to overloaded runners
09:54 bentiss: well, the runners do not seem to be in a pretty good shape, so a reboot might help
09:57 MrCooper: didn't seem to affect ccache from F34 though, or I would have expected screaming on #dri-devel
09:59 MrCooper: still hanging on fdo-equinix-m3l-12: https://gitlab.freedesktop.org/daenzer/mesa/-/jobs/36124305
09:59 bentiss: anyway, m3l-11 is now rebooting, ww'll see if this one fails
10:00 bentiss: m3l-12 still not updated/rebooted FWIW
10:00 bentiss: I prefer not killing 2 out of the 3 runners at the same time
10:03 bentiss: MrCooper: mind if I kill that job?
10:03 MrCooper: not at all
10:03 bentiss: k, thanks
10:10 bentiss: damn, it seems that the reboot did not help: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/jobs/36124753
10:10 ishitatsuyuki: it did get my radv job through though
10:11 ishitatsuyuki: https://gitlab.freedesktop.org/mesa/mesa/-/jobs/36124526 for reference
10:13 bentiss: yeah some are passing, most are not
10:20 bentiss: I am tempted to just dump m3l-11 and respin a new one
10:20 bentiss:just doesn;'t have the time to debug this today
10:30 bentiss: OK, so all 3 x86 runners have been updated, but m3l-11 is still behaving badly, so I disabled it. I'll respin a new one this afternoon when I get a little bit more time
10:34 bentiss: MrCooper: ^^ please tell me if m3l-12 is still acting badly
10:34 MrCooper: still hanging on -13: https://gitlab.freedesktop.org/daenzer/mesa/-/jobs/36125582
10:35 MrCooper: I don't think it's related to the other runner issues
10:35 bentiss: MrCooper: it could be a f37 issue
10:35 MrCooper: some kind of bad interaction between f37 and the CI environment, yeah
10:36 bentiss: have you tried running it locally in a container?
10:36 MrCooper: I'll try if it happens with f36 as well
10:37 bentiss: I got to go for an errand, bbl
10:48 MrCooper: bentiss: doesn't hang on my personal gitlab-runner (which is an old version though due to Debian, 13.3.1): https://gitlab.freedesktop.org/daenzer/mesa/-/jobs/36125610
10:50 MrCooper: also note that it uses docker, not podman
12:53 bentiss: Alright. I have now spun up m3l-14 and will burn with fire m3l-11 asap
13:11 zmike: https://gitlab.freedesktop.org/mesa/mesa/-/pipelines/804310 🤔
13:11 zmike: is it normal for sanity job to take 40 minutes to start
13:18 daniels: no
13:18 daniels: see ongoing fire above
13:18 zmike: k just checking if same issue
17:36 zmike: hmm are ci disks out of space? https://gitlab.freedesktop.org/mesa/mesa/-/pipelines/804523
18:31 bentiss: indeed, that runner doesn't have disks properly set :(
18:49 bentiss: alright, I respun a new one, the cloud-init config file was completely wrong, and fixing it would take more time than just bringing in a new one
19:13 alatiera: one of the gst runners is out of space too
19:14 alatiera: fixing