08:20 bentiss: FWIW, I still haven't receive feedback from fastly regarding bot protection, so I'll try to enable anubis and a more strict caching policy (only `^/assets`), and see how it goes
08:20 bentiss: this will be switched on during the day I think
08:45 karolherbst: good luck
09:32 bentiss: either I broke everyone's workflow, either this had a nice impact on the servers :)
09:33 bentiss: Which image pasting service I can use to show a couple of graphs???
09:34 bentiss: imgur maybe
09:35 bentiss: https://imgur.com/a/umfMB3G
09:40 karolherbst: bentiss: nah, that's the usual impact with anubis
09:40 karolherbst: well..
09:41 karolherbst: usually the impact is bigger 🙃
09:41 karolherbst: reduction of 90% load is kinda the norm
09:41 bentiss: I'm still happy with the impact on gitaly, as that means faster queries for legitimate clients
09:42 karolherbst: bentiss: maybe turn down the difficulty a bit :D
09:42 bentiss: git pull over https still works so... we are good?
09:42 karolherbst: yeah.. git pull seems to work here
09:42 bentiss: karolherbst: you mean turn up? from 4 to 5?
09:42 karolherbst: nah. maybe starting with 3.. 4 can already be kinda slow
09:42 bentiss: 5 was barely usable in the tests
09:43 karolherbst: like on a beefy laptop it even might take a second or two
09:43 karolherbst: took 5 seconds here
09:43 bentiss: oh, it took 400 ms here
09:44 bentiss: with a 5 year old desktop
09:44 karolherbst: I get 40 kH/s but I think something is up with my laptop...
09:44 karolherbst: or the power profile stuff is just broken
09:45 bentiss: anyway, lunch here, I'll monitor over the afternoon
09:45 karolherbst: have fun
09:45 bentiss: FWIW, it seems that part of the traffic is still not using anubis, which might explain the little gain
10:27 bentiss: finally, 0R/s on the gitlab endpoint... and still have 126 R/s on the anubis one. We used to have 150 on gitlab so I guess it's better, but not absolutely fabulous
10:53 eric_engestrom: bentiss: some jobs are running without outputting any logs and seem to run forever without making progress, is that a possible fallback from enabling anubis?
10:53 eric_engestrom: eg. https://gitlab.freedesktop.org/mesa/mesa/-/jobs/75852279
10:55 eric_engestrom: there are also jobs that are no longer getting picked up, but it might be a problem with the runner, rebooting it now
11:01 valentine: yes, jobs on the shared runners appear to be fine, but jobs on external farms aren't outputting anything on gitlab
11:14 eric_engestrom: it's more than just no output actually, they run forever (past timeout) so I'm guessing they get stuck and don't actually run
11:25 mupuf: lovely :D
11:36 dj-death: eric_engestrom: same here, I have an MR not making any progress
11:49 bentiss: looks like I was missing the PATCH request and anubis returned 405 for them
11:51 bentiss: (and yes, the fdo shared runners have direct access to gitlab, bypassing fastly and anubis)
11:56 bentiss: that seemed to have a positive impact, but the jobs need to be restarted
11:57 bentiss: (just did that for https://gitlab.freedesktop.org/mesa/mesa/-/jobs/75857007)
12:09 bilboed: bentiss: the API endpoints aren't protected by anubis, right ?
12:10 bentiss: bilboed: yeah, only GET requests are
12:10 bilboed: 👍️
12:10 bentiss: and it's not entirely protected. I forward the request once to anubis, and if the client gets the cookie, I validate it at the fastly level and bypass anubis entirely
12:12 valentine: bentiss: Thanks!
12:13 bentiss: for anyone interested: https://gitlab.freedesktop.org/freedesktop/terraform-gitlab-fastly/-/blob/main/anubis/src/main.rs?ref_type=heads#L88 this is my poor rust implementation of that
12:14 bentiss: if anyone wants to help, it's in rust, so that's a lot of fun... :-)
12:25 mupuf: Berenguer1931[m]: do you think we'll need to cancel all the jobs thast were running on hardware farms?
12:25 mupuf: bentiss: ^
12:26 bilboed: oh wow, all requests at fastly go through rust code ?
12:45 bentiss: bilboed: yep, it's compiled in webasm and fastly executes that at the edge... (/me learned a lot of new terms)
12:46 bentiss: mupuf: maybe?
12:46 mupuf: bentiss: hehe, ok
12:48 mupuf: bentiss: seems like they got themselves unstuck
12:48 mupuf: so, all good!
12:48 bentiss: \o/
12:49 mupuf: congrats for entering the 22nd century, with rust compiled to webasm :D
12:49 bentiss: I know... this is a quite a feat :)
12:50 bentiss: Also, FWIW, the anubis config in itself is properly unconfigured, so maybe someone else could help on that
12:50 bentiss: (i.e. just pulling latest image and run)
12:52 mupuf: seems to work well. Was just a bit slow on my phone (22 s)
15:26 eric_engestrom: thanks bentiss for the fix, and for putting anubis in place 🙏
15:48 DemiMarie: bentiss: I know Rust!
15:50 bentiss: heh
16:11 karolherbst: bentiss, mupuf: I figured it out... it's _super_ slow only on firefox. Same machine with chromium it's pretty much instant
16:12 karolherbst: but no idea why it's so slow with firefox :D
16:13 karolherbst: maybe xe has any ideas?
16:13 karolherbst: or maybe firefox users just get punished or something :P
16:16 xe: karolherbst: i've been trying to figure that out myself
16:16 xe: i'm going to prioritize the wasm port for the checker as a stopgap
16:16 karolherbst: yeah.. maybe the JS engine is just bad or you hit bad patterns and the JIT can't make it go fast
16:17 xe: my guess is that my intent on going from JIT to highly optimized C++ browser internals is a bet that did not pay off lol
16:17 karolherbst: heh
16:17 xe: additional fun part
16:17 xe: i do all my development on firefox
16:18 xe: so this just registers as "normal" for me
16:18 karolherbst: mhhh
16:18 karolherbst: it's quite fast on chromium
16:18 xe: yeah, i am willing to bet that chromium does what i expect firefox to do
16:19 karolherbst: which algo is used anyway?
16:19 karolherbst: or what's the math problem
16:19 karolherbst: I never checked the details :D
16:23 xe: naïve sha256 get leading number of zeros :D
16:24 karolherbst: mhhh
16:25 karolherbst: sounds like something that an optimizer could mess up if it doesn't use sha instructions
16:25 dwfreed: basically xe implemented bitcoin :D
16:25 dwfreed:ducks
16:25 xe: dwfreed: i mean, i implemented hashcash but bitcoin implemented hashcash
16:25 dwfreed: right
16:26 xe: also "design" is a very strong word for this implementation, the rust one is a lot more designed than this is lol
16:26 karolherbst: anyway.. I hope the wasm port is going to help there a lot
16:27 xe: another stopgap is to port the thing to use an in-JS sha256 library
16:28 xe: but i trust javascript about as far as i can throw it, i have muscle weakness due to medication, and i'm unable to grasp or throw concepts
16:28 xe: er, pure JS cryptographic code*
16:29 karolherbst: I think the argument in favor of a widespread used in-JS library is, that the JS JIT developers probably used it to optimize their JIT
16:29 xe: as soon as I ship v1.18.0 later today i'm gonna rescue the wasm checker port and reduce scope to square peg -> round hole into existing anubis
16:30 karolherbst: and I wouldn't be surprised that firefox' pattern matching to detect where the sha instructions could be used, might trigger more reliably there
16:30 karolherbst: heh
16:30 karolherbst: have fun
16:30 xe: the 28KB of wasm uses SIMD128 so it's fast enough that it's a logistical difficulty lol
16:30 karolherbst: mhhh
16:30 xe: it also does bit-wise difficulty scaling so i'm gonna have to figure out a migration path :)
16:31 karolherbst: I'd look at it like this, if anubis gets big enough, that scrappers will try to get around it, they'll probably replace your code with something that's super fast
16:31 xe: yeah, tbh, sha256 like this is bait
16:31 xe: i'm waiting for an AI company to make a bypass by doing GPU offloading
16:31 xe: then tweak things slightly and gain herd immunity :)
16:31 karolherbst: heh
16:32 xe: long term i'm going to have 64 variants of proof of work and tweak things so that proof of work is shown less often
16:32 xe: the kinda cool part about the implementation in wasm is that it runs the same binary on both the client and the server so that everything is in lockstep
16:33 karolherbst: it's kinda a pain how this residential business model works, because you can't even trust that a second connection from the same IP isn't malicious...
16:34 xe: i've been looking at options, but sadly there's difficulties there :(
16:34 karolherbst: yeah...
16:35 daniels: karolherbst: residential IP reputation stopped being viable as a concept 20 years ago
16:36 karolherbst: heh fair
16:37 daniels: CGNAT, your uncle's inexplicable attachment to his Android 4.4 phone, your IoT toaster, etc
16:38 karolherbst: tbf, if my toaster wants to access gitlab, who am I to judge
16:39 eric_engestrom: fraking toasters
16:41 eric_engestrom: (battlestar galactica reference)
16:43 karolherbst: I'm curious if this also solves our bot sign up problem...
16:43 xe: from what i've seen elsewhere: yes
16:44 karolherbst: mhhh
16:44 xe: karolherbst: could you get me in contact with the person that set up the gitlab k8s manifest? I'd like to see what ingress controller you're using so I can see if a Terrible Idea™ works
16:45 karolherbst: I'm sure bentiss set it all up
16:51 DemiMarie: does anyone see this message?
16:51 xe: DemiMarie: yes
16:52 DemiMarie: xe: your reliance on browser crypto really helps people with hardened browser configs, including what I believe is (or might be) MS Edge when visiting a website that one hasn't visited recently
16:53 xe: no problem! glad it helps :)
16:54 DemiMarie: xe: one of the most common and effective hardening measures is disabling JIT
16:54 DemiMarie: it's the default in GrapheneOS
16:55 DemiMarie: Lockdown mode on iOS enables it too, and I believe MS Edge disables JIT for websites one hasn't visited before
16:55 xe: I haven't been able to test things on GrapheneOS due to not having hardware that can run it
16:56 DemiMarie: Lockdown on iOS also disables WebGL and WebGPU
16:56 xe: but could you file a bug about that? https://github.com/TecharoHQ/anubis
16:56 DemiMarie: xe: It works on GrapheneOS right now
16:56 DemiMarie: the reason I am letting you know is to try to avoid any regressions
16:58 eric_engestrom: bentiss: ci jobs are broken again, docker registry pull fails with:
16:58 eric_engestrom: Head "https://registry.freedesktop.org/v2/mesa/mesa/debian/x86_64_pyutils/manifests/20250422-ci-fairy--c6aeb16f86e32525fa630fb99c66c4f3e62fc3cb": Get "https://gitlab.freedesktop.org/jwt/auth?account=gitlab-ci-token&scope=repository%3Amesa%2Fmesa%2Fdebian%2Fx86_64_pyutils%3Apull&service=container_registry": context deadline exceeded
16:58 DragoonAethis: Also, visiting https://gitlab.freedesktop.org/mesa/mesa currently returns "Service Unavailable"
16:58 xe: DragoonAethis: you do realize that it's hard for me to ensure there aren't regressions without the ability to test it, right?
16:58 xe: er oops DemiMarie sorry DragoonAethis
16:59 DragoonAethis: xe: no worries, thanks for your work <3
16:59 xe: <3
17:01 xe: probably gonna end up having to buy a Pixel 9A for the integration jungle though
17:03 eric_engestrom: fyi docker pull works again
17:43 bentiss: heh, glad it works again when I come back (though it's late already)
17:59 DemiMarie: xe: you can test with stock Chromium by disabling JIT
18:09 DemiMarie: xe: not sure if Google Chrome exposes this on Android, though
18:13 xe: do you see why i want to cut out the middleman and do testing on the exact environment that fails? :)
18:13 xe:goes to file a taxable business expense
18:25 xe: DemiMarie: my pixel 9a comes on friday, please enable JIT for websites that use anubis until then
18:26 DemiMarie: xe: it works just fine, which is why I didn't already file a bug report
18:26 DemiMarie: I mentioned this to prevent future regressions only
18:27 xe: there's been other people that have complained about grapheneos and vanadium being inconsistent, i'm just adding it to my test suite
18:29 DemiMarie: xe: thank you for Anubis
18:30 xe: no prob! it's surreal to see it take off so much
18:30 xe: never thought the shitpost would end up with a wikipedia page
18:37 hakzsam: Is the "Oh noes" Anubis stuff new? I can't access mesa fdo from my phone now
18:41 xe: what error code are you getting hakzsam?
18:45 xe: and what phone / browser / os are you using
20:54 ocrete: I'm seeing random failures when uploading CI artifacts in the GStreamer CI. is that a known issue ?
20:56 ocrete: It,s getting 503 errors