00:59fdobridge_: <gfxstrand> Kicked off an Ada run on my laptop. I'll start Ampere after supper
01:34fdobridge_: <gfxstrand> The bad news is that Ada fails. The good news is that it fails consistently and quickly. Looks like exactly the same as the WSI fail.
04:50fdobridge_: <gfxstrand> Okay, this bug is utterly insane. If I delete a bunch of tests that skip from the test list it passes...
04:51fdobridge_: <gfxstrand> My mind is boggled.
04:53fdobridge_: <gfxstrand> Also, reducing the test list is a bear. I've got it down to 218859 tests to reproduce (about 12.5 seconds) but reducing further is proving difficult.
05:22fdobridge_: <gfxstrand> Alright, I'm calling it. This is a kernel bug. It's something with context creation, I'm pretty sure. When I run my repro list with `NVK_DEBUG=push`, I get exactly the same thing for the hanging case as the passing case. When I run with `NAK_DEBUG=print`, the only differences are some randomly generated constants in shaders that get written to buffers. They change from run to run so there's no way those are the culpret given that it's 100%
05:22fdobridge_: <gfxstrand> @airlied dakr: ^^
05:38fdobridge_: <gfxstrand> https://gitlab.freedesktop.org/drm/nouveau/-/issues/315
05:38fdobridge_: <gfxstrand> @airlied Back to you. Have fun!
05:39fdobridge_: <gfxstrand> If you want to tell me where to hack nouveau to bump various ring sizes, I'm happy to try stuff out.
05:46fdobridge_: <gfxstrand> Oh, and if it helps motivate anyone, this is the bug standing between us and 1.3 conformance on Turing+. 😝
07:36fdobridge_: <airlied> Once I figure out the correct fix for the sync irqs I'll take a look, or maybe while I'm getting annoyed at that I will have time
15:04fdobridge_: <!DodoNVK (she) 🇱🇹> https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27319 :triangle_nvk:
16:56fdobridge_: <gfxstrand> Thanks! I keep forgetting those... 🤦🏻♀️
17:09fdobridge_: <karolherbst🐧🦀> @gfxstrand random thought: with `mme` we could actually dump the subchannel context and dump all set values on each method. Could be useful to debug some annoying issues and diff between different runs
17:10fdobridge_: <karolherbst🐧🦀> the annoying part would be to write that code, as I bet you can't just read out everything as the GPU will complain about unknown methods :/ and I don't know how that works on methods being data dumps
17:13fdobridge_: <gfxstrand> Yeah, I've thought about that. I've also thought about trying to do something at a CPU SW level to prevent us from using state we don't explicitly initialize. IDK that I really like that, though, as there's a lot of state that doesn't matter until we use it.
17:15fdobridge_: <gfxstrand> And, like, with this one... What the hell state is it using?!? It's a really dumb draw call. It's not even using a texture. I know we disable all the shaders on init.
17:16fdobridge_: <gfxstrand> Maybe there's a stale bound CBuf?
17:26fdobridge_: <gfxstrand> Yeah, it's not that. I'm now disabling all CBufs on init and it still crashes
17:28fdobridge_: <gfxstrand> If I could just get my fault address, this might be debuggable
17:30fdobridge_: <karolherbst🐧🦀> could also be some post draw thing.. like I dunno..
17:32karolherbst: TimurTabi: weren't you working on something debugging related to GSP or was that somebody else?
17:32fdobridge_: <gfxstrand> We're really starting to run out of ways that draw could possibly get at memory to fault
17:32fdobridge_: <karolherbst🐧🦀> it might not be the draw
17:33fdobridge_: <gfxstrand> I'm running with `NVK_DEBUG=push_sync`
17:34fdobridge_: <gfxstrand> With everything I've eliminated, it's starting to look like corrupt page tables or weird GSP shit
17:34fdobridge_: <karolherbst🐧🦀> huh.. that push-fail thing doesn't even have a draw?
17:34fdobridge_: <gfxstrand> It does. It's the MME at the bottom
17:34fdobridge_: <gfxstrand> But it's a really dumb draw. Doesn't use textures or anything
17:35fdobridge_: <karolherbst🐧🦀> at least the one you uploaded doesn't have an MME
17:35fdobridge_: <karolherbst🐧🦀> only a `NVC1B5_LAUNCH_DMA`
17:35fdobridge_: <gfxstrand> Oh, that's the wrong one
17:35fdobridge_: <gfxstrand> I had to fix pushbuf dumping
17:35fdobridge_: <gfxstrand> It's a draw that's crashing
17:36fdobridge_: <karolherbst🐧🦀> but yeah... there is random stuff going on, but that's since forever and I'm sure we also have pre GSP kernel bug somewhere...
17:36fdobridge_: <karolherbst🐧🦀> I've ran into similar issues with the GL CTS, just that those faults happened after hours
17:37fdobridge_: <karolherbst🐧🦀> but that's just using a single context, because GL
17:37fdobridge_: <karolherbst🐧🦀> @gfxstrand can you make the CTS stop reusing `VkDevice`s?
17:38fdobridge_: <karolherbst🐧🦀> I wonder if that changes anything
17:38fdobridge_: <karolherbst🐧🦀> (or makes it even worse)
17:38fdobridge_: <gfxstrand> No
17:39fdobridge_: <gfxstrand> I mean maybe?
17:39fdobridge_: <gfxstrand> IDK if there's a flag for that or not
17:39fdobridge_: <karolherbst🐧🦀> yeah.. not sure either, I think there might be something, but I'm also only 5% sure
17:40fdobridge_: <gfxstrand> https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27320/diffs?commit_id=d440eb96ce01d7a44ef365ea9f5957b32f075fb6
17:40fdobridge_: <gfxstrand> There isn't a flag.
17:42fdobridge_: <karolherbst🐧🦀> maybe somebody had a patch somewhere...
17:42fdobridge_: <gfxstrand> I mean... I could do a crazy driver hack to create a new context for every command submit
17:42fdobridge_: <karolherbst🐧🦀> mhhhh
17:42fdobridge_: <karolherbst🐧🦀> I mean.. if you invalidate all state it _should_ work, shouldn't it?
17:43fdobridge_: <karolherbst🐧🦀> but it doesn't really tell us anything anyway
17:43fdobridge_: <karolherbst🐧🦀> state dumping sounds more promising tbh
17:44fdobridge_: <gfxstrand> State dumping doesn't do much if I don't know what I'm looking for
17:44fdobridge_: <karolherbst🐧🦀> I mean like.. dumping everything
17:44fdobridge_: <gfxstrand> That's just a bigger haystack. Without even knowing the fault address, how am I supposed to know if anything is wrong.
17:45fdobridge_: <karolherbst🐧🦀> diff it with the working run
17:45fdobridge_: <gfxstrand> Yeah, maybe?
17:45fdobridge_: <karolherbst🐧🦀> yeah, I dunno
17:46fdobridge_: <gfxstrand> I wonder how long a "dump everything" MME would take to run...
17:46fdobridge_: <karolherbst🐧🦀> state dumping is just a pain, because I'm sure nvidias internal description of that stuff would allow you to auto generate dumping code :ferrisUpsideDown:
17:46fdobridge_: <karolherbst🐧🦀> mhh actually
17:46fdobridge_: <gfxstrand> I think my first attempt would be to literally dump everything and see if that works.
17:47fdobridge_: <karolherbst🐧🦀> probably enough to only dump stuff `border_swizzle` touches
17:47fdobridge_: <gfxstrand> Maybe it complains but maybe it doesn't?
17:47fdobridge_: <karolherbst🐧🦀> mhhh
17:47fdobridge_: <karolherbst🐧🦀> yeah.. dunno what happens on reads actually
17:47fdobridge_: <gfxstrand> Nah. The border swizzle tests don't actually run
17:47fdobridge_: <karolherbst🐧🦀> ahh...
17:47fdobridge_: <karolherbst🐧🦀> pain
17:47fdobridge_: <gfxstrand> They just probably create and destroy enough contexts to get the moons to align
17:48fdobridge_: <gfxstrand> (I guess NVKland now canonically has multiple moons...)
17:48fdobridge_: <karolherbst🐧🦀> figures
17:50fdobridge_: <gfxstrand> https://tenor.com/view/thats-no-moon-obi-wan-kenobi-starwars-death-star-gif-10170550
17:57fdobridge_: <gfxstrand> Hrm... There are only about 1k states on Turing. That should totally be dumpable.
18:18fdobridge_: <tom3026> Use the force, gfxstrand.
18:21fdobridge_: <karolherbst🐧🦀> yeah.. the it's not too bad, and 3D is already quite big compared to the others
18:21fdobridge_: <karolherbst🐧🦀> @gfxstrand you know what we could do with state dumping in place? Find aliasing methods between the sub channels...
18:40fdobridge_: <gfxstrand> Yeah, maybe
19:13Sid127: oh, heh, looks like a bouncer went offline
19:14Sid127: oh wait these are all matrix connections
19:35karolherbst: matrix moments
21:21fdobridge_: <pavlo_it_115> HelIo everyone! Who can tell in what condition the NVK for kepler? Can it be used?
21:23fdobridge_: <Sid> Kepler does work if you set `NVK_I_WANT_A_BROKEN_VULKAN_DRIVER=1` but it's a bit sketchy.
21:24fdobridge_: <pavlo_it_115> Sketchy?
21:24fdobridge_: <pavlo_it_115> why
21:25fdobridge_: <mhenning> It's only partially implemented
21:26fdobridge_: <mhenning> It needs more work before it will really be usable, although some stuff might work already
21:36fdobridge_: <redsheep> Here's an example of where Kepler is at https://gitlab.freedesktop.org/mesa/mesa/-/issues/10447
22:10fdobridge_: <gfxstrand> I really need to get around to fixing that but I think it'll have to wait until after Vulkanised
22:10fdobridge_: <gfxstrand> Or maybe I'll plug my pikvm into my Kepler card and fix it during the MoltenVK talks. 😅
22:12fdobridge_: <gfxstrand> So... I dumpped everything on the 3D engine... No differences until... the MME faults.
22:12fdobridge_: <gfxstrand> Yeah, the state dumping MME does part-way through.
22:14fdobridge_: <redsheep> Will Kepler ever be able to support enough for zink to work well, or will zink need special paths for it?
22:14fdobridge_: <gfxstrand> Kepler should be able to support must stuff.
22:14fdobridge_: <gfxstrand> Just not memory model
22:20fdobridge_: <gfxstrand> @karolherbst https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27321
22:21fdobridge_: <pavlo_it_115> Can I be more detailed please? But what does it affect?
22:22fdobridge_: <karolherbst🐧🦀> wait.. all methods are the inputs to that macro? impressive
22:24fdobridge_: <gfxstrand> Yup
22:24fdobridge_: <gfxstrand> It just dumps everything in clc597.h
22:25fdobridge_: <gfxstrand> A lot of it is probably useless but whatever
22:27fdobridge_: <gfxstrand> Some of the details around memory ordering on global memory access. Global loads and stores work, they just don't have tight enough ordering guarantees on Kepler.
22:27fdobridge_: <dadschoorse> also no vulkanMemoryModel -> no vulkan 1.3
22:27fdobridge_: <gfxstrand> Yup
22:27fdobridge_: <gfxstrand> Unless I can somehow be more clever than the NVIDIA engineers.
22:28fdobridge_: <gfxstrand> But I know the people who did the memory model stuff at NVIDIA. They're *very* clever.
22:32fdobridge_: <karolherbst🐧🦀> what's the limitation here anyway?
22:32fdobridge_: <gfxstrand> Something with memory ordering in the hardware
22:32fdobridge_: <gfxstrand> Something with memory access ordering in the hardware (edited)
22:32fdobridge_: <karolherbst🐧🦀> mhhh
22:32fdobridge_: <karolherbst🐧🦀> I see
22:39fdobridge_: <gfxstrand> This has me really confused. The only reason I could imagine that happening is if my page tables are being ripped out from under me somehow.
22:40fdobridge_: <redsheep> What about Fermi? I know Nvidia never tried as it was already pretty much EOL at the time but wasn't Fermi pretty close to Kepler?
22:41fdobridge_: <redsheep> Just naively reading the white papers it's hard to see what Kepler really added that would matter here except being much wider
22:41fdobridge_: <gfxstrand> I think so? 🤷🏻♀️
22:41fdobridge_: <gfxstrand> I've heard that they had a Fermi driver they decided not to release.
22:42fdobridge_: <gfxstrand> But the hardware gets more annoying the further back you go
22:44HdkR: Oldest architectures also had the annoying split data and texture caches. So if you were modifying textures through a buffer and then sampling you would need some cache clear operations in the middle. Nightmare :|
22:45fdobridge_: <redsheep> Yeah for now it certainly makes sense to focus on the hardware with the features and performance to make NVK work really well.
22:45fdobridge_: <gfxstrand> IDK. I worked on Intel. 😂
22:47fdobridge_: <gfxstrand> We had separate caches for everything
22:48fdobridge_: <redsheep> I know it's probably been asked a thousand times but I don't feel like I've ever got a satisfying answer, but is there any hope at all for Maxwell? I'm sure that before long Nvidia will be looking to axe support and it would be awesome for mesa to pick up where Nvidia left off there.
22:48fdobridge_: <gfxstrand> And image layouts. 🤡
22:48fdobridge_: <redsheep> Like I know the basics, it can't reclock, the firmware we have sucks
22:48fdobridge_: <redsheep> Is it just up to Nvidia to publish firmware we can actually use?
22:48fdobridge_: <gfxstrand> Maxwell should mostly be working. A few people are trying to bring up the new compiler right now but and it's already passing a pile of tests.
22:50fdobridge_: <redsheep> So gm100 chips could theoretically reclock and make use of that, but gm200 just never will with current firmware right?
22:50fdobridge_: <gfxstrand> Pretty much
22:52fdobridge_: <tom3026> are you talking about same pages as https://github.com/torvalds/linux/commit/23baf831a32c04f9a968812511540b1b3e648bf5 because im just about to try revert that commit it breaks my dma_pools but i dont get why im the sole person on this planet getting a stacktrace on boot because of it heh
22:52fdobridge_: <airlied> And NVIDIA will not release Maxwell fw we can use probably ever
22:53fdobridge_: <redsheep> For all the Nvidia people reading here, a great marketing story would be resolving that blocker at the same time you guys announce dropping Maxwell support. Turn a bad story good.
22:54fdobridge_: <airlied> Nope they have no case for doing it, it's a lot of engineering work
22:57fdobridge_: <tom3026> or well just noticed it landed a kern version before it broke bleh i was so close.. oh well time to git bisect a gazillion MM commits then
23:02fdobridge_: <gfxstrand> I mean, it's possible I guess but I have no evidence that that would help.
23:02fdobridge_: <tom3026> yeah it probably wont noticed the tags its in before it began borking
23:03fdobridge_: <tom3026> i have faith if i bisect this find the offending commit suddenly all my software runs null pointer deref free and allocates memory like never before
23:04fdobridge_: <pavlo_it_115> Anyone in the community asking for firmware from nvidia?
23:04fdobridge_: <pavlo_it_115> everyone seemed resigned
23:05fdobridge_: <pavlo_it_115> everyone seemed resigned..? (edited)
23:06fdobridge_: <pavlo_it_115> everyone seemed... resigned? (edited)
23:06fdobridge_: <airlied> We only asked since Maxwell 2 released 🙂
23:06HdkR: The funny joke is, even if you ask more, it won't change
23:07fdobridge_: <airlied> Yes they have no interest in giving redistribution rights for the fw they use, so even if you RE it all it's a pain in the ass
23:08fdobridge_: <redsheep> Sorry to bring it up, I know it's a sore spot.
23:10fdobridge_: <pavlo_it_115> I remembered something (but vaguely). It seems some director of nvidia linux drivers left the company a long time ago and he was actively accepting applications from the nuvo community etc. Because of his resignation, now there is no "connection" with nvidia or what?
23:10fdobridge_: <airlied> No we at Red Hat have regular meetings with them and engage with community concerns
23:11fdobridge_: <gfxstrand> Yeah, the problem isn't a lack of people asking. It's a lack of motivation on their part. They don't really have a reason to do the work required to do a competent firmware release.
23:12fdobridge_: <pavlo_it_115> As I understand it, he cannot influence this situation in any way?
23:13fdobridge_: <airlied> They have no business case for the amount of engineering effort
23:15fdobridge_: <airlied> Even getting the fw we have pre gsp was very painful and expensive engineering wise
23:16fdobridge_: <airlied> Since it was bespoke nouveau fw and their engineering was always slow to react to new hw updates
23:16fdobridge_: <gfxstrand> And asking them to put in all that work for hardware that they've already sold and won't ever make another penny on... Yeah, that isn't going to happen. Especially with as close to EOL as it is. Their solution is "go buy a new GPU"
23:17fdobridge_: <airlied> Or use the last binary driver forever
23:18fdobridge_: <pavlo_it_115> And what did they answer?)
23:18fdobridge_: <gfxstrand> If they'd answered "yes", would we be having this conversation right now?
23:18fdobridge_: <pavlo_it_115> -
23:19fdobridge_: <airlied> There was a lot of looking into it, but nobody could work out a supportable strategy that wasn't let's get GSP done
23:20fdobridge_: <redsheep> I remember some news in the last couple of years that somebody managed to get those cards to ignore the check for signed firmware, but I assume based on what you're saying that doesn't really provide any new hope
23:22fdobridge_: <redsheep> I'm sure the RE work is immense any way you slice it
23:23fdobridge_: <pavlo_it_115> You intrigued me
23:24fdobridge_: <airlied> There was some key leak at some point, but I forget the details
23:24fdobridge_: <conan_kudo> Well at this point, most pre Turing users could upgrade affordably as long as it's not a laptop
23:24fdobridge_: <conan_kudo> GTX 16 series cards are very cheap now
23:25fdobridge_: <conan_kudo> And I'm saying that as a Maxwell owner
23:27fdobridge_: <pavlo_it_115> https://forums.developer.nvidia.com/t/can-nvidia-contribute-to-open-source-nouveau-driver-directly/270264
23:27fdobridge_: <pavlo_it_115> Silence)
23:27fdobridge_: <airlied> https://gist.githubusercontent.com/plutooo/733318dbb57166d203c10d12f6c24e06/raw/15c5b2612ab62998243ce5e7877496466cabb77f/tsec.txt was some stuff but no idea if any of that is similar off the switch
23:29fdobridge_: <airlied> Public forums aren't going to have any influence
23:35fdobridge_: <pavlo_it_115> 🤬
23:36fdobridge_: <pavlo_it_115> I have no words this
23:36fdobridge_: <pavlo_it_115> I have no words (edited)
23:37fdobridge_: <pavlo_it_115> Maybe try writing to them on github?
23:37fdobridge_: <karolherbst🐧🦀> they already do, but uhh... not much 😄
23:38fdobridge_: <gfxstrand> It's not going to have any effect
23:38fdobridge_: <karolherbst🐧🦀> the problem isn't engineers seeing that stuff
23:38fdobridge_: <karolherbst🐧🦀> they know
23:38fdobridge_: <karolherbst🐧🦀> it's purely dysfunctional on a management level
23:38fdobridge_: <airlied> Yeah not sure why you think we don't talk to the people who make the decisions and have decided aleeady
23:39fdobridge_: <karolherbst🐧🦀> nvidia just doesn't want to
23:39fdobridge_: <karolherbst🐧🦀> it's really as simple as that
23:41fdobridge_: <karolherbst🐧🦀> there are of course exceptions.. but that's mostly engineers working around the situation
23:41fdobridge_: <marysaka> Pretty sure that kind of exploit isn't going to work on non Tegra stuffs, but the keys are common between desktop GPU and Tegra
23:41fdobridge_: <marysaka> I think the best approach would be to get HS execution on Maxwell/Pascal via the Falcon's bootrom, but that requires quite the work to get something working (ROP chain against execute only memory is fun...)
23:41fdobridge_: <marysaka> But even if that happen, you would still need a way to research the actual blobs to know what to write (so back to square one, need the keys...)
23:44fdobridge_: <pavlo_it_115> Decided what? Is this classified information?
23:44fdobridge_: <pavlo_it_115> Decided what? Is this private information? (edited)
23:45fdobridge_: <redsheep> He's saying that management decided not to allocate the engineering effort. It's not lack of communication, and it's not the engineer's fault.
23:45fdobridge_: <karolherbst🐧🦀> yes
23:46fdobridge_: <gfxstrand> That, too.
23:47fdobridge_: <karolherbst🐧🦀> we certainly talk with nvidia about stuff, and e.g. the headers they publish is one result of those efforts
23:47fdobridge_: <karolherbst🐧🦀> it's just...
23:47fdobridge_: <karolherbst🐧🦀> nvidia wants to see $$$
23:47fdobridge_: <karolherbst🐧🦀> for anything they do
23:47fdobridge_: <karolherbst🐧🦀> so if contributing to nouveau doens't bring them any money, they won't do it
23:47fdobridge_: <redsheep> As somebody dealing with support for a software company, I can totally understand getting burned out by tons of people asking for something where the engineering effort won't ever make sense on an EOL product.
23:47fdobridge_: <karolherbst🐧🦀> so if you want nvidia to contribute, make a billion dollar contract with them only using nouveau
23:47fdobridge_: <karolherbst🐧🦀> that's how you get nvidia to contribute
23:49fdobridge_: <karolherbst🐧🦀> and the number is probably in the right ballpark
23:50fdobridge_: <karolherbst🐧🦀> and even then I have my doubts
23:51fdobridge_: <karolherbst🐧🦀> google tried, they failed, sooo....
23:51fdobridge_: <redsheep> They have an obligation to the shareholders to do things that make money, and not do things that won't, so it makes sense. Not saying I like it, but it's the reality we all live with.
23:52fdobridge_: <karolherbst🐧🦀> I mean.. on top of that is that they just don't want to open up 😄
23:52fdobridge_: <karolherbst🐧🦀> probably
23:55fdobridge_: <pavlo_it_115> And what about nouveau-fw?
23:56fdobridge_: <pavlo_it_115> If I'm not mistaken, the binaries were simply extracted from the nvidia installer
23:56fdobridge_: <pavlo_it_115> and likewise someone here extracted binaries for reclocking
23:57fdobridge_: <redsheep> As airlied mentioned earlier the nouveau firmware is bespoke, meaning it required its own engineering effort to make it exist. It is special firmware.
23:59fdobridge_: <karolherbst🐧🦀> only the GSP stuff is something nvidia published
23:59fdobridge_: <karolherbst🐧🦀> and that's also something we've talked about with them for years