03:02DemiMarie: mareko: why is gfx11 unlikely to be useful?
03:04DemiMarie: To me, this solves a very serious stability problem: on AMD GPUs, any process that times out typically kills the whole desktop session, as most compositors cannot recover from a context loss.
03:20DemiMarie: Is the problem that these devices cannot handle preemption when a page fault is pending?
03:20DemiMarie: That can be solved with "no page faults"
03:21DemiMarie: For context, GTK developers consider GPU resets to be a hardware/driver bug and aren't interested in supporting recovery.
08:40MrCooper: colour me skeptical that's a tenable position though
16:38DemiMarie: MrCooper: Complain to the GTK developers, not me. They want to treat GPU buffers as immutable.
16:40DemiMarie: MrCooper: Personally I think that innocent-context resets *are* a driver bug.
16:40DemiMarie: Or a hardware bug.
16:42DemiMarie: The GPU and hardware should ensure that contexts cannot interfere with each other.
16:43DemiMarie: Actually, I think in my use-case (Qubes OS) this might not matter, because Qubes OS will only support AMD GPUs with process isolation for security reasons. With process isolation, there is only ever one context running on the GPU at a time. Of course, dGPUs losing VRAM is still a problem but I think that is just a hardware/firmware/etc bug.
17:00MrCooper: I did express my opinion to them
17:36DemiMarie: MrCooper: My hope is that a Red Hat customer pays Red Hat to make GTK and Mutter robust to device loss.
18:21mupuf: DemiMarie: innocent context resets will be greatly diminished in a near future
18:23DemiMarie: mupuf: is that the user queue work?
18:23mupuf: we had two meetings this week with Christian, pepp, and more AMD and RADV people on this very topic
18:24DemiMarie: nice
18:24mupuf: userspace command submission is orthogonal to the handling of resets
18:24DemiMarie: Ah, okay
18:24DemiMarie: I thought userspace command submission was needed for per-queue reset. I’m glad to know I was wrong.
18:24mupuf: well, not fuuuuuullly
18:24mupuf: yeah, no, we can and will do much better across the board
18:25DemiMarie: That is great.
18:25mupuf: well, not sure how far back it will go, but we'll see
18:26DemiMarie: RDNA1 and later would be enough I think
18:27mupuf: it will never be perfect because the hardware is deeply pipelined and more than one context is in flight at all times, but there is plenty we can do to kill the hung context and keep the rest running
18:27mupuf: I would prefer Polaris10+, but we'll see
18:29mupuf: we have a plan to make it happen, and it will take many hands to succeed
18:30mupuf: IGT tests, CI, continuous bug filing, and bug fixing
18:30DemiMarie: mupuf: actually, in my use-case there should not be more than one context in-flight
18:31mupuf: on top of the first implementation
18:31DemiMarie: Qubes OS will be forcing process isolation on for security reasons
18:31DemiMarie: hopefully that will make things better
18:31DemiMarie: still, I am very glad that you want to improve this.
18:31mupuf: DemiMarie: oh, that's a separate feature, yeah
18:32DemiMarie: mupuf: LeftoverLocals and Whispering Pixels are very in-scope for the Qubes OS threat model, which is why I was so glad to see that the process isolation feature arrived.
18:34mupuf: yeah, for desktops, isolation is a must. Screw the bubble in the pipeline
18:35mupuf: hardware should evolve in this direction though, so the perf hit is only temporary... unlike the IOMMU's...
18:35DemiMarie: how big a hit is the IOMMU?
18:36mupuf: depends on the app. Nothing to 50%
18:36DemiMarie: What is it determined by?
18:36DemiMarie: Also, does huge pages on the CPU side change anything?
18:36mupuf: the size of the dataset and access pattern
18:36mupuf: this is mostly problematic for APUs
18:37mupuf: discrete GPUs shouldn't be so affected... although with resizeable bar 🫣
18:38mupuf: yeah, huge pages could help, but GPUs don't support them AFAIK... unless your name is broadcom
18:38DemiMarie: Christian wrote that AMD GPUs do have huge pages in their own MMUs, and the IOMMU also supports them I believe.
18:39DemiMarie: Whether the driver supports them is obviously another matter
18:39mupuf: oooooooh, that is interesting
18:39DemiMarie: That is one reason that Xen PV + GPU = sadness
18:39mupuf: we disabled it for the deck because of the perf hit... and the limited security concern
18:39DemiMarie: no huge pages in Xen PV
18:39DemiMarie: Security concern?
18:40DemiMarie: Ah, “it = IOMMU”, not “it = huge pages”
18:41mupuf: well, a gaming console doesn't need much confidentiality between apps... especially when they all run with the same UID
18:41mupuf: IOMMU
18:41mupuf: just sucks that we can't catch driver bugs
18:41DemiMarie: mupuf: A gaming console should have confidentiality between applications, and this should be enforced by a sandbox (perhaps provided by Steam)
18:42DemiMarie: whether or not they do is another question
18:42DemiMarie: Also firmware bugs
18:43mupuf: but FYI, we run in CI with the IOMMU enabled, IIRC
18:43mupuf: to catch some of these bugs
18:43DemiMarie: nice
18:43DemiMarie: My favorite thing about GPUs is probably that they do not execute speculatively, because they have no need to.
18:44DemiMarie: So no Meltdown and no Spectre outside the firmware and kernel driver.
18:45DemiMarie: I'm really glad that AMD GPUs will be losing their reputation for bringing down desktop environments.
18:47mupuf: there are plenty more side channel attacks, but at least no.speculation
18:47mupuf: perf counters, are insanely powerful on gpus
18:48mupuf: but, what you should worry about is leaking memory between processes
18:49mupuf: amdgpu doesn't zero out VRAM pages before handing them o another process, IIRC
18:50mupuf: same with registers, and other. They've been designed for throughput, not sharing resources
18:53DemiMarie: mupuf: that is what process isolation is for
18:53DemiMarie: not zeroing memory is something that is going to be dealt with
18:53DemiMarie: on the kernel side
18:53DemiMarie: is there a way to just disable the perf counters?
18:54DemiMarie: or wipe them at context switch?
19:45mupuf: On Intel and nvidia GPUs, the global counters are mediated by the kernel. Intel adds some random noise and limits the refresh rate to mitigate the problem. For per-context counters, yeah, i'm sure they are reset in the preambles