02:40AndrewR: ..it seems blender 2.79 performance settings->system tab can freeze my g92 by using mesa git. I tried few older version and it seems problem started between 40a554c8dd06d24cbf56b919ea0a08ffdc7a4e5b and 8ecace073ea68e3819ea7c5637bd479067e1420b (9-13 Jan 2021). I hope to bisect but it may take some time ...
02:40imirkin: woohoo, i must have broken it with all my fixes
02:41imirkin: i definitely see my name in there :(
02:42imirkin: sorry :(
02:43imirkin: AndrewR: might be this guy? ec668e2fd142db27dfa9ea1084005db328889721
02:43imirkin: or this? c0171c4626319ae6822f9c490d9118d5caf43246 ? but seems super-unlikely
02:44AndrewR: imirkin, not sure yet, I just started compile of first bisection step ...let it finish?
02:44imirkin: sure
02:47AndrewR: imirkin, I had old dri install on liveDVD, but was surprized just replacing nouveau_dri.so there with new version doesn't introduce bug. Still moving in nouveau_pipe.so did this ... so first I looked for st/mesa commits .... may be I was wrong.
02:47imirkin: let me know what broke - happy to (try to) fix
03:01AndrewR: imirkin, https://cloud.mail.ru/public/ww6k/mhfLh5kh6 - there is apitrace (3 mb) from working mesa version - but be aware it may hang your gpu... sometimes I was able to kill blender and sgutdown, but if it hangs for first time with dmesg complain it usually will hang again harder on second try.
03:01imirkin: AndrewR: ok thanks. i have a G84 but it's not my primary
03:01imirkin: but if i disconnect, i hope you'll understand :)
03:02AndrewR: imirkin, yeahh ....if you need your machine for something this is not urgent .....
03:02imirkin: it's night
03:02imirkin: i'll need it for work tomorrow
03:03imirkin: but hopefully it'll recover by then ;)
03:03imirkin: AndrewR: replays fine fwiw
03:04imirkin: with the latest master
03:04imirkin: + a couple patches, mostly unrelated
03:05AndrewR: imirkin, good ... do you use 64-bit mesa? I have weird setup with 64b kernel + 32b mesa compiled with clang ..... it may complicate something
03:05imirkin: yes, 64-bit userspace
03:05imirkin: doing 32-bit would not be trivial for me
03:05imirkin: achievable, but non-trivial
03:06AndrewR: imirkin, then lets see where my bisect lands .....
03:06imirkin: AndrewR: does replaying the trace trigger the issue for you?
03:07AndrewR: imirkin, I did it on working mesa just now, so not yet tried to reply on known bad mesa ...let me make cook (compile) mesa again :}
03:08imirkin: ok
03:08imirkin: could be something that doesn't end up causing a problem when replaying the trace
03:08imirkin: or could be something that only happens in your setup
03:13AndrewR: ..X restart ...
03:20AndrewR: ..so, " iris: Delete iris_resolve_conditional_render" seems to be good ... now testing 4439757db23490b9a4b75487d699470cdd7ebcf4 (it seems to be slightly above those nv50 commits .. )
03:21imirkin: actually i didn't check it with today's commits
03:21imirkin: only yesterday's i think? or day before?
03:24AndrewR: imirkin, I think bug was around (for me) for few days already ..I was checvking another tab usually :}
03:25imirkin: oh no, it was this morning
03:25imirkin: i rebased on test out the drm shim
03:45AndrewR: so, 4439757db23490b9a4b75487d699470cdd7ebcf4 seems to be fine as well ..... (and so all commits before it, incl. those nv50- ones .... ?) There is one more nouveau-related commit just below cut line ....mmm..intrigue!
04:16AndrewR: sadly replaying this trace (blender.bin) doesn't trigger bug, while real Blender run hang GPU now ....
04:17imirkin: =/
04:17imirkin: that's what i kinda figured
04:17imirkin: also isn't blender 2.79 super-old now?
04:19AndrewR: imirkin, sure, but I rarelyy use it, mostly for looking into files my friend send me from time to time (he uses 2.79 + 2.80x ... on win)
04:19imirkin: not a problem, just pointing it out
04:20imirkin: the one in portage is 2.83
06:23AndrewR: imirkin, sorry: reverting 28a781323fba87e6e338cfecb0b6fe25a08f61a4 ("nouveau: change fence destruction logic on screen destroy") seems to fix my blender hang ... (note, this machine also has two nv50-class gpus)
15:31imirkin: AndrewR: gah, ok
15:50imirkin: AndrewR: ok, we need to quickly revert that
15:50imirkin: it was for drm-shim's benefit... sigh
15:50imirkin: i guess this stuff is more subtle
15:50imirkin: there's some case i didn't consider
15:51imirkin: AndrewR: that's very weird though ... this gets triggered at, essentially, program exit
15:51imirkin: although blender might do something weird
15:51imirkin: and create/destroy a few screens first
15:51imirkin: hm
15:53imirkin: skeggsb: does anything bad happen if we destroy a channel while stuff is running on it? will the kernel "handle it", or does userspace need to be careful about it?
16:31karolherbst: imirkin: if you submit a new pushbuf the kernel returns an error
16:31imirkin: karolherbst: what about the old ones?
16:31karolherbst: well, you already submited them
16:31imirkin: that were already submitted and are in the middle of executing
16:31karolherbst: well, they won't finish
16:31imirkin: does destroying the channel screw anything up for them
16:32imirkin: what if you start GEM_CLOSE'ing stuff
16:32imirkin: etc
16:32karolherbst: I was planning on handling all that much nicer
16:32karolherbst: but we also have a way to "sync" on a submission, but that's mainly for debugging
16:32imirkin: this has nothing to do with multi-threading
16:32karolherbst: but we kind of have to handle the case where we know the context is gone
16:32imirkin: this has to do with screen shutdown
16:33imirkin: anyways, i sorta assume you don't have a precise answer. that's why i was hoping skeggsb would answer.
16:55AndrewR: imirkin, if it works on your side or with different gpus - then I can live with local revert - I have 72 patches for patch am anyway :} (cl stuff and some random series I like to test today)
16:57AndrewR: imirkin, (testing/bisecting for this problem was done without any of my patches, apart from two related to buildability )
17:05imirkin: AndrewR: if it's a problem for you, it's a problem for others
17:05imirkin: i don't understand what the problem is, and i don't run this code live.
17:06karolherbst: AndrewR: what's the thing breaking btw?
17:06imirkin: blender 2.79
17:06karolherbst: ahh.. :/
17:06imirkin: (maybe other versions too, but that's the one he uses)
17:08AndrewR: karolherbst, not just blender, but attempt at looking at 'system' preferences pane .... where I already set some aa and stuff ..... may be this is important. but apitrace replay of very same actions doesn't hang like real thing here .....
17:09imirkin: yeah, it's probaby creating/destroying contexts
17:11karolherbst: probably
17:11AndrewR: imirkin, but does 2.83 works for you? Or 'not compiled yet' due to surprizingly-heavy dependencies? :}
17:13imirkin: well, definitely not compiled yet
17:13imirkin: but due to not wanting it ;)
17:18AndrewR: imirkin, I was intrigued by how Openshot tries to create animated titles via blender, but as it turned out this process is ...slow .... surprizingly slow. (at my amd fx 4300)
17:43AndrewR: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4223
17:43AndrewR: just so it will not lost
20:46skeggsb: imirkin: the drm submits a fence and waits on it before telling nvkm to tear the channel down
20:47imirkin: skeggsb: ok, so modulo bugs, userspace can submit a pushbuf, and then immediately tear down the universe, and all should be well, right?
20:47skeggsb: that's where those "failed to idle channel" messages come from on dead channels
20:47skeggsb: yes, in theory
20:48imirkin: AndrewR: to confirm - does blender hang, or does the computer hang?
20:48skeggsb: in an ideal world we'd just tear the channel down without waiting, but at least on some chipsets, nvkm doesn't do enough to ensure that's safe right now
20:49imirkin: skeggsb: the change that seems to trigger issues is when tearing down nouveau in screen_destroy
20:49imirkin: it used to wait for all (user) fences to complete before triggering work / tearing things down
20:49imirkin: but i changed it to not do that anymore
20:50imirkin: so now it instantly triggers all the work and destroys the context/etc
20:51skeggsb: does it still blow up if you don't delete the engine objects and just kill the channel? there's some potential issues there
20:51imirkin: and i think some (esp older) software does a funny dance with GLX which ends up causing screens to be created/destroyed on start
20:51imirkin: skeggsb: unfortunately i can't repro the issues, but i could try to write a patch which leaks those
20:52imirkin: (obv not a permanent solution, but it's a thing to test)
20:52skeggsb: (that, also, *should* work, but probably doesn't on some chipsets for the same reasons we need a fence before channel destroy)
20:52skeggsb: hopefully fixed "soon"...
20:52imirkin: this is on G92
20:52imirkin: in case it matters
20:52skeggsb: yeah, very likely that one's broken
20:52imirkin: so why don't i see the breakage on a G84?
20:53skeggsb: yeah, those use identical code for everything relevant there, so i'm not sure
20:53imirkin: hm, i'll try some stuff maybe a bit more directed
20:53imirkin: e.g. just like do the bringup, submit something, destroy, repeat
20:54skeggsb: yeah, could just be timing, the g84 perhaps gets through the pushbuffer faster for whatever reason
20:54imirkin: hrm
20:54imirkin: i'll try submitting lots of stuff then ;)
20:54imirkin: big draws
20:54skeggsb: worth a try, who knows though, could be a bunch of things
20:55imirkin: yeah. would definitely prefer having a repro
20:55imirkin: than just 'guess and check'
20:56skeggsb: in any case, it *should* work fine without causing the world to end (in the case of deleting engine objects that are still in use, channel errors would be expected, but they *should* be harmless to the system, but quite likely not on some chipsets)
20:56skeggsb: working on it...
20:58imirkin: ok cool
21:13AndrewR: imirkin, sorry was distracted by other things. Blender hangs GPU, usually - i can move mouse cursor but not much else. acpi poweroff work
21:14imirkin: ok, so that's consistent with a hung gpu + failed recovery
21:14AndrewR: imirkin, _sometimes_ I was able to switch to ttry and see dmesg/kill blender. But not lately :(
21:14imirkin: yea
21:14imirkin: check my suggestion in the bug you filed
21:14imirkin: if you're willing to experiment some more
21:16AndrewR: imirkin, I assume I should comment this block with hangy (unreverted) mesa ?
21:16imirkin: yes
21:16AndrewR: I guess russian support will not answer too fast ( around midnight here) so I'll risk :}
21:17AndrewR: (was buying tickets online)
21:17imirkin: skeggsb: btw, assuming that not destroying the engines works ... what's the recourse?
21:17imirkin: AndrewR: well, no big rush
21:17imirkin: skeggsb: i guess there's nothing we can really do without a kernel change...
21:18skeggsb: imirkin: just free() them instead, the kernel will clean them up
21:18imirkin: aha right
21:18imirkin: and we'll "leak" them for the duration of the process, but ... wtvr
21:18skeggsb: ah yeah that's a potential problem
21:18imirkin: well, only a problem if you create tons and tons of these
21:18imirkin: not sure what would cause that
21:19imirkin: maybe can do something clever with the loader
21:20skeggsb: even if the kernel didn't screw up and cause a hang, you'd still risk channel errors in dmesg, it's userspace's responsibility to make sure stuff isn't being used before it kills them
21:20imirkin: hrmph ok
21:20imirkin: annoying
21:20imirkin: ooh, can i do a nouveau_bo_wait on something?
21:20imirkin: here's the problem i'm trying to solve:
21:20imirkin: drm-shim is basically "emulating" some of these ioctl's
21:20imirkin: obviously doing any sort of fence wait with the shim ain't gonna work
21:21imirkin: so my solution to that problem was "just delete stuff instead of waiting"
21:21imirkin: is there a way to wait for things to quiesce via kernel api's?
21:21imirkin: (since then the shim would just return "yup, it's done" immediately)
21:21skeggsb: mmmm.... perhaps waiting on the pushbuf bo
21:22imirkin: hmmmmmm. yeah, that's a good idea. unfortunately not easy to implement since that's not exposed iirc.
21:22imirkin: but maybe a libdrm bump
21:23karolherbst: ohhh, that might be indeed a fine addition
21:24imirkin: AndrewR: i'll have some more patches for you tonight. sorry for the troubles =/
21:25AndrewR: imirkin, sorry for discovering this that late!
21:26imirkin: thanks for discovering it at all =]
21:34AndrewR: it hanged .....
21:34imirkin: bleh
21:34imirkin: sorry
21:34imirkin: well, there could be other problems
21:34AndrewR: but again, I have reclocking patch in kernel .... so, it partially reclocked GPU (may play role too) will add to bug
21:34imirkin: i'll try to make something more robust
21:35imirkin: do you increase or decrease clocks?
21:41AndrewR: i think g92 only can go up .....
21:42AndrewR: *my* g92 - thabnks to vbios being different on different cards ...
21:43Lyude: hey btw -if any folks here are interested are interested https://lore.kernel.org/dri-devel/CAHUNapTB1tt6T931LfBWVWreXGFwd6tTPqH58i7s3WKivCDT4g@mail.gmail.com/ if anyone has ideas for shorter GSoC projects for this year, we're still looking for responses to the emails we sent out about this
21:46imirkin: AndrewR: ah ok. ben's theory was that slower boards would be more susceptible to this
21:46imirkin: you G92 should be better in every way than my G84 though
21:47imirkin: although maybe you're only reclocking some things and not others. dunno
21:47imirkin: i'm going to run some experiments to see what all i can repro
22:15imirkin: AndrewR: ftr, that memory reclock doesn't seem to work
22:16imirkin: note that the memory clock stays at 500mhz, not the desired 850mhz
22:16AndrewR: imirkin, yeah .... but I have no knowledge to fix this :} :(
22:16imirkin: i bet RSpliet has been *dying* to fix this...
22:17AndrewR:hopes this is metaphor
22:17imirkin: heh
22:17imirkin: more like needling him
22:17imirkin: it's something he looked at a while back
22:17imirkin: like ... 5y ago
22:17imirkin: heh
22:18imirkin: this stuff is all beyond old
22:18imirkin: the heyday of these GPUs was late 2000's
22:18imirkin: i.e. 2007-2009
22:19AndrewR: imirkin, yeah, but this consequence: you can buy one for 1000 rubles and not for say 5000 :}
22:19imirkin: heh, yeah
22:19imirkin: sounds like about the same as the rates in USD
22:20imirkin: $10-ish
22:25AndrewR: imirkin, I recall problem with mem reclock was due to now hard it was/is to capture all those subtle operations between engines, controlled via firmware (memory link training?). Same for instability problem on fermi ... sadly there is no ext. x-scope you can point at gpu :}
22:26imirkin: it's not so hard on the earlier gpu's
22:27AndrewR: imirkin, I still can find some way to boot livecd + blob, if there are ideas for testing ....
22:28imirkin: AndrewR: i'll have some patches tonight
22:28imirkin: not this second, sorry
22:28AndrewR: imirkin, ok, I'll probably will stay 'up' for some quite time because I spend day sleeeping
22:29imirkin: ehm
22:29imirkin: might be easier if you sleep at night
22:29imirkin: your call ;)
22:44RSpliet: imirkin: day sleeping tends to not be ones call funny enough
22:45imirkin: don't i know it.
22:45RSpliet: I had no say in the fact that I just slept from 19:00-22:30
22:45RSpliet: Now what have I been dying to fix?
22:45RSpliet: oh, DRAM reclocking on G93
22:45imirkin: reclock on earlier tesla
22:45RSpliet: ehh
22:45RSpliet: -1
22:46RSpliet: Yeah that just doesn't work. Don't think the DRAM reclocking code is even hooked up?
22:47imirkin: it's not
22:50RSpliet: Figured. Anyway, it's just like all the other DRAM reclocking code. Not difficult, just tedious.
22:50RSpliet: Biggest blocker for these <G98 cards is that it needs to be selective about which bits in which registers it touches. If a column of bits in the pstate table contains either all-0 or all-1, we shouldn't touch it.
22:50RSpliet: Not set it once, or unset it once, just leave it as it is
22:50RSpliet: Need a surprising amount of code for something that sounds so trivial
22:50imirkin: boo
22:51imirkin: can't you just add if's before emitting the memx_* stuff?
22:51skeggsb: the gk104 code handles that stuff somehow, though i don't remember how exactly
22:51RSpliet: I was thinking of a solution where the if's are done once, and decide the masks of bits that can be touched by memx_*
22:51skeggsb: as does the incomplete fermi stuff
22:52imirkin: sounds complicated
22:52imirkin: if's are cheap.
22:52RSpliet: yeah, GK10x handles it, NVAx didn't need it (mainly because we got lucky)
22:54skeggsb: if (ram->diff.rammap_11_0b_0400) {
22:54skeggsb: data |= cfg->bios.rammap_11_0b_0400 << 5;
22:54skeggsb: mask |= 0x00000020;
22:54skeggsb: }
22:54skeggsb: like that :P
22:55imirkin: skeggsb: btw, i assume any fermi plans are as stalled as they have been in the past?
22:55RSpliet: But a hundred times :-D Yeah
22:55skeggsb: yes, there's an inconvenient number of knobs and dials to deal with
22:55skeggsb: imirkin: tbh, i forgot it even existed
22:55imirkin: hehe
22:55imirkin: so even more stalled than before ;)
22:57skeggsb: dealing with that stuff requires a level of concentration and willingness to accept pain that is... rare lol
22:57imirkin: i don't blame you!
22:57imirkin: just checking
22:57skeggsb: keeping all the interactions paged in at once is difficult
22:58skeggsb: with hw that old, i really don't see why nvidia couldn't just go "here's the RM code as a reference, go for it"
22:58skeggsb: buuuut wishes be fishes
22:58RSpliet: I'm thinking the same thing about RM code for current gen HW, but then who am I...
22:59skeggsb: :P