14:42 imanho: :karolherbst is this the interesting parts of the driver for me?
14:42 imanho: https://github.com/torvalds/linux/blob/88808fbbead481aedb46640a5ace69c58287f56a/drivers/gpu/drm/nouveau/nouveau_mem.c#L36
14:43 karolherbst: imanho: this is just how we do it, nvidia has different ioctls
14:43 karolherbst: but yeah.. generally if the driver allows you to mmap that stuff then yes
14:43 karolherbst: _but_
14:44 karolherbst: generally all of this is very driver specific
17:06 karolherbst: imirkin: I bet that the difference between init_on_free and init_on_alloc is, that this is purely for kernel memory and with init_on_free=1 you'd still get random values from userpace maybe?
17:07 imirkin: like i said - needs investigation.
17:07 karolherbst: yeah..
17:07 karolherbst: anyway.. I assume we have some if check which does the right thing with random values in it :p
17:08 karolherbst: this happens on suspend/resume, right?
17:08 imirkin: yes.
17:08 karolherbst: nice... let me try my tesla GPUs then
17:09 karolherbst: got a new mobo, hopeing that those legacy BIOS card still work :D
17:11 karolherbst: ehh.. not cool
17:12 imirkin: ?
17:12 imirkin: i'm guessing it's less-than-fully-working?
17:12 karolherbst: it does boot at least
17:13 imirkin: just no display until nouveau loads?
17:13 karolherbst: seems like it
17:13 karolherbst: efifb does load though
17:14 imirkin: makes sense
17:14 imirkin: efifb doesn't know anything about displays
17:14 imirkin: it just talks to the efi firmware
17:15 imirkin: which is there :)
17:15 karolherbst: ehh.. seems like my display was just annoying afterall
17:15 karolherbst: now it works after unplugging it
17:16 imirkin: oh heh
17:17 karolherbst: the serial console on this board is strange
17:20 karolherbst: mhhh
17:20 karolherbst: at least on my G80 it does seem to work over DVI
17:21 karolherbst: imirkin: anything specific what those systems have in common coming to your mind?
17:21 imirkin: you're testing with a literal G80?
17:21 karolherbst: yes
17:21 imirkin: lol
17:21 imirkin: well, fwiw i think all the failures were on like G92 or G96's or later
17:22 karolherbst: okay
17:22 imirkin: at least the GPIO stuff changed around a bunch
17:22 imirkin: also G80 didn't have a PCIPHER
17:22 karolherbst: ahh nice.. passively cooled G92 here
17:22 imirkin: whereas G84+ do (with a handful of weird laptop exceptions where it's fused off)
17:23 imirkin: and it's used for copying stuff around, esp on resume
17:23 imirkin: also are you doing suspend-to-ram or disk?
17:23 karolherbst: to ram
17:23 imirkin: ok. i think that's what the other people were doing
17:25 karolherbst: mhhh
17:25 karolherbst: something is up
17:25 karolherbst: gnome stopped doing... anything
17:25 karolherbst: as in it doesn't login completely
17:27 karolherbst: ahh yeah
17:27 karolherbst: imirkin: kasan complains loudly
17:27 imirkin: yay
17:27 karolherbst: ehh wait
17:27 karolherbst: it's not kasan
17:27 karolherbst: soft lockup
17:27 imirkin: whatever it is, it's loud ;)
17:27 karolherbst: https://gist.github.com/karolherbst/87224e5e01de344a7fb0f08583bb01b6
17:29 karolherbst: but gnome already doesn't start, so I assume there is something up for real
17:30 karolherbst: can also be this backlight regression ...
17:30 karolherbst: let's see what git bisect says
17:35 karolherbst: ehhh... I hope my CPU even boots on older kernels :O
17:38 karolherbst: mhh [ 395.589275] nouveau 0000:01:00.0: fb: trapped read at 4f31764800 on channel 3 [0f8c4000 Xorg[7789]] engine 00 [PGRAPH] client 0a [TEXTURE] subclient 00 [] reason 00000000 [PT_NOT_PRESENT]
17:38 karolherbst: fun
17:38 karolherbst: G80 _did_ work
17:38 imirkin: G80 uses m2mf to do copies, fwiw
17:38 imirkin: rather than PCIPHER
17:39 karolherbst: sure
17:39 imirkin: (which is a copy engine part of video unit, which can also do encryption)
17:39 karolherbst: but something mm related is completely broken as it seems
17:39 imirkin: yeah, people have been changing stuff around there and making lots of changes to nouveau, with what i assume is zero testing
17:40 imirkin: this is why i keep an older kernel on my box -- upgrading generally causes things to not work
17:40 karolherbst: wouldn't be the first regression
17:40 imirkin: i'm on 5.6 now which has proven fairly solid
17:40 karolherbst: uhhh
17:40 karolherbst: I am sure I won't even be able to boot on that one :D
17:41 karolherbst: well maybe boot
17:41 karolherbst: but...
17:41 karolherbst: got a new i7-12700 here and I could imagine that the x86 code isn't ready for it
17:41 imirkin: yea no clue
17:41 imirkin: in general these things are pretty backwards-compatible
17:41 karolherbst: right... but mono also broke on bit.LITTLE configs due to weirdo cache line assumptions
17:42 karolherbst: not sure how much the cores are difference on alderlake, but...
17:43 karolherbst: at least the L2 cache is different in size
17:49 karolherbst: mhhh...
17:49 karolherbst: now I got a NULL pointer access on 5.14...
17:49 karolherbst: bbut we also had this other weirdo ttm bug
18:42 karolherbst: imirkin: ehh... I tested on the wrong branch :(
18:42 karolherbst: I was missing the ttm bugfix
18:42 imirkin: oops
18:42 karolherbst: yeah.. I thought drm-misc-next would be good, but...
19:30 karolherbst: mhhh.. my machine doesn't suspend though...
19:30 karolherbst: ahh now it did.. that took long
19:31 karolherbst: mhh "[ 127.352945] nouveau 0000:01:00.0: fb: trapped read at 0000000d00 on channel 3 [0f8c4000 Xorg[6330]] engine 00 [PGRAPH] client 05 [CCACHE] subclient 00 [CB] reason 00000002 [PAGE_NOT_PRESENT]"
19:31 imirkin: that's ... a very low address
19:31 imirkin: feels like where the PTE's live?
19:32 karolherbst: might be
19:32 karolherbst: trying without init_on_alloc=1 now
19:32 imirkin: er, PDE's rather
19:32 karolherbst: I hope init_on_alloc doesn't affect device memory in weird ways...
19:33 karolherbst: wondering if kasan even triggers on init_on_alloc...
19:35 karolherbst: imirkin: the thing is.. I think the kernel aborts suspending
19:36 karolherbst: https://gist.githubusercontent.com/karolherbst/40a0c4e0f196800780244501a86d0df2/raw/7ff98f97c4668b275e93a69cb22f0cadaf74b06c/gistfile1.txt
19:36 karolherbst: not sure if this is purely informational or if something gives up
19:36 imirkin: hehe
19:36 imirkin: no, that's something that can def happen
19:36 imirkin: you're probably the first person trying suspend-to-ram with a G80
19:37 imirkin: and only like the second trying nouveau with a G80 :)
19:37 karolherbst: ehh it's a G92 now
19:37 imirkin: oh
19:37 karolherbst: the G80 worked fine
19:37 imirkin: interesting
19:37 imirkin: what if you boot with nouveau.config=cipher=0
19:37 karolherbst: welll.. I think.. didn't dry on 5.17-rc2
19:37 imirkin: that should make it use m2mf for copies too
19:38 karolherbst: our error handling is catastrophic :(
19:38 karolherbst: I think we need to be better about when to give up.. like if a channel is throwing out errors constantly every second for minutes, maybe we should just let it bet and kill it :D
19:42 karolherbst: imirkin: sure that this option still exists?
19:42 imirkin: i'm sure that an option like it exists
19:42 imirkin: but perhaps not spelled precisely like that
19:42 karolherbst: mhh
19:42 imirkin: maybe cypher?
19:42 imirkin: or something else?
19:43 imirkin: or crypt maybe?
19:43 karolherbst: ahh maybe
19:43 imirkin: yeah. crypt sounds more like it actually
19:43 imirkin: dunno. should be able to rtfs to sort it out
19:43 imirkin: there's also a wiki page that covers it
19:43 karolherbst: at least it rings a bell
19:44 imirkin: oh, but it hasn't been updated for the new terminology
19:46 karolherbst: then it should be chipher, as the engine is called cipher
19:47 karolherbst: but booting with it didn't change anything
19:47 imirkin: ok. that's what i thought it got updated to
19:47 imirkin: ok
19:47 imirkin: on boot iirc it says what engine it uses for copies
19:47 imirkin: does it still say that it uses cipher for copies?
19:47 imirkin: hopefully not :)
19:47 karolherbst: "[ 16.146750] nouveau 0000:01:00.0: DRM: MM: using M2MF for buffer copies"
19:47 imirkin: ok. so it worked.
19:48 karolherbst: yeah... okay, but suspend taking like a minute is wrong regardless
19:48 karolherbst: but kasan doesn't print anything, so most likely a valid bug somewhere
19:52 karolherbst: imirkin: but it seems to be quite the old regression :/
19:53 karolherbst: "Downgrading to 4.19 (4.19+105+deb10u11) works. I had similar problems with 5.8." :O
19:53 imirkin: well, the init_on_* thing got introduced in 5.2
19:53 imirkin: (or thereabouts)
19:53 karolherbst: mhhh
19:53 imirkin: which is what "caused" the problem
19:53 imirkin: (obv not, but made it apparent)
19:53 karolherbst: so might be broken since forever
19:53 imirkin: yes.
19:53 imirkin: which is why i was more keen on understanding how the new feature works precisely
19:54 imirkin: to understand what things it can affect
19:54 imirkin: to hopefully reduce the scope from "everything"
19:54 karolherbst: ohh wait...
19:54 karolherbst: some user was smart
19:54 karolherbst: apparently it stalls inside nv50_disp_atomic_commit_tail
19:54 imirkin: there's a user who wrote a book-length report on his findings
19:54 karolherbst: yeah
19:54 imirkin: tbh i didn't have the time to go through it all
19:55 imirkin: but it seemed like he made progress in investigating stuff
19:55 imirkin: if not actual progress in figuring out what the issue is
19:56 karolherbst: thing is.. I don't get the eviction failed message
19:59 karolherbst: but it feels like that memory is trashed
20:00 karolherbst: "TRAP_MP_EXEC - TP 0 MP 0: 00000010 [INVALID_OPCODE] at 07ff80 warp 0, opcode fffeffff ffffffff" :)
20:01 imirkin: certainly doesn't look _great_
20:01 imirkin: it's consistent with his findings about ttm stuff somewhere
20:01 imirkin: feels like we're not restoring stuff correctly?
20:01 karolherbst: no
20:01 karolherbst: I think we do something terrible at suspend
20:01 karolherbst: why does it take a minute for starters?
20:02 karolherbst: if it takes like 5 seconds.. okay, not great, but not terrible
20:02 karolherbst: but it shouldn't take longer than that
20:22 karolherbst: and something is up with the linux serial console driver :(
20:22 karolherbst: at random points it starts printing garbage
22:15 karolherbst: imirkin: mhhhh.. I suspect it's something channel related
22:16 karolherbst: sooo.. suspending without anything running besides the console seems fine
22:16 karolherbst: maybe we mess up halting/stopping channels or they continue too soon or something stupid like that?
22:16 karolherbst: but that still doesn't explain why suspending takes so long if something is running
22:23 imirkin: sounds like a whole barrel of fun you just poked
22:23 imirkin: esp since none of this is reproducing the original issue people were having :)
22:24 karolherbst: esp since my serial console really messes up... well... on the software side :(
22:27 karolherbst: ohhhh... the heck is it doing
22:28 karolherbst: https://gist.githubusercontent.com/karolherbst/336120bd3f63a298a0f131a62a7f9830/raw/41ea96f47582baad333378a06286421a412b4c7b/gistfile1.txt
22:29 karolherbst: mhhh.. maybe some pins are wrongly connected after all?
22:31 karolherbst: mhhh
22:31 karolherbst: no.. seems alright
22:31 karolherbst: weird
22:32 imirkin: could be control flow setting?
22:32 imirkin: 8N1 is the common thing
22:32 karolherbst: console=ttyS0,115200n8 on the host
22:32 imirkin: hardware vs software control flow? dunno
22:32 karolherbst: both disabled
22:32 imirkin: maybe the cable can't handle it?
22:32 karolherbst: it usualy works
22:32 karolherbst: I just get random crap
22:32 imirkin: 115.2 is high
22:32 imirkin: try 56k?
22:33 imirkin: or even, *gasp*, 9600
22:33 karolherbst: well.. it does work perfectly fine up to a point where my client interprets random crap in a way the client breaks
22:33 karolherbst: but as you see I get dmesg putout
22:33 karolherbst: *output
22:33 karolherbst: and also sysrq
22:33 karolherbst: and random stuff
22:33 imirkin: right. it works perfectly until it doesn't
22:34 karolherbst: in minicom it looks like this e.g.: https://gist.github.com/karolherbst/6efb57f95248188b94d88e81f43bf86e
22:34 karolherbst: and I used the same settings on the old mobo
22:34 imirkin: yeah
22:34 imirkin: welcome to the world of hardware
22:34 imirkin: hardware sucks.
22:34 karolherbst: anyway, this one has a proper plug so I don't even have to rewire
22:35 karolherbst: and if the pins were mirrored I shouldn't even get anything
22:38 karolherbst: I am trying with flow control enabled
22:38 karolherbst: maybe that works better
22:41 imirkin: i've always done it with hardware flow control
22:41 imirkin: coz i use the fancy new-fangled 16650 UARTs :)
22:41 karolherbst: heh.. now it blocked boot :D guess I can't hit random keys
22:43 karolherbst: something is up though as I can't send anything ...
22:44 ccr: I remember running into odd case with some HP network switches, if they were already powered on, the serial interface would not sometimes work. or at least it didn't work with linux PC box + minicom. sometimes it started working when banging random keys/data, sometimes it required booting the switch while the serial cable was connected.
22:44 karolherbst: ccr: yeah.... it's weird
22:45 karolherbst: I think my laptop is sending random crap... or the usb thingy
22:46 ccr: never know about those, somehow I have a gut-feeling that the modern cheap usb-to-serial implementations are probably worse than "real" integrated serial chips
22:47 karolherbst: well...
22:47 karolherbst: sooo
22:47 ccr: but perhaps there are good ones and bad ones, shrug
22:47 karolherbst: the botherboard has a real one on it
22:47 ccr: botherboard :D
22:47 karolherbst: but to communicate with it I have a RS232 usb thingy
22:47 karolherbst: I receive everything correctly as it seems, but sending stuff doesn't really work
22:49 ccr: I have one of those usb-serial converters as well, due to laptop not having a rs232 port. I can't remember which chip it has but it is supported by kernel.
22:52 karolherbst: mhhh maybe getty is causing issues here
22:52 imirkin: make sure the getty config matches
22:52 imirkin: iirc the default for getty used to be like 9600 or 38400
22:53 karolherbst: I do see the proper login prompt though
22:53 imirkin: oh
22:53 imirkin: perhaps it's not expecting you to send with some setting? dunno
22:53 karolherbst: no idea
22:54 karolherbst: ehhh....
22:54 karolherbst: now ssh doesn't start
22:54 karolherbst: *sshd
22:57 karolherbst: nope.. it was nm
22:58 karolherbst: okay...
23:01 karolherbst: or maybe it's systemd writing to the console...
23:15 karolherbst: yeah... I think it's getty