00:22 imirkin: kherbst: did you play at all with the kms nvidia thing? i.e. can i just use kms and not worry about X for the mmiotrace?
00:24 gnarface: i don't think the nvidia binary drivers work with kms
00:24 gnarface: they don't appear to work
00:25 imirkin: they claimed they did
00:25 imirkin: or i understood that ;)
00:27 gnarface: like, recently?
00:27 gnarface: maybe it changed
00:31 imirkin: past year or so
00:36 kherbst: yeah, I think it should just work
00:37 kherbst: stuff like dracut doesn't work on my system for whatever reason
00:37 kherbst: soo I have no output until I get to X
00:37 kherbst: no idea if that is my fault or nvidias though as I also have efifb enabled
00:37 kherbst: some claim nvidia kms + efifb = fail
00:37 imirkin: ok. no efifb here.
00:37 imirkin: i guess i'll find out.
00:38 kherbst: do you have dm-crypt?
00:39 kherbst: could be challanging to put in your password with a display outputing nothing
00:43 imirkin: well, i'm not really planning on loading it until the system has booted...
00:47 kherbst: ahh
00:48 kherbst: imirkin: anyway, how much do you know about PCIe + turning off devices?
00:48 imirkin: zero.
01:04 kherbst: imirkin: mhh, sad. If we are really unlucky, we might have to RE most of the PCI mmio regs to fix suspend/resume issues :(
01:37 imirkin: kherbst: have you played with nvidia blob + kms at all?
01:37 imirkin: does it work?
01:37 kherbst: imirkin: yes, on my laptop with a gm206
01:37 kherbst: afaik it should just work
01:37 imirkin: you can run like modetest and it works?
01:37 imirkin: k
01:38 kherbst: well, nvidia has their own implementation
01:38 imirkin: sure
01:38 imirkin: but modetest uses the kms api's
01:39 imirkin: or is the nvidia kms separate from the upstream kms?
01:40 kherbst: I didn't really looked into the details
01:40 kherbst: apperantly you also need to set "nvidia-drm.modeset=1"
01:40 imirkin: k thanks
01:41 kherbst: and there is no fbdev :)
03:20 rhyskidd: does anyone have a mmt run file from Maxwell or Pascal? wanted to verify some things
03:41 zerothis: my laptop with NVIDIA GeForce GTX doesn't seem to know about the HDMI port, plugged or not
03:46 zerothis: NVIDIA Corporation Device 1c8d (rev a1)
06:54 Riastradh: imirkin: The sync buffer is getting assigned ttm buffer object offset #x60000. The bus address that we're mapping into the CPU, however, is BAR(1) + 0, not BAR(1) + #x60000. Does this seem suspicious to you?
06:58 Riastradh: nyef: ^
11:10 nyef: Riastradh: I'm unfamiliar with how things are supposed to work with memory mapping and DMA and whatnot.
12:37 pendingchaos: rhyskidd: "mmt run file"? do you mean a mmt trace?
12:38 pendingchaos: I have a bunch of them with my Pascal card
12:38 rhyskidd: yes, a mmt trace
12:38 pendingchaos: do you want one of anything particular?
12:38 rhyskidd: (not mmio trace, i have heaps of them)
12:39 rhyskidd: possibly one that is just driver bringup for a simple gl app, and a compute workload if you have them
12:45 pendingchaos: rhyskidd: should have some traces and their shader tests: https://drive.google.com/file/d/1pNGxY7ajdo3oyINlHdlCldwgOzEm95yF/view?usp=sharing
12:46 pendingchaos: though I think I can create something smaller than the fragment_shader_interlock.mmt
12:46 pendingchaos: it's slightly more than just driver bringup
12:50 rhyskidd: that's great, thanks
12:50 pendingchaos: https://drive.google.com/file/d/1L5stiLHQOMMOaelwxija18q11P3ZsVJp/view?usp=sharing < with a smaller gl app trace
13:32 kherbst: rhyskidd: you can't currently trace compute workloads with mmt
13:32 kherbst: I have some patches which parse the nvidia-uvm ioctl though by including nvidias headers
13:33 pendingchaos: ah, forgot about that
13:59 mslusarz: those traces don't use uvm and seem to decode fine using demmt from master though
14:53 imirkin: alright. phase 1 complete -- GP108 in computer.
14:54 HdkR: Almost done then :)
14:54 imirkin: Riastradh: have a look at how imem works
14:55 imirkin: HdkR: ... and nvidia driver installed. just have to LOAD the nvidia driver now, and convince it to modeset 4k@60
14:55 imirkin: [over hdmi]
14:55 HdkR: :D
14:57 imirkin: [ 53.064044] nouveau 0000:02:00.0: gr: intr 00000040
14:57 imirkin: skeggsb: --^ expected on a GP108? got 2 of them shortly after boot
15:01 imirkin: Lyude: i don't need this info *now*, but with DP hotplug ... i know you sent a bunch of patches, but that's all around the edges right? in general nouveau should work ok in simple scenarios, like plugging/unplugging a screen?
15:05 imirkin: zerothis: pastebin dmesg
15:07 Lyude: imirkin: tbh: I wouldn't be surprised if it was somewhat broken everywhere, but it seemed to work mostly for normal desktop setups
15:15 imirkin: Lyude: ok. did you ever consider adding some more software way of adding/removing connectors?
15:15 imirkin: or i guess you were focused on making the real thing work...
15:15 imirkin: whereas i just want connector hotplug to work :)
15:58 karolherbst: imirkin: I just noticed today, that kms wasn't actually enabled on my machine, but it works nethertheless and dracut also magically started to display stuff on nvidia, which uses the kms API for that
16:02 imirkin: karolherbst: so it makes a card node and everything?
16:02 karolherbst: imirkin: inside /sys/class/drm: card0 card0-DP-1 card0-DP-2 card0-eDP-1 card0-HDMI-A-1 renderD128 version
16:02 imirkin: very cool
16:12 karolherbst: imirkin: they don't have a fb device though, so if you want to use a tty with it, I think you actually need efifb or something
16:12 Lyude: imirkin: yeah, that was actually part of what I've been so preoccupied doing
16:12 Riastradh: imirkin: Anything in particular about how it works? Is that just a `yes, that's expected'? Is there a particular part of envytools documentation or the nouveau source code you meant me to look at?
16:13 Lyude: (the P50 stuff is one of many sets of issues I've fixed in the last month or two)
16:13 Lyude: Erm, *MST stuff. I say P50 since that was all work needed to make that ThinkPad work Ok
16:20 imirkin: Riastradh: that's a "i don't remember, but have a look at imem, and you might find it instructional"
16:21 Riastradh: imirkin: nouveau/nvkm/subdev/instmem/*?
16:21 imirkin: sounds right
16:21 imirkin: basically there are many ways of accessing vram from the cpu
16:21 imirkin: at least one of them is a hole somewhere, and 600000 sounds like a familiar offset.
16:21 imirkin: or 60000 or something
16:22 imirkin: iirc it's the peephole method
16:22 karolherbst: yeah, 0x600000 sounds like the offset inside the mmio reg space
16:22 imirkin: i learn this stuff when i need to, and then promptly forget it
16:22 Riastradh: I'm looking at <https://envytools.readthedocs.io/en/latest/hw/bus/bars.html#bar1-vram-aperture> but I don't see anything obvious there.
16:22 imirkin: 600000 is where the VGA stuff lives
16:22 Riastradh: karolherbst: mmio reg space or BAR 1?
16:22 imirkin: isn't mmio regs on bar0?
16:22 Riastradh: mmio regs is BAR 0, yes, I thought.
16:23 imirkin: right, on BAR1 it's the sliding window
16:23 imirkin: so that's the thing -- you configure somewhere where that window points at
16:23 karolherbst: ohh right, pramin is at 700000
16:23 imirkin: which is an offset to any accesses in that window
16:23 Riastradh: What I'm seeing is that the sync buffer is mapped with nvbo->bo.offset = 0x60000, but the physical bus address is choosing to map the sync buffer's va to is 0.
16:23 imirkin: so ... iirc bo.offset == the VA address
16:24 imirkin: which means that the right VM has to be loaded in for that to work
16:25 imirkin: the PA can't be 0... what do you mean by that?
16:25 imirkin: the PDE lives at 0
16:25 Riastradh: Rather, is BAR(1) + 0.
16:25 imirkin: (well, 0x200 or so)
16:25 Riastradh: I.e., 0xe000...00 on this machine.
16:25 imirkin: right
16:26 imirkin: but i bet the BAR1 offset is 0x60000
16:26 Riastradh: `BAR1 offset'?
16:26 imirkin: i don't remember how that's programmed though
16:26 imirkin: well, BAR1 is a view into memory
16:26 imirkin: it has an offset for that view
16:26 imirkin: (and a size, which is determined by ... some stuff)
16:26 imirkin: this is why skeggsb is going to be a lot better at answering these questions
16:27 imirkin: i just know how this stuff works in theory
16:27 Riastradh: skeggsb: Halp!
16:27 imirkin: but have to go search lots of stuff to find the practical answer
16:27 Riastradh: The symptom I'm seeing is that evo_sync always fails, so I'm trying to find whether something is getting mapped wrong or something.
16:27 Riastradh: Rather: the _earliest_ symptom.
16:27 Riastradh: Obviously the main symptom is a black screen.
16:28 imirkin: right, so evo_sync failing == disp is stuck
16:28 Riastradh: That happens very early on, before we even try to do a mode switch.
16:29 imirkin: and it's already stuck... heh.
16:29 imirkin: oh, silly question
16:29 imirkin: do you have some kind of vesa thing going on?
16:29 Riastradh: Yes.
16:29 imirkin: can you kill it?
16:29 imirkin: preferably with fire
16:29 Riastradh: ...I was going to ask `how', but I'm not sure where the kill_with_fire subroutine is in NetBSD.
16:29 imirkin: ;)
16:29 imirkin: should be!
16:30 karolherbst: imirkin: but that shouldn't cause evo to fail, or should it?
16:30 imirkin: i dunno how netbsd works, but on linux it's as simple as "don't build it in"
16:30 Riastradh: OK. So, it makes sense that if NetBSD is still printing to the VGA character array console, evo_sync will fail?
16:30 imirkin: karolherbst: well, it coudl cause disp to get confused
16:30 karolherbst: mhh, yeah, maybe
16:30 imirkin: Riastradh: well VGA is a different animal
16:30 Riastradh: OK, maybe I'm confused.
16:31 imirkin: but if you look at the linux nouveau, it first "kicks out" the other drivers
16:31 imirkin: because doing VESA calls at the same time as driving the gpu directly
16:31 imirkin: is going to end poorly
16:32 imirkin: so your BIOS dumps you into vga, and you just feed characters to 0b800h and magic happens
16:32 imirkin: with VESA, you can actually switch modes
16:32 imirkin: and do some other fancy things
16:32 imirkin: (fancy for the year 1991)
16:33 karolherbst: does the VESA stuff actually works on efi booted systems? Never actually tried that
16:33 imirkin: VGA emulation happens through pure magic. no one at any of these GPU companies has any idea how it works, much less us.
16:33 imirkin: karolherbst: no. it's installed as a bios service.
16:33 karolherbst: makes sense
16:33 imirkin: iirc it's all hooked into int 10h
16:33 imirkin: but it's been a little while since i've twiddled it myself
16:33 karolherbst: anyway, nvidia is fine with efifb being active, but I have no idea about efi support inside netbsd
16:34 imirkin: seems unlikely that the machine which shipped with a G84 mobile chip is going to have efi.
16:34 karolherbst: ohh, true
16:34 imirkin: efi = weird new thing no one has. it'll stay that way for another 5 years.
16:34 karolherbst: :D
16:35 karolherbst: efi isn't _that_ uncommon
16:35 imirkin: depends who you ask
16:35 imirkin: in the right circles, windows is pretty uncommon
16:35 karolherbst: okay sure, but we talk about x86 here
16:35 imirkin: i don't have efi
16:36 imirkin: Riastradh doesn't have efi
16:36 mslusarz: btw, last time I looked at this (which is >3 years ago), there was a very short window when both nouveau and vesa touched the hardware
16:36 imirkin: it works for n = 2, therefore it works for all n
16:36 karolherbst: your motherboards have to be very old for that
16:36 imirkin: karolherbst: not old at all. from 2010.
16:37 mslusarz: even with all that kicking out "other" drivers stuff
16:37 imirkin: mslusarz: yeah, that makes sense.
16:37 imirkin: since there's no real way to *kick* vesa out
16:37 imirkin: esp if it's in the middle of something
16:37 Riastradh: I'm fuzzy on how the whole VGA and VESA business works, but what we have is the VGA registers mapped at bus adddress 0x3c0, and we stop using them before we try the nouveau mode switch.
16:37 mslusarz: I mean vesafb
16:38 imirkin: mslusarz: ah. that's doubly unfortunate.
16:38 imirkin: Riastradh: right. those are the legacy VGA registers. VESA works a little differently.
16:39 imirkin: Riastradh: anyways ... before nouveau loads, do you have a "low" resolution or a "high" resolution?
16:39 Riastradh: If VESA is what you get by mapping the PCI display-class device's BAR 1 or whatever as a framebuffer, then we're not using that.
16:39 Riastradh: low
16:39 imirkin: ok. probably no vesa loaded then.
16:39 imirkin: VESA is a standardized way of accessing features beyond what VGA offers
16:39 Riastradh: This is also not different now.
16:40 imirkin: lets you set modes beyond 640x480x16 or 320x200x256
16:40 Riastradh: Right.
16:40 imirkin: and lets you have your own fb, i think, rather than having to use the shadowed memory at 0a000h / 0b800h
16:40 imirkin: [for graphics / text modes]
16:41 imirkin: and i think vesa dumps text modes entirely
16:41 imirkin: anyways
16:41 Riastradh: So, anyway, the disp is stuck eaaaaarly at boot.
16:41 imirkin: evo_sync failing BEFORE setting a mode is certainly odd.
16:41 Riastradh: (or I have something mapped wrong)
16:42 imirkin: how come i didn't see that earlier?
16:42 Riastradh: (and I'm just waiting for an idle page of system RAM to become nonzero after poking it)
16:42 karolherbst: we actually use VRAM to talk with evo, maybe something while mapping the buffer went wrong?
16:42 Riastradh: karolherbst: Heh. That's what prompted this in the first place!
16:43 karolherbst: well you need VRAM access, otherwise no evo for you
16:44 Riastradh: https://www.NetBSD.org/~riastradh/tmp/20180830/dmesg.20180830.6
16:44 Riastradh: I added lots of debug prints in there, some with stack traces
16:44 Riastradh: imirkin: Probably because evo_sync didn't print anything on success or failure.
16:45 imirkin: ah.
16:45 Riastradh: The only evidence of the early failure was a 2sec delay at boot.
16:45 Riastradh: Which, when the early-boot clock is screwed up for reasons I haven't looked into, doesn't really manifest here!
16:45 Riastradh: I mean doesn't manifest in the written dmesg log.
16:47 Riastradh: For reference: BAR 1 is e0000000.
16:49 karolherbst: "[ 1.037569] kern error: 0088 1 nv50_display_init" that is kind of bad
16:50 Riastradh: karolherbst: Yes, but the evo_sync failure happens long before that.
16:50 Riastradh: ...errr, wait, why is that bad?
16:50 Riastradh: `kern error' just means it was printed with DRM_ERROR.
16:50 Riastradh: or printk(KERN_ERR) or something.
16:50 karolherbst: because nv50_display_init shouldn't fail
16:50 Riastradh: It's not failing.
16:51 Riastradh: That's the evo_push/evo_data/evo_mthd/evo_kick trace.
16:51 karolherbst: ohh, I see
16:51 karolherbst: it is just an print you added?
16:51 Riastradh: Yes. Well, unifdef'd.
16:51 Riastradh: #define evo_mthd(p,m,s) do { \
16:51 Riastradh: const u32 _m = (m), _s = (s); \
16:51 Riastradh: printk(KERN_ERR "%04x %d %s\n", _m, _s, __func__); \
16:51 Riastradh: *((p)++) = ((_s << 18) | _m); \
16:51 Riastradh: } while(0)
16:52 Riastradh: (There's an #if 1 <silent version> #else <the above noisy version> #endif, and I switched it to #if 0.)
16:52 Riastradh: But anyway, the first evo_sync failure happens looooong before that.
16:53 Riastradh: ...oh, wait, never mind, maybe it doesn't.
16:53 karolherbst: ;)
16:53 karolherbst: yeah
16:53 karolherbst: the evo fail happens on the second kick
16:54 Riastradh: Well, it's the first evo_sync.
16:54 karolherbst: the mapped memory kind of acts as a ring buffer
16:54 Riastradh: nv50_display_init doesn't evo_sync, just evo_kicks.
16:55 Riastradh: So I gathered.
16:55 karolherbst: we don't have evo_sync anymore with master
16:55 Riastradh: My suspicion is that we're just moving the ring buffer forward and the device isn't doing anything with it because it's wired wrong.
16:55 Riastradh: Yes, so I noticed...
16:55 karolherbst: yeah, something like that
16:56 Riastradh: If I could confirm that this is just a problem with the mode-setting, then I might just give up on this and move on to the next update.
16:56 karolherbst: I would expect something to be messed up with how the buffer is handled or something there
16:56 Riastradh: But this is smelling like a deeper problem in the wiring, in which case it's not likely to go away with the change to atomic-only mode set.
16:56 Riastradh: Which is why I printed all the bus space mappings, instobj addresses, bo offsets, &c.
16:57 karolherbst: what is the condition that evo_sync fails?
16:58 Riastradh: Waits for syncbuf[0] to become nonzero.
16:58 karolherbst: mhhh
16:58 karolherbst: "[ 1.037569]" those are timestamps, right?
16:58 Riastradh: Specifically, sets syncbuf[0] = 0, does some evo_mthd/data/kick, then waits for syncbuf[0] to become nonzero.
16:58 karolherbst: it kind of looks like there is no wait at all
16:58 Riastradh: They are supposed to be, but whoever put them in recently didn't bother to make sure a functioning timecounter was ready, so they're not actually useful; disregard them.
16:59 karolherbst: yeah... okay
16:59 Riastradh: I can attest that there actually is a 2sec delay here, from watching the console output.
16:59 karolherbst: okay
17:00 karolherbst: so basically we get no answer from evo
17:00 Riastradh: Yes.
17:00 karolherbst: in the new code, this is done inside "core507d_update"
17:01 Riastradh: New code? Newer than Linux master as of the past month or so?
17:01 Riastradh: linus master
17:01 karolherbst: nouveau master
17:01 Riastradh: Oh, OK.
17:01 Riastradh: Haven't looked at that at all.
17:02 karolherbst: looks like nv50_disp_atomic_commit_core is the actual thing being called
17:03 karolherbst: I mean, no idea how that is called in your version
17:03 Riastradh: Well, it all got changed some time between 4.4 and 4.18 in the switch to atomic-only mode set.
17:04 Riastradh: Not actually sure which one is this first call to evo_sync.
17:04 Riastradh: Either nv50_display_flip_next or nv50_crtc_disable, but that's a little weird because I don't think anything has requested a mode set by that point, and the VGA console is still in use.
17:04 karolherbst: okay, so what happens is, that we set the head of the section to o
17:05 karolherbst: "nouveau_bo_wr32(bo, offset / 4, 0x00000000);"
17:05 karolherbst: inside core507d_ntfy_init
17:05 karolherbst: uhm, whatever is NV50_DISP_CORE_NTFY
17:05 karolherbst: which is actually 0
17:05 Riastradh: Sounds like
17:05 Riastradh: nouveau_bo_wr32(disp->sync, EVO_MAST_NTFY, 0x00000000);
17:05 karolherbst: yeah
17:06 Riastradh: where EVO_MAST_NTFY = EVO_SYNC(0, 0x00) = (0 * 0x100 + 0) = 0.
17:06 karolherbst: then we do the push
17:06 karolherbst: +kick
17:06 karolherbst: and now we wait on that value to change
17:06 karolherbst: I mean
17:06 Riastradh: Right. And here it's not changing.
17:06 karolherbst: bo->offset + 0x0
17:06 karolherbst: uhm
17:06 Riastradh: if (nouveau_bo_rd32(data, EVO_MAST_NTFY) != 0x00000000) {
17:06 Riastradh: printf("%s:%d -- success!\n", __func__, __LINE__);
17:06 karolherbst: bo->offset + offset + 0
17:07 karolherbst: Riastradh: mind printing the addresses?
17:07 karolherbst: when writing to 0 and when waiting?
17:07 Riastradh: Which ones?
17:07 Riastradh: [ 1.037569] evo_sync:603: wr32(bo=0xfffffa1e843fd010 va=0xffffa2806f2ad000 phys=0 off=60000, reg=0, value=0)
17:07 karolherbst: Riastradh: the syncing stuff
17:07 karolherbst: "nouveau_bo_wr32(disp->sync, EVO_MAST_NTFY, 0x00000000);"
17:07 karolherbst: and the wait
17:08 karolherbst: _then_
17:08 karolherbst: just don't write 0, but write 1 and just sleep for one second
17:08 karolherbst: and see what happens then
17:08 Riastradh: bo is disp->sync, va is disp->sync->kmap.virtual, phys is the paddr of the first physical page in disp->sync->bo, off is disp->sync->bo.offset
17:09 Riastradh: Oh, hm. phys doesn't make sense there because I printed the wrong thing.
17:09 Riastradh: Shoulda printed vtophys(virtual) instead; this doesn't have a backing page allocated by ttm.
17:10 Riastradh: karolherbst: Wait one second after it fails?
17:10 karolherbst: no, when you replace writing 0 with writing 1
17:10 karolherbst: just as an experiment
17:10 karolherbst: or maybe just remove the check
17:10 Riastradh: And do what after one second? Treat evo_sync as success?
17:10 karolherbst: and continue after 1 or 2 seconds
17:11 karolherbst: mhh, I am actually wondering
17:11 Riastradh: Nothing actually checks the return code of evo_sync, so success/failure is reflected only in what I print there.
17:11 karolherbst: ahh, I see
17:11 Riastradh: But I'll try printing 1 (and printing the right physical address).
17:11 karolherbst: Riastradh: by any chance, your display connected doesn't try to do interlaced modes, does it?
17:11 Riastradh: (Conveniently there is no distinction between bus addresses and physical addresses here, on x86.)
17:12 Riastradh: karolherbst: Umm... I don't know what interlaced modes are. I certainly haven't asked it to do that!
17:12 Riastradh: (I'm pretty much completely clueless about _graphics_ per se; all I know how to do is wire up and debug operating systems.)
17:14 Riastradh: Screen still blanks; lemme get dmesg...
17:15 Riastradh: Oh, oops, I waited for it to become nonzero, and I had set it to nonzero so it presumably sped right ahead without waiting.
17:15 Riastradh: But at least I confirmed phys=e0000000 as I expected this time.
17:15 karolherbst: mhh
17:16 Riastradh: Should I just wait the full two seconds each time, and/or print what is at syncbuf[0] when done (i.e., nouveau_bo_rd32(disp->sync, EVO_MAST_NTFY))?
17:19 karolherbst: it doesn't look like all evo_syncs are actually failing
17:20 karolherbst: which is a bit odd
17:20 karolherbst: ohh no, they all fail
17:20 Riastradh: Why do you say that?
17:20 Riastradh: OK.
17:21 karolherbst: I mean, writing to the bos seems to work, right?
17:21 karolherbst: (and reading back values)
17:21 Riastradh: Sure, writing to the bos works; the device just doesn't respond to them.
17:22 karolherbst: so either evo doesn't read them (either because it reads from somewhere else) or evo isn't active at all
17:22 karolherbst: or maybe it just got suck?
17:22 Riastradh: I can double-check, I guess, but certainly some values seem to be stored, as shown in the display error interrupt.
17:22 Riastradh: I'd guess the first two (doesn't read or isn't active) because I can't imagine what could have stuck it so early.
17:22 imirkin: it'd be nice if nouveau had some sort of self-test for making sure that common operations work
17:23 imirkin: like all the vram access methods work, etc
17:23 karolherbst: Riastradh: soo, the first evo write is where the actual sync handle is
17:23 imirkin: can't capture everything, but should help a lot in debugging odd cases
17:23 karolherbst: "evo_data(push, core->chan.sync.handle);"
17:23 imirkin: either by demonstrating the failures, or by proving that the basic constructs work
17:24 karolherbst: inside core507d_init
17:24 karolherbst: method 0x0088
17:24 karolherbst: so
17:24 karolherbst: "f0000000" is written there as a value
17:24 karolherbst: Riastradh: where does that "f0000000" comes form?
17:24 imirkin: he's on a 4.4 kernel
17:24 karolherbst: *from
17:25 karolherbst: in nv50_display_init
17:26 karolherbst: "evo_data(push, nv50_mast(dev)->base.sync.handle);" in 4.4
17:27 Riastradh: I think it comes from this:
17:27 Riastradh: ret = nvif_object_init(&dmac->base.user, 0xf0000000, NV_DMA_IN_MEMORY,
17:27 Riastradh: &(struct nv_dma_v0) {
17:27 Riastradh: .target = NV_DMA_V0_TARGET_VRAM,
17:27 Riastradh: .access = NV_DMA_V0_ACCESS_RDWR,
17:27 Riastradh: .start = syncbuf + 0x0000,
17:27 Riastradh: .limit = syncbuf + 0x0fff,
17:27 Riastradh: }, sizeof(struct nv_dma_v0),
17:27 Riastradh: &dmac->sync);
17:27 imirkin: Riastradh: please use pastebin / hastebin
17:27 Riastradh: OK, sorry.
17:28 karolherbst: this list_for_each_entry before setting that sync handle also looks interesting
17:29 Riastradh: Better: https://nxr.netbsd.org/xref/src/sys/external/bsd/drm2/dist/drm/nouveau/nouveau_nv50_display.c#344
17:29 karolherbst: sadly I am not familiar at all with the 4.4 display code, I just started to look into it with recent kernels
17:32 imirkin: karolherbst: they're just fixed handles
17:33 karolherbst: yeah, looks like it
17:33 Riastradh: So, I could update again and get the new display code, but I suspect the problem isn't in the display code per se and an update is not likely to help find where the deeper problem is.
17:33 karolherbst: mhhh
17:34 karolherbst: what we could do is, to check with pramin if the content actually arrives
17:34 karolherbst: in vram
17:34 Riastradh: How can I do that?
17:34 karolherbst: read nvaforcebios code
17:34 karolherbst: uhm
17:34 karolherbst: nvafakevbios?
17:34 Riastradh: Grepped -i for `nva.*bios', nothing.
17:34 karolherbst: nvafakebios
17:34 karolherbst: not inside nouveau
17:34 karolherbst: envytools
17:35 karolherbst: nvafakebios is a tool to upload a vbios into VRAM
17:35 karolherbst: same code can be also used to actually read from VRAM
17:35 karolherbst: should be inside nvagetbios
17:35 karolherbst: the PRAMIN stuff inside nvagetbios
17:35 karolherbst: but you actually deal with physical addresses there
17:36 Riastradh: OK. What am I looking for? Is this basically just reading from VRAM through another aperture or something?
17:36 karolherbst: through mmio+0x700000
17:36 karolherbst: it is basically a window into (V)RAM you can configure
17:36 Riastradh: That should be easy enough, thanks. How do the offsets line up?
17:36 karolherbst: read nvagetbios code ;)
17:36 karolherbst: vbios_extract_pramin
17:36 Riastradh: vbios_extract_pramin?
17:36 Riastradh: OK.
17:37 karolherbst: you have to adjust the offsets to whatever you are actually interested in
17:37 karolherbst: but
17:37 Riastradh: Just wondering about the offset 60000 business above.
17:37 karolherbst: it may help you track down some of those VRAM issues
17:37 imirkin: Riastradh: well, i think page tables get allocated low
17:37 imirkin: i suspect it's just the next available address
17:37 Riastradh: imirkin: Sounds plausible -- know how I can confirm?
17:37 imirkin: leak a bo
17:37 imirkin: i.e. allocate some bo
17:37 imirkin: and ignore it
17:37 imirkin: somewhere early on
17:39 Riastradh: OK, will try that: read VRAM through MMIO in BAR0 rather than through aperture in BAR1, and leak a BO before display initialization to see what the offsets look like.
17:39 Riastradh: I assume I'll need to pin and map the BO too or something?
17:40 karolherbst: well you should pin it to VRAM and write something into it to verify you can read it out via bar0
17:40 karolherbst: I expects that works
17:40 karolherbst: but maybe something is indeed funky there
17:40 imirkin: ttm generates a good bit of indirection with all this too
17:41 Riastradh: I feel like about a third of my time is spent navigating ttm indirections and the other two third navigating nouveau indirections...
17:41 karolherbst: can't have enough layers
17:41 imirkin: yeah, i'm not a fan of how indirect nouveau has gotten
17:41 imirkin: i'd rather have a bunch of if/else than jumping between files
17:41 Riastradh: I think it got a little better between 3.15 and 4.4...
17:42 Riastradh: At least, fewer void *'s and generic nouveau_objects everywhere.
17:42 imirkin: yeah, 4.3 had a giant rewrite which got rid of the void *'s
17:42 imirkin: everything is typed now
17:42 imirkin: which is indeed much nicer
17:42 Riastradh: Indeed.
17:43 Riastradh: OK, time for lunch!
17:43 Riastradh: *pooF*
17:43 karolherbst: imirkin: well, I think it is still cleaner if we really continue to support that many devices
17:43 karolherbst: also it is easier to swap things out
17:43 imirkin: easier to swap things out. harder to understand what's going on for one given device.
17:44 karolherbst: right
17:44 imirkin: and since most of the time is spent debugging rather than writing code...
17:45 imirkin: it's just hard to navigate for someone who isn't intimately familiar with all this stuff
17:45 karolherbst: maybe we should write a vi plugin, where you can set the chipset and it resolves all func pointers accordingly :p
17:45 imirkin: hehe
17:45 imirkin: and the thing is ... only skeggsb is intimately familiar at this point
17:46 karolherbst: mhh maybe more a job for ctags actually
17:46 imirkin: i suspect some docs which explain all this would make it all a lot more accessible
17:47 karolherbst: convince skeggsb to write those :p
17:47 imirkin: i've tried.
17:47 imirkin: but it's his job and he has things which he's actually paid to do
17:48 imirkin: much like you, nowadays
17:49 karolherbst: well, my situation is a bit weird
17:49 karolherbst: as I am not there mainly for doing nouveau stuff
17:50 imirkin: and i think he got sick of working on nouveau on the weekends quite a while back
17:50 karolherbst: but in the past there was just a lot of things I had to do which involved working on nouveau
17:50 karolherbst: yeah, understandable though
17:55 karolherbst: Lyude: are you there?
17:55 karolherbst: Lyude: and do you have access to a laptop with runpm issues right now?
18:12 imirkin: alright. time to see if i remember how to operate mmiotrace. bbl.
19:46 karolherbst: Riastradh: and did you got something to work?
19:55 Riastradh: karolherbst: Just got back a bit ago, debugging a wifi driver right now, will take a shot in a moment.
20:09 Riastradh: So, if I understand correctly...
20:10 Riastradh: Writing a VRAM offset to MMIO+0x1700 causes MMIO+0x700000 through MMIO+0x700000+something to be a window into VRAM at that offset?
20:10 Riastradh: something=0x10000 or 0x20000, depending on devicE?
20:11 Riastradh: And nvif_rd/wr32 is for mmio registers?
20:11 Riastradh: Echhhh, there's something shifty going on here.
20:12 Riastradh: nva_wr32(cnum, 0x1700, vbios_vram >> 16);
20:27 imirkin: skeggsb: ok, so a few fun facts. #1 - i seem to be getting DP_LINK_BW_5_4 set on the hdmi2.0 link (haven't tested with a hdmi 1.4 device) in 612300
20:27 imirkin: #2 - there's some SCDC thing which we're allegedly supposed to play with to mess with dividers - needs more investigation
20:27 imirkin: #3 - the div in 612300 follows a weird pattern:
20:28 imirkin: [0] 602.318491 MMIO32 R 0x612b00 0x009b7080 PDISPLAY+0x2b00 => 0x9b7080
20:28 imirkin: [0] 602.318500 MMIO32 W 0x612b00 0x00d37080 PDISPLAY+0x2b00 <= 0xd37080
20:28 imirkin: [0] 602.318509 MMIO32 R 0x612b00 0x00d37080 PDISPLAY+0x2b00 => 0xd37080
20:28 imirkin: [0] 602.318518 MMIO32 W 0x612b00 0x00d37180 PDISPLAY+0x2b00 <= 0xd37180
20:28 imirkin: we always set to div << 8 | div, while here it has 1 << 8 | 0
20:31 imirkin: more info here - https://people.freedesktop.org/~imirkin/traces/gp108/
20:31 karolherbst: Riastradh: yeah, mind the shift
20:31 imirkin: plugged into both DVI and HDMI
20:31 imirkin: i could have thrown some MARK's in there but ... sorry
20:32 karolherbst: imirkin: did you find where nvidia does the link training bits?
20:32 imirkin: for hdmi?
20:32 karolherbst: yes
20:33 karolherbst: apperantly for hdmi 2.0 we need to do link training
20:33 imirkin: didn't know there was any link training.
20:33 imirkin: i did see some SEQ writes
20:33 karolherbst: somebody brought it up
20:33 imirkin: so there's some SCDC thing where you're supposed to figure out if you're dividing by 10 or 40
20:33 imirkin: and also some sort of scrambling
20:34 karolherbst: imirkin: "<mupuf> hdmi 2.0 requires link training"
20:35 imirkin: https://hastebin.com/pacesiqema.cs
20:35 imirkin: nfc what that is
20:35 karolherbst: imirkin: those regs should be documentat afaik
20:35 karolherbst: *documented
20:35 karolherbst: but hum
20:35 airlied: not sure if hdmi link trainig is sw controlled
20:36 airlied: but i915 or dal should have anything needed if it is
20:37 Riastradh: karolherbst: Do you know what the units involved are?
20:37 karolherbst: Riastradh: nvaforcebios is uploading a vbios at the end of the vram
20:38 karolherbst: Riastradh: so what it does is: 1. figure out how big the VRAM is 2. open window at the end (max VRAM - X) 3. upload vbios
20:38 karolherbst: uhm, nvafakebios
20:38 karolherbst: Riastradh: 0x619f04 is an unreliable register to get the vram size
20:38 Riastradh: Size in...? Bytes? Pages? Words?
20:39 karolherbst: it kind of is set by the driver or something
20:39 karolherbst: not quite sure
20:39 imirkin: airlied: i haven't tried anything yet. just trying to figure out what all needs to be done
20:40 imirkin: i wonder if we need to set the VIC via evo, or if the one in the infoframe is enough
20:41 karolherbst: Riastradh: let me check
20:42 karolherbst: Riastradh: 0x1700 is 0x00017ff0 on my gm206
20:43 karolherbst: which when shifted by << 16 seems to be the size in bytes
20:43 Riastradh: OK, so 0x1700 contains a byte offset in VRAM?
20:43 karolherbst: no
20:43 karolherbst: bytes >> 16
20:43 Riastradh: Err, right.
20:44 Riastradh: So I want to set it to nvbo->bo.offset >> 16.
20:44 karolherbst: physical address
20:44 karolherbst: bo.offset is vm, no?
20:45 Riastradh: Oh. How do I find VRAM physical address of nvbo->bo?
20:46 karolherbst: Riastradh: also make sure that the high bits are 0
20:46 Riastradh: nvbo->bo.mem.bus.base + nvbo->bo.mem.bus.offset?
20:46 karolherbst: 0:23 are safe to set, above... not so much
20:46 karolherbst: Riastradh: I really don't know
20:46 karolherbst: I know that the MMU knows
20:46 karolherbst: but that's basically it
20:47 Riastradh: No, that can't be right, that's the bus address we pass to ioremap.
20:48 imirkin: ttm knows.
20:48 imirkin: all hail the mighty ttm.
20:48 karolherbst: Riastradh: you need to be a bit careful with that pramin window as you can just overwrite system memory as well
20:49 Riastradh: Errrrrrrrrr...isn't this a window into physical VRAM?
20:49 karolherbst: high bits
20:49 Riastradh: High bits?
20:49 karolherbst: bits 24:27 or something control the type
20:49 Riastradh: Oh. I see.
20:49 karolherbst: 0: VRAM, 1:?? 2: sysram 3: SYSRAM_NO_SNOOP
20:49 karolherbst: ohh only two bits
20:50 karolherbst: so 24:25
20:50 Riastradh: Where are you reading this?
20:50 karolherbst: nva
20:50 karolherbst: uhm
20:50 karolherbst: rnndb actually
20:50 Riastradh: I'm looking at <https://envytools.readthedocs.io/en/latest/hw/bus/pbus.html#space-pbus> and I don't see the breakdown in bits.
20:50 karolherbst: "lookup -a gm107 0x1700 0x02000000"
20:51 karolherbst: mhh
20:51 karolherbst: wait a sec
20:51 imirkin: that's for the PTE's
20:51 imirkin: 0x1700 configures the hole
20:51 imirkin: i believe it uses PA's
20:51 Riastradh: aughhhhhh
20:51 Riastradh:brain fire
20:52 imirkin: but not 100% sure
20:52 imirkin: if you check the driver source, should become apparent
20:52 imirkin: iirc imem uses it
20:52 karolherbst: Riastradh: https://github.com/envytools/envytools/blob/master/rnndb/g80_defs.xml#L29
20:53 Riastradh: imirkin: I'm checking, but nouveau/nvkm/subdev/bios/shadowramin.c, but the units aren't clear to me.
20:53 Riastradh: s/checking, but/checking/1
20:53 karolherbst: Riastradh: and https://github.com/envytools/envytools/blob/master/rnndb/bus/pbus.xml#L323
20:53 imirkin: Riastradh: usually the units are pages.
20:53 karolherbst: select the right variant
20:53 imirkin: i.e. 4k
20:53 karolherbst: imirkin: that one is 65k though
20:53 imirkin: yeah, could be 64k
20:53 karolherbst: ohh, right. :)
20:54 imirkin: nv50_instobj_wr32_slow
20:54 imirkin: so that's an example of how to use the peephole
20:54 Riastradh: Heh. I don't seem to have that...
20:55 Riastradh: But, nv50_instobj_wr32 does seem to use it.
20:55 imirkin: drm/nouveau/nvkm/subdev/instmem/nv50.c
20:55 Riastradh: However, I'm not sure which numbers here are VRAM offsets, which are bus addresses, which are VM addresses...
20:55 imirkin: this only works for PA's in VRAM
20:56 Riastradh: OK. How do I get the VRAM PA of a pinned mapped bo?
20:56 imirkin: you configure the hole at 0x1700 (the base)
20:56 imirkin: and then you get 64K of MMIO which lets you read that stuff
20:56 imirkin: starting at 0x700000
20:57 Riastradh: Right, I got that, just not sure what number to put into 0x1700, other than that bits >=24 must be zero.
20:57 karolherbst: and phys addrs >> 16 ;)
20:57 Riastradh: Right.
20:57 karolherbst: so you want to know the phys address of a bo
20:57 imirkin: yeah. gimme a minute...
20:57 Riastradh: I can put in nvbo->bo.offset, but it sounds like that's the GPU VA offset, not the VRAM PA offset.
20:57 imirkin: it'll be set by the stupid ttm thing
20:57 imirkin: somewhere
20:59 Riastradh: nouveau_ttm_io_mem_reserve?
21:00 karolherbst: ttm_get_physical_addr would be a too obvious choice for a name to implement such a function :)
21:00 Riastradh: Heh.
21:00 imirkin: problem is that none of us understand the logical model under which this is all done
21:00 Riastradh: ttm_mem_io_bus_reserve_address_placement_caching_buffer
21:01 imirkin: it's not designed to be used this way
21:01 imirkin: which is why it's tricky
21:01 Riastradh: Where else do we use VRAM PAs?
21:01 imirkin: nowhere.
21:01 Riastradh: ...Isn't there a page table that we program them into?
21:01 imirkin: yes
21:01 Riastradh: For the GPU virtual address space?
21:01 Riastradh: OK, where do we do that?
21:01 imirkin: but it's like 20x indirected, like everything else
21:01 Riastradh: Is that subdev/mmu?
21:02 imirkin: that or vm or vmm
21:02 Riastradh: I thought vm got renamed to mmu.
21:02 imirkin: there's the nvkm_memory stuff
21:02 imirkin: but i have no idea how that correlates to nouveau_bo's
21:04 Riastradh: Well, it's gotta happen either in nouveau_bo_pin or in nouveau_bo_map, right?
21:04 imirkin: like i said, it's 20x indirected
21:04 imirkin: nouveau_bo stuff all goes through ttm
21:04 Riastradh: Guessing nouveau_bo_pin -> nouveau_bo_validate -> ttm_bo_validate...
21:04 imirkin: so like
21:04 imirkin: look at nv50_dma_push
21:05 imirkin: it finds the bo in the client vmm
21:05 imirkin: and then vma->addr i think is the VA
21:05 imirkin: but vma may have a PA in it too
21:05 imirkin: no. of course not. it has a reference to a vm
21:05 imirkin: which in turn bla bla bla
21:07 Riastradh: nouveau_vram_manager_new?
21:08 Riastradh: nv50_ram_get
21:09 Riastradh: mem->offset = (u64)r->offset << NVKM_RAM_MM_SHIFT;
21:09 Riastradh: (here mem is struct nvkm_mem, not struct ttm_mem_reg, just to keep you on your toes)
21:10 Riastradh: mem->start = node->offset >> PAGE_SHIFT;
21:10 Riastradh: (here mem is the ttm_mem_reg and node is the nvkm_mem)
21:10 imirkin: skeggsb: would you accept a patch which renames every method to have a typename in it? so that instead of foo->ctrl it'd be foo->hdmi_ctrl or whatever?
21:10 Riastradh: Can't really tell whether this is vram or va...)
21:10 Riastradh: imirkin: Yes please!
22:03 imirkin: so based on intel and amd gpu's, we definitely have to do this SCDC write for the divider and the scrambler
22:03 imirkin: but it's unclear how the local device is configured for this
22:05 imirkin: anyone remember how i2c is done for e.g. edid stuff?
22:05 imirkin: (i.e. what to look for in the logs)
22:08 imirkin: oh fk me... 1 bit at a time ... this isn't fun.
22:09 Riastradh: OK, wifi driver fixed; back to nouveau VROOM mappings.
22:39 Riastradh: So, if I understand correctly:
22:39 Riastradh: - nvkm_vm_get(vm, size, pgshift, access, vma) allocates a VM range in the VM space vm, and stores it in vma.
22:40 Riastradh: - nvkm_vm_map_at(vma, delta, node) programs the page tables of vma's VM space to point at the physical addresses in node, which is a list of VRAM(?) physical address ranges.
22:44 imirkin: sounds plausible
22:44 Riastradh: Hm. But it looks like ttm sets bo->offset to be bo->mem.start << PAGE_SHIFT, where bo->mem.start = node->start >> PAGE_SHIFT, and on the left side I thought we had a VM address, while on the right side I thought we had a VRAM address.
22:45 Riastradh: So now I'm just as confused as I was before.
22:45 imirkin: hrmph
22:46 Riastradh: It definitely looks lke nvkm_vm_map_at treats node->regions as a list of physical addresses, though.
22:46 Riastradh: list_for_each_entry(r, &node->regions, rl_entry) {
22:46 Riastradh: u64 phys = (u64)r->offset << 12;
22:47 Riastradh: Oh, wait, head(node->_regions_)->offset versus node->offset.
22:48 Riastradh: ...also bo->mem.mm_node is a struct nvkm_mem, not a struct nvkm_mm_node.
22:48 Riastradh: @!&#^
22:49 imirkin: ok, so ... vma.addr is definitely a VA
22:50 Riastradh: Hooray! Something definite!
22:50 Riastradh: You mean a CPU VA?
22:50 Riastradh: Wait, what's vma.addr?
22:50 imirkin: because e.g. nvc0_bo_move_m2mf takes it and feeds it via pushbuf
22:50 Riastradh: nvkm_vma doesn't have an addr member?
22:50 imirkin: no, a GPU VA
22:50 imirkin: struct nouveau_mem *mem = nouveau_mem(old_reg);
22:50 imirkin: u64 src_offset = mem->vma[0].addr;
22:51 imirkin: struct nvif_vma
22:51 Riastradh: Oh, nvif_vma.
22:51 Riastradh: Err, I don't have nvif_vma.
22:51 imirkin: probably a new thing
22:51 Riastradh: (nor nouveau_mem)
22:51 imirkin: hehe
22:51 imirkin: well look at nvc0_bo_move_m2mf
22:51 imirkin: you definitely have that
22:51 imirkin: src_offset/dst_offset in there are GPU VA's
22:51 Riastradh: struct nvkm_mem *node = old_mem->mm_node;
22:51 Riastradh: u64 src_offset = node->vma[0].offset;
22:53 Riastradh: OK, so struct nvkm_vma::offset (in 4.4, addr in master) is a GPU VA.
22:54 Riastradh: It sounds like nvkm_mem::regions is likely to be physical addresses.
22:54 Riastradh: It appears only in subdev/fb/ram*.c and in subdev/mmu/base.c.
22:54 imirkin: so the thing to remember
22:54 imirkin: is that a bo may live in vram, system memory, or paged out
22:55 Riastradh: It also looks like nvkm_mem::offset is the first (physical?) region offset.
22:55 imirkin: so the abstractions have to be able to refer to all this stuff
22:55 imirkin: a GPU VA may refer to vram backing or sysmem backing
22:55 imirkin: etc
22:55 Riastradh: imirkin: Yes -- I'm creating one just like the sync buf, pinning it, and mapping it, so it _should_ be in VRAM.
22:55 imirkin: i get that
22:55 imirkin: but when you're navigating all the abstractions
22:55 imirkin: this is why they exist
22:55 Riastradh: Sure.
22:56 imirkin: realistically, you'll have to wait until you and skeggsb are in here at the same time
22:56 imirkin: he's generally around on weekdays during the regular workday for western australia.
22:57 imirkin: er, make that eastern
22:57 Riastradh: OK.
22:57 imirkin: same timezone as sydney iirc
22:57 Riastradh: Gotta run for a while now!
23:15 rhyskidd: karolherbst: thanks for the current status of compute in demmt. it's something i'd like to work on though
23:17 karolherbst: rhyskidd: well, I got mmt to trace it, it just depends on nvidias header for now. We kind of want to have support for multiple versions
23:19 karolherbst: rhyskidd: problem is, nvidia-uvm doesn't have a free license, so we can't just copy those headers
23:19 karolherbst: but nvidia-uvm is afaik fully open source