01:07gnurou: skeggsb: just to make sure, you haven't made any progress on the GM20B texture lookup always black issue?
01:08gnurou: I will spend some time on it now
01:55RSpliet: mupuf: to me it makes sense to "arm" the load monitor; it's no use having it raise interrupts if there's no perflvl lower and/or higher, so even just a bitmask stating what you want it to do is useful
01:56RSpliet: and by demanding the host to arm it, you don't have to worry about needlessly repeating your interrupts
01:56RSpliet: (I think we can already assume a perfect link where no interrupts go unnoticed, right?)
02:21mupuf: oh, you are entirely right
02:22mupuf: my solution was to ack an IRQ when the frequency got changed
02:22mupuf: and if the host refuses, then do nothing
02:22mupuf: there would be 2 hysteresis windows
02:22mupuf: one for small changes and one for bigger ones
02:23mupuf: and each of them can only re-emit an IRQ if the load dropped to the other side of the threshold or if the perf was changed
02:23mupuf: and the perf changed event is the host sending an ACK
02:23mupuf: being able to change the thresholds from the host side sounds useful but more like a pre-configuratoin rather than run time
02:23karolherbst: mupuf: what would happen if the target is like 50%, the pmu sends a request at 65%, the host doesn't ack, and the load grows to 75%?
02:23mupuf: but hey, let's keep this open
02:24mupuf: that's why you have 2 windows
02:24mupuf: karolherbst: ^^
02:24karolherbst: but let's say, the do a big change is at 85%
02:24karolherbst: what happens between 65% and 84%?
02:24mupuf: then may be it. The host must have had a good reason not to change the clocks
02:25karolherbst: maybe the load icnrease isn't enough to actually bother changing the clcoks
02:25mupuf: we do not have an unlimited amount of perf levels
02:25karolherbst: maybe increase at 65% means, the load will drop to 30% after reclock
02:25mupuf: I doubt that :D
02:25karolherbst: and then we would upcloack <=> downclock in cycles
02:25karolherbst: mupuf: there are some fancy kepler vbios
02:25karolherbst: some with only 3 cstates in total
02:25karolherbst: one for each pstate
02:25mupuf: oh, that is bad
02:25karolherbst: it is
02:26mupuf: well, the lower usage threshold is super low for nvidia
02:26mupuf: like 5 or 10%
02:26mupuf: so, no probs
02:26mupuf: and yes, coming up with the thresholds is going to require a lot of experiments
02:26karolherbst: yeah, but I think we need to think about more when the pmu is allowed to resend a request
02:26mupuf: and you know what we need to do before doing this work :p
02:26mupuf: I think we need to ignore the PMU entirely until we have something that works well
02:27mupuf: then we will think about optimising power usage on the cpu side
02:27mupuf: any other approach to this problem is just going to lead to more frustration
02:27karolherbst: I don't think we should even start having a check every 0.1 seconds on the cpu one way or the other
02:27mupuf: yes, we should, as an experiment
02:27mupuf: everything in the userspace
02:27mupuf: quick prototyping
02:28mupuf: running experiments
02:28karolherbst: yeah okay, but I have such thing already in userspace
02:28karolherbst: I thought you meant kernel space
02:28karolherbst: ahh yeha okay, then it's fine
02:28karolherbst: allthough my daemon currently messes up my kernel big time :/
02:29mupuf: that is a bug in the kernel, the userspace should never be able to mess up the kernel
02:29karolherbst: don't know, drm just completly breaks
02:29mupuf: ack, that is a bug worth investigating!
02:29karolherbst: mupuf: well I poke into my pstate and cstate interface files
02:29karolherbst: mupuf: ohh nouveau often messes up drm so big, that even intel won't do anything anymore
02:30mupuf: yep, playing with clocks is dangerous
02:30karolherbst: but why should i915 be effected by this?
02:30mupuf: but hey, instead of thinking it is your daemon
02:30mupuf: ah ah, put the gpu off bus
02:30mupuf: and the entire machine will be down
02:30mupuf: anyway, how stable is reclocking for you?
02:30karolherbst: pretty much
02:31mupuf: tried reclocking in a loop for hundred thousands of times?
02:31karolherbst: as long as you don't do it too often in a second
02:31mupuf: then it is unstable
02:31mupuf: and this is the first thing to address
02:31karolherbst: I would say reclocking every <0.1 seconds is a bit problematic
02:31mupuf: it should not be
02:31mupuf: and it means there are stuff we are not handling
02:31karolherbst: the locking part is a bit bricked somewhere
02:31karolherbst: it hangs on a lock somehwere
02:31karolherbst: the gpu still runs
02:31mupuf: oh, very good then!
02:31karolherbst: but the echo > pstate call is in a hang state
02:32mupuf: an actual bug that is easy to fix :)
02:32karolherbst: let me write a bash script :D
02:32mupuf:was running at 60 reclocks/s for an entire night for testing purposes
02:32mupuf: and then disabling vsync and doing the same even faster
02:33mupuf: after 3 millions of reclock, I can say it is stable enough :D
02:33karolherbst: I reclock very 0.01 seconds now
02:33karolherbst: and running glxspheres64
02:33mupuf: are you sure about that?
02:33karolherbst: mhh too stable
02:34karolherbst: every 0.001 seconds now
02:34karolherbst: aaaan hang
02:34mupuf: how are you checking you actually do it that fast?
02:34karolherbst: I trust bash :D
02:34karolherbst: stupid fifo
02:34mupuf: and you are wrong to do so
02:34karolherbst: got a nouveau 0000:01:00.0: fifo: PBDMA0: 00000002 [MEMACK_TIMEOUT] ch 2 [00bf890000 glxspheres64] subc 0 mthd 001c data 00000002
02:34mupuf: the write to the pstate file is blocking, isn't it?
02:35karolherbst: it blocks until everything is done
02:35karolherbst: and now it inifnite blocks for me
02:35mupuf: ok, and then you added a sleep before the loop?
02:35karolherbst: and a kworker is dead
02:35karolherbst: while true; do echo "try reclock to 07" ; echo 07 > pstate ; sleep 0.001 ; echo "try reclock to 0f"; echo 0f > pstate ; sleep 0.001; done
02:36mupuf: well, this does look like a hw issue here. As in, we screw some things up
02:36mupuf: so, unstable :D
02:36mupuf: would be nice to check if limiting the frequency but running longer helps
02:36karolherbst: the kworker hangs inside nvkm_pmu_send
02:37karolherbst: let me check where exactly
02:37mupuf: reclocking too fast makes it hard on the voltage controller
02:37mupuf: as for your script, you do not check how many reclocks per second you make
02:38mupuf: you assume the reclocking does not take any time, which is wrong since it waits for vblank
02:38mupuf: that's for this week end, see you guys!
02:39karolherbst: it waits on wait_event(pmu->recv.wait, (pmu->recv.process == 0));
02:39karolherbst: yeah okay, cu
02:47karolherbst: mhh something on the pmu is messed up
02:58karolherbst: yay, found an error
02:59karolherbst: usually the last thing is "R[10f200] = 02cf9800"
03:00karolherbst: rarely also R[10f200] = 02cf1800
03:00karolherbst: ohh no, it is right
03:01karolherbst: but there is no reply or maybe the reply get's lost somehow?
03:12RSpliet: mupuf, karolherbst: well, in the case of a "MEMACK_TIMEOUT", I would actually believe it if some DMA engine just hits a timeout because memory is constantly taken off-line
03:12RSpliet: (doesn't get its work done any more)
03:12karolherbst: RSpliet: mhhh no, I think the memory reclock stuff doesn't come to an end
03:13karolherbst: the echo > pstate calls are blocking until everything is done
03:13mupuf: RSpliet: could be a good explanation
03:13karolherbst: and I get two waits actually in a hang state
03:13RSpliet: mupuf: depends on the other symptoms :-)
03:13karolherbst: it waits inside nvkm_pmu_send for a reply from the pmu
03:14karolherbst: and it was called by gk104_ram_prog => nvkm_memx_fini
03:14karolherbst: so I think the memory isn't enabled at this time
03:14karolherbst: and therefore every request to memory will timeout
03:15RSpliet: but bottom line is, don't think "the tightest possible loop" is a good test-case. Rather do an endurance test within a small factor from rates you expect to change clocks at, and leave that running endlessly
03:15RSpliet: (while going to bed, having dinner with your grandparents, work your way through a box of tissues, that kind of endless)
03:16karolherbst: no, I think this has to be stable that way too
03:16karolherbst: I tested it without doing anything else now
03:16karolherbst: only pstate change
03:16RSpliet: only if you're 100% sure the DMA engine is supposed to do absolute fuckle
03:16karolherbst: the pmu communication has to be rock solid, because otherwise the card will die with no obvious reason
03:17karolherbst: even then we have to handle error cases so that the driver doesn't die
03:17karolherbst: it's in an dead lock situation
03:17RSpliet: yes, agreed; but that should be tested by stressing the PMU communication channel, ideally while taking out "reclocking" and "monitoring" from the equation
03:18karolherbst: okay, I could just ask my pmu for the current load data
03:18karolherbst: it is just a message to the pmu, reading out some data and replying
03:18RSpliet: think big, work small :-)
03:19karolherbst: mupuf: could a pmu timer mess things up?
03:19RSpliet: the obvious pitfall is overflow
03:19karolherbst: so maybe a pmu stack ran full?
03:20RSpliet: well, this timer probably relies on a counter
03:20karolherbst: not really
03:20karolherbst: just wait and a call
03:21karolherbst: but the timer calls perf_recv and I don't know how the interrupt handling and stuff works there exactly :/
03:21karolherbst: could be that my changes messes up the pmu
03:22RSpliet: is that what you're referring to?
03:22RSpliet: or is it a host-side time-out?
03:22karolherbst: I use ticks_from_us
03:22karolherbst: and then call(timer)
03:23karolherbst: the question I have is, what happens when the timer calls perf_recv, doesn't return and get a host side interrupt to do stuff
03:24karolherbst: also in perf_recv
03:25karolherbst: anyway, the kworker hangs here: https://github.com/karolherbst/nouveau/blob/master_karol_no_touchy/drm/nouveau/nvkm/subdev/pmu/memx.c#L88
03:27karolherbst: mhh, don't know if that helps
03:27karolherbst: trying to get a bit more up in the stack
03:27RSpliet: there's a kernel macro/function to print the stack
03:28RSpliet: for userspace there's either GDB or some copy-past fiddlies to be found on the internet
03:29karolherbst: yeah, I already use addr2line to parse the source code line
03:29karolherbst: but there is some inlining which messes it up a little
03:29karolherbst: it is called somewhere inside gk104_ram_prog
03:29karolherbst: but I really don't know where
03:29RSpliet: ohh addr2line, we meet again...
03:29karolherbst: the address brings me to ramfuc_exec
03:29RSpliet: yes, simple little tool
03:29karolherbst: but that doesn't help .d
03:30karolherbst: even gdb can'T help there
03:30karolherbst: ohh I can simply decrease the address a little
03:31karolherbst: and then the ram_exec call it is
03:31karolherbst: after that
03:33karolherbst: and the stack ends then here: https://github.com/karolherbst/nouveau/blob/master_karol_no_touchy/drm/nouveau/nvkm/subdev/pmu/base.c#L92
03:33karolherbst: so I assume that either the request wasn't transfered without harm
03:34karolherbst: or the response from the pmu got lost
03:34karolherbst: either way, the memory should be in a messy state here
04:23karolherbst: ohhh I think I messed up my kernel interrupt stuff somehow
04:23karolherbst: all I did was poking some stuff into the pmu :/
04:32karolherbst: ohh I am wondering why loading nouveau takes soo long after boot, but the module is 100MB big because of debug symbols :D
04:34karolherbst: RSpliet: plain pmu communication seems to work without any issues
04:35karolherbst: with waiting on a reply
04:35karolherbst: ohhh wait
04:35karolherbst: it doesn't
04:35karolherbst: got a hang now
04:38karolherbst: same place inside nvkm_send_pmu
04:38karolherbst: okay, so somehow the reply doesn't trigger the wake_up
04:44hakzsam: imirkin, right, this was for pmoreau and his spirv branch. I forgot to remove it
07:59imirkin_: hakzsam: can you take a look at the patch series that was just sent, ' [PATCH 0/9] gallium: batch query objects and related cleanups'
07:59imirkin_: since you've been messing with all that stuff
08:17karolherbst: RSpliet: the reply from the pmu can take more than 10msecs :/
08:18karolherbst: I replaced wait_event with wait_event_timeout( ... msecs_to_jiffies(10))
08:18karolherbst: and it fails really often
08:22karolherbst: having a timeout there makes the driver really stable here
08:24karolherbst: and even after the pmu is completly messed up, accel still works
08:27karolherbst: mupuf: https://github.com/karolherbst/nouveau/commit/bd1328b4a7868160263c88438f9bce795cda72ef what do you think about this?
08:29karolherbst: at least this makes developing suck less, because you can actually remove the driver without the need of a reboot
08:38karolherbst: even reclocking still works
08:38karolherbst: except a bit of memory reclocking
08:50RSpliet: karolherbst: for mem reclocking the upper bound should be around 17ms
08:51karolherbst: RSpliet: this is too low :/
08:51karolherbst: you also have kernel scheduling and everything involved here
08:51karolherbst: sadly 50ms wasn't enough
08:51karolherbst: so I want with 500
08:52RSpliet: does that mean that if wait_event_timeout is preempted, it will fail regardless of whether the reply was received?
08:52RSpliet: because that implies that the wait() routine is flawed
08:52RSpliet: and should be non-preemptible
08:52RSpliet: or detect pre-emption and spin it one more cycle before returning failure
08:52karolherbst: mhhh, don't think so, I mean the condition is updated inside a kworker
08:52karolherbst: and this can be started whenever
08:53karolherbst: we will get an interrupt from the pmu at some point, which will triger the working thread
08:53karolherbst: which updates the conditions and calls wakeup
08:53RSpliet: hmmno, the mutex should disable preemption, doesn't it?
08:53karolherbst: nvkm_pmu_intr calls schedule_work
08:54karolherbst: and the working thread is bound to nvkm_pmu_recv
08:54RSpliet: okay, erm... I don't have the time right now to work out what could happen
08:54karolherbst: yeah, it works though
08:55karolherbst: with a timeout of 500ms the working calls are all faster
08:55karolherbst: and the non working ones are not
08:55karolherbst: I don't know what happens with the pmu though
08:55karolherbst: sometimes I can read out the load from the pmu, sometimes not
08:55karolherbst: and the load seems to be right
08:55RSpliet: but it is *very* important that the timeout does not fail due to host-side preemption
08:56karolherbst: what about full preemptive kernels?
08:56RSpliet: preemption shouldn't be the cause of a timeout error
08:56karolherbst: anyway I want to use wait_event_hrtimeout instead anyway
08:56karolherbst: which uses ktime_t
08:57karolherbst: mhh I will check the source I think
08:57karolherbst: wait_event_timeout puts the task to TASK_UNINTERRUPTIBLE
08:58RSpliet: which means the scheduler doesn't try to schedule anything else iirc
08:58karolherbst: there are multiple version of wait_event
08:58karolherbst: so I guess this is the one which can't get interrupted :D
08:58RSpliet: well, double-check that
08:59RSpliet: interrupts might still come in, but wouldn't cause a reschedule
08:59karolherbst: wait_event calls schedule internally
09:00karolherbst: between each check
09:02RSpliet: which enforces a check after being rescheduled, right? :-)
09:02karolherbst: well, it has to do something after checking the condition
09:03karolherbst: busy loop would be kind of ugly
09:04karolherbst: RSpliet: https://stackoverflow.com/questions/11012406/race-condition-between-wait-event-and-wake-up
09:04karolherbst: mayb that helps?
09:04karolherbst: ohh it is interuptible
09:04karolherbst: not timeout, my bad
09:05karolherbst: ohhhh mhhh
09:05karolherbst: last comment
09:07karolherbst: ohhh that makes soe much sense now
09:07karolherbst: pmu->recv.process is shared across all interrupts
09:08karolherbst: and if we get a reschedule between the time where the condition was true and where it is checked again, and pmu->recv.process is set, we might mess up
09:09karolherbst: same goes for wait_event
09:09karolherbst: between line 266 and 267
09:09karolherbst: if we get a reschedule there
09:15karolherbst: but it doesn't make sense somehow
09:28karolherbst: okay, a call to nvkm_pmu_recv is totally missing in the timeout case
09:31karolherbst: not even nvkm_pmu_intr is called
09:32karolherbst: maybe the interrupts are really going into thin air for whatever reason :/
09:32imirkin_: neat. got bin/arb_shader_storage_buffer_object-rendering working with nvc0... soon, the world
09:34karolherbst: that's for 4.3, right?
09:35imirkin_: that ext is core in 4.3, yes
09:35karolherbst: mupuf: so I don't know. either the pmu isn't getting the message or the interrupt goes ops
09:35hakzsam: imirkin, done, this took me a while
09:36imirkin_: gr, stupid unsized arrays... the write tests want them, need to implement more.
09:36imirkin_: hakzsam: thanks :)
09:37hakzsam: imirkin, are you going to make more comments about the compute support?
09:37imirkin_: just the ones on-list and the ones i made on irc yesterday
09:38hakzsam: fine by me
09:38karolherbst: RSpliet: I hit the timeout after aroung 67800 reclocks, which no sleep in between
09:45imirkin_: hakzsam: it seems like in general you went with the "let's copy nvc0" approach, which is fine by me
09:46hakzsam: imirkin, yeah, cool
09:46imirkin_: and in case, this code is fairly little used, so as long as it's not actively breaking stuff, i don't really care
11:43imirkin_: gnurou: i think the running theory is that we need to actually compute the sched codes for maxwell rather than using the fixed ones
11:44imirkin_: gnurou: some information has been RE'd and made public about them by the MaxAs guy, but i can't say i fully grasp it
11:44imirkin_: gnurou: https://trello.com/c/zW06zeI6/116-maxwell-sched-codes
11:45imirkin_: although perhaps it's something much simpler
12:10karolherbst: mupuf: what is the best idea to debug the pmu? mess with some regs?
14:11mlankhorst: does tegra have a watchdog?
14:13imirkin_: CONFIG_TEGRA_WATCHDOG -- suggests that yes
14:14imirkin_: ... and fail. tegra30-timer
14:29mlankhorst: yeah no luck
14:30imirkin_: the dts does have a timer
14:30imirkin_: i dunno how that correlates into watchdogs
14:31imirkin_: [which is actually a tegra20-timer, oddly enough]
14:31imirkin_: i assume this is for tegra124?
14:31imirkin_: do you have CONFIG_TEGRA_TIMER enabled?
14:31mlankhorst: clocksource it seems
14:32mlankhorst: so not the same
14:32imirkin_: there's a tegra124-car
14:33mlankhorst: needs a declaration tegra-wdt
14:35mlankhorst: hm, or tegra30-timer
14:35imirkin_: right, but that's not in the dts
14:35imirkin_: only tegra20-timer
14:36imirkin_: there's a reset button
14:36mlankhorst: yeah but would be nice, mine seems to lock up
14:36mlankhorst: lets see if I can fudge it to work..
14:40mlankhorst: looks like kernel docs say it should work with a patch to dtb
14:42mlankhorst: lets see if the naive solution of adding compatible=tegra30-timer works..
14:47mlankhorst: meh :p
15:00mlankhorst: ok manually did some binding, will it reboot?
15:02mlankhorst: yes :D
15:09mlankhorst: ill send a patch for it soon then
15:09mlankhorst: my tegra's hanging way too much right now :(
15:10imirkin: what are you doing to it?
22:19gnurou: imirkin: thanks, I will have a look at this