01:07 gnurou: skeggsb: just to make sure, you haven't made any progress on the GM20B texture lookup always black issue?
01:08 gnurou: I will spend some time on it now
01:55 RSpliet: mupuf: to me it makes sense to "arm" the load monitor; it's no use having it raise interrupts if there's no perflvl lower and/or higher, so even just a bitmask stating what you want it to do is useful
01:56 RSpliet: and by demanding the host to arm it, you don't have to worry about needlessly repeating your interrupts
01:56 RSpliet: (I think we can already assume a perfect link where no interrupts go unnoticed, right?)
02:21 mupuf: oh, you are entirely right
02:22 mupuf: my solution was to ack an IRQ when the frequency got changed
02:22 mupuf: and if the host refuses, then do nothing
02:22 mupuf: there would be 2 hysteresis windows
02:22 mupuf: one for small changes and one for bigger ones
02:23 mupuf: and each of them can only re-emit an IRQ if the load dropped to the other side of the threshold or if the perf was changed
02:23 mupuf: and the perf changed event is the host sending an ACK
02:23 mupuf: being able to change the thresholds from the host side sounds useful but more like a pre-configuratoin rather than run time
02:23 karolherbst: mupuf: what would happen if the target is like 50%, the pmu sends a request at 65%, the host doesn't ack, and the load grows to 75%?
02:23 mupuf: but hey, let's keep this open
02:24 mupuf: that's why you have 2 windows
02:24 mupuf: karolherbst: ^^
02:24 karolherbst: but let's say, the do a big change is at 85%
02:24 karolherbst: what happens between 65% and 84%?
02:24 mupuf: then may be it. The host must have had a good reason not to change the clocks
02:25 karolherbst: maybe the load icnrease isn't enough to actually bother changing the clcoks
02:25 mupuf: we do not have an unlimited amount of perf levels
02:25 karolherbst: maybe increase at 65% means, the load will drop to 30% after reclock
02:25 mupuf: I doubt that :D
02:25 karolherbst: and then we would upcloack <=> downclock in cycles
02:25 karolherbst: mupuf: there are some fancy kepler vbios
02:25 karolherbst: some with only 3 cstates in total
02:25 karolherbst: one for each pstate
02:25 mupuf: oh, that is bad
02:25 karolherbst: yes
02:25 karolherbst: it is
02:26 mupuf: well, the lower usage threshold is super low for nvidia
02:26 mupuf: like 5 or 10%
02:26 mupuf: so, no probs
02:26 mupuf: and yes, coming up with the thresholds is going to require a lot of experiments
02:26 karolherbst: yeah, but I think we need to think about more when the pmu is allowed to resend a request
02:26 mupuf: and you know what we need to do before doing this work :p
02:26 mupuf: I think we need to ignore the PMU entirely until we have something that works well
02:27 mupuf: then we will think about optimising power usage on the cpu side
02:27 karolherbst: mhhh
02:27 mupuf: any other approach to this problem is just going to lead to more frustration
02:27 karolherbst: I don't think we should even start having a check every 0.1 seconds on the cpu one way or the other
02:27 mupuf: yes, we should, as an experiment
02:27 mupuf: everything in the userspace
02:27 mupuf: quick prototyping
02:28 mupuf: running experiments
02:28 karolherbst: yeah okay, but I have such thing already in userspace
02:28 karolherbst: I thought you meant kernel space
02:28 mupuf: nope
02:28 karolherbst: ahh yeha okay, then it's fine
02:28 mupuf: :)
02:28 karolherbst: allthough my daemon currently messes up my kernel big time :/
02:28 mupuf: why?
02:29 mupuf: that is a bug in the kernel, the userspace should never be able to mess up the kernel
02:29 karolherbst: don't know, drm just completly breaks
02:29 mupuf: ack, that is a bug worth investigating!
02:29 karolherbst: mupuf: well I poke into my pstate and cstate interface files
02:29 karolherbst: mupuf: ohh nouveau often messes up drm so big, that even intel won't do anything anymore
02:29 karolherbst: :D
02:30 mupuf: yep, playing with clocks is dangerous
02:30 karolherbst: but why should i915 be effected by this?
02:30 mupuf: but hey, instead of thinking it is your daemon
02:30 mupuf: ah ah, put the gpu off bus
02:30 mupuf: and the entire machine will be down
02:30 mupuf: anyway, how stable is reclocking for you?
02:30 karolherbst: pretty much
02:31 mupuf: tried reclocking in a loop for hundred thousands of times?
02:31 karolherbst: as long as you don't do it too often in a second
02:31 karolherbst: mhhh
02:31 mupuf: then it is unstable
02:31 mupuf: and this is the first thing to address
02:31 karolherbst: I would say reclocking every <0.1 seconds is a bit problematic
02:31 mupuf: it should not be
02:31 mupuf: and it means there are stuff we are not handling
02:31 karolherbst: the locking part is a bit bricked somewhere
02:31 karolherbst: it hangs on a lock somehwere
02:31 karolherbst: the gpu still runs
02:31 mupuf: oh, very good then!
02:31 karolherbst: but the echo > pstate call is in a hang state
02:32 mupuf: an actual bug that is easy to fix :)
02:32 karolherbst: let me write a bash script :D
02:32 mupuf: yop!
02:32 mupuf:was running at 60 reclocks/s for an entire night for testing purposes
02:32 mupuf: and then disabling vsync and doing the same even faster
02:33 mupuf: after 3 millions of reclock, I can say it is stable enough :D
02:33 karolherbst: I reclock very 0.01 seconds now
02:33 karolherbst: and running glxspheres64
02:33 mupuf: are you sure about that?
02:33 karolherbst: mhh too stable
02:34 karolherbst: every 0.001 seconds now
02:34 karolherbst: aaaan hang
02:34 mupuf: how are you checking you actually do it that fast?
02:34 karolherbst: I trust bash :D
02:34 karolherbst: stupid fifo
02:34 mupuf: and you are wrong to do so
02:34 karolherbst: got a nouveau 0000:01:00.0: fifo: PBDMA0: 00000002 [MEMACK_TIMEOUT] ch 2 [00bf890000 glxspheres64[6857]] subc 0 mthd 001c data 00000002
02:34 mupuf: the write to the pstate file is blocking, isn't it?
02:34 karolherbst: yes
02:35 karolherbst: it blocks until everything is done
02:35 karolherbst: and now it inifnite blocks for me
02:35 mupuf: ok, and then you added a sleep before the loop?
02:35 karolherbst: and a kworker is dead
02:35 karolherbst: yes
02:35 karolherbst: while true; do echo "try reclock to 07" ; echo 07 > pstate ; sleep 0.001 ; echo "try reclock to 0f"; echo 0f > pstate ; sleep 0.001; done
02:36 mupuf: well, this does look like a hw issue here. As in, we screw some things up
02:36 mupuf: so, unstable :D
02:36 mupuf: would be nice to check if limiting the frequency but running longer helps
02:36 karolherbst: the kworker hangs inside nvkm_pmu_send
02:37 karolherbst: let me check where exactly
02:37 mupuf: reclocking too fast makes it hard on the voltage controller
02:37 mupuf: as for your script, you do not check how many reclocks per second you make
02:38 mupuf: you assume the reclocking does not take any time, which is wrong since it waits for vblank
02:38 mupuf: anyway
02:38 mupuf: that's for this week end, see you guys!
02:39 karolherbst: mhhh
02:39 karolherbst: it waits on wait_event(pmu->recv.wait, (pmu->recv.process == 0));
02:39 karolherbst: yeah okay, cu
02:47 karolherbst: mhh something on the pmu is messed up
02:58 karolherbst: yay, found an error
02:59 karolherbst: https://gist.githubusercontent.com/karolherbst/b8d49084699179073342/raw/30eb0a3638bb01d080bf2e32a82d0c34187fec16/gistfile1.txt
02:59 karolherbst: usually the last thing is "R[10f200] = 02cf9800"
03:00 karolherbst: mhh
03:00 karolherbst: rarely also R[10f200] = 02cf1800
03:00 karolherbst: weird
03:00 karolherbst: ohh no, it is right
03:01 karolherbst: but there is no reply or maybe the reply get's lost somehow?
03:12 RSpliet: mupuf, karolherbst: well, in the case of a "MEMACK_TIMEOUT", I would actually believe it if some DMA engine just hits a timeout because memory is constantly taken off-line
03:12 RSpliet: (doesn't get its work done any more)
03:12 karolherbst: RSpliet: mhhh no, I think the memory reclock stuff doesn't come to an end
03:13 karolherbst: the echo > pstate calls are blocking until everything is done
03:13 mupuf: RSpliet: could be a good explanation
03:13 karolherbst: and I get two waits actually in a hang state
03:13 RSpliet: mupuf: depends on the other symptoms :-)
03:13 karolherbst: it waits inside nvkm_pmu_send for a reply from the pmu
03:14 karolherbst: and it was called by gk104_ram_prog => nvkm_memx_fini
03:14 karolherbst: so I think the memory isn't enabled at this time
03:14 karolherbst: and therefore every request to memory will timeout
03:15 RSpliet: but bottom line is, don't think "the tightest possible loop" is a good test-case. Rather do an endurance test within a small factor from rates you expect to change clocks at, and leave that running endlessly
03:15 RSpliet: (while going to bed, having dinner with your grandparents, work your way through a box of tissues, that kind of endless)
03:16 karolherbst: mhhh
03:16 karolherbst: no, I think this has to be stable that way too
03:16 karolherbst: because
03:16 karolherbst: I tested it without doing anything else now
03:16 karolherbst: only pstate change
03:16 RSpliet: only if you're 100% sure the DMA engine is supposed to do absolute fuckle
03:16 karolherbst: the pmu communication has to be rock solid, because otherwise the card will die with no obvious reason
03:17 karolherbst: even then we have to handle error cases so that the driver doesn't die
03:17 karolherbst: it's in an dead lock situation
03:17 RSpliet: yes, agreed; but that should be tested by stressing the PMU communication channel, ideally while taking out "reclocking" and "monitoring" from the equation
03:18 karolherbst: okay, I could just ask my pmu for the current load data
03:18 karolherbst: it is just a message to the pmu, reading out some data and replying
03:18 RSpliet: think big, work small :-)
03:19 karolherbst: mupuf: could a pmu timer mess things up?
03:19 karolherbst: "timer
03:19 karolherbst: "
03:19 RSpliet: the obvious pitfall is overflow
03:19 karolherbst: mhhh
03:19 karolherbst: so maybe a pmu stack ran full?
03:20 RSpliet: well, this timer probably relies on a counter
03:20 karolherbst: not really
03:20 karolherbst: just wait and a call
03:21 karolherbst: but the timer calls perf_recv and I don't know how the interrupt handling and stuff works there exactly :/
03:21 karolherbst: could be that my changes messes up the pmu
03:21 RSpliet: http://cgit.freedesktop.org/~darktama/nouveau/tree/drm/nouveau/nvkm/subdev/pmu/fuc/kernel.fuc#n100
03:22 RSpliet: is that what you're referring to?
03:22 RSpliet: or is it a host-side time-out?
03:22 karolherbst: I use ticks_from_us
03:22 karolherbst: and then call(timer)
03:23 karolherbst: the question I have is, what happens when the timer calls perf_recv, doesn't return and get a host side interrupt to do stuff
03:24 karolherbst: also in perf_recv
03:25 karolherbst: anyway, the kworker hangs here: https://github.com/karolherbst/nouveau/blob/master_karol_no_touchy/drm/nouveau/nvkm/subdev/pmu/memx.c#L88
03:27 karolherbst: mhh, don't know if that helps
03:27 karolherbst: trying to get a bit more up in the stack
03:27 RSpliet: there's a kernel macro/function to print the stack
03:28 RSpliet: for userspace there's either GDB or some copy-past fiddlies to be found on the internet
03:29 karolherbst: yeah, I already use addr2line to parse the source code line
03:29 karolherbst: but there is some inlining which messes it up a little
03:29 karolherbst: it is called somewhere inside gk104_ram_prog
03:29 karolherbst: but I really don't know where
03:29 RSpliet: ohh addr2line, we meet again...
03:29 karolherbst: the address brings me to ramfuc_exec
03:29 RSpliet: yes, simple little tool
03:29 karolherbst: but that doesn't help .d
03:29 karolherbst: well
03:29 karolherbst: inlining
03:30 karolherbst: even gdb can'T help there
03:30 karolherbst: ohh I can simply decrease the address a little
03:30 RSpliet: yep
03:31 karolherbst: https://github.com/karolherbst/nouveau/blob/master_karol_no_touchy/drm/nouveau/nvkm/subdev/fb/ramgk104.c#L1237
03:31 karolherbst: and then the ram_exec call it is
03:31 karolherbst: after that
03:33 karolherbst: and the stack ends then here: https://github.com/karolherbst/nouveau/blob/master_karol_no_touchy/drm/nouveau/nvkm/subdev/pmu/base.c#L92
03:33 karolherbst: so I assume that either the request wasn't transfered without harm
03:34 karolherbst: or the response from the pmu got lost
03:34 karolherbst: either way, the memory should be in a messy state here
04:23 karolherbst: ohhh I think I messed up my kernel interrupt stuff somehow
04:23 karolherbst: all I did was poking some stuff into the pmu :/
04:32 karolherbst: ohh I am wondering why loading nouveau takes soo long after boot, but the module is 100MB big because of debug symbols :D
04:34 karolherbst: RSpliet: plain pmu communication seems to work without any issues
04:35 karolherbst: with waiting on a reply
04:35 karolherbst: ohhh wait
04:35 karolherbst: it doesn't
04:35 karolherbst: got a hang now
04:38 karolherbst: same place inside nvkm_send_pmu
04:38 karolherbst: okay, so somehow the reply doesn't trigger the wake_up
04:44 hakzsam: imirkin, right, this was for pmoreau and his spirv branch. I forgot to remove it
07:59 imirkin_: hakzsam: can you take a look at the patch series that was just sent, ' [PATCH 0/9] gallium: batch query objects and related cleanups'
07:59 imirkin_: since you've been messing with all that stuff
08:04 hakzsam: yeah
08:17 karolherbst: RSpliet: the reply from the pmu can take more than 10msecs :/
08:18 karolherbst: I replaced wait_event with wait_event_timeout( ... msecs_to_jiffies(10))
08:18 karolherbst: and it fails really often
08:22 karolherbst: mhhh
08:22 karolherbst: having a timeout there makes the driver really stable here
08:24 karolherbst: and even after the pmu is completly messed up, accel still works
08:27 karolherbst: mupuf: https://github.com/karolherbst/nouveau/commit/bd1328b4a7868160263c88438f9bce795cda72ef what do you think about this?
08:29 karolherbst: at least this makes developing suck less, because you can actually remove the driver without the need of a reboot
08:38 karolherbst: even reclocking still works
08:38 karolherbst: except a bit of memory reclocking
08:50 RSpliet: karolherbst: for mem reclocking the upper bound should be around 17ms
08:51 karolherbst: RSpliet: this is too low :/
08:51 karolherbst: you also have kernel scheduling and everything involved here
08:51 karolherbst: sadly 50ms wasn't enough
08:51 karolherbst: so I want with 500
08:52 RSpliet: does that mean that if wait_event_timeout is preempted, it will fail regardless of whether the reply was received?
08:52 RSpliet: because that implies that the wait() routine is flawed
08:52 RSpliet: and should be non-preemptible
08:52 RSpliet: or detect pre-emption and spin it one more cycle before returning failure
08:52 karolherbst: mhhh, don't think so, I mean the condition is updated inside a kworker
08:52 karolherbst: and this can be started whenever
08:53 karolherbst: we will get an interrupt from the pmu at some point, which will triger the working thread
08:53 karolherbst: which updates the conditions and calls wakeup
08:53 RSpliet: hmmno, the mutex should disable preemption, doesn't it?
08:53 karolherbst: nvkm_pmu_intr calls schedule_work
08:54 karolherbst: and the working thread is bound to nvkm_pmu_recv
08:54 RSpliet: okay, erm... I don't have the time right now to work out what could happen
08:54 karolherbst: yeah, it works though
08:55 karolherbst: with a timeout of 500ms the working calls are all faster
08:55 karolherbst: and the non working ones are not
08:55 karolherbst: I don't know what happens with the pmu though
08:55 karolherbst: sometimes I can read out the load from the pmu, sometimes not
08:55 karolherbst: and the load seems to be right
08:55 RSpliet: but it is *very* important that the timeout does not fail due to host-side preemption
08:56 karolherbst: mhhh
08:56 karolherbst: what about full preemptive kernels?
08:56 RSpliet: preemption shouldn't be the cause of a timeout error
08:56 karolherbst: anyway I want to use wait_event_hrtimeout instead anyway
08:56 karolherbst: which uses ktime_t
08:57 karolherbst: mhh I will check the source I think
08:57 karolherbst: ohh
08:57 karolherbst: wait_event_timeout puts the task to TASK_UNINTERRUPTIBLE
08:58 RSpliet: which means the scheduler doesn't try to schedule anything else iirc
08:58 karolherbst: yeah
08:58 karolherbst: there are multiple version of wait_event
08:58 karolherbst: so I guess this is the one which can't get interrupted :D
08:58 RSpliet: well, double-check that
08:59 RSpliet: interrupts might still come in, but wouldn't cause a reschedule
08:59 karolherbst: wait_event calls schedule internally
09:00 karolherbst: between each check
09:02 RSpliet: which enforces a check after being rescheduled, right? :-)
09:02 karolherbst: well, it has to do something after checking the condition
09:03 karolherbst: busy loop would be kind of ugly
09:04 karolherbst: RSpliet: https://stackoverflow.com/questions/11012406/race-condition-between-wait-event-and-wake-up
09:04 karolherbst: mayb that helps?
09:04 karolherbst: ohh it is interuptible
09:04 karolherbst: not timeout, my bad
09:05 karolherbst: ohhhh mhhh
09:05 karolherbst: last comment
09:07 karolherbst: ohhh that makes soe much sense now
09:07 karolherbst: ...
09:07 karolherbst: pmu->recv.process is shared across all interrupts
09:08 karolherbst: and if we get a reschedule between the time where the condition was true and where it is checked again, and pmu->recv.process is set, we might mess up
09:09 karolherbst: same goes for wait_event
09:09 karolherbst: http://lxr.free-electrons.com/source/include/linux/wait.h#L262
09:09 karolherbst: between line 266 and 267
09:09 karolherbst: if we get a reschedule there
09:09 karolherbst: ...
09:15 karolherbst: but it doesn't make sense somehow
09:28 karolherbst: okay, a call to nvkm_pmu_recv is totally missing in the timeout case
09:31 karolherbst: mhhh
09:31 karolherbst: not even nvkm_pmu_intr is called
09:32 karolherbst: maybe the interrupts are really going into thin air for whatever reason :/
09:32 imirkin_: neat. got bin/arb_shader_storage_buffer_object-rendering working with nvc0... soon, the world
09:33 karolherbst: nice
09:34 karolherbst: that's for 4.3, right?
09:35 imirkin_: that ext is core in 4.3, yes
09:35 karolherbst: mupuf: so I don't know. either the pmu isn't getting the message or the interrupt goes ops
09:35 hakzsam: imirkin, done, this took me a while
09:36 imirkin_: gr, stupid unsized arrays... the write tests want them, need to implement more.
09:36 imirkin_: hakzsam: thanks :)
09:36 hakzsam: np
09:37 hakzsam: imirkin, are you going to make more comments about the compute support?
09:37 imirkin_: just the ones on-list and the ones i made on irc yesterday
09:38 hakzsam: fine by me
09:38 karolherbst: RSpliet: I hit the timeout after aroung 67800 reclocks, which no sleep in between
09:45 imirkin_: hakzsam: it seems like in general you went with the "let's copy nvc0" approach, which is fine by me
09:46 hakzsam: imirkin, yeah, cool
09:46 imirkin_: and in case, this code is fairly little used, so as long as it's not actively breaking stuff, i don't really care
10:03 hakzsam: yep
11:43 imirkin_: gnurou: i think the running theory is that we need to actually compute the sched codes for maxwell rather than using the fixed ones
11:44 imirkin_: gnurou: some information has been RE'd and made public about them by the MaxAs guy, but i can't say i fully grasp it
11:44 imirkin_: gnurou: https://trello.com/c/zW06zeI6/116-maxwell-sched-codes
11:45 imirkin_: although perhaps it's something much simpler
12:10 karolherbst: mupuf: what is the best idea to debug the pmu? mess with some regs?
12:54 mlankhorst: T/1
12:54 mlankhorst: oops
14:11 mlankhorst: does tegra have a watchdog?
14:13 imirkin_: CONFIG_TEGRA_WATCHDOG -- suggests that yes
14:14 imirkin_: tegra_wdt.o
14:14 imirkin_: ... and fail. tegra30-timer
14:29 mlankhorst: yeah no luck
14:30 imirkin_: the dts does have a timer
14:30 imirkin_: i dunno how that correlates into watchdogs
14:31 imirkin_: [which is actually a tegra20-timer, oddly enough]
14:31 imirkin_: i assume this is for tegra124?
14:31 mlankhorst: yeah
14:31 imirkin_: do you have CONFIG_TEGRA_TIMER enabled?
14:31 mlankhorst: clocksource it seems
14:32 mlankhorst: rtc
14:32 mlankhorst: so not the same
14:32 imirkin_: there's a tegra124-car
14:33 mlankhorst: needs a declaration tegra-wdt
14:35 mlankhorst: hm, or tegra30-timer
14:35 imirkin_: right, but that's not in the dts
14:35 imirkin_: only tegra20-timer
14:36 imirkin_: there's a reset button
14:36 imirkin_: :p
14:36 mlankhorst: yeah but would be nice, mine seems to lock up
14:36 mlankhorst: lets see if I can fudge it to work..
14:36 imirkin_: doh
14:36 imirkin_: bbl
14:40 mlankhorst: looks like kernel docs say it should work with a patch to dtb
14:42 mlankhorst: lets see if the naive solution of adding compatible=tegra30-timer works..
14:47 mlankhorst: /4
14:47 mlankhorst: meh :p
15:00 mlankhorst: ok manually did some binding, will it reboot?
15:02 mlankhorst: yes :D
15:02 pmoreau: \o/
15:09 mlankhorst: ill send a patch for it soon then
15:09 mlankhorst: my tegra's hanging way too much right now :(
15:10 imirkin: what are you doing to it?
22:19 gnurou: imirkin: thanks, I will have a look at this