12:42 tomeu: and the basics of the kernel driver for rockchip work now :)
12:46 phh: cool. do you plan on keeping support for vendor kernel driver? 0:-)
12:46 tomeu: I'm afraid I should, as it has been useful in the past
12:46 tomeu: for VSI
12:47 tomeu: oh, actually, I cannot submit it to mesa because of the license
12:47 tomeu: so I will just keep a patch in some branch I guess
12:47 phh: the license of the header?
12:47 tomeu: yep
12:47 tomeu: it's GPL only
12:47 phh: gosh that's sad. I'm pretty sure this could be discussed with rockchip
12:48 tomeu: I think so, yeah, along a few "gems" I have found along the way...
12:48 phh: you mean the standard vendor one security flaw per 3 lines of code? :P
12:49 tomeu: guess we could have support in mainline for some time as long as it is minimally safe, but they have at least two big security holes in their UABI
12:50 tomeu: well, it's not as much bugs in code as purposeful circumventing of kernel security mechanisms
12:50 phh: yup perfectly normal vendor driver :D
12:50 tomeu: hehe
12:52 phh: are there other delegates than teflon in progress mesa-side? (it's not that I don't like tflite... well yes I actually dislike tflite)
12:52 tomeu: not really
12:53 tomeu: guess next will be the Android NN HAL
12:53 tomeu: which one would you like to see?
12:55 phh: well in my dreams I could directly use all the pytorch models I can get on the internet. though i understand that's probably the hardest target
12:56 tomeu: my understanding is that there is no mechanism in pytorch for external delegates
12:56 phh: my non-egoistic self says that Android NN HAL sounds like a good next target. Another reasonable target would be ONNX I think
12:57 tomeu: but I think it can use the Android NN HAL? not sure how that could work on regular Linux though
12:57 tomeu: yeah, didn't find with ONNX a external delegate thing either though, only in-tree backends
12:58 phh: yeah sure ONNX can use Android NN HAL
12:59 phh: but yeah my taget usage of my rk3588 isn't android, but rather a server. I'm not sure what's the situation with Android with mainline Linux on rk3588, I don't think anyone started working on that
13:01 tomeu: yeah, I think I will wait until somebody approaches me with an offer :)
13:01 phh: you mean you'll do whatever you get paid for?
13:02 phh: (just checking if i'm understanding what you're saying)
13:03 tomeu: I meant that the NN HAL work will probably wait until somebody offers money for it
13:03 phh: makes sense
13:03 tomeu: because I have no idea of how useful it would be atm, eg. how it would help with increasing adoption
13:03 tomeu: so I would instead increase HW support than adding the NN HAL
13:05 phh: actually the most popular target would probably be llama.cpp/whisper.cpp which could be done with teflon
13:06 tomeu: you mean via tflite?
13:06 phh: yes
13:06 tomeu: glad to hear that, I had been wondering
13:06 phh: I don't mean that llama.cpp has support for it -_-' I'm pretty sure it doesn't, you'd need to do that
13:07 tomeu: haha
13:07 tomeu: so, what would be the best way of supporting LLM models above the Gallium layer?
13:15 phh: I'd say ONNX ( https://github.com/microsoft/onnxruntime-genai ) , but it's super recent. Like I looked at it 3 weeks ago? onnxruntime-genai (the onnx lib for LLMs) didn't have the Android part yet
13:15 phh: on smarpthones some people use MLC, but https://github.com/mlc-ai/mlc-llm says that it supports only OpenCL which looks like a very bad fit for rockchip npu
13:26 phh: though I don't know to which point it makes sense to do on NPU, maybe it's already bottlenecked by the RAM on GPU
13:49 tomeu: hmm, not sure why it would need more bandwidth per operation when executing on the NPU than on the CPU or GPU
13:50 tomeu: I would expect rather less
13:50 tomeu: the size of the models is the main problem on mobile, but if people find a way of running them on the CPU or GPU, then on those devices it should be worth it running them on the NPU
13:51 tomeu: (or more realistically, a combination of those)
14:04 phh: i'm rather saying that NPU wouldn't help
14:04 phh: well it would still leave cpu/gpu to do other things, and reduce power consumption
14:27 tomeu: yes, but I still think there should be an advantage
16:02 f_: welcome back tomeu !
16:02 f_: <tomeu> and the basics of the kernel driver for rockchip work now :)
16:03 f_: tomeu: that's awesome :D