16:03 tagr: anholt_, karolherbst: looking at the latest data from our internal test farms I don't see any failures on Jetson Nano or Jetson TK1
16:04 tagr: oops... anholt ^^
16:04 tagr: I do have those devices locally so I can try to reproduce, perhaps something in the farm causes these to be hidden
16:05 tagr: anything in particular that I would need to do to reproduce the PCIe and/or IOMMU issues?
16:05 anholt: hmm. I've got two different boards here, L4T 21.8 flashed, tftp booting to a kernel using defconfig+https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/.gitlab-ci/container/arm.config+CONFIG_ARM_TEGRA_DEVFREQ=y
16:07 tagr: huh... "does not exist on main" ...
16:07 anholt: flashing was moving jetson-tk1_extlinux.conf.nfs aside and "sudo ./flash.sh -s pxeboot.sh -N 10.42.0.1:/nfs jetson-tk1 eth0" where pxeboot.sh is "run bootcmd_pxe"
16:07 anholt: sorry, https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/.gitlab-ci/container/arm.config
16:08 anholt: https://paste.centos.org/view/68d34d9d
16:08 tagr: anholt: oh wait... so this uses the U-Boot version that shipped with L4T?
16:09 anholt: yeah
16:09 tagr: that could be an issue
16:10 tagr: in fact, that could be a cause for the IOMMU as well
16:10 tagr: IOMMU issue, that is
16:10 tagr: I typically grab a recent version of U-Boot from upstream, build and flash that
16:11 tagr: though admittedly I haven't done that on Jetson TK1 in a while... let me find out what version I have currently flashed
16:12 tagr: ha! mine's even older
16:13 tagr: I've got 2017.11-rc1 with 3 local patches on top
16:13 tagr: but it's an upstream one and downstream U-Boot sometimes does funky stuff that messes with upstream Linux
16:14 tagr: for Jetson TK1 specifically I vaguely recall some issue with secure PMC and that would explain why the power-ungating of PCI fails
16:14 tagr: because if Linux thinks that the PMC is not secure it might just read bad register values
16:14 anholt: flashing now
16:15 tagr: not sure what the status is for U-Boot on Jetson TK1 in mainline, but in my experience it's pretty stable
16:17 anholt: replaced bootloader/ardbeg/u-boot.bin with that, get nothing from serial upon supplying power
16:17 anholt: (jetson-tk1_defconfig from u-boot)
16:17 tagr: did you use u-boot-dtb.bin from the build?
16:17 anholt: where does that go?
16:18 tagr: that's the one that you should flash
16:18 anholt: ah
16:18 tagr: that's basically u-boot.bin but with a control DTB appended, if I recall correctly
16:18 tagr: (it's been a while)
16:18 anholt: still nothing
16:19 tagr: by the way, there's a more light-weight mechanism to update U-Boot, which I think I documented somewhere at some point
16:19 anholt: oh, they're bit-identical
16:19 tagr: so maybe revert to the original u-boot.bin that you have from L4T to see if you can get it back up
16:19 anholt: perhaps u-boot-dtb-tegra.bin?
16:19 tagr: possibly...
16:21 anholt: reflashing back to stock u-boot.bin gets me serial again
16:21 tagr: looks like this was changed since I last looked at it
16:22 tagr: so you can update U-Boot without actually using the flash scripts, though it requires a few more dependencies
16:22 anholt: 2022.04 u-boot tag is what I'm using
16:24 tagr: it's possible that that's broken
16:25 tagr: u-boot-dtb-tegra.bin is the right one to use according to some old scripts that I found
16:26 tagr: maybe an easier thing to try is to execute U-Boot from RAM until you've found a version that works
16:26 tagr: if you have tegrarcm and the right BCT file you should be able to do that fairly easily
16:26 tagr: I suspect L4T has all those files
16:27 anholt: I apparently need to find the right tegrarcm? 21.8 doesn't have one, and the version I have for jetson didn't have the args that imirkin was suggesting to me
16:27 tagr: do you have this file: PM375_Hynix_2GB_H5TC4G63AFR_RDA_924MHz.bct
16:27 tagr: ?
16:27 anholt: yeah
16:28 anholt: this release uses "nvflash"
16:30 anholt:needs to take off
16:31 tagr: ah... nvflash...
16:31 tagr: alright, I can drop you an email with the details on how to flash this
16:36 anholt: thanks
16:37 tagr: I've got a couple of minutes left, so I'll see if I can find a good version
16:45 tagr: anholt: latest U-Boot from upstream works fine for me with the instructions that I just sent you
16:45 tagr: latest for me is: 2022.07-rc2-00080-g15a1d6924abd
16:45 tagr: let me check 2022.04
16:47 tagr: 2022.04 works as well, so it's possible that something goes wrong with the nvflash process and upstream U-Boot
16:48 tagr: if you want to make the U-Boot install persistent, you may want to try these instructions: https://github.com/NVIDIA/tegra-uboot-flasher-scripts/blob/master/README-developer.txt
16:48 tagr: Stephen wrote those a long time ago, but I think they should still work
16:49 tagr: one could possibly extract some useful snippets from that and make it a bit less convoluted
16:50 tagr: those scripts are meant to make it as easy as possible if you've got nothing set up at all, but if you've got a U-Boot build and everything already it's just a matter of generating the right file and uploading it
17:20 tagr: anholt: so yeah, you can follow the tegra-uboot-flasher-scripts instructions and you'll eventually get there
17:21 tagr: but it's a bit convoluted so it can get complicated if you deviate from the exact instructions (like if you want to inject a custom U-Boot build)
17:21 tagr: if you don't care about the exact version then the instructions should be sufficient
17:21 tagr: let me know if you need any help
17:28 tagr: anholt: fwiw, Linux v5.18-rc7 boots fine for me on Jetson TK1, though I haven't exhaustively tested it
17:28 tagr: no IOMMU errors or PCI errors though with that U-Boot 2022.04
20:25 airlied: karolherbst: btw your wierd fence stuff isn't perhaps msi related?
20:25 airlied:assumes you get getting interrupts
20:25 karolherbst: I do get interrupts, yes
20:25 karolherbst: and it could be indeed msi related... no idea :)
20:28 airlied: back in those times, disabling msi was always the first thing I tried :-P
20:29 karolherbst: he
20:29 karolherbst: h
20:29 karolherbst: I might actually try that
21:31 anholt: tagr: thanks, tegra-uboot-flasher and the uboot it built got me past pcie fail.
22:00 Lyude: Does TTM have any kind of way to handle having to acquire a lock for a bo move? I ask because I'm trying to fix this lockdep regression: https://paste.centos.org/view/fcc04219 . The issue seems simple enough, lockdep inversion between a TTM ww lock and &cli->mutex is happening. I think the solution should just be making sure we grab any relevant ttm locks before grabbing &cli->mutex,
22:00 Lyude: but that doesn't seem possible since it appears that ttm lock is per-bo.
22:01 Lyude:hopes that the mutex_lock_nested() call in nouveau_bo_move_m2mf isn't related, but i'm about to test to find out
22:08 Lyude: Ah - yes, it's definitely related
22:25 anholt: tagr: so, which u-boot should I be using for the nano, do you think?
22:32 Lyude: oh hm, I might have a better idea to fix this