07:50 Venemo: Lynne: sorry for the late reply I was away from the keyboard. I am the right person to ping about radv mesh shaders, and I will look into it after I return from my holidays. until then please open a mesa issue if you haven't already so it won't get forgotten
09:36 mareko: DavidHeidelberg: it's still broken, temporarily removing radeonsi from the CI is being considered https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26643
09:47 mareko: and radv
10:15 mareko: This is an announcement that radeonsi and all jobs that depend on it will be removed from the CI on December 29, 2023 due to a libdrm upgrade issue. 4 days should be more than enough time for interested parties to resolve it.
10:33 Venemo: mareko: what exactly is the issue with libdrm?
10:46 mareko: Venemo: the latest libdrm is required, but the CI was changed to use libdrm from the distro, which was obviously a mistake
10:47 Venemo: ouch
10:47 Venemo: we should just revert that CI change then, no?
10:48 mareko: too many conflicts and it's like 5 changes in 1 commit, so not revertable
10:49 mareko: I switched it back to builds from source manually, but some jobs still have errors
10:51 mareko: radeonsi and radv are good in there, but swrast and layered ones can't find that libdrm, this is the pipeline: https://gitlab.freedesktop.org/mesa/mesa/-/pipelines/1064179
10:52 mareko: usually when we release a new libdrm, we do it because we want to require it in Mesa 30 seconds later
10:53 mareko: we never release libdrm if it's not required by Mesa immediately
10:58 Venemo: ouch
10:59 Venemo: I think such a change should not have been made during the holidays
11:12 mareko: it'll be remembered for years to come though
11:40 mareko: I'm entertaining the idea of using gallium/rtasm to convert preamble NIR to x86 bytecode and run it on the CPU
12:18 Venemo: whoah
14:28 alyssa: mareko: surely interpreting NIR would be easier and not much more expensive?
14:28 alyssa: though JIT'ing it is cooler :D
14:29 DavidHeidelberg: mareko: just keep the DRM as is and wait until I fix it. On christmas day I was not really excited to start changing CI.
14:33 DavidHeidelberg: *Christmas day and weekend.
16:07 DavidHeidelberg: Ok, since we have libdrm under our control, we could you generate artifacts or put the build into S3 and just install it from our CI.
16:08 DavidHeidelberg: I understand we want fresh libdrm in the CI, on other hand, it's dependency for multiple projects and it make sense to keep it within packaging. This way we could use the package we generate in mesa/drm repository.
16:08 DavidHeidelberg: mareko: what do u say?
16:08 DavidHeidelberg: is it acceptable to you?
21:57 karolherbst: DavidHeidelberg: can it be done in a way, that any developer can dump the version of those packages?
21:59 karolherbst: could we even CI it? Like something checking if the requested version is available and if not, trigger a build and package it (and keeping the other versions)?
21:59 karolherbst: (unless we don't care if we accidentally bump the required version on stable branches)
21:59 karolherbst: (then the latest would do)
22:00 DavidHeidelberg: karolherbst: could it look like this https://gitlab.freedesktop.org/gfx-ci/ci-deb-repo/-/blob/main/libdrm.yml?ref_type=heads ?
22:01 DavidHeidelberg: I'm not saying it's that straightforward as we had it in CI, but on other hand our CI isn't one small script with two deps to compile but pretty huge monster
22:01 karolherbst: DavidHeidelberg: no, I meant that mesa requests a version and triggers an external pipeline to wait for the build to finish rather
22:02 karolherbst: also
22:02 karolherbst: I don't think we can go with "always latest" unless stable maintainer say it doesn't matter
22:02 DavidHeidelberg: hmm, can you rephrase, not sure about your idea
22:03 karolherbst: CI should also catch that backporting certain commits could require a newer libdrm without us noticing
22:03 karolherbst: like.. in mesa CI we check if a _specific_ libdrm version is in the repo, if not, we ask an external CI pipeline to build it for us and add it to the repo
22:04 karolherbst: but as I said, it's only needed if we can't go with "always latest" also on stable branches
22:05 karolherbst: but anyway.. personally I don't see the point of adding this complexity
22:05 karolherbst: what's the benefit here anyway?
22:05 DavidHeidelberg: we could do manually dpkg -i _libdrm-2.4.118_, but building infra for libdrm with minimal changes seems to be overkill
22:05 karolherbst: using deb files also seems overkill to me honestly
22:06 DavidHeidelberg: not if other software depends on it
22:06 DavidHeidelberg: or do we plan to compile all the software above libdrm?
22:06 karolherbst: why? is that an apt limitation?
22:07 karolherbst: other build systems have ways of marking something as installed without actually installing that package so deps are resolved, but other packages depending on it can be installed regardless
22:07 karolherbst: *package
22:07 DavidHeidelberg: I understand we compile by hand few last-tier test suites and utils. I don't understand why mess with core-system packages which are relied on by other installed packages
22:07 DavidHeidelberg: if we want to manage everything by hand, we can invent new distro for our CI
22:07 karolherbst: what are those packages?
22:08 DavidHeidelberg: 2-3 years back, when we did some simple tests, pointed our builds to /usr/local/lib... I get it. Now we have 10 scenarios, 4 different containers with different package set, multiple architectures
22:09 karolherbst: sure, but what packages do we need which depend on libdrm?
22:09 DavidHeidelberg: I know there are. I had my fun like year ago and it was a lot of pain
22:10 karolherbst: couldn't we just build the deb files inside mesa CI and install it there?
22:11 DavidHeidelberg: now I'll get to my main point and that's not spending 3x 1 hour for build armhf/arm64 and x86_64 containers, but just install prepared packages from one repo
22:11 DavidHeidelberg: it's really not fun to work with CI (as a CI developer).
22:11 karolherbst: we don't have to build all deb packages in one go
22:11 DavidHeidelberg: and yet, libdrm CI gets downloaded compiled in 30 seconds
22:12 karolherbst: the deb file could be the artifact of one job and then the container building ones just pull in all those deb files
22:12 DavidHeidelberg: but when we have other packages, which can be installed in few seconds, we can do same for libdrm
22:13 DavidHeidelberg: I was preparing this for last few months, but this "let's bump libdrm and reintroduce all ugliness" made me enjoy my Christmas with hacking up on this
22:13 mareko: DavidHeidelberg: if there is a timeframe for implementing the solution, I'll drop my earlier statements
22:13 DavidHeidelberg: yes, I have the libdrm done in the repo, but I'm doing some final tests
22:13 karolherbst: DavidHeidelberg: well, nobody force you to work on this, or did somebody?
22:13 DavidHeidelberg: https://gitlab.freedesktop.org/gfx-ci/ci-deb-repo/-/tree/bookworm?ref_type=heads
22:14 DavidHeidelberg: right
22:15 DavidHeidelberg: I need to do few more force pushes, since currently the repo doesn't contain hash in commit msg from which it was created
22:15 karolherbst: nothing against a deb repo in itself, I just prefer that we have all the logic inside mesa, as otherwise we could run into other issues we haven't before (like e.g. mismatched between used libdrm APIs and declared dependency version)
22:15 DavidHeidelberg: karolherbst: yeah, problem is we compile & build already double digit of projects
22:15 DavidHeidelberg: of course, we could do better ccaching
22:16 DavidHeidelberg: but then we would have to spend lot of time preparing some nice framework for each project to be properly ccached
22:16 karolherbst: yeah.. so that was my building the deb package idea
22:16 karolherbst: just have a job for each package
22:16 DavidHeidelberg: so it's easier just rebuild one package which gets affected
22:16 DavidHeidelberg: yes. Welcome in ci-deb-repo
22:16 mareko: distro-provided libdrm can be overwritten by our own, just build it and ninja install
22:16 DavidHeidelberg: https://gitlab.freedesktop.org/gfx-ci/ci-deb-repo/-/pipelines/1064687
22:16 karolherbst: but that's outside of mesa
22:16 DavidHeidelberg: mareko: alpine has it with alpine:lastest so that can be quickly resolved
22:17 DavidHeidelberg: for Fedora... hmm. I have to check
22:17 DavidHeidelberg: karolherbst: yeah, but as you can see there is like 8 packages. But in reality in Mesa we have like 20
22:17 mareko: why are we talking about other packages?
22:18 DavidHeidelberg: mesa CI should be machinery for building 20 packages. it should install stuff, run few scripts and give us the containers
22:18 karolherbst: the thing is just.. what do we then do with the fedora/other containers?
22:18 karolherbst: what if we come to the situation we have to bump it in all distributions?
22:18 DavidHeidelberg: it doesn't matter. Fedora just do build. Alpine just does build. All the other testing is done on Debian.
22:18 karolherbst: sure
22:18 karolherbst: but they might also need updated libdrm
22:19 DavidHeidelberg: we just bump to recent versions. The testing will be unaffected (except build test, which isn't very sensitive to new libs & stuff)
22:19 mareko: when you update libdrm, you don't need to update any other packages
22:19 DavidHeidelberg: I can bump alpine in upstream in matter of minutes
22:20 karolherbst: so maybe we shouldn't use debian then?
22:20 DavidHeidelberg: mareko: that's why you update only libdrm
22:20 DavidHeidelberg: you don't need touch other packages
22:20 DavidHeidelberg: karolherbst: enjoy rewriting our CI :)
22:21 karolherbst: ohh, I would totally do it to get rid of debian
22:21 DavidHeidelberg: guess why we have Debian. Because it's like immutable
22:21 DavidHeidelberg: we don't have to care that something there will change
22:21 karolherbst: ?
22:21 DavidHeidelberg: except what we change
22:21 karolherbst: how is that different with other repos
22:21 karolherbst: *distros
22:22 karolherbst: sure they update packages more often, but debian also could pull in some update breaking us
22:22 DavidHeidelberg: the difference how often. in Debian this happen rarely
22:23 karolherbst: okay, so getting rid of debian isn't an option anymore?
22:23 DavidHeidelberg: Also what I was working on was improving our situation to have stable repo for mesa-ci
22:23 karolherbst: yeah....
22:23 DavidHeidelberg: I mean, it'll cost you enormous amount of work with little to zero benefit
22:23 karolherbst: probably
22:24 karolherbst: but debian annoys me personally
22:24 DavidHeidelberg: what is annoying with debian is packaging. and I tried to "workaround it" with the repo I was working on
22:24 DavidHeidelberg: as you could see the yaml file I sent into chat, I think it's much better than f**** with debian/ directory :D
22:24 karolherbst: yeah...
22:24 karolherbst: debian packaging is pure pain
22:25 DavidHeidelberg: Mesa has few people involved in Debian, if you need quick bump, you'll just increase the number, should work like ... let's be pessimistic 90% of time.
22:26 DavidHeidelberg: when it'll not, you can ping me or someone from Debian packaging and we'll fix asap :) (it'll have to be fixed for debian anyway later, so...)
22:26 karolherbst: "quick bump" never worked for me in debian
22:26 karolherbst: even for critical bug fixes
22:26 DavidHeidelberg: the trick is we have the yaml and some automated magic behind this in the ci-deb-repo
22:27 karolherbst: does debian package allows us to have multiple versions of the same package in the repo and choose which version to install?
22:29 karolherbst: anyway.. I'm a little concerned with the idea of managing a single repo (probably used by multiple projects) but not being able to install a specific version
22:29 DavidHeidelberg: karolherbst: I thought about it 😉
22:29 karolherbst: and just installing it inside `/usr` and forcing the package system to assume it's installed is the solution with the lowest friction imho
22:30 DavidHeidelberg: developer laptop: "deb [trusted=yes] https://gitlab.freedesktop.org/gfx-ci/ci-deb-repo/-/raw/bookworm/ bookworm main"
22:30 DavidHeidelberg: CI scripts: "deb [trusted=yes] https://gitlab.freedesktop.org/gfx-ci/ci-deb-repo/-/raw/$HASH/ bookworm main"
22:30 karolherbst: but if we can't have like multiple versions in the same deb repo, we could just have a http server where you download specific versions from
22:30 DavidHeidelberg: so we'll have the repo always fixed at some specific software-set
22:30 karolherbst: mhhhhh
22:30 DavidHeidelberg: and when you want to bump, you just update the hash
22:30 karolherbst: I'd rather download deb files then
22:31 DavidHeidelberg: again, 3 archs, 50 packages (libdrm has for example multiple)
22:31 DavidHeidelberg: you don't want to go one by one
22:31 karolherbst: I mean.. it's just differs how you pull the packages, right?
22:31 DavidHeidelberg: yeah, you can still browse the GitLab UI and see the packages in repo
22:32 DavidHeidelberg: but the CI will pull them as needed
22:32 karolherbst: instead of adding them into a deb repo, you upload them somwhere, and instead of installing them, you download those, sure it's more painful, but again, how would we implement "install a _specific_ versions of X" then?
22:33 karolherbst: what if the version of ilibdrm inside that deb repo is 2.4.119 in the deb repo, but I want to CI that it still builds with 2.4.110 (because that's the required version in the build system)?
22:33 karolherbst: s/nside that deb repo//
22:33 DavidHeidelberg: you can for ci-deb-repo
22:33 DavidHeidelberg: then you edit libdrm.yml to old version
22:34 DavidHeidelberg: and you point CI to your fork @your_hash
22:34 karolherbst: so in the worst case, every project using that, creates a fork pinning specific versions?
22:34 karolherbst: what if you also want to juggle with stable/main branches?
22:34 karolherbst: so you mirror that branching in your work as well?
22:34 DavidHeidelberg: the stable branches will be fixed @some_commit
22:34 DavidHeidelberg: you can make stable branch
22:35 DavidHeidelberg: so, let's say I create branch from hash_a; then I'll push new commit; and I'll point to the new hash from the bookworm_23_1 branch
22:35 DavidHeidelberg: it's much better than what we have now
22:36 DavidHeidelberg: and the trick is, you can make stable branch only when you need some bump
22:36 karolherbst: couldn't we simplify this with external pipelines?
22:36 karolherbst: like
22:36 karolherbst: you have your config file of all packages as your input, the downstream deb repo CI does it magic only building what it needs to, and has a mapping of config_file -> deb repo container
22:37 karolherbst: mhhh or maybe other idea
22:37 DavidHeidelberg: I see an issue, how can you use it externally (for example to test on your laptop)?
22:37 DavidHeidelberg: git checkout $hash_a; git checkout -b bookworm_23_3; # do change; git push
22:37 DavidHeidelberg: how can you beat this?
22:38 karolherbst: why would I want to use it locally?
22:38 DavidHeidelberg: well, why would someone run any tests locally
22:39 karolherbst: I didn't mean running tests, I mean install deb packages from that repo
22:39 DavidHeidelberg: but these packages contains libs, llvm, tests, whatever :)
22:39 karolherbst: for llvm we should just use the upstream repo honestly
22:40 DavidHeidelberg: also debugging CI isn't very exciting with all waiting and abusing of FDO
22:40 karolherbst: but anyway, not everybody uses debian locally
22:40 DavidHeidelberg: sure
22:42 DavidHeidelberg: I'm not saying my solution solves all the problems of universe, but on other hand gives us software @versions which we need, while we keep stable system underneath
22:43 DavidHeidelberg: and gives us very precise control which version is in which pipeline, while we don't have to download,compile and install it by hand
22:44 DavidHeidelberg: and also allows us nice rollbacks without rebuilding stuff (for example new package causing problems, but we also did some other changes to CI meanwhile.. we just change hash and keep the changes in our CI untouched).
22:47 karolherbst: right...
22:47 DavidHeidelberg: While I get this is not that strong argument, but I spend like 2 years (incl. people from my team) playing with CI and I believe I see painpoints which can be solved. I wouldn't want to make my life worse... of course I can overlook something, but I hope this improves things for better.
22:48 DavidHeidelberg: there are other solutions which yield also nice improvements, maybe a bit better, maybe a bit worse, but so far this seems to me doable and workable :D
22:48 karolherbst: yeah, I just kinda wished that managing multiple versions wouldn't be soo annoying, but apparently there are patches to reprepro to do that...
22:49 DavidHeidelberg: karolherbst: I'm thinking I could eventually keep the old version inside the repository 4ever
22:49 DavidHeidelberg: then you could just pin it in our CI
22:49 karolherbst: yeah..., that would make it easier at least
22:50 karolherbst: maybe have a config which lists which versions to keep or something..
22:50 karolherbst: or rather...
22:50 DavidHeidelberg: we could keep everything, it's like git lfs
22:50 karolherbst: each project build jobs checks if all versions exists and only builds the missing ones
22:50 karolherbst: or we rely on storage and to never mess it up
22:50 DavidHeidelberg: it's just the reference
22:50 karolherbst: mhh git lfs might indeed be an option
22:50 karolherbst: but we might also have to rebuild versions
22:52 DavidHeidelberg: I see, experimental reprepro has it :)
22:53 DavidHeidelberg: that's why I like the whole repo hash downgrading mechanism, you have all packages build against each other (and we can be sure the Debian as a base won't suprise us)
23:03 karolherbst: well, worst case we'll see if it checks out or not
23:03 daniels: karolherbst: (yes you can have multiple versions of the same package present in a single repo, or overlapping in multiple repos, and select either a specific version, or latest from repo foo)
23:04 daniels: apt install libdrm1=2.4.119-1
23:05 daniels: or apt install libdrm1/karol-weird-experiment-repo
23:05 DavidHeidelberg: daniels: yup, the reprepro thou has it only in recent (this year) experimental version :P
23:05 daniels: DavidHeidelberg: really?
23:05 DavidHeidelberg: yup, I just checked the https://packages.debian.org/experimental/reprepro
23:05 karolherbst: is there another tool to manage repos, or would have done it through this "take this directory and make it into a repo" tihng?
23:06 daniels: we can just install reprepo from experimental?
23:06 DavidHeidelberg: https://askubuntu.com/questions/1377057/how-can-i-enable-publish-of-multiple-package-versions-in-reprepro
23:06 DavidHeidelberg: daniels: yup :)
23:07 DavidHeidelberg: anyway, I'm going to sleep, any idea how to reset gitlab-ci GIT caching?
23:07 daniels: huh?
23:07 daniels: I’m missing some context for the last question
23:08 DavidHeidelberg: I do force-push to git repo where later CI pipeline pushes... but it forget about my force push all the time
23:08 DavidHeidelberg: so it look like I didn't do git push --force from my PC
23:09 DavidHeidelberg: daniels: does it make sense?
23:27 daniels: DavidHeidelberg: sorry I was just getting home but also -EPARSE … I assume you don’t mean GitLab CI itself pulling the repo the pipeline is in, but some kind of internal cache?
23:41 DavidHeidelberg: yup, it seems gitlab caches the remote (I do git remote add; git fetch; git checkout)
23:44 DavidHeidelberg: daniels: there should be 2 commits (initial and push), nothing between
23:44 DavidHeidelberg: https://gitlab.freedesktop.org/gfx-ci/ci-deb-repo/-/commits/bookworm/?ref_type=heads
23:45 fluix: I'm also confused. what are these gitlab-ci bot (?) commits
23:45 fluix: like this just looks like you pushed and then whatever gitlab bot is pushed more commits
23:45 DavidHeidelberg: fluix: it's from the CI job
23:46 fluix: and what's wrong
23:46 DavidHeidelberg: the trick is, when I push locally 1st empty commit (git push --force) it'll ignore it on another push from CI and restore previous ones)
23:46 DavidHeidelberg: and it just appending to what I force pushed into initial commit
23:47 DavidHeidelberg: if you look at the same link, this is how the branch should look like
23:48 fluix: "it'll ignore it" what's the first it
23:48 DavidHeidelberg: now it's empty, then +1 commit from CI (but without these old ones)
23:48 fluix: the link right now only has your empty commit
23:48 DavidHeidelberg: yes. it should. but problem is, when CI pushes, it restores the "removed" commits
23:49 DavidHeidelberg: even when it shouldn't
23:49 fluix: example gitlab ci log?
23:50 fluix: however it's cloning, it means it has these commits. unless you explicitly configure some cache I don't think it's doing anything wrong. if you want your behaviour it sounds like you need a git reset
23:50 DavidHeidelberg: https://gitlab.freedesktop.org/gfx-ci/ci-deb-repo/-/jobs/53124278
23:51 daniels: DavidHeidelberg: yeah that does get cached, there’s a YAML variable to explicitly disable that iirc, but you really just want to fetch a branch and checkout a specific revision
23:52 DavidHeidelberg: good idea for a workaround :)
23:52 DavidHeidelberg: thanks!
23:53 DavidHeidelberg: even better, recalled I could remove the branch
23:54 fluix: gitlab auto caches repositories? like you're saying https://gitlab.freedesktop.org/gfx-ci/ci-deb-repo/-/jobs/53124278#L36 already pulls in the extra commits?
23:54 fluix: anyways, glad you solved it. this does look like a rather complex setup though
23:55 DavidHeidelberg: fluix: yup, GitLab loves to do some magic to "improve stuff"
23:55 daniels: fluix: the working directory gets cached by default
23:56 DavidHeidelberg: it's not that bad in complexity terms, anyway I don't want to pollute in future heavy used repo with some testing :)
23:58 fluix: how does `git remote add` not fail with `error: remote origin_gitlab already exists.
23:58 fluix: then?
23:59 fluix: nvm, there's a git remote remove
23:59 fluix: gotcha, thanks!
23:59 DavidHeidelberg: fatal: couldn't find remote ref bookworm ... 2 lines ...fatal: a branch named 'bookworm' already exists
23:59 DavidHeidelberg: F$#@