12:18mannerov: so the memfd solution works to reduce virtual memory usage. I now have ME2 with the texture mods running well. Still a few things to figure out (for now it's csmt_force=0), but going well
12:47LiquidAcid: mannerov, nice, i guess it's time to replay the trilogy then!
12:59mannerov: yep :-)
13:03mannerov: but fixing everything first
13:40mannerov: seems to be working as expected. Only remaining thing is the thread safety for the csmt thread :-(. I probably cannot do without mutexes (I hoped at some point I could use atomics only), however I have to decide what do I protect with the mutexes.
13:41mannerov: the csmt thread can release textures. That's the only thing that needs the thread safety. But it involves updating many structures to update the infos needed to reuse the space freed
17:52iive: mannerov, what was the problem and how does memfd solves it?
17:57mannerov: problem: running out of ram memory
17:57mannerov: solution: allocate more than 4GB on 32 bits via memfd.
18:02mannerov: basically on linux you can allocate memory using mmap
18:03mannerov: I heard that basically it's what behind malloc on linux
18:03mannerov: the problem is that it takes space in your 4GB (obviously)
18:04mannerov: what if you know you don't need the data for a long time, and that for accesses the user has to call lock/unlock functions. Basically you could store the data somewhere else, and retrieve it when needed.
18:48mannerov: memfd it's basically mmap/munmap except that it is associated to a ram file, so when you munmap you don't lose the content and you can read it back with mmap
18:49mannerov: thus you can use it to allocate more than 4GB while using less virtual memory space
19:00iive: how often is that data needed? once per frame? once in a while?
19:01mannerov: either it's used only a few times shortly after the data is allocated, and maybe reused much later (like level change)
19:01mannerov: or in addition to those usages, it's used once in a while randomly during gameplay
19:02mannerov: but only for a few of these per frame
19:02mannerov: (basically this usage, which happens with ME2 is uploading a higher resolution texture and copying the low res texture to the lower mipmap samples of the high res one)
19:04mannerov: if not for this use case, the simplest would be to retrieve the data from the gpu memory version of the texture
19:04mannerov: however this usecase would cause stalls if doing that
19:04iive: what i'm kind of asking is if we need to keep a copy of the data at all. OK, you just answered that.
19:04mannerov: wined3d and dxvk (optionally for the latter) do that