[1.3.0.6] Shaders causing performance drops in specific areas

I’ve decided to make this separate thread even though I’ve been updating this one. I’ll start with a summary, then some findings, some guesses, and a plea.

With Fangs, you get increases in frametimes, FPS drops, and sluggish rendering at specific areas, when Shadows are enabled on any quality level. As soon as shadows are Off, none of this happens.
I do not, at all, refer to the microstutters/microfreezes the game suffers in general. My best guess about those is that a single thread is burdened with decompressing and loading-in resources from ARC archives, and you get a microstutter if the same thread happens to be performing something computationally heavy at the same time. Offloading resource load-in to a separate thread would be my first line of attack, but of course I have no knowledge of how feasible multi-threading this engine is. Excuse the digression.

Two areas which consistently reproduce the issue are (1) the bridge just after Fort Ikon and before the first Fangs area: you consistently get awful frametime jitter when walking to and fro the new world (I say world when I mean a newly loaded area as indicated by the overlay text that you’re entering a new region). Remarkably, playing the game on DirectX 9 does not reproduce this, initially indicating something to do with shaders.

At any rate, the second area (2) is Kurnhold itself. Most of the city, but especially the spawn area and the circular dwelling where the riftgate and bounty table are. At spawn, you rarely get more than 60-70 FPS (where the game normally manages an average of 140ish) but around the riftgate it always, consistently, drops to 52-56ish. The drop corresponds to a steady, measurable increase in frametime while the GPU never gets pushed beyond 70%. You feel the framerate return to normal as soon as you leave north of spawn, around the entrance, to the waterfall vista, or if you go up the stairs east of spawn towards the vendors. Walking west of spawn towards the altar is still burdened with the poor frametime. The area around the cauldron and vendors suffers the same issue, as do the steps you enter the city from and Yigra’s area. It is only after you’ve gone down the entry stairs and exit Kurnhold that framerate shoots to 120+. Performance appears to be better on DirectX 11 there rather than 9, although both suffer the same issues.

The first things I did was try to mitigate via DXVK configuration. Setting dxvk.tilerMode to True was the only thing that made a measurable difference, granting a few FPS. But that’s a tiny band-aid.
The real breakthrough was gutting each individual shader (or group of shaders) from Shaders.arc to see whose absence makes the framedrops go away. I’ve been able to narrow it down to standardstaticdetail.ssh and standardstaticdetail_nofade.ssh. When both of those shaders are removed, framerate remains stable at least around Kurnhold spawn and riftgate. (Removing either of them individually retains the issue.) I happen to keep the GOG offline installers for version 1.2.1.2, so I tried replacing just those two shaders, and then the entire Shaders.arc archive just to see what happens. The issue persists. So, it doesn’t appear to be a change in those two shaders, but rather what they do in the affected area(s) that causes problems. Almost as if there’s a calculation running haywire…

Clearly, it’s to do with shadows (as switching them Off alleviates this). Now, precompiled shaders I have no idea how to debug, but running strings to see if my shadowy hypothesis might be correct, proves there’s several shadow manipulation functions in (at least) standardstaticdetail.ssh. "worldToShadowMatrix" sounds very suspicious to me. It makes me imagine there’s shadow calculations happening onto, or applied to, a given area. Then, something like that would explain why specific areas trigger framedrops when shadows are enabled. It makes me imagine the Kurnhold area has something unique that causes the shadow calculations to go haywire. What’s also interesting is that setting shadows to Low makes the shadow coverage in Kurnhold very unstable and weird, which also points to this hypothesis. Although I admit that last part might be something on my end as I haven’t yet asked if anyone’s able to reproduce it.

And so my plea. Please review these two shaders. There is verifiable, reproducible, measurable and felt performance loss which has to be addressed. I’m willing to record video evidence of this even though I’m very bad at it. If it isn’t clear I’m doing all this because I love the game so much, all I want is to have a stable rendering experience. My only concern is this might be swatted off as not reproducible and not addressed. I’d happily fiddle with the shaders myself, but as I said to my knowledge you can’t decompile precompiled shaders. And I don’t fancy playing without shadows. Thank you in advance.

Computer specs for reference:
Ryzen 5500 boosting to 4400MHz,
16GB 3200 MHz DDR4,
RX 6600 8GB stock,
PCIe 3 NVMe.

6 Likes

Amazing work. I’d be really, really disappointed if even this is not worth an acknowledgment/response from the devs.

1 Like

Again, thank you kindly. Does anyone happen to know if this persists on 1.3.0.7? I have not yet had a chance to test as I require the GOG offline installers.

1 Like

Yes, these are new shaders and new tech for FoA and used in the new assets, which is why they are more intensive and why turning down shadows makes such a difference for you.

Switching off. Not turning down. If we have shadows on at any setting, the performance tanks. Not “more intensive” load, tanking.

2 Likes

I’m concerned the nature of the bug report might willingly or unwillingly get misunderstood. Which is why I was as thorough as I can in the OP.

But to be perfectly clear: we are not reporting performance being more taxing and thus requiring better hardware. We’re reporting what we’ve confirmed to be a product issue independent from hardware, which manifests at the mentioned areas (and most certainly more than the mentioned) whenever shadows are at all enabled. Resulting in less than half of normal average FPS.

2 Likes

Shadows with anything other than Off makes the FPS drop from ~180 to ~60 running at Ryzen 5800x with PBO, RTX 5070ti, 3440x1440@120Hz gsync (totally very much over High of the game requirements). It completely maxes-out just one core of the 8. I have this kind of fps dips in Kurnhold and other areas with vast Z-axis range.

Please see my posts here:

3 Likes

But also it cannot be that new shaders are causing these issues. If I replace the entire Shaders.arc from version 1.2.1.6 into 1.3.0.6, the same issue persists. It has to be the way shaders interact with the affected areas.

2 Likes

Update. I tried gutting every Kurnhold mesh one by one to see if it’ll make a difference. I couldn’t land on a specific mesh or mesh group causing the issues. I know if I gut enough of them, performance is fine - but that’s not really helpful. (Looking at you Zantai: I’m not dumb enough to believe getting rid of 40 meshes is a fix.) I did learn a few things while doing that though; not to mention while trying to gut terrain meshes.

Changing the angle matters:

Here’s us having a framerate drop to 51.2 FPS…

And here’s us having no framerate lag when shifting the view angle.

Again up next to the cauldron…

Oh boy it gets worse…

Ah but then it’s fine! Notice the frametime graph…

(Again: I am not taking screenshots while rotating. I shift my angle, wait a few seconds, then record a screenshot.)

And here’s just another example of the view angle fixing the performance crap. Keep in mind, I’m on a 75 fps frame cap, to fix other performance issues with the game… Here is clearly visible the smoothing out of the framerate graph when shifting the perspective.

Another interesting this was getting rid of the Kurnhold platform meshes:

Do we do any culling at all? Do we render the cliff_granite* meshes all the time? Why would we do that?

Now, this all makes me rather sad. I was hoping to present to you one or series of meshes whose shadow calculations don’t work properly, as per my initial hypothesis. I wasn’t able to. But, for the life of me, I don’t know why we hadn’t tried changing the camera angle first. Surely that’s a strange phenomenon! Is it about a mesh out of view being culled, and fixing the frametime? Is it a shader that’s affected by the angle? Is it the height of the cliff textures being changed depending on the viewing angle? (Zooming in makes no difference.) Not to mention knowing, for certain, that it’s not down to the asset intensity as per Zantai’s proposal:

It isn’t their intensity causing issues:

  • because changing angles fixes performance;

they aren’t issues for me:

  • because we have people replicating this,

nor is their being new shaders:

  • because the offending shaders taken from pre-Fangs versions of the game make no difference; in fact replacing the entire Shaders.arc - thus having NO new shaders - makes no difference,

and as we’ve established toning down the quality of shadows makes no difference. It is down to there being new assets not getting along with the old shaders. So either the shaders need tuning, the assets need revisioning, or the culling could not be happening as consistently as necessary.

2 Likes

There’s something very wrong with shadows in the town of KurnHold, and possibly the other new areas. They’re causing a massive drop in performance/FPS. I’m playing with fps capped at 60fps/vertical sync, and my game started to stutter heavily when I reached the boss just before the rift at Ulo’s Spring.

In KurnHold with Shadow OFF 447 FPS, shadow to LOW ~200 FPS, and Shadow ULTRA still ~200 FPS but using more power to render. I can’t post multiple links or embedded image since I’m a new user to the board, check the imgur link before to see the album with the screenshots. I also added Devil’s Crossing for comparison

https://imgur.com/a/o56080b

edit: My autocorrect is misbehaving.

3 Likes

My own anecdote and might not be related but on Linux there has been huge temperature issue since 1.3.0.6 with Vulcan. Playing the game with OpenGL reduced the temperature from 75 to 50 Celcius.

2 Likes

I would love to test on 1.3.0.7 but the GOG offline patch is still not available.

I’ve noticed the same thing. How are you forcing OpenGL?

I’m guessing he’s using Wine’s default WineD3D layer rather than DXVK. I find performance terrible with it.

Let us not discuss Linux performance workarounds here. This might steer the conversation in the wrong direction. I want performance problems which are clearly not OS-dependent to be addressed.

3 Likes

Wow. My buddy @christo2211 here, working his ass off for free, doing your work for you, and this is all you have to say after all of this, after two whole threads of information data and evidence?
Wow.

It’s not just for him, but for many others across different systems and configurations, including me.

It’s not turning down shadows, it’s turning them completely OFF.

But does this even matter, I must wonder. After all that, just to get dismissed like this ?

Again, huge hats off to you man. Amazing work and patience. From how I see it, and when I actually read what you report, you present good and thorough evidence, comprehensively too. I fear tho, that in the end, whatever you find out, it’s up to Crate to actually code it in, fix it, implement it, and judging by their approach I’m starting to be even less optimistic.

Sorry if I’m being too doomy about this, but Zantai’s response really rubbed me the wrong way. Seems disrespectful to you especially. I’d wager to say that you’ve spent the most time on this out of all of us.

2 Likes

Now, what’s unfortunate is this doesn’t only happen in Kurnhold. You were right about Malmouth areas, and I do occasionally see this in Gods areas too, especially last time in the Lost Oasis. I haven’t thoroughly inspected those areas. My thoughts are if the stuff in Kurnhold can be enough for troubleshooting, perhaps whatever Crate does there will have an effect across the board.

I don’t know about you being doomy, but I’m optimistic this is something that must be repaired one way or another. Perhaps I’m incredibly off and nothing of what I’ve found is related to the issue – but we can start by admitting to an issue. I’m hopeful we’ll get an official statement about it soon. Acknowledgement would be a fantastic first step.

And once again, thank you for the kind words!

1 Like

I really mistyped the FoA, I didn’t mean Malmouth areas but Fangs. I get those two acronyms mixed all the time, for some reason.
But yeah, Malmouth city areas like Steelcap, always ran choppy for me. Didn’t test it much on 1.3.0.x. I think it still ran the same, but I don’t remember too well…
Sorry for the mixup.

This is my last hope as well.

Fully agree. That’s also why I’m frustrated with Zantai’s reply, no acknowledgement, nothing.
I of course, hope your optimism is well placed.

1 Like

I already acknowledged that Kurnhold runs poorly and we are looking into optimizing it.

The shader discussion here is not the magic bullet fix you’re hoping for, regardless of how much thought and research the OP has put into it.

Not in this thread you haven’t. Nor in a patch note thread or a development update thread.
I’m glad you’re acknowledging it here now though, thank you. I shall graciously be looking forward!

I’m hoping performance gets fixed, I’m not claiming to have found the root cause(s). I did call those “guesses”, didn’t I? Only sharing what I think might be enough proof for you to consider it being an issue. Which, judging by your first response here, did not seem you were willing to do. Again, I am glad it is now considered an issue.

1 Like

On Steam launch options
PROTON_USE_WINED3D=1 %command%

1 Like