Disabling RTSS hooking for .NET apps

Brad Moore

• 3 min read

TL;DR; Source code for this post is available here.

RivaTuner Statistics Server (RTSS) is a great tool for monitoring system performance while playing PC games, however sometimes it hooks applications it shouldn't. In my case it seems in some users systems RTSS is hooking DLSS Swapper and preventing the app from launching.

Looking into the issue I found several cases of RTSS misbehaving with WinUI applications (but not every WinUI/WindowsAppSDK application 🤷‍♂️)

As of RTSS v7.3.4 Unwinder (RTSS developer) has added measures to try prevent this from happening.

Improved hook engine:
- Added WinUI3 runtime libraries to injection ignore triggers list
- Added 64-bit VK Play overlay library to the list of trigger modules for delayed injection engine
- Now injection ignore triggers list can specify modules, dynamically loaded by hooked application during runtime. This feature can be used to the exclude applications using delayed load of WPF/WinUI3 runtimes (e.g. Microsoft Power Toys)

RTSS v7.3.5 beta 5 also comes with a default exclusion for DLSS Swapper (thanks!).

But I wanted to see what else I could do to fix the issue for users who hadn't updated and as recommended by Unwinder in this comment I can add an export to prevent hooking.

extern "C"
{
    __declspec(dllexport) DWORD RTSSHooksCompatibility = 0x00000000;
}

The problem is this needs to be part of the executable and as of .NET 8 there is no way to do this nicely. The solution is to link that above code in a static library into a NativeAOT C# app.

Creating a static library was easier than I thought. A single C file and small build script. The source of both can be found here. Assuming this builds on your system it will produce a RTSSHooksCompatibility.obj file.

I am not able to detect that RTSS is disabled for DLSS Swapper (or even a new WinUI3 project) as it was not causing any crashing for me originally. I could not find any logging from RTSS so instead I decided to make a game, a very boring and cornflower blue looking game.

The simplest dotnet/C# based game I can make is with XNA, but that was discontinued almost a decade ago. So I turned to the next best thing, MonoGame! Using the default desktop OpenGL template I was able to get an application up and running showing the performance indicators as I had hoped.

MyAwesomeGame showing an RTSS overlay

Knowing I was going to use NativeAOT I had to update my app to .NET 8 which was incredibly easy. Cracking open the csproj file I had to change

<TargetFramework>net6.0</TargetFramework>

to

<TargetFramework>net8.0</TargetFramework>

and knowing I was going to enable NativeAOT, below this I added

<PublishAot>true</PublishAot>

To add embedding of my obj I also added

<ItemGroup>
  <NativeLibrary Include="../../RTSSHooksCompatibility/RTSSHooksCompatibility.obj" />
</ItemGroup>

To publish my game I then need to run,

dotnet publish ./MyAwesomeGame/MyAwesomeGame.csproj --self-contained -r win-x64 -c Release -o .\publish\

And the final result is a game that does not have RTSS enabled. Again, do NOT do this if you have valid reason. Don't lock your users out from having access to RTSS, they can just re-enable it by setting ForceHooking=1 in [Hooking] section in either the global or game specific profile.

MyAwesomeGame without an RTSS overlay

Using a tool like Dependencies you can take a peek at the exe and see that RTSSHooksCompatibility is indeed present and doing what we expect.

Picture of the Dependencies application after having loaded MyAwesomeGame

Next steps, NativeAOT compile DLSS Swapper đź’€