[Tool] GD-DPS : Grim Dawn Data Patching System

Alternate download link on moddb.com:
GD-DPS with Grim Dawn EU Lite MOD

Already available since early November 2020 on the “MOD Project” section of this site, I thought it could benefit from a bit of visibility in the “Utilities” section too. So here it goes:

What is this? How does it work?

The GD-DPS or “Grim Dawn Data Patching System”, consist of several .AHK and .BAT files that are designed to operate together. GD-DPS patching operations flows look something like this:

1 - Starting from a working copy of the game, a backup “source” is created. These source archives files, ARC and .ARZ, are then extracted each in their respective, separate folders.

2 - Scripts are then executed to “patch” the game databases, templates or resources contents. These patching scripts determine what is actually changed, or modded, in the game. Using AutoHotkey powerful regular expression evaluator, anything, from changing the attack sound of an item to completely recomputing the attributes bonus from a class mastery, can be done. Resources file too can be manipulated: Textures, Meshes, Conversations, Sound files, etc… Binary patching, by providing an offset from the start of the file and a data buffer, can be done to modify non text files.

3 - During the above process, nothing from the source backup is changed in any way whatsoever. This means that complete restoration of the original game files, or parts of it, can be done at a later time.

4 - Once all the patches are applied, the game archives files are then reconstructed, automatically, from the modified content.

5 - Finally, the game can be played in its new state, with or without additional MODs provided by the community.

How can I get started moding Grim Dawn using this system?

The easiest way would be to familiarize yourself with the way the games Templates, Databases and Resources work together. Trial and error or looking for information on the Grim Dawn official forum are a good ways to learn. Once you have a basic understanding of ARZ and ARC files, then you can start experimenting with GD-DPS.

Double-click on the Windows batch script file named “GD-DPS.bat” from a Windows explorer view. This will open a GD-DPS console window. From there, read the simple instructions provided. Of course, there is a moderate to steep learning curve associated with moding a game, any game. However, a lot can be learned about the intricacies of the GD-DPS by opening the various .BAT and .AHK files in a text editor and inspecting how different command operates. The core patching function of the game are designed with AutoHotkey high level language. DBR, the records that the game use to implement its data structures, are text file. They are, by default, encoded as ANSI with Unix style line endings. Since they are text files, they can be modified easily by a powerful search and replace algorithm known as “Regular Expression” or RegEx for short. GD-DPS comes with some simple functions that leverage RegEx to find, insert and modify DBR entries. Using AutoHotkey, very advanced patch can be scripted, for example, to read a DBR entry, perform arbitrarily complex calculation on its components and replace them with newly computed data.

Resources, sounds, textures, world “levels” to name just three, can also be manipulated. GD-DPS uses compact “Packages” to deliver, delete and/or replace any extracted resources. So, for example, say you would like to add more variety to and replace some of the guns sounds: The creation process is up to you, the moder. You can start with existing sounds or edit new ones with the audio software of your choice. Once you have the new “resources”, the sound files in our example, GD-DPS can handle the rest. First you create a “GD-DPS Package” using the aptly named command “Package.bat”. Simply put, you replace the games assets files that you want to change then you provide the “locations” of these files to “Package.bat” and it does the rest. Should you need to also remove some files from the game resources pool, post-extraction action, POSEXAC, can be programmed into the “package” to perform whatever needs to be done after insertion of any new files. Finally, to deliver that GD-DPS Package in a controlled (hopefully!) manner, there is a core function dedicated to that that can be implemented in an AutoHotkey, or AHK, script file.

Binary patching of a non text file is also possible. This works by providing the target file you want to modify as well as the offset, from the beginning of said file, where you want to overwrite the content with your own provided “array of data”, anything from 1 byte to the whole file length worth of bytes. Again, a core GD-DPS function handle this, both in pure binary form or in text string form. CNV conversation files are good candidate for this operation.

To conclude, GD-DPS can be seen as a tool set. Different tools perform different actions on the game files. Used together, they will allow you to deconstruct, patch records, add new ones, modify the game world, inject new resources or replace default ones and, finally, reconstruct the data into archives files that the game can use. Just like the default ones.

So, this is a “base game” moding system?

Yes. The way Crate designed the default tool set and mod system, creating “custom games” that can be loaded on top of the base game, as many problems. It is great for simple “add-on” type mods but cannot be used, as is, for more advanced moding; You need better tools for this. I believe the GD-DPS toolbox to provide these new, improved tools.

2 Likes

Just letting you know that your file is flagged as “ suspicious “ in nexus.

1 Like

@Galenmacil looks very interesting am building a mod atm so will def give this a go, will give you some feedback if you want?
Question: Is there a ‘heavy’ / ‘full’ version or just the lite (being named for a lightweight editor not because there is another version etc.)?

1 Like

tippytam: Yes. This is very annoying. I keep sending them email to ask for download unlock, which they provide, but they told me “until you get to 1000 downloads, locking and marking a new upload as suspicious is standard procedure when the mod contains .BAT or .EXE”.

I have to roll with the punches on this. With the recent “revelation” that, oh surprise!, Microsoft have some serious problems with their NTFS file system, I cannot blame them. It is so shamefully simple to crash a Windows system it is almost ridiculous:

So, that is why I started providing a link to the MOD on ModDB.com instead.

1 Like

Khalarin: No there is only one version of the mod. The name might be a bit confusing, I can now see that but it originally stood for: “Elite/Ultimate Lite” as in lighter elite and ultimate experience.

I am welcoming feedback of course! If you intend to use GD-DPS for your mod, you will be the first to do it as far as I know, so. I am honored…

Galen, I thought the name was something like that just wanted to check just in case =)
Oh I will def give it a go, AM has a few issues for some reason it refuses to import records so I have to do it manually which is really not ideal hopefully this tool will help!
Wont be for a few days though as am decorating atm but its installed and waiting to go =)

Will report back soon =)
-Khal

1 Like

Hi,

Just wanted to say that your rework on the sound is phenomenal!
Now the game feels somewhat spooky & has a sense of dread which makes it more immersive. :clap:t4: :raised_hands:t4::pray:t4:

1 Like

Thanks tippytam.

A lot of work went into sound improvements. I am glad you noticed how much they improve immersion. :upside_down_face:

1 Like

I just released a new version of GD-DPS.

It is now possible to patch custom game MODs. Currently, EU Lite MOD comes with “Dawn of Masteries” compatibility patches built-in. Other mods may get their own compatibility patches if there a demand for them.

3 Likes

:open_mouth: Wow! A mod within a mod??
This is absolutely astronomical! :star_struck:
The best is yet to come! Thank you so much for this! :clinking_glasses:

1 Like

The installation of the mod got stuck with this wording:
"All patches applied. Press any key to begin reconstruction process…"
what should I do?
…and above all, why?

Hello Piale. This is most likely due to a “feature” of the windows console windows that gets “paused” whenever you click or click and drag inside the window. To unpause it, make sure it is the active window and press any key.

If this is not solving the problem, here a few other things you could try:

  • Make sure your account has administrative privileges.

  • Make sure that you confirmed/answered any pop-up windows that appears. There are 3: One ask if you want to scale the movement speed of pets, one ask if you want to activate >100 level progression and the last one ask if you want to set the maximum devotion point to 200. The installation waits for the 3 dialog windows to close before continuing.

  • “Protection software” can potentially cause issue with installation by silently blocking script execution. You could try to disable any protection software in effect for the duration of the installation.

  • Do you use, on your computer, custom made “AutoHotkey” script?

  • Can you confirm that there is enough free space on your drive? You need a minimum of 50GB free space or the installation risk failing.

  • If you want to retry to install the mod, before doing so, open the file called “Setup.bat” in a text editor and search for a line that reads “SET Debug=False” and change it to “SET Debug=True”. This will change several aspect of the installation:

1 - You will receive extra warnings and continue confirmation along the way. Pay attention to the “Press any key to continue…” prompts.
2 - During the patching stage, in the notification section of the windows task bar, you will see, for each currently running script, an “H” letter shaped icon. This can be useful to determine which, if any, script is stuck and why. You can right click on an “H” icon and terminate a misbehaving script.
3 - Pay attention, in particular, to the now visible console windows that shows the progression of “FFMPEG.exe” sounds modifications. It will rapidly list what sound files are edited and it should progress all the way until it ask you to “Press a key to conitnue…”.

If the installation still fails to complete, post the last message shown on the console windows when it hung here and I’ll try and guide you thru other possible fix.