GD modding in R

Grim Dawn Modding - Projects
1

This has been WIP for a while. I think it might be close enough to get some eyes on. It explains a lot about modding, but I highly recommend, if you have any programming in you, to install R and try it out.

There might be bugs, I think some code was written during the documentation process without testing.

These two attachments need to be downloaded and put in your Working folder. The Working folder is where you had AssetManager extract the game files and has the database and resources folders in it. Open run.R in a text editor and make sure src.R is there too. Read run.R and try to follow along with a live R session.

I will break the contents of run.R over multiple posts below.

Attachment: run.R.txt
Attachment: src.R.txt

2

This will take a bit of time.

It is more reading than coding, at least at first.

But I hope to pop some knowledge on you that makes modding GD

both easy and fun.

First of all, I feel I should qualify the language. R is a scripting

language, aka runs in a live interpreter. You type something, it happens.

You can also store script in a file and run batch jobs.

I have C/C++/Java/Objective-C/Qt/Tcl/Tk/Javascript as languages I can

code in. But I work exclusively in R (some C/C++ but for R packages) now.

It is a really nice language. Julia looks fun too, but I have dependencies

for work in R and so, for now, R is it.

So before we move into R, lets talk about what GD modding is about.

GD has a database which is a collection of records. Each record has a name.

The recordName is basically a path to a dbr file.

Inside that dbr file is a series of fieldName = fieldValue pairs.

The values can be all sorts of crazy stuff from strings to

ints,floats,boolean,arrays of these types and strings that are

recordNames to other dbrs.

This method uses an array of strings to hold all recordNames

Each DBR is a list. A list in R is NAME -> OBJECT pairs.

The name is always a string (the fieldName).

The value is a mixed data type but can be stored as a string.

Thus each DBR can be stored as a list.

This list is indexed in another list using the

recordName as the index key.

Thus, to start modding GD, all we need is a list of lists.

recordNames -> LIST_1 -> DBR ( LIST_2 [ fieldName -> fieldValue ] )

To make a mod, you need two locations on your file system.

Working refers to the location where you have placed run.R and src.R

and where you have the database and resources folders from the

extraction of the game files that you did using the AssetManager.

BEFORE YOU USE ASSETMANAGER, READ THE PDF GUIDE FROM CRATE.

The second folder is the output folder, where you want to write the

modded records to. You can then use AssetManager to build the mod.

In fact, you should use AssetManager to make the mod (empty) before

you write the records with this script.

Before you can use any of the script in this file you will need

to have R installed (preferably use the 64-bit version).

https://cran.r-project.org/bin/windows/base/

Comments are lines in code that do not execute, they are there to

aid the reader in understanding what the code does.

such lines in R start with the ‘#’ symbol.

If you paste such lines into an R session, it is meaningless

the comments will have no effect.

First thing you have to do is identify your working directory.

This is the location that the GD-Toolset extracted all the game files.

For example, on my windows 7 system it is here:

C:/Users/jiaco/Documents/My Games/Grim Dawn/Working

Before I show you a single line of R code, you must promise that you

have R open. Just reading this will not impress you at all.

When you have R open, any function that you see has help builtin

and can be accessed but ?function at the R prompt.

If you do not understand the concepts of variable, function, operator,

operator overloading, string, etc… There is probably a wall nearby

that you could bang your head against and it would be faster and less

painful for you. This is programming, and certain things are assumed

to be known to the reader.

Here comes some more introduction and then actual R code.

Before I set a string variable to hold my Working directory path,

There are these points to know.

1) in R, single strings as well as arrays of strings have the same type

which is a vector of character. See ?vector

2) in R on windows, you set a path in unix format not in windows format

(unix:slash(/) vs windows:back-slash())

work.dir = “C:/Users/jiaco/Documents/My Games/Grim Dawn/Working”;

there, actual R code. That line above needs to be copy/pasted

into your R session. You now have the work.dir variable in your

environment and it holds the length 1 vector string(character)

pointing to the location where we READ the extracted game records from.

We also need the target location, where to WRITE to. The place

where the modded records can be then built into a mod with AssetManager.

out.dir = “C:/Users/jiaco/Documents/My Games/Grim Dawn/Working/mods/Grimmest/database”;

Last warning. Everytime there is a line that does not start with ‘#’

you need to copy/paste that line into the R session.

Besides the variables that you set, there are environment variables

inside the session. One of them holds the current working directory.

You want to change that to move into the Working directory.

setwd( work.dir );

Be sure you put the src.R file you downloaded into the same location

as you set your work.dir to.

this next command will load all the little functions I wrote for GD

into your R session so we can use them.

source( “src.R.txt” );

If the above command fails, the command before that failed.

this is common in R. You get stuck at step N but the error is at

step N - X. Find X, back up and fix that step. This is pipeline code

meaning that you can not jump around and do something on line 50 without

possibly also needing to have done things on line 1, 5, 12, 25, 44 and 49.

This brings us to:

You communicate with a live R interpreter via a programming language R

When you start a session, you get an environment, where you work

and make variables and run functions on those variables.

here we want an array of strings, which in R is called a

vector( mode=“character”, length = NUMBER_OF_DATABASE_RECORDS )

this number of database records is going to be determined by the

function that recursively scans your file system starting at work.dir

for dbr files

records = listRecords();

listRecords() is a function in src.R, check it out now.

you see, it recursively list.files() in the “database” folder in the

current working folder which was set before with setwd()

the vector of character is then named with itself.

This become important in the read step later.

If you reached this point without doing the following in R:

?list.files

?setwd

?vector

then you are doing this wrong. At least prove to yourself

that you can ask R for help 3 times now please.

3

LESSON 1: Making and using a Binary Serialized Object.

You have to have the tags. Tags in GD transform tag-variables

that are found in fieldValues of database records into language

specific text strings that are the descriptions found in the game.

tags = loadTags( “resources/text_en” );
tags.save( tags );

Then next time, instead of running loadTags,

you can get the data back like this.

tags = tags.load();

because tags.save() dumps the R-object to a file in your Working dir

if tags.load() cannot find it, it will not work, use loadTags();

you can do this with the entire database too, but not now.

THIS STEP CAN TAKE TIME

You should be using 64 bit R GUI in Windows if you can.

Be patient, once you do it once, it is faster to load the BSO

But there are a lot of file operations and it can be long.

The rest of the tutorial DOES NOT really depend on this.

So I comment here

dbrs = readRecordS( records );

dbrs.save( dbrs );

the two commands that make it happen.

dbrs = dbrs.load();

and the command to load it back into memory for a fresh session.

again, look at src.R and see the code

4

LESSON 2 : Filtering your set of records/DBRs

In a lot of projects, you will not need all the records loaded

there are 3 main ways to filter a set of dbrs. But first, we have to

look into what we did above with the naming of the records vector.

Inside readRecordS is a critical R builtin called lapply().

This is standard fare in R. You have a vector of something

and for each something, you want something else, of variable nature.

so the class used is called a list() {hence the l in lapply()}

a list is like an array but each element of the array can be anything

including a list.

by naming the records vector, we ensured that the dbrs object

back from the lapply() call in readRecord() is a named list

where now the record name does not just index a vector of charater that

contains the same data as the index string, but now contains the

contents of each dbr file represented by that record name (path)

the basic 3 ways to filter are

by the path, or the record name. For example proxies live here:

proxy.records = findByRecordName( records, “records/proxies” );

the next 2 ways, Class and templateName require the contents of the DBR

in order to search.

proxy.mixed = readRecordS( proxy.records );

if you have dbrs obj from a dbrs.load() you can do a similar subsetting

proxy = dbrs[ findByName( records, “records/proxies” ) ];

again, this is because dbrs is indexed on the record name (path).

findByRecordName() returns the string of the path matching the search string

see ?list

learn the difference between dbrs[[ recordName ]] and dbrs[ recordNameS ]

now that the records are loaded, we can search on the class

proxy = subsetDBRsBy( proxy.mixed, “Class”, “Proxy” );

or templateName

proxypool = subsetDBRsByTemplate( proxy.mixed, “ProxyPool” );

so remember before, a list can be made up of lists? well that is the

data type I use. A dbr.list is a named list of dbrs. The name at this

level is the record name (path).

inside, at the second level, the named list is named with fieldName

and each field contains fieldValue obj which is a string

the fieldValue string might be character or int or float or boolean or

a string that is another dbr record path or an array of any of those

separated by ‘;’ in the string.

you have proxy.mixed, proxy and proxypool now. 3 dbr.lists

try length( x ) where x is one of the dbr.lists

names( x )

lapply( x, length )

lapply( x, names );

and of course try ?lapply

5

LESSON 3 : REMOVE MONSTER LEVEL SCALING

so in grimmest, there is this trick that removes level scaling:

proxy.set = findByRecordName( records, “records/proxies/area001” );
proxy.set = subsetDBRsByTemplate( proxy.set, “Proxy” );
proxy.mod = lapply( proxy.set, addField, “difficultyLimitsFile”,
“records/proxies/limit_unlimited.dbr” );

remember the out.dir we set a million lines ago?

well you do this and all the records get dumped with the new field

in AssetManager you can now build your mod with these batch modded records

writeRecords( proxy.mod, out.dir );

6

lesson 4 : Extract data to plan your mod

wip

7

lesson 5 : Automate the creation of your mod

wip

8

wip about 10 little characters