[WIP] ArzBaller - the CLI for packing dbrs

Grim Dawn Modding - Projects
81

I did a lot of testing and checking this time around before posting. again using the grimmest sources and arz for all testing since its a small folder and is available for everyone :slight_smile:
using the source it crashes…
using the arzBaller source it works fine, just like before

so I copied the gameengine.dbr from the source over to the extracted folder to see if it crashes… it does. In addition the gameengine.dbr from the source folder was 2kb larger than the one extracted by arzBaller.
I checked the lines… 660 compared to 540. In other words arzBaller is skipping lines and crashes when it gets to a field it skips when extracting.
I went one step further and edited the arzBaller gameengine.dbr with the DBR-Editor, added a space to the fileDescription and removed it just so I could save the dbr without actually changing anything - now its 831 lines and of course crashing the arzBaller packing.

I uploaded the files for comparison and the only working file is:
gameengine_grimmest_arzBaller.dbr
and thats the one I got from extracting grimmest.arz with arzBaller.

82

off topic, I love the graph we get out of this lol.

yCSWGoJ.png1001×668

83

Honestly, at this point, there are soo many arzs and folders of dbrs on my system, I am not even sure who is who or what is what.

Going to start from scratch tomorrow and setup a complete set of tests. You might not hear from me for awhile…I have to clean the place I am at and drive back home tomorrow as well. So I doubt I will post a new version before that.

84

v0.6 should not longer seg fault (crash) but I have no choice but to write a complete template parser. so this version may work for some things, but not everything, for example, Omega has Dagger, which should be boolean, but the code will guess int.

Source:

Binary

I will work on the templates next time I get time, for now, I am packing/cleaning/driving/unpacking/cleaning and then going to have to do some serious gardening…as the jungle has not been touched in ~3 weeks.

85

You can pre parse the file, search for , and thus mark that as bool, instead of fixing templates. No?

Btw, along with experienceEquation I recall you mentioned another record saying “this might be it” do you remember what was it?

Ty

86

after trying it again with the grimmest source (which worked) I tried my mod to see what isnt working. Ill do some more testing but for now:
I created an inst kill the player mod with arzBaller!
the moment I log in I die… and die… and die again, an endless loop of death :smiley:

on templates… I dont think it would be the worst you could do, to make a template parser - when Crate updates/adds a template you dont need to worry about it.

87

Insta death is likely all related to the same issue.

I am back home, things do not look so bad here, but we brougt home an abandoned kitten and already had two cats, which look fucking HUGE now…and this is not going easy.

I should have the parser soon though, will start tomorrow at 6 am, while the rest of these monkeys sleep.

88

Just to prove the parser is done.


setwd( "c:/users/jiaco/documents/templateWork" );
t.path = "templates";
t.suffix = "*.tpl";
lsd = list.files( t.path, t.suffix, recursive = TRUE );

file.path = sprintf( "%s/%s", t.path, lsd );

parseTpl = function( path )
{
	not.used = c( "list", "system" );
	rv = list();
	lines = readLines( path );
	lines = gsub( "	", "", lines );
	lines = gsub( "\"", "", lines );
	toks = strsplit( lines, " = " );
	oi = 1;
	for( i in 1:length(toks) ) {
		if( length( toks[[ i ]] ) == 2 ) {
			rv[[ oi ]] = toks[[ i ]];
			oi = oi + 1;
		}
	}
	trv = rv;
	rv = list();
	t.name = "";
	t.type = "";
	t.class = "";
	oi = 1;
	for( i in 1:(length(trv)) ) {
		if( trv[[i]][1] == "type" ) {
			t.type = trv[[i]][2];
			rv[[ oi ]] = c( t.name, t.class, t.type );
			oi = oi + 1;
			t.name = "";
			t.type = "";
			t.class = "";
		} else if( trv[[i]][1] == "name" ) {
			t.name = trv[[i]][2];
		} else if( trv[[i]][1] == "class" ) {
			t.class = trv[[i]][2];
		}
	}
	rv = do.call( rbind, rv );
	colnames( rv ) = c( "name", "class", "type" );
	return( rv[ !( rv[,3] %in% not.used ), ] );
}
names( file.path ) = file.path;
result = lapply( file.path, parseTpl );

But damn, getting this Qt project up and running on my desktop has proven to be a mess and wasted more than an hour on that and it still does not import the existing project correctly.

In any case, the idea is to make a 4 column csv file with templateName,fieldName,fieldClass,fieldType from the templates and then arzBaller will load this in and use it to create arzs. This way, it should be pretty easy to extend and modify as needed. Luckily, R made parsing this file format quite trivial. But they (family, human and otherwise) start to awaken, and my peace is over.

89

Ok, down to the final issues…

Using Grimmest source, I see that some templates have changed between when I stopped working on Grimmest and today.

For example, I have sHandedSpecialAnim36, which I guess got axed.

In addition, there is survivalAdjustment in gameengine, which is now 3 fields for Normal/Elite/Ultimate.

There is an issue with fileNameHistoryEntry which seems pointless anyway.

And I found a couple of records with ‘,’ (commas) in the FileDesription, which are bugs I entered in AssetManager that never got caught.

Still some more tests to run, but I am pretty sure that arzBaller should not write any fields to the arz that are not found in templates, nor should it write any fields that have stray commas, as they are not going to help the arz function in any way…and may break stuff.

Will continue to dump these instances to stderr so that you can see them as you use it. But honestly, with all the templates now integrated, if arzBaller chokes on something, the modder should look into fixing the records.

Just a few more tests and then I post this.

90

NICE! :slight_smile:

yeah crate changes templates every now and then, new features or even just new masteries will need a new template.

fileNameHistoryEntry is I believe just something the dbrEditor adds and I have no idea what it does. Since Im working with the dbrEditor and Sublime Text to edit dbrs depending on what I want to do, for some things the dbrEditor is actually better than using a text editor :slight_smile: anyways, when I notice that line with a texteditor I usually remove it.

If you dont know what a dbr file looks like outside the dbrEditor, using “,” seems like a good idea for a fileDesc, but yeah it breaks the dbr :smiley:

91

It would be nice if the templates could auto update or something. :confused: The only way I know how to update them is to just re-extract the entire game content after every big patch.

92

Here is v0.9. Yes, we skipped a couple numbers, that is how much I am hoping this is pretty much final, maybe a real log file or something to finish at 1.0 release.

Test the heck out of it. Working for me just fine.

Source:

Binary:

Once I get no bad news, will be closing this thread and posting a release.

93

yep would be great if they were part of the modding tools :slight_smile: everytime I see a patch or hotfix Im like “Great I can extract the game files again for the next 10 min, afk” :smiley:

a log file would be nice

now is the time for davood to try this :smiley: if it works for him it’ll work for 90% of the existing mods.

94

What a crazy assumption you make. :D:D:D

95

first report:
not working - I tried removing my /mod_wanez/ folder so only the /records/ is left, but the stash isnt showing up properly, also remove the (what Im guessing) chinese from the fileDesc to see if thats messing it up, but still nothing.

Ill do some more testing later and extract the arz made by arzBaller to see what happened to the files.
But for now… not ready for 1.0, sorry :frowning:

96

Can you PM a link with what you are trying to pack, all I have is what I unpack with my tool, there must be differences somewhere.

97

While I could confirm that the source you sent does not pack well with my tool, I just loaded it up into Assetmanager and it spent more than half of the build time doing something “Updating record: blahblah.dbr”…

But still, packing the source after Assetmanager built a functional mod, did not load in game.

Back in assetmanager, I found a check records option in the database menu…it took almost 6 minutes, but reported 0 error(s). packing the source records after this showed just as many “errors” - fields not in templates, as before, I am not sure what transpired during those 6 minutes, but it did not help.

So I rewrote how unhandled fields are reported and these are the only fields that were skipped during packing. Cannot see how they broke everything, I mean the game does not even load with this arz.


 FOUND 46 fields that could not be typed
 0 dHandedSpecialAnim36
 1 dHandedSpecialAnim37
 2 dHandedSpecialAnim38
 3 dHandedSpecialAnim39
 4 dHandedSpecialAnim40
 5 dHandedSpecialAnimRef36
 6 dHandedSpecialAnimRef37
 7 dHandedSpecialAnimRef38
 8 dHandedSpecialAnimRef39
 9 dHandedSpecialAnimRef40
 10 dHandedSpecialAnimSpeed36
 11 dHandedSpecialAnimSpeed37
 12 dHandedSpecialAnimSpeed38
 13 dHandedSpecialAnimSpeed39
 14 dHandedSpecialAnimSpeed40
 15 melee2hSpecialAnim36
 16 melee2hSpecialAnim37
 17 melee2hSpecialAnim38
 18 melee2hSpecialAnim39
 19 melee2hSpecialAnim40
 20 melee2hSpecialAnimRef36
 21 melee2hSpecialAnimRef37
 22 melee2hSpecialAnimRef38
 23 melee2hSpecialAnimRef39
 24 melee2hSpecialAnimRef40
 25 melee2hSpecialAnimSpeed36
 26 melee2hSpecialAnimSpeed37
 27 melee2hSpecialAnimSpeed38
 28 melee2hSpecialAnimSpeed39
 29 melee2hSpecialAnimSpeed40
 30 sHandedSpecialAnim36
 31 sHandedSpecialAnim37
 32 sHandedSpecialAnim38
 33 sHandedSpecialAnim39
 34 sHandedSpecialAnim40
 35 sHandedSpecialAnimRef36
 36 sHandedSpecialAnimRef37
 37 sHandedSpecialAnimRef38
 38 sHandedSpecialAnimRef39
 39 sHandedSpecialAnimRef40
 40 sHandedSpecialAnimSpeed36
 41 sHandedSpecialAnimSpeed37
 42 sHandedSpecialAnimSpeed38
 43 sHandedSpecialAnimSpeed39
 44 sHandedSpecialAnimSpeed40
 45 fileNameHistoryEntry
COMPILE database to output arz: dga.arzarzBaller: compiling 5572 records from database to dga.arz

At a total loss at this point. Not even going to continue to try…this is no longer fun at all.

98

so I made a new mod for arzBaller, just a new NPC, nothing special about it.

Extracted the game files again in case there was an old one using an older template and it would mess it up. Copied over one of the DC guards loaded the worldEditor, made a map, placed him there.
Made a few more things to create more different NPC files, to get it from the arzBaller extract, edit it with the dbrEditor and so on, since previously it seemed to have worked with extracted files by arzBaller.

Not this time tho, packing it with arzBaller did not at all work, NPCs didnt show up in the editor or in game.
BTW the files (from extract core game files and the one I got from arzBaller) are identical, same number of lines, same size, and they both work if I pack it with AssetManager. When I edit the npc file with the dbrEditor it adds about 2k lines and more than doubles its size, than I unpack it again with arzBaller and its back to its old size and lines and once again identical to the file extracted by AssetManager :smiley:
The arz from arzBaller is roughly 20% smaller than the one created by AssetManager.

Part II:
using arzBaller to pack it again using one of the dbrEditor edited NPCs with the 2k+ lines. The resulting mod.arz is certainly much larger, but still not working in game. unpacking it with arzBaller results in giving me the file with 2k+ lines.

But it is working for CrucibleLT, so whats the difference? :eek: or is 0.9 not working for it anymore either?

EDIT:
Im realizing something, before the files extracted by arzBaller were much different than the ones used, maybe now arzBaller is extracting the files better but also removing entries from each record the AssetManager created so the game would actually be able to use it.
an arz packed by AssetManager and than unpacked by arzBaller, changed with a text editor and packed again with arzBaller worked… now arzBaller gets the same result as AssetManager does when extracting the files (based on the solid template parser you made). So there are no additional lines for each record that would make them work in game when packing it again with arzBaller.

EDIT2:
so I used 0.4 again to extract and repack it - its working. Using the real source however doesnt. Ill keep testing and comparing the files, there has to be a difference somewhere Oo

99

I think my lack of comprehension of what you wrote is indicative of the fact that I am burnt out on this project atm.

I need to not think about it for a day and revisit the source code fresh, there must be something utterly stupid in the code, that I just cannot see for now.

Thanks for trying. I am pretty sure this is all just bugs in my code.

100

I found the bug and figured out why dga is not working (caravan too)

You cannot use arzBaller with “Class,” in your dbr.

You have never set this field and I have no idea how assetmanager gets it right cause the string in the Class field is encoded (as a string) in the arz binary format.

This might be possible through the templates…but as of now, I was not parsing for that.

Anyway, there was also bugs in the code so here is v0.9.1

But I doubt this helps you WareBare, as I cannot see how you are going to get all these records a Class field…I am busy today, but next time I look at this, I will try to see if I cannot use templateName to obtain a Class string from the tpl files.