Custom Save Blocks

After some research, I’ve figured out how to customize the save file header information for both AWL and AnWL.

Each game has a file named wlp1.rel, which is a relocatable module (similar to how .dll files function for Windows programs). This file has the information which dictates the header info when creating or overwriting a save file.

This information includes the chapter names, the save filename, the game / save title, and the banner / icon to be used.

By manipulating this file, I’ve changed the game / save title for each game to HM: APL (for the wlw mod) and HM: AnPL (for the mlm mod).

I’ve considered changing the filename (originally bokujyo4.dat or bokujyoA.dat, depending on the game), but changing this would invalidate any previous save files for users of earlier alpha/beta builds of the mod.

Save Block Textures

In addition to modifying names / variables in this file, we can also modify the textures used. While I had already modified banner_card.tpl, I decided to try my hand at modifying card_icon.tpl. This file is a bit trickier since it uses a texture format that utilizes pallettes.

The important thing to note hear is that all of the individual textues must utilize the same pallette for the textures to render properly when displayed by the console hardware.

It’s also important to note that the order the textures are displayed is in bouncing order. That is to say that they go in order of 1->2->3->2->1->2->3, as opposed to a normal GIF loop order of 1->2->3->1->2->3. This is most noticable when seeing how the dog mouth opens / closes on the otiginal save icon

Standard Loop
Bounce Loop

Creating a new Save Block Texture

I started by deciding what I wanted to make for the new save icon. After some internal debate, I decided on using the blue feather (the symbolic item for proposing to one of the romance options) with a LGBT flag background. I created a version with a standard 6-stripe rainbow flag (1979 6-stripe variant), as well as one with the lesbian flag (2018 5-stripe variant).

I also created versions with the Philadelphia pride flag (2017 8-stripe variant) and Progress flag (2018 variant). While these show up ok with increased internal resolutions (i.e. emulator), they appears blurry / pixelated on original interlaced resolutions (i.e. original hardware). For this reason, I have made them available as optional, which can be used to replace the card_icon.tpl texture file on the disc.

Blue Feather (from commonall.arc.clz\symbol.tpl\Texture5), 1979 Rainbow Flag, 2018 Lesbian Flag, 2017 Philadelphia Flag, 2018 Progress Flag
Progress and Philadelphia Pride Flag Mockups

Philadelphia Flag card_icon.tpl [DOWNLOAD]
Progress Flag card_icon.tpl [DOWNLOAD]


To ensure that all images used the same pallette, I saved each texture as a single-frame 256-colour .gif file. When saving the first texture, I saved the generated colour table as a Photoshop palette. Then, when I saved the second and third textures, I loaded the colour table from the previously saved palette file. This would ensure that all three saved .gif image files had the same embedded colour table / palette.

When loading a .gif with an embedded colour palette into BrawlBox, it will give the option to import the palette. By selecting this option when replacing each texture in the card_icon.tpl file, it ensures they’ll all match.

Once the game is loaded with the edited wlp1.rel, card_icon.tpl, and banner_card.tpl files, and the game diary is re-saved, the changes will take effect.

AWL/AnWL vs APL/AnPL

Introducing a New Side Project: GC/Wii Disc Filesystem Catalog

Throughout my research for Harvest Moon: A Proud Life, I’ve come across numerous odd files.

Many of these file types are common across various other Gamecube and Wii games. Some are less common.

That’s why I’ve decided to start a new project. I’ll be analyzing any GC/Wii dumps I can get my hands on so that I can log their disc contents. I plan on analyzing/posting a different game each day.

Each post will be tagged as GCN or Wii. All applicable file types (e.g. .tpl, .arc, .txt, etc.) will also be added as tags.

My goal is that other modders can easily check if there are other games that use a specific file type. I hope that this project will serve the community as a useful reference.

This new project will be located at gcwiifst.home.blog.

Here’s an example of what each disc analysis will look like…

Save File Checksums

After some research, we’ve determined that the save files for A(n)WL use a checksum to verify data integrity.

If the save data is modified and this checksum isn’t accounted for, a data corruption error will be presented when attempting to load the data.


This checksum is a 4-byte (32-bit) value stored at 0x40

Note: save files can be when using Dolphin emulator are located in “C:\Users\{USER}\Documents\Dolphin Emulator\GC\USA\Card A\”

The value can be calculated by taking all of the data after the checksum position, i.e., 0x44 to the end of the save file, and then calculating the checksum using the CRC32_bzip2 algorithm.

I’ve found that the easiest way to accomplish this is by saving the save data (from 0x44 onwards) to a seperate file (in this case, I’ve named it as hashtest.bin)

Once created, the save data (hashtest.bin) can be analyzed using a supported tool such as Jacksum. For Jacksum, I used the following hash command:

java -jar jacksum.jar -a all -F "ALGONAME{i} (#FILENAME) = #CHECKSUM{i}" hashtest.bin

Then just look at the output of the crc32_bzip2 hash and voila! It should match the previous 0x40 checksum value.


Using this knowledge, we can now edit various aspects of the save file including:

  • Character names
  • Animals
  • Items
  • Buildings (e.g. milking room, seed maker)
  • Money

The process for making these edits should be fairly straightforward

1. Extract the save data from your GCI file (from 0x44 onwards) and save it to a seperate file (e.g. hashtest.bin)

2. Find the value(s) you want to edit (e.g. money) and edit them

In this case, my money value was previously 999999 (hex = 0F423F). The value seems to be stored at 0x1962e (in hashtest.bin). We’re gonna try changing this to 123456 (hex = 01E240).

3. Calculate a new checksum of your save data using Jacksum (as shown above) by analyzing your seperate save data file (i.e. hashtest.bin)

CRC32_bzip2 checksum of our new data (money = 123456) is 34efd17e

4. Open your original GCI, overwrite with your new checksum at 0x40

5. Overwrite the save data fro 0x44 onwards with your newly edited data (copied from hashtest.bin)

6. Save your new GCI file

Voila!

Update

As an alternative to Jacksum, you can actually calculate the checksum right from within Hex Editor Neo (if you’re using a different hex editor, you’ll need to follow the Jacksum directions above).

Simply go to View -> Tool Windows -> Checksum

Then add, select the “Add Algorithm button to create an algorithm with the following settings:

  • CRC-32
  • Initial Value: ffffffff
  • Polynomial: 4c11db7
  • XOR Out: ffffffff
  • Uncheck Reflection In
  • Uncheck Reflection Out
  • Name: CRC-32/BZip2

Once saved, simply highlight the affected data in your .gci save file (i.e. 0x44 to the end of the file). Then, select the “Selection Only” option from the Checksum tool window, and the appropriate value should be generated in the CRC-32/BZip2 row.

Voila! You can now modify any data directly within the GCI file, regenerate the checksum (by selecting data from 0x44 to the end off the file), and then insert the new checksum (at 0x40).

Special thanks to James for figuring this out.

Tutorial: Extracting Raw Texture Data and Calculating Hash Values From TPL Files + Naming PNG Image Files As Per Dolphin Standards

I’ve been playing around with the “Dump Texture” feature in Dolphin recently.

When the emulator dumps textures, it creates a file name that includes a hash of the texture data. This hash uses the xxhash64 algorithm.

I’ve found that, using information about the TPL file format, I’m able to extract the raw image data from the TPL files.

Once I extracted this data (via manually copy/pasting in a hex editor), that I’m was able to feed it into the xxhash64 algorithm. I used QuickHashGUI for this.

To my surprise, this actually worked and the hashes matched!

Extracting the image data from the first image contained in the logo.tpl file (the Marvelous logo) and comparing hash values to the dumped Dolphin texture filename.

I’m going to try creating a QuickBMS script to extract raw image data from TPL files to somewhat automate the process.

Eventually this means that I should be able to generate hashed filenames for every texture on the game’s disc. This bypasses the need to typically play through and manually come across every texture in-game.


Update

I’ve created a QuickBMS script that can take any input TPL file, and output the extracted raw texture data.

endian big

idstring "\x00\x20\xAF\x30"

get filesize asize
get openedFile filename

get numberOfImages long
get imageTableOff long

if numberOfImages == 1
    goto imageTableOff
    get imageHeaderOff long
    goto imageHeaderOff
    get height short
    get width short
    get imageFormat long
    get imageAddress long
    
    set name string openedFile
    string name - ".tpl"
    string name + "_Texture0_"
    string name + width
    string name + "x"
    string name + height
    string name + "_"
    string name + imageFormat
    string name + ".bin"
    
    math filesize - imageAddress
    
    log name imageAddress filesize
else
    for i = 1 < numberOfImages
        goto imageTableOff
        get imageHeaderOff long
        get null long
        get nexImageHeaderOff long
        
        goto imageHeaderOff
        get height short
        get width short
        get imageFormat long
        get imageAddress long
        
        goto nexImageHeaderOff
        get null long
        get null long
        get nexImageAddress long
        
        set imageSize long nexImageAddress
        math imageSize - imageAddress
        
        set name string openedFile
        string name - ".tpl"
        string name + "_Texture"
        set texNum long i
        math texNum - 1
        string name + texNum
        string name + "_"
        string name + width
        string name + "x"
        string name + height
        string name + "_"
        string name + imageFormat
        string name + ".bin"
        
        log name imageAddress imageSize
        
        math imageTableOff + 0x08
    next i
    
    goto imageTableOff
    get imageHeaderOff long
    goto imageHeaderOff
    get height short
    get width short
    get imageFormat long
    get imageAddress long
    
    set name string openedFile
    string name - ".tpl"
    string name + "_Texture"
    math numberOfImages - 1
    string name + numberOfImages
    string name + "_"
    string name + width
    string name + "x"
    string name + height
    string name + "_"
    string name + imageFormat
    string name + ".bin"
    
    math filesize - imageAddress
    
    log name imageAddress filesize
endif

Once extracted, these files can be hashed with a tool (e.g. QuickHash GUI) using the xxHash64 algorithm.

Combining the data from the script with the hash will provide you with the full filename that Dolphin would normally generate.

Comparison of BMS-extracted texture data vs Dolphin-dumped files

Once you’ve determined the file hashes, you can extract the textures from your TPL as PNG files using a tool like BrawlBox or Wexos’s Toolbox.

The naming convention will be as follows:

  • Naming Type – denotes the texture naming scheme used by Dolphin, should always be “tex1”
  • width x height – dimensions of the original texture, can be found in the file output from the BMS script above
  • hash – xxHash64 of the raw original texture data (as output from the BMS script once fed through a tool such as QuickHash GUI)
  • format – original texture image format as named in output from the BMS script above
  • Output file extention – output image file, should always be “.png”

Here’s a visual demonstration of the file naming convention, and where the different parts can be found.

Extracting a texture to a png image using BrawlBox, using naming information provided by QuickBMS + QuickHash

Update #2

After going through every CLZ, ARC, and TPL file, I beleive I’ve been able to dump/rename (almost) every texture in the game.

There are a total of 4115 unique texture images from Harvest Moon: A Proud Life (lesbian version).

I’ve compiled all of these dumped textures into a Dolphin-compatible pack, which can be downloaded [here].

Cutscenes / Trigger Events

Cutscene / Event Files

After a bit of digging into the file formats of A Wonderful Life, I was able to figure out which files contain data for cutscenes.

The files in question are .sb script files. The versions of these files utilized in-game are compressed using CLZ-compression, resulting in .sb.clz files.

There are a few places where these files can be found.

  • .sb debug scripts at disc:/test/Script/
  • Unnamed CLZ-compressed versions of the above debug scripts (i.e. .sb.clz files) contained in disc:/test/Script/test.arc
  • Unnamed CLZ-compressed chapter-specific events (e.g. romance cutscenes, son cutscenes) in disc:/Chapter#.arc
  • Unnamed CLZ-compressed common events (i.e. can be triggered in any chapter) in disc:/Common.arc

The unnamed Chapter and Common events can be determined by looking at the legend in disc:/test/text/EventOCC.txt. As an example, here is disc:/Chapter2.arc, with it’s corresponding event legend.

Replacing Events with Debug Scripts

Using this knowledge, we can compress some of the debug scripts using Sukharah’s CLZ Tool, then can import them into the Chapter or Common files, replacing a previously-defined event. This is done by right-clicking the intended event in the destination .arc file, then selecting “Replace” from within Wexo’s Toolbox. Then, when the game wants to trigger the normal event, it will trigger the newly replaced debug event instead.

Here is an example where I replaced 8009_link.sb.clz in disc:/Common.arc with the (compressed) debug script 0000_Teleport.sb.clz.

0000_Teleport.sb loading instead of 8009_link.sb

The main downside of the debug scripts is that almost all of the debug text is untranslated Japanese. The above example was translated by myself by modifying debug.mes.

Here is an example using the default untranslated dialog.

Accessing Galen/Nina’s house during Chapter 2

Heart Event Cutscenes

While I could manually extract every bachelorette cutscene from Chapter1.arc and import it to replace 8009_link as above, this would be fairly time consuming. Fortunately, there’s another way.

The test script folder contains a handy script, Launcher_Love.sb.

When compressed, imported, and triggered, this script will open up a menu, from which you can select any of the possible bachelorette cutscenes including heart events, matchmaker scenes, reverse proposals, and successful marriage events.

Here’s an example loading one of Muffy’s heart events.

Muffy Heart Event 4

Note that these cutscenes can only be loaded during Chapter 1. Attempting to load them in any of the later chapters will result in a crash.

CLZ Decompression / Recompression

Sukharah’s CLZ Compression Tool

Recently a wonderful soul who goes by Sukharah has managed to reverse-engineer the CLZ compression format used by AWL and AnWL.

They’ve used this knowledge to create a CLZ Compression Tool, which can be used to decompress clz files, as well as recompress them.

This is beyond amazing and opens up the project to begin work on models, animations, textures, and so much more.

Compiling / Downloading the Tool

The tool is meant to be compiled using a C++ compiler (e.g. g++). You can follow the instructions on Sukharah’s GitHub to compile it yourself.

Precompiled Version

For those not experienced in compiling programs, I’ve compiled a version that can be downloaded from [here]. I’ll be posting any newly compiled versions on the HMAPL Discord as Sukharah continues to develop the tool.

Note that this pre-compiled build will only work on Windows 10 and up, since that’s what I used to compile it.

Using the Tool

The utility is fairly straightforward.

I found the easiest method was to put your desired CLZ file in the same directory as CLZ.exe.

Then, open up a command prompt in that directory (e.g. type “cmd” in the Windows Explorer navigation bar).

Then, in the command prompt, execute the command:

 clz unpack input.file.clz output.file
clz unpack mainchapter0.arc.clz mainchapter0.arc

In the above example, I decompressed mainchapter0.arc.clz into it’s native arc format.

Quirks of Decompressed U8 Archives

The decompressed U8 archives (.arc files) seem to display improperly when viewed in BrawlBox (my former go-to arc tool).

Data sequence mismatching, resulting in seemingly duplicate data

As we can see from the above screenshot, the contents of boy_0.arc (embedded in mainchapter0.arc) are displayed before the actual boy_0.arc descriptor. This can cause mismatch/corruption issues when attempting to edit the files using BrawlBox.

After a bit of trial and error, I’ve found the best tool to work with these decompressed U8 archives is Wexos Toolbox

No more weird data sequence mismatches

Now that we’re able to decompress CLZ into ARC and modify these ARC files using the appropriate tools, we can play around with models, textures, and so much more.

Recompressing CLZ Files

While decompressing is one step, we need to recompress the files back into CLZ archives for any hope of the game reading them.

Fortunately, Sukharah built this functionality into their tool as well.

Unfortunately I had some issues using the “pack” function (output files would crash when loaded into AWL), although I believe this issue has been resolved in a recent revision of the tool.

So I ended up using the memory-optimized pack method.

clz pack2 input.file output.file.clz
clz pack2 mainchapter0.arc mainchapter0.arc.clz

Files compressed using the pack2 function worked perfectly, albeit with slightly longer load times than the original AWL files.

A Brief Rundown of CLZ Files Thus Far

CLZ-Compressed files remain one of the biggest hurdles for the Proud Life mod.

For this reason, I’ve decided to document everything currently known about these files.

Archives vs. Compression

The easiest way to understand how the game handles compression is to understand the difference between “archiving” and “compression”.

Most compression tools on Windows will do both of these in the same step (e.g. .zip files).

However, archiving is the act of taking multiple files, and putting them into one file.

Compression is when this “archive” is compressed to save space.

Both A Wonderful Life and Another Wonderful Life separate these two steps by creating archives (in the form of U8 archive or .arc files), then compressing them (using clz compression).

Note how the U8 Archive is the same size as it’s parts (5+2+7 = 14) , and it’s the CLZ file that’s actually smaller in size

Essentially, we need to figure out how to undo the compression step, do whatever we want to the uncompressed data, then redo the compression step.

File Header

All CLZ files will follow a similar format.

  • File header (“CLZ”)
  • The size of uncompressed data in hex (twice). This is useful since we can use it to confirm if an attempt at uncompressing a CLZ file was successful or not.
  • Compressed data
In this example, an uncompressed mainchapter0.arc file should be 6.4MB

Compression Algorithm

While I haven’t been able to find a match for the compression used, the file extension alludes to it being a Custom Lempel-Ziv (CLZ) algorithm.

The files utilize a big endian byte encoding.

They seem to use some sort of dynamic dictionary generation. Which can be most seen when viewing the chapter arc.clz files. It’ll display the first occurrence of a pattern, but not subsequent occurrences. In this case _0.arc is listed for “boy0_0.arc” but not the files afterwards.

Note _o.arc only appears for the first entry

It seems to be generating some sort of dynamic dictionary.

Comparing Compressed / Decompressed Files

Certain data on the disc can be used to see both compressed and decompressed versions of some files.

The disc:\test\Scripts folder contains numerous script (.sb) files.

Of note, this folder also contains a U8 archive test.arc.
While this U8 archive is formatted in such a way that you can’t directly see the filenames, they can be viewed in a hex editor.

In this case, we can see that the first file in the U8 archive is a compressed CLZ file that, when decompressed, would yeild a sb script of 2a79 bytes in size.

By comparing the file size in the CLZ header, we can match up these CLZ files with the uncompressed sb files.

In this case, we can find that the first compressed “@” clz file in test.arc actually matches up with the uncompressed 0000_Animal_Navigation.sb (2a79 bytes)

I’ve managed to match up the compressed/uncompressed files (and have renamed them accordingly). They can be found [here] for comparison/research purposes.

The Search for Help

I’ve tried searching on various forums (e.g. ZenHAX, XeNTaX, and Romhacking.net), but haven’t really come up with much.

If anyone has any ideas on how to tackle this compression algorithm, it would be greatly appreciated.

Endian Differences between the Gamecube and PS2

Upon review, it seems that the Gamecube primarily stores data using big-endian (a type of data encoding), whereas the PS2 primarily stores data as little-endian.

This means that some game data (e.g. dialogue files) might appear to have their data “flipped” when comparing the GCN and PS2 versions.

As an example, a section of data might be “cdc3b0b0” for a file in the PS2 version, but “b0b0c3cd” in the GCN version.

This means that certain files were re-encoded when the game was ported to the PS2 to avoid them from being read as corrupt.

Shoutout to Dahni in the Discord chat for making this discovery.

It’s unclear at this time exactly which files were re-encoded. It seems that dialogue files were altered at the very least, but this could extend to other file types (textures, models, etc).

With that said, I’ll be focusing exclusively on the GCN versions of the game files to prevent any confusion of endian encoding going forward.

Discovery of SceneInit function potentially related to reading data from CLZ files

After opening the game’s main executable (Start.dol) in BrawlBox’s memory editor, I was able to find references to a SceneInit.cpp (C++) file.

Based on the name, I assume that SceneInit would be called whenever loading (i.e. initializing) the game (scenery). This function would then need to load certain files (models, textures, etc) from a mainchapter file (e.g. mainchapter2.arc.clz).

In addition, there are also references to common.arc.clz and preload.arc.clz, possibly related to a FileUpdater.cpp function.

I’m going to post some of my findings on ZenHax. Hopefully someone there will be able to provide some additional insight.

CLZ Compression Help

I have some (potential) good news.

While doing some work on the mod, I recently came across a forum of users that specialize in game research, including file identification and extraction.

It also seems like I’m not the only one who’s come across this, as there was already a thread with someone looking into A Wonderful Life’s CLZ files.

I’ve since posted on that thread and will hopefully be able to get some further help with compressing/decompressing CLZ files.

I’ll update as I get more info.