Tutorial: Extracting Raw Texture Data and Calculating Hash Values From TPL Files + Naming PNG Image Files As Per Dolphin Standards

I’ve been playing around with the “Dump Texture” feature in Dolphin recently.

When the emulator dumps textures, it creates a file name that includes a hash of the texture data. This hash uses the xxhash64 algorithm.

I’ve found that, using information about the TPL file format, I’m able to extract the raw image data from the TPL files.

Once I extracted this data (via manually copy/pasting in a hex editor), that I’m was able to feed it into the xxhash64 algorithm. I used QuickHashGUI for this.

To my surprise, this actually worked and the hashes matched!

Extracting the image data from the first image contained in the logo.tpl file (the Marvelous logo) and comparing hash values to the dumped Dolphin texture filename.

I’m going to try creating a QuickBMS script to extract raw image data from TPL files to somewhat automate the process.

Eventually this means that I should be able to generate hashed filenames for every texture on the game’s disc. This bypasses the need to typically play through and manually come across every texture in-game.


Update

I’ve created a QuickBMS script that can take any input TPL file, and output the extracted raw texture data.

endian big

idstring "\x00\x20\xAF\x30"

get filesize asize
get openedFile filename

get numberOfImages long
get imageTableOff long

if numberOfImages == 1
    goto imageTableOff
    get imageHeaderOff long
    goto imageHeaderOff
    get height short
    get width short
    get imageFormat long
    get imageAddress long
    
    set name string openedFile
    string name - ".tpl"
    string name + "_Texture0_"
    string name + width
    string name + "x"
    string name + height
    string name + "_"
    string name + imageFormat
    string name + ".bin"
    
    math filesize - imageAddress
    
    log name imageAddress filesize
else
    for i = 1 < numberOfImages
        goto imageTableOff
        get imageHeaderOff long
        get null long
        get nexImageHeaderOff long
        
        goto imageHeaderOff
        get height short
        get width short
        get imageFormat long
        get imageAddress long
        
        goto nexImageHeaderOff
        get null long
        get null long
        get nexImageAddress long
        
        set imageSize long nexImageAddress
        math imageSize - imageAddress
        
        set name string openedFile
        string name - ".tpl"
        string name + "_Texture"
        set texNum long i
        math texNum - 1
        string name + texNum
        string name + "_"
        string name + width
        string name + "x"
        string name + height
        string name + "_"
        string name + imageFormat
        string name + ".bin"
        
        log name imageAddress imageSize
        
        math imageTableOff + 0x08
    next i
    
    goto imageTableOff
    get imageHeaderOff long
    goto imageHeaderOff
    get height short
    get width short
    get imageFormat long
    get imageAddress long
    
    set name string openedFile
    string name - ".tpl"
    string name + "_Texture"
    math numberOfImages - 1
    string name + numberOfImages
    string name + "_"
    string name + width
    string name + "x"
    string name + height
    string name + "_"
    string name + imageFormat
    string name + ".bin"
    
    math filesize - imageAddress
    
    log name imageAddress filesize
endif

Once extracted, these files can be hashed with a tool (e.g. QuickHash GUI) using the xxHash64 algorithm.

Combining the data from the script with the hash will provide you with the full filename that Dolphin would normally generate.

Comparison of BMS-extracted texture data vs Dolphin-dumped files

Once you’ve determined the file hashes, you can extract the textures from your TPL as PNG files using a tool like BrawlBox or Wexos’s Toolbox.

The naming convention will be as follows:

  • Naming Type – denotes the texture naming scheme used by Dolphin, should always be “tex1”
  • width x height – dimensions of the original texture, can be found in the file output from the BMS script above
  • hash – xxHash64 of the raw original texture data (as output from the BMS script once fed through a tool such as QuickHash GUI)
  • format – original texture image format as named in output from the BMS script above
  • Output file extention – output image file, should always be “.png”

Here’s a visual demonstration of the file naming convention, and where the different parts can be found.

Extracting a texture to a png image using BrawlBox, using naming information provided by QuickBMS + QuickHash

Update #2

After going through every CLZ, ARC, and TPL file, I beleive I’ve been able to dump/rename (almost) every texture in the game.

There are a total of 4115 unique texture images from Harvest Moon: A Proud Life (lesbian version).

I’ve compiled all of these dumped textures into a Dolphin-compatible pack, which can be downloaded [here].

Exploring the GameCube Service Disc gpl models

Preliminary Analysis of gpl files on the Service Disc

After a bit of research on gpl models, I’ve been lead to the Gamecube Service Disc v1.0/03.

Upon analysis of this disc, I’ve found that it contains gpl files created using version 12012000 of the geometry library (same as the Dolphin SDK), as well as version 6012001 (same as A Wonderful Life).

When booting the disc normally, it seems to load the v12012000 gpl files (e.g. disc:/gxDemos/CarDemo/f50.gpl) for the default Car Demo program (disc:/CarDemo/cardemo.dol).

Attempting to Swap Models

However, I can get it to load the 6012001 versions (e.g. disc:/CarDemo/f50.gpl) by replacing the main executable on the disc (disc:/main.dol; naming might vary depending on the tool used) with a special version of the Car Demo program (disc:/ntd/cardemoN.dol).

With this in mind, I should be able to (in theory) replace some of the models in the cardemo with files from AWL.

E.g. If I were to look at bass.arc from AWL, it contains all of the needed files (an act, a gpl, and tpl).

If I were to extract these files and adjust the files on the service disc as follows:

  • rename disc:/CarDemo/f50.act to disc:/CarDemo/f50.act.backup
  • rename disc:/main.dol to disc:/main.dol.backup
  • copy disc:/ntd/cardemoN.dol to disc:/main.dol
  • copy bass.act to disc:/CarDemo/f50.act
  • copy bass.gpl to disc/CarDemo/bass.gpl
  • copy bass.tpl to disc/CarDemo/bass.tpl

It should load the bass model when booting the disc, instead of the default ferrari.

However, it seems that the new model (in this case, bass.gpl) does not load correctly.

The bass model should be above the rectangular shadow in the middle.

I’m not sure why this is currently the case, but I plan on doing some further tests as I can.

Why a lot of AWL models won’t work

A lot of gpl files on the root of the AWL disc (which, it’s worth mentioning, aren’t actually used by the game) don’t seem to be configured correctly.

As an example, let’s look at boy_0.arc.

While this file contains a valid act and gpl, it has 3 seperate texture (tpl) files for the body (b0), eyes (f0_e), and mouth (f0_m).

This is a problem because most gpl files are coded to load one tpl file. In this case, we can open boy_0.gpl in a hex editor and see that it’s supposed to load all of it’s textures from a nonexistent Boy_0.tpl file.

As such, attempting to load the boy_0 model using the above method I tried for bass, will not work. The program will simply give an error “Could not find CarDemo/Boy_0.tpl” and freeze/crash.

Attempting to rename any of the other texture files (e.g. renaming boy_0_b0.tpl to Boy_0.tpl) also doesn’t work, it just causes a crash. This is likely because the elusive Boy_0.tpl file would include all body, eye, and mouth textures in one single tpl file.

More Research Required

I plan on doing more testing with the Service Disc and the CarDemoN program.

There’s also other demos, disc:/ntd/zebraA.dol and disc:/ntd/zebraS.dol, that could be useful for viewing models/animations.

There’s also a program disc:/ntd/preview.dol that might be useful for previewing models.

Hopefully I’ll be able to come up with something over the next few weeks.

Fun with Textures in Dolphin

I’ve recently tried playing with the texture dump and load tools in Dolphin (fan emulator).

I dumped the boy textures from A Wonderful Life and swapped them with the girl textures from Another Wonderful Life.

The results were… interesting.

It’s worth noting that this is just a temporary solution until I can find a way into the game’s CLZ file format to manually replace the texture image files on the disc.

It’ll mainly serve as a way to identify the main character during test playthroughs as a female/male via their face/clothing.

Also, I really need to find a better way of differentiating Dolphin (Gamecube Development Codename) and Dolphin (fan emulator). Maybe I should refer to the former as Nintendo Dolphin and the latter as Dolphin-emu.

Retail Serial Console Output Using Dolphin

As a follow-up to my previous post, I’ve now figured out how to view serial console/debug output using Dolphin (fan emulator).

This can be enabled by toggling the View -> Log and View -> Log Configuration options. Then, enabling OSReport in the Log Configuration.

Console output will appear in the Log tab.

I tried running some of the problematic gcm images I created from Previewer + GPL models from AWL (version-modified), and got the following output:

(wildplant.gpl) doesn’t match standard 8.3 format.

The error above is likely due to a long filename (wildplant). Early computing systems used a standard known as 8.3, restricting filenames to 8 characters or less, followed by the file extension. Since this file is used in the game, I presume that the notification was fulfilled and the restriction was removed in a later iteration of the Dolphin Character Pipeline.

I tried a few different files with shorter file names. However, they all also gave errors.

Unknown state setting for Display Object

It’s difficult to know what is causing the above error, but my best guess is that, since AWL was presumably built using an updated version of the Character Pipeline, it uses certain settings for gpl models that were only supported by the updated version.

If these findings are correct, then the models from the game could only be viewed using an updated build of Previewer from a newer version of the Character Pipeline. I’ll do some digging to see if I can find anything.


On a related note, now that I know about the Dolphin log feature, I’m going to try playing through some of AWL and AnWL to see if the games happen to have any serial console output as part of their code.

Huge thanks to Video James (sonic15783)#6150 on the Discord! This wouldn’t be possible without him!

Preparing and Running Previewer on Retail Hardware (+ Tutorial)

After my research in getting previewer running on my iBook, I found that the models from A Wonderful Life could only be viewed on HW2 systems (retail hardware or equivalent dev kits).

Through some more research, I am now able to compile a version of previewer to run on retail hardware.

Here’s how I did it…

1. Create a folder to serve as the dvd root, in this case I used C:\defroot. This folder will contain the files to be previewed in a folder called “preview”, including the required prevload.txt file.

2. Convert the HW2 version of previewer.elf to .dol format using makedol.

3. Create a new file in Notepad the following settings and save as
previewerD.ddf.

4. Use the NPDP program OdemMakeDlf to compile our ddf file into a dlf file.

5. Download a retail-patched version of makegcm.dll and put the appropriate SDK folder C:\DolphinSDK1.0\X86\bin, overwriting the original makegcm.dll file.

6. Run makegcm.exe to compile the dlf file to a functional gcm (gamecube disc image).


The resulting gcm file can run on a retail Gamecube, or be opened in the Dolphin fan emulator.

Zebra, now on Gamecube

I haven’t tried out any of the AWL files yet, but that will be the next step.

Overall, I’m leaps and bounds beyond where I was just a couple months ago.


Update

I’ve tried using the above method to preview some gpl files from A Wonderful Life. Unfortunately, the program freezes on a black screen immediately after the Gamecube logo.

I’m not sure what the error is since the retail method of loading doesn’t provide any output. To get that, I’d need to be using a development system (e.g. GDEV, TDEV) that could be hooked up to a PC for console output.

The alternative would be to try using an entirely different piece of software, dspin.

This software (I have version 0.94) allegedly has experimental gpl import support.

I’ll do some tests with this software tomorrow to see if I can get anywhere.

If dspin doesn’t work, I’ll just have to shift focus back to dialogue editing until a workaround can be found, such as a more updated previewer for Mac (i.e. newer than the version included with the April 2001 Character Pipeline).


Update II:

dspin didn’t seem to work. I renamed the file and put it in the dvdroot folder, but the program didn’t seem to do anything with it.

I’m not sure what I’m doing wrong. Further research needed.

Windows-Compatible .gpl Models

After having success exporting 3DS Max projects to gpl format, I decided to dive further.

The Dolphin Emulator e28 from the Gamecube SDK comes with a modified version of the 3DS Max plugin CPExport. This new version, called MaxConv.dle, is capable of exporting little-endian files.

I exported the Zebra demo from 3DS Max using the new plugin.

I opened up previewerD.exe and it was actually able to load the newly exported gpl.

Tiny .gpl zebra, now on Windows!

I was even able to navigate the previewer (somewhat) using my Logitech F310 controller. With that said, only some of the buttons would work:

  • Primary stick (mapped to my left analog stick)
  • Trackball/light control switch (mapped to Y on my controller)
  • Zoom out (mapped to A on my controller).

Most notably, the zoom in function is not working with any of the buttons on my F310.

I’ll try my controller with the iBook to see if it works. I also have a USB PlayStation 3 controller that I’ll try out on both the Windows and Mac versions of Previewer.

Even though it seems helpful to have models opening on the Windows version, it will only accept custom-exported gpl files. I might try looking at the gpl files in a hex editor to find exactly how the data is organized compared between Windows (little-endian) and PowerPC (big-endian) versions of the plugin.


Update: If I unplug my controller, the emulator defaults to keyboard controls:

Key Controller
Arrow keys Main Stick
`A’, `S’, `D’, `W’ Sub Stick
NumPad 2 A button
NumPad 4 B button
NumPad 6 X button
NumPad 8 Y button
NumPad 7 Trigger Left
NumPad 9 Trigger Right
NumPad 0 Menu button

This means that we can finally zoom in on our model using the Windows previewer (provided that the gpl file has been exported as big-endian).

Not-so-tiny zebra

An update on the Windows version of “Previewer”

After setting up a full software development environment in a Windows XP Virtual Machine using a combination of Visual Studio 6.0 and software from a recent AssemblerGames dump, I was able to make some progress with the SDK files.

I managed to delete the version-check code from C:\pcemu\build\charPipeline\geoPalette\src\geoPalette.c (a module used when previewing gpl geometry files).

I was then able to recompile geopalette.obj and charpipeline.lib (using the newly modified geopalette.c code) by opening C:\pcemu\build\charPipeline\win32\makeall.bat.

Once those were created, I could finally rebuild Previewer.exe and PreviewerD.exe (debug) by running the makefile located at C:\pcemu\build\demos\charPipeline\pcmakefile.

This means that the program will bypass the previous version mismatch error I was getting.

However, the program still will not actually load the model, and will crash shortly after displaying the “Loading” message.

After looking at the documentation that came with the SDK files, it seems that this issue is because the Windows version of Previewer needs gpl files exported specifically using little-endian, as opposed to big-endian (typical format for exporting for Gamecube/Mac test systems).

My best bet at this point would to continue my search for an OS 9 mac that I can run further tests on, or to find a way to convert the big-endian gpl files to little-endian for viewing on PC.

A Windows version of “Preview” existed this whole time!

After some digging around in the Dolphin/Gamecube SDK files I had, I’ve discovered that there was a version of the “preview” program (for viewing .gpl 3D models) for Windows this entire time!

I feel like an idiot for somehow not noticing this sooner.

The app launches just fine. Unfortunately, I got a version-mismatch error when trying to load any actual files (including the examples that came with the SDK).

With that said, the source code for the program included with the SDK. My plan of action right now is to try removing the version-check code, then recompile the program.

If this works, we might finally have a way of viewing 3D models from the game, without me resorting to trying an old 90’s-era Macintosh.

Getting Closer to 3D Model Previews

Namely, there was a “previewer” program compiled for old macs (old macs used a PowerPC architecture, similar to the Gamecube and Wii)

I’ve been fiddling around with some of the files from the Dolphin SDK.

I had tried running this before, but got stuck on the program looking for a “prevload.txt” file in a specific directory.

After some digging around in various documentation and script files, I finally figured out what I needed to do.

The program looks for an inserted disc with the volume label “cp”. It then looks for the folder “cp:/cpdata/preview/”. This folder must contain the file to be previewed (in this case, Zebra.gpl) as well as a text file called “prevload.txt” (which just contains the name of the file to be previewed).

With all of these conditions met, the program should display the requested file.

Unfortunately my virtual machine setup seems to freeze (or in some cases crash) at this point. This seems to be an issue with SheepShaver not supporting 3D hardware rendering.

Ultimately, the next (and only) step to get this software running would be to try it out on a real old-school mac. And even if that works, I’m not sure whether I’ll be able to navigate the program (since it seems to use some sort of game controller).

As always, I’ll keep you all updated.

SDK Software

I managed to figure out some of the software from the Gamecube SDK.

It looks like some of the SDK includes binaries built for the legacy mac platform.

Old Macintosh computers used the same PowerPC architecture (as opposed to other computers that used the Intel x86 architecture) as the Gamecube. This means that running software for the Gamecube would require either the actual dev kit hardware (known as HW1 or HW2), or a Mac.

There was also the official Dolphin emulator (not to be confused with the current fan project Dolphin Gamecube/Wii emulator), but that could only emulate certain aspects of the hardware.  You’d still need a mac or dev kit to run most things.

So, after some trial and error, I managed to get an old copy of Mac OS 9 running in a virtual machine known as SheepShaver.

I then got the preview program (an app from the SDK to view the character models/textures) to extract and open.  But when trying to run it, it fails after saying a needed file (prevload.txt) couldn’t be found. Documentation on getting this to run is sketchy at best.

I’ll need to do more digging to figure out the full configuration needed to get the app to run.  I read something about setting “CP_Root environment variables” in the SDK documentation, so that might be my next step.

I also tried getting my hands on a trial copy of 3DS Max (the software used by devs to create models). But then I found out that 3DS Max Plugins are version specific, so I’d need the old 3DS Max v3.1 to use the Gamecube plugins from the SDK to import/export models. And this will likely also require I use a virtual machine with an old version of Windows (e.g. Windows 98) to run it.

One step forward, two steps back.