PS4 [Tutorial] PSPHD - Texture Dumping and Replacement

[mexxpower]

Video showcasing GTA: Chinatown Wars with some replacement textures from the Android Version (courtesy of SonofUgly), running on PSPHD.

Spoiler: Preface

Making this method of texture replacement work required quite a lot of trial and error, as well as modifying the executable. This method uses two different PSPHD versions, a modified Echochrome emulator for dumping and the LocoRoco Midnight Carnival emulator for actually running the game with replacement textures. While this slightly complicates things, it is necessary since neither of the emulators seem to support both features.

From looking at the changes between both emulators, I believe Sony is/has been working on the texture replacement system recently. (Command line option changes, features being broken, etc.)
Also, bear in mind that both games shipped with these emulators do NOT use any texture replacement. So I'm happy this works at all.

This hopefully means that in the future we will see new PSP games released to PS Plus which use texture replacement. If that happens, we might be able to improve upon this current method of texture dumping and replacement.

TL;DR:
This allows you to replace original textures with new high resolution ones.
Texture dumping and replacing works surprisingly well in my experience.
Textures can either be included inside the pkg or in an external folder.

Now for the actual tutorial.

What you will need / Downloads:

• The PSP .iso you want to use
• A tool for generating PSP-fpkgs, such as Jabu's PSP-FPKG tool [1]
• A tool for extracting fpkgs, such as PS4-PKG-Tool [2]
• A tool for repacking extracted fpkgs, such as fPKG_Maker_GUI [3]
• PSPHD_Texture_Dump_Replace_v1.1.zip
  Folder Dumper contains the patched and pre-configured Echochrome emulator.
  Folder ReplaceFiles contains required files of the patched and pre-configured LocoRoco Midnight Carnival emulator.

Recommended Workflow:
Setup and execute Asset Dumper

1. Create a fpkg of your PSP game [1]
   
2. Extract the generated fpkg [2]
   
3. Extract PSPHD_Texture_Dump_Replace.zip to a separate folder.
4. Copy Image0/data/USER_L0.IMG from the extracted fpkg to the Dumper/ULUS99999-app/data/ folder of the asset dumper.
5. Configure the asset dumper's config-title.txt to your needs. Please read the comments in the config file for extra details.
6. Build the asset dumper fpkg [3]
7. Install and run it on your PS4 (make sure to create all folders set in the config beforehand)

Create replacement texture

1. Textures are dumped when they are loaded into memory.
   
   Spoiler: Details and Hints

   This means that you will have to play the game until the texture you want to replace is loaded. You can utilize game saves or the emulator's State Saving/Loading feature for this.
   For example, you could install the Asset Dumper with --texsave disabled, save states where your desired textures are loaded, and then enable --texsave and reinstall the Asset Dumper. This way you can easily load your saved states without having to worry about too much lag and countless undesired dumped textures.
2. Once done exporting, analyze the dump folder and identify the textures you want to replace.
3. Create a replacement texture of the same size, 4x or 8x the size (other sizes might work, I have not tested them). Make sure that the filename and type of the replacement texture is IDENTICAL to the dumped one (regardless of the size of the new texture)!

Apply replacement textures

1. Prepare your original extracted fpkg for repacking:
   Rename the folder Image0 to [title_id]-app, where you replace [title_id] with the title_id of your game (e.g. ULUS10490).
   Move all files from Sc0 to the sce_sys folder inside your extracted fpkg.
   You should end up with a directory structure similar to the one inside the Dumper folder.
   
2. Copy all files from ReplaceFiles to the root of your extracted fpkg and overwrite all files.
   Note: Replacing the eboot is essential! The default LocoRoco Midnight Carnival emulator has a bug which causes flaky replacement texture loading.
   
3. Set desired settings in the config-title.txt. Please read the comments in the config file for extra details.
4. Move replacement textures to the folder you specified in the config.
5. Rebuild the fpkg [3] and install.

General notes:

• I highly recommend to use an external directory (like /data/texreplace) for replacement textures while testing. This allows you to change the textures without reinstalling the fpkg.

• I have personally tested this method with GTA: Chinatown Wars (ULUS10490) and Patapon (UCUS98711). New problems and issues may arise with other games.
  
  
• The emulator included in the Dumper folder should have all features and functionality of the Echochrome emulator, just with working asset dumping options.
  
  
• The emulator included in the ReplaceFiles folder should have all features and functionality of the LocoRoco Midnight Carnival emulator, with flaky replacement texture loading fixed.
  However, I cannot guarantee that this is indeed the case. See spoiler below for explanation.
  
  
• If you are having trouble identifying an issue, have a look at the logs. Just check "Enable TTY Redirect" in the KLog Settings of GoldHEN 2.2.4+. Then you can listen to the log using telnet (telnet <PS4 IP> 3232).
  
  
• Spoiler: Context and details on Echochrome patch

  After looking at the logs, I noticed the line <118>(PSP) Package: disabling --texsave, --texmissingsave, --dumpvideos, --dumpaudio.
  This made me believe that the emulator dynamically disables asset dumping when detecting it's being run from a pkg.
  
  I then used UnfSelf.exe and IDA Freeware to have a look inside the eboot. I noticed that after printing the aforementioned log message, all four option strings are being addressed by a lea instruction. Furthermore, I had previously found that this log message is not printed if we pass malformed command line options which are missing a value (such as --debug).
  
  Using this information, I then thought about how I would implement dynamically disabling cl options and concluded that after parsing all options to a map, I would simply remove the unwanted ones afterwards based on some condition. So something like optionsMap.removeKey("--texsave").
  Now, if you misspell the option string you want to remove from the map it obviously doesn't get removed...
  
  And that's what I tried. I just shifted the address referenced by the lea instructions to an empty string (of course in the fself eboot.bin). And lo and behold, I was correct. After this slight modification those options are working as expected.
  
  However, while the same modification seems to work for LocoRoco Midnight Carnival, the dumping feature itself just crashes the application for me.
  
  
• Spoiler: Context and details on LocoRoco Midnight Carnival patch

  While initially experimenting with texture replacement, I noticed that replacment textures had a random chance to simply not be loaded. Interestingly, it wasn't an issue where certain textures would sometimes not load. It was either all texture or no textures.
  
  Inspecing the logs, I noticed that whenever textures were loaded, four different log messages relating to texture loading were printed. When the loading failed, none of them were printed.
  
  As with the Echochrome patch, I had a look at the eboot in IDA Freeware and could not find anything plainly obvious which could be causing this behaviour. I then just followed the condition branches (jump instructions) leading to the texture loading code and was able to identify the single jnz instruction responsible for skipping the whole section.
  
  I tried to follow the branches back even farther to try and identify the condition deciding whether to skip the texture loading or not. I was not able to find anything. So in the end, I just forced the texture loading branch by replacing the jnz with a bunch of nop instructions.
  
  I was surprised to not only make texture loading work seemingly 100% reliably, but also to not notice any negative side effects.
  So, while I seem to have fixed the problem, I still have no idea what the actual issue was. Therefore, I cannot with good conscience say that there are no unintended side effects yet to be discovered.

Potential for Improvement:
As mentioned above, this method is not perfect. Here are some things that we might be able to improve:

• Patch the emulator to enable --texswitch. This features is currently supposedly only available on Neighborhood or Windows.
  
• Use the same emulator for dumping and texture replacement. Maybe we can look at different PSPHD versions for that.
  
• Maybe using the .pak format for texture archives, which is mentioned in the emulator logs. Could potentially improve performance for large replacment texture packs.

Changelog:

Spoiler

Jul 4, 2023 - Removed dead link to PS4PKGViewer by theleecherman

Dec 1, 2022 - Fixed flaky texture loading + added additional details on replacement options added to config.
Changed files: ReplaceFiles/eboot.bin and ReplaceFiles/config-title.txt

Nov 29, 2022 - Initial release

Finally, I want to thank the community for all their prior work making this method possible.
Specifically, @Jabu for his easy-to-use fpkg converter tools, pearlxcore and TheLeecherMan for their pkg viewing and extracting tools, and codemasterv for their fpkg builder tool.
And of course the whole community, for your enthusiasm and dedication to creating such a plethora of awesome stuff!

I hope we can find uses for this texture replacement method and improve upon it.
I look forward to hearing your results and findings in the comments!

mexxpower

 

[Stayhye]

Great work!

 

[JSimesen]

Great Work!

 

[Stayhye]

Would this be something you could take a look at in the PS1HD emu as well? There are references to "--texreplace" in the EBOOT...

 

[mexxpower]

Would this be something you could take a look at in the PS1HD emu as well? There are references to "--texreplace" in the EBOOT...

I actually had a brief look at PS1HD while experimenting, but unfortunately didn't find the same code structure as Echochrome uses to disable texture dumping.
Maybe I was looking at the wrong version though... (the one PSX-FPKG v0.3 uses)

Do you know if there are any official PSX "remasters" running on PS1HD which actually use texture replacement?
Even older ones running on the pre-PS Plus versions might help.

 

[Stayhye]

AFAIK, the only PS1HD emu before echochrome was the one that was with Medievil.

 

[mexxpower]

AFAIK, the only PS1HD emu before echochrome was the one that was with Medievil.

I've now had a look at both the older Medieval and current Syphon Filter PS1HD versions. Things don't look promising I'm afraid.

At first sight, both emulators seem to support texture replacement. The option --texreplace is present and active in both emulators, but I couldn't generate any meaningful log output with it. When I used the same options like they work on the Patapon PSPHD emu (--host="/data/" and --texreplace="host0:texreplace") I got a "path was not found" log message even though the directory existed.
When I tried with an absolute path like LocoRoco Midnight Carnival PSPHD uses (--texreplace="/data/texreplace") I got no log output at all. Which leads me to believe that the option does not use absolute paths.
Looking at the eboot in IDA, you can see that there are definitively some checks for texture replacement implemented (like "unexpected filename", etc). However, I am not sure if they are accessible or dead code.

But more importantly, the --texdump option only exists in the Medieval emu and has been entirely removed in the Syphon Filter version.
And even in the Medieval version there are no addressing references to --texdump. This means that the emulator never checks if the option was set, and consequently cannot activate any functionality based on the option.
Also, no additional strings relating to texture dumping exist in the eboot at all. (PSPHD contains something like "Wrote texture '%s'").

So, even if we were to make --texreplace work there is most likely no way to dump textures. Which is absolutely necessary to obtain the correct replacement texture properties.

In conclusion, I do not believe current PS1HD versions actually support texture replacement. If we get a PS1HD game using texture replacement in the future, I will absolutely have a look again. But until then it's a bust in my opinion.

 

[Stayhye]

Thank you for putting this to rest for now.

 

[mexxpower]

Update: I've fixed flaky replacement texture loading.

If you have already downloaded the zip archive, please download the updated version which now includes a patched LocoRoco MC emulator.
I've also added some more details to the tutorial and config-title.txt.

 

[1321 Studio]

So I've started dumping textures of SMT: Persona and used the textures from the HD UI Texture pack for ppsspp in the texreplace folder. So good news is that the new textures fit perfectly and is not crashing the replace fpkg.

Here's my problem, In order for the new textures to not glitch out, the texture must be loaded to memory, resulting in multiple dumps of the same file. I simply copied multiple copies of the updated textures and renamed the copy after the original and placed it in the texreplace folder.(for example XXXXXXXX_81c353f7_128_128_0 for all protag bigger portraits, 'XXXXXXXX' being the random 8 Digit code the appears I guess when dumped from memory. Same with icons, Example - # 8dae17cc_XXXXXXXX_128_16_0 for controls2.png. And there would be like 10-20 copies within just the first hour of gameplay)

Is there any way to have texreplace pull from designated folders similar to ppsspp is what I'm asking?
​

 

[mexxpower]

So I've started dumping textures of SMT: Persona and used the textures from the HD UI Texture pack for ppsspp in the texreplace folder. So good news is that the new textures fit perfectly and is not crashing the replace fpkg.

Here's my problem, In order for the new textures to not glitch out, the texture must be loaded to memory, resulting in multiple dumps of the same file. I simply copied multiple copies of the updated textures and renamed the copy after the original and placed it in the texreplace folder.(for example XXXXXXXX_81c353f7_128_128_0 for all protag bigger portraits, 'XXXXXXXX' being the random 8 Digit code the appears I guess when dumped from memory. Same with icons, Example - # 8dae17cc_XXXXXXXX_128_16_0 for controls2.png. And there would be like 10-20 copies within just the first hour of gameplay)

Is there any way to have texreplace pull from designated folders similar to ppsspp is what I'm asking?
​

Glad to hear that the replacement is generally working for you.

Personally, I was mostly experimenting with GTA Chinatown Wars, where I also noticed the problem you describe. It happened mostly with environment textures, and since I was only focused on replacing UI elements, I unfortunately did not investigate it further.

I have no idea why this is happening or how to fix it. The only thing I can think of is that maybe emulator options related to texture hashing could help eliminate duplicate texture dumps. Specifically, --texclutmode (full or filter) and --texrecent (true or false) might somehow affect how the emulator dumps textures. However, this is pure speculation, so you'll have to try and see if you can get it to work somehow.

Since my original post, I have stopped looking into PSPHD's texture replacement, as the implementation, while functional, seems immature and a bit unreliable. I have been waiting for an official release that uses texture replacement, but no luck so far.

 

[andshrew]

Hi @mexxpower - this was an interesting read. I don't know if you're still interested in this all these years later, but did you ever look at the earlier versions of PSP HD (eg. Castlevania Requiem, Locoroco Remastered, PaRappa The Rapper Remastered).

I've been playing around with Castlevania, the texture dumping function works with this version of the emulator out of the box. It's also an example of an official release loading replacement textures when those textures have been bundled into a .PAK file (texreplace.pak and texreplace-4x.pak). Locoroco and PaRappa have them as individual files in the texreplace folder.

I still don't think there has been any more recent releases which are using this, so perhaps it's a semi abandoned feature now.

 

[mexxpower]

Hi @mexxpower - this was an interesting read. I don't know if you're still interested in this all these years later, but did you ever look at the earlier versions of PSP HD (eg. Castlevania Requiem, Locoroco Remastered, PaRappa The Rapper Remastered).

I've been playing around with Castlevania, the texture dumping function works with this version of the emulator out of the box. It's also an example of an official release loading replacement textures when those textures have been bundled into a .PAK file (texreplace.pak and texreplace-4x.pak). Locoroco and PaRappa have them as individual files in the texreplace folder.

I still don't think there has been any more recent releases which are using this, so perhaps it's a semi abandoned feature now.

Hi, I'm glad to see that people are still interested in this topic and are even experimenting with texture dumping/replacement themselves.

To answer your question: No, I never tried those emulator versions. And to be completely honest, I don't really remember why I chose the ones I did. I remember trying different ones and eventually sticking with the two in my post, so they were probably the ones I had access to and where this feature worked. However, I've given my PS4 away, so unfortunately I can't test anything related to that anymore.

And frankly, I don't feel motivated to do so. This experimental project was a lot of fun to work on, but it never really became anything meaningful. The replacement textures were just too much of a hassle to set up in the way I wanted. Eventually I got tired of trying to work with this, as you suggested, abandoned and somewhat unofficial feature. I feel I have done what I can without access to the source code. Anything more would have to be officially supported by newer versions of the emulator.

At the end of the day, I'm a gamer who enjoys playing games. So when I reached the point where it became tedious and not fun, I stopped. I'm not much into older games these days, but if I get the itch again, I'll probably go with a well-supported mainstream emulator like PPSSPP.

 

[Charles_n_town]

maybe I misunderstood or something, but afaik, castlevania requiem is based on the psp version of symphony of the night. I own both medievil and requiem. in fact, medievil is the first/only game I ever bought after playing a demo at the store. how is psp emulation on the ps4/5? I'm assuming it's better than the ps3 considering the ps3's psp emulator is designed for "minis."

 

[mexxpower]

maybe I misunderstood or something, but afaik, castlevania requiem is based on the psp version of symphony of the night. I own both medievil and requiem. in fact, medievil is the first/only game I ever bought after playing a demo at the store. how is psp emulation on the ps4/5? I'm assuming it's better than the ps3 considering the ps3's psp emulator is designed for "minis."

The PS4 does not have a PSP emulator (or any emulator for that matter) baked into its firmware. Instead, emulators come bundled with the emulated games. These emulators are generally based on the same base emulator (internally called PSPHD for PS4), but often modified to support different (hardware) features required for the specific game they are bundled with. That's why there are many different versions of PSPHD that all have their own (mostly undocumented) peculiarities. For instance, texture dumping/replacement is supported by some versions, broken in others and completely missing for the rest. That's why @andshrew was suggesting to try different emulator versions that come bundled with the games they mentioned.

When trying to emulate a new game that was not officially released for/with PSPHD, you first have to find an emulator version that uses similar features and consequently works well with the new game. For PS2 emulation, there are some emulator versions that work well with most games. However, the official PSP library is significantly smaller, so there is not that much choice. In addition to the emulator version, you also have to set up some game-specific emulator configs to try to iron out glitches and performance issues.
Ultimately, these bundled emulators are really not meant for general purpose use with different games. They often seem to only support the requirements absolutely necessary for the target game, with some unused features broken and others implemented as a "hack".

From my experience, the performance was not all that great. I played emulated GTA: Chinatown Wars and had some significant stutters and and frame drops. And sure, that is a quite taxing game for the PSP, but the PS4 HW should be able to perform better. I don't have any experience with PS5, but I suspect that it performs better due to the greater performance overhead the more powerful HW provides. As mentioned in my previous reply, I would personally go with PPSSPP if you're looking for good PSP emulation.

 

[Stayhye]

The PS4 does not have a PSP emulator (or any emulator for that matter) baked into its firmware. Instead, emulators come bundled with the emulated games. These emulators are generally based on the same base emulator (internally called PSPHD for PS4), but often modified to support different (hardware) features required for the specific game they are bundled with. That's why there are many different versions of PSPHD that all have their own (mostly undocumented) peculiarities. For instance, texture dumping/replacement is supported by some versions, broken in others and completely missing for the rest. That's why @andshrew was suggesting to try different emulator versions that come bundled with the games they mentioned.

When trying to emulate a new game that was not officially released for/with PSPHD, you first have to find an emulator version that uses similar features and consequently works well with the new game. For PS2 emulation, there are some emulator versions that work well with most games. However, the official PSP library is significantly smaller, so there is not that much choice. In addition to the emulator version, you also have to set up some game-specific emulator configs to try to iron out glitches and performance issues.
Ultimately, these bundled emulators are really not meant for general purpose use with different games. They often seem to only support the requirements absolutely necessary for the target game, with some unused features broken and others implemented as a "hack".

From my experience, the performance was not all that great. I played emulated GTA: Chinatown Wars and had some significant stutters and and frame drops. And sure, that is a quite taxing game for the PSP, but the PS4 HW should be able to perform better. I don't have any experience with PS5, but I suspect that it performs better due to the greater performance overhead the more powerful HW provides. As mentioned in my previous reply, I would personally go with PPSSPP if you're looking for good PSP emulation.

The later PSP emus are much better as far as compatibility and performance. But, I don't think any of the later emus use texture replacements.

 

[Charles_n_town]

oh, yeah, I had forgotten about the fact that they used different emulators. not sure why I made that mistake, but thanks for reminding me. I didn't realize some could or could not use texture replacements. I'm not all that well versed in fixing psp/ps2 games for the ps3/ps4/ps5 though.