PS3 Apollo save tool (development thread)

[Algol]

Je confirme que c'est très long comparé au travail que fait mmCM en écriture sur de l'USB même avec une clef USB2 rapide.
Je suppose que c'est lié au "DevKit" de SONY pour cette interface ...

I confirm that it is very long compared to the work that mmCM does in writing on the USB even with a fast USB2 key.
I guess it's related to SONY's "DevKit" for this interface ...

 

[bucanero]

Btw, probably you already know it, but worths to be mentioned...
If the "source" and "target" users are in the same PS3 and are not registered in PSN (in other words, have an account_id filled with zeroes) the trophy import/export is as simple as this procedure:
1) copy files from source account
2) paste files in target account
3) rebuild database
4) profit

Is just a copypaste, this allows to share trophies in between user accounts, and allows to format or replace the hdd, create a new account and just "paste" the trophy files in the new user account

@sandungas , do you know if you need to actually have the game installed and run it before copying the trophy files?
I want to confirm if you can just copy a trophy folder from an export backup, and then run a DB rebuild. That's enough to import?

 

[bucanero]

"Import option" is the last thing, related to user data backup, to be added to Apollo. If it's possible, it will be fantastic . (Apollo is already a must have application, but... this feature will add the "icing on the cake" on Apollo).
Another thing I noticed is about copy performance. When exporting trophies (or saves), the copy takes too long to complete (to export about 6MB of data, it took about 2~3 minutes). Probably a PSL1GHT limitation...

yes, the file copy function to USB is quite slow, and probably related to the psl1ght implementation.
I should upgrade the code and use the "async" copy functions that are also available... I implemented them at some point in PKGi, so I just need to recycle the code (it was based on Irisman or managunz source)
That should give an improvement over the regular fopen/fread/fwrite functions that are really slow when using USB. If you access dev_hdd0 the performance is much better.

I honestly didn't care too much at first, because I was focusing on the main Apollo features, but now I think there's room to do these little tweaks and improvements, because all the big stuff is already there and working.

 

[sandungas]

...if someone can confirm that copying a trophy folder (and rebuilding the DB) is enough to import a trophy-set, then I'll add an import option so every step of the "trophy road" is covered.

I made this process many times
1) copy the trophy folder with a file manager
2) delete the trophy folder with a file manager
3) rebuild database (to cleanup all the info related with that trophy installation)

At this point you can do 2 different things
1) boot the game (so it creates a new trophy installation)
2) paste the trophy folder and rebuild database

Actually, you can do both things consecutivelly, first allow the game to create the new trophy installation... and then overwrite it with the backup (and rebuild database)

----------------
If at some point you boot the game with incorrect info in the database (as example, the trophy installation is indexed in the database, but the files doesnt exists)... the game is going to display an error message at boot time and exits to XMB (this cleanups the database)... and after that if you boot the game again the trophy installation is performed normally

 

[sandungas]

@sandungas , do you know if you need to actually have the game installed and run it before copying the trophy files?

Is not needed

I want to confirm if you can just copy a trophy folder from an export backup, and then run a DB rebuild. That's enough to import?

Yes, if the trophy files are valid (because was backuped from the same PS3/account, or because you are making them valid with apollo) the only thing you need to do is rebuild the database

Lets say... i have a game i never played... but i boot it one time to allow it to perforrm the trophy installation (and savedata generation, and gamedata installation, the games are dependant of that 3 things)

After that i backup the trophy installation, delete the saves and gamedata, and rebuild database... with the goal of cleaning all the records of that game in my PS3 (in other words, the next boot of the game is going to do the same like if it was booting for first time)

At that point (before booting the game for first time), if you restore the trophies instalation the game loads them fine... technically the game needs to perform all the actions that happens when you boot it for first time, but the trophies installation is bypassed because the trophy files you restored are indexed in the database and are valid, so the game loads them like if it was booted before

 

[DeViL303]

and then run a DB rebuild. That's enough to import?

If you wanted you could add an optional tick box for "force database rebuild on next reboot" , all it requires is to copy this db.err file to dev_hdd0/mms/ https://www.psdevwiki.com/ps3/Recovery_Menu#4._Rebuild_Database

 

[sandungas]

If you wanted you could add an optional tick box for "force database rebuild on next reboot" , all it requires is to copy this db.err file to dev_hdd0/mms/ https://www.psdevwiki.com/ps3/Recovery_Menu#4._Rebuild_Database

Instead a checkbox.... i would display a "full screen message" inmediatly before exiting apollo with a text like this:

Some of the changes you made in this apollo session requires a database rebuild, do you want to enable it automatically in the next boot ?
YES/NO​

Behind that are required some functions to identify if the changes made in the actual apollo session requires the database rebuild (otherway dont display the warning)

 

[DeViL303]

Yes, but IMO instead of when exiting, It would be better to give the option at the time of applying a change that required a database rebuild. Because the user can always exit with the PS button.

 

[bucanero]

thanks @DeViL303 @sandungas , nice idea!

for these trophy actions Apollo can create that db.err file so the user just needs to reboot and get the trophy updates working.
I can ask when the user makes a trophy change, or at the exit... probably at the trophy change is easier.

 

[haxxxen]

@bucanero
just want to leave my "5 cents" here, but your app has matured pretty well and I can't wait to try it by now
why don't you try your luck with the plugin-side of Artemis? downside is, it is only possible with sdk

afterall you are more experienced than me with coding. unfortunately, my current degree/level only let me modify stuff, though, I will try my best to get better

 

[sandungas]

thanks @DeViL303 @sandungas , nice idea!

for these trophy actions Apollo can create that db.err file so the user just needs to reboot and get the trophy updates working.
I can ask when the user makes a trophy change, or at the exit... probably at the trophy change is easier.

You could force the reboot too, there are syscalls for it (irisman does it when you select one of the options availables from the setting "BD emulation type")

I suggested to display the warning automatically when exiting apollo because this way apollo is doing some "smart" detection to identify if is needed to rebuild the database
The implementation could be tricky because i guess would be needed to "flip" the status of it in every option that requires it
Lets say, in the source code you could start with a global variable named db_rebuild=0 (disabled by default), and the actions that requires the rebuild are going to "flip" it to the state 1

---------
I was also thinking that apollo can be used to perform many changes in the same session, lets say... we need to do 10 changes and 4 of them requires the database rebuild
Adding a chekbox (or info message) to all them could complicate the look of the interface, or the usage

 

[bucanero]

@bucanero
just want to leave my "5 cents" here, but your app has matured pretty well and I can't wait to try it by now
why don't you try your luck with the plugin-side of Artemis? downside is, it is only possible with sdk

afterall you are more experienced than me with coding. unfortunately, my current degree/level only let me modify stuff, though, I will try my best to get better

regarding Artemis plugin, I actually did some tests, but I think my setup/SDK is not correct or I'm missing something, because I've built the plugin .sprx (without any modification) and when I tested it on my PS3, the plugin shows "Artemis attached" but no cheats are applied to the game.

I mean, I can build the source, I get a .sprx with the same file size of the original r5 plugin by Dnawrkshp, I execute it correctly on the PS3 (so the binary is valid), then I run a game, press Start to attach, the plugin shows "artemis attached", but the cheats are never applied.
I assume it's failing to attach, or failing to apply the memory cheats after attachment.

Again, I don't have too much experience with the original SDK so I got tired, because I honestly don't understand why this is happening, since I'm using the un-modified original source. Maybe there's some other requirement during build, I don't really know.
Since you have built some artemis plugins in the past, maybe you could help me validate, if building the sprx source just as it is in my Github repository works properly or not.
If it works (cheats get applied), then the issue is in my building process or sdk.

btw, if you want to try the latest Apollo beta, just use replace with this EBOOT:
https://www.mediafire.com/file/ghyfeio4qa171xx/Eboot_b3.zip/file

 

[bucanero]

Another thing I noticed is about copy performance. When exporting trophies (or saves), the copy takes too long to complete (to export about 6MB of data, it took about 2~3 minutes). Probably a PSL1GHT limitation...

I've been doing some tests using the async copy functions from the PS3, the copy to USB speed gets improved by a good margin.
Still, the improvement seems better when copying saves than copying trophies. Trophy folders have too many files (like many PNG for each trophy image) and copying many files seems to degrade performance.
For trophies, it seems that you get a much better result if you select "export to Zip", as the system only writes one big file with all the stuff in it, and avoids writing many small files separately.
I'll try to add a "bulk export trophies to Zip", it might help users that have many trophy-sets installed.

 

[jcorrea]

I've been doing some tests using the async copy functions from the PS3, the copy to USB speed gets improved by a good margin.
Still, the improvement seems better when copying saves than copying trophies. Trophy folders have too many files (like many PNG for each trophy image) and copying many files seems to degrade performance.
For trophies, it seems that you get a much better result if you select "export to Zip", as the system only writes one big file with all the stuff in it, and avoids writing many small files separately.
I'll try to add a "bulk export trophies to Zip", it might help users that have many trophy-sets installed.

Nice feature . After we discover whats wrong with the resign process of "copied" trophies from another PS3 (if we discover and if it's possible to implement at Apollo), I think we can add the import option. I'm installing VS 2019 right now to debug TrophyIsGood to check a problem with a specific trophy-set.

 

[bucanero]

Nice feature . After we discover whats wrong with the resign process of "copied" trophies from another PS3 (if we discover and if it's possible to implement at Apollo), I think we can add the import option. I'm installing VS 2019 right now to debug TrophyIsGood to check a problem with a specific trophy-set.

From what we can check in the wiki, the trophy files (TROPCONF, TROPUSR, TROPTRNS) have many signatures and hash checks, so I can imagine that files coming from a different PS3 could fail those checks and fail to sync online.
https://psdevwiki.com/ps3/Trophy_files

I imagine that the process works fine when you edit your own installed trophies because any hash related to the console will still be the same, but when "importing" files from a different PS3 or account, checks would fail.

 

[bucanero]

talking about trophies, I saw that the dev-wiki page doesn't have any info about the structure of TROPUSR.DAT and TROPTRNS.DAT:
https://psdevwiki.com/ps3/Trophy_files

@sandungas , I think that you're one of the editors there, right? you can add or link the structure information here:
https://github.com/bucanero/apollo-ps3/blob/master/include/trophy.h

also, some block type information can be found here:
https://github.com/bucanero/apollo-ps3/blob/master/source/trophy.c

(un)related fun fact: why I called "lollipop chainsaw" the trophy release branch?
while I was doing some early research/tests on trophy editing, I broke my Lollipop chainsaw trophy-set and lost all the trophies (forgot to make a backup)
I replayed the game, and I was able to unlock many trophies, except for 1 (the one related to looking under Juliet skirt ) . No matter how much I looked down there, the trophy didn't show up.
So, to fix it and recover that trophy I completed the development of Apollo's trophy edit function.

in the end.... next time you edit a trophy using Apollo, remember that it's mostly thanks to Juliet's skirt

 

[sandungas]

talking about trophies, I saw that the dev-wiki page doesn't have any info about the structure of TROPUSR.DAT and TROPTRNS.DAT:
https://psdevwiki.com/ps3/Trophy_files

@sandungas , I think that you're one of the editors there, right? you can add or link the structure information here:
https://github.com/bucanero/apollo-ps3/blob/master/include/trophy.h

also, some block type information can be found here:
https://github.com/bucanero/apollo-ps3/blob/master/source/trophy.c

Thx, the page was created years ago, and i created an "skelleton" of sections in it (waiting for being completed) but other people was editing it (stuff related with encryption, signatures, etc...) and i lost control of the page. I think the sections needs to be reordered/renamed, and the whole page needs a rewriting, but i cant do it because is over my skills
So i just added the links to github/apollo, and i hope eventually someone will rewrite the page and will take your code in consideration

 

[bucanero]

I recently added some small improvements,
- now Apollo can display the icon.png from the savegame or trophy set (even the ones from the online database)
- if you apply trophy edits, apollo will ask if you want to rebuild database on reboot (app will create the db.err file for you)

I don't have room for big features, so I guess I'll try to add a few more details or fixes for a quick release.

 

[Algol]

@bucanero (Buk).

Bonjour mon ami.

Aurais-tu l'amabilité de me faire un lien pour charger cette toute dernière amélioration de ton excellent HomeBrew s'il te plait ?
Jeu viens d'aller sur ton site GitHub et je n'ai chargé que celle datée du 3/04/2021.

Merci par avance et à très bientôt.


Algol "le papy".


Hello my friend.

Would you kindly link me to upload this latest upgrade to your excellent HomeBrew please ?
Game just went to your GitHub site and I only uploaded the one dated 03/04/2021.

Thank you in advance and see you very soon.

 

[bucanero]

@bucanero (Buk).
Would you kindly link me to upload this latest upgrade to your excellent HomeBrew please ?
Game just went to your GitHub site and I only uploaded the one dated 03/04/2021.

Thank you in advance and see you very soon.

the source code changes are already on Github, but I haven't released the new version yet.
Don't worry, I'll post a link when the release is ready.