Mass installation of game updates

[Yo_Yo]

Has this been done before? I have been reading conflicting information: That there is no best way to install updates, that it depends on the game, and it doesn't matter just install it. I also heard that some games only need the last update, while some need updates installed in order. I'm not sure what is true.

My issue is I have updates backed up for 167 games with 331 files totaling 57.9GB. I would prefer to do this offline and all at once. instead of loading 167 games individually and selecting each one at a time and/or redownloading it all.

I did notice when mass installing DLC. I had all of my DLC sorted by size and transferred it to an external. When I selected to "install all" it started installing them by size in the same order I transferred them to the external.

I'm wondering if I sorted files alphabetically and transferred them alphabetically this would work. Since I think the PS3 will sort by the files by what was placed on the drive first "creation date".

This is just a theory but does anyone know if this is a bad idea or not before I try anything.

 

[DeViL303]

Yes, they do install in the order that you copy them to the drive. So if you name them correctly, then arrange them alphabetically and copy them to the drive in the same order, then they will install correctly.

Normally you do need to install each update, not just the last one. If you have time free you can combine the updates from any game into 1 pkg. To do this you extract them all, then paste them all into 1 folder from first to last , over writing files if asked.

 

[Yo_Yo]

Yes, they do install in the order that you copy them to the drive. So if you name them correctly, then arrange them alphabetically and copy them to the drive in the same order, then they will install correctly.

Normally you do need to install each update, not just the last one. If you have time free you can combine the updates from any game into 1 pkg. To do this you extract them all, then paste them all into 1 folder from first to last , over writing files if asked.

Very Interesting! Thanks for the very helpful info as always @DeViL303 !

I thought you had to install them all. but is saw this here about Dark Souls 2 :
https://www.reddit.com/r/ps3hacks/comments/763hvn/must_update_game_in_order/

I can't see why it would hurt installing them all in the correct order either way.

I also don't know if the game needs to create the game data before patching/updating or if installing them before they are installed/loaded matters. Since I don't think sony would expect people to retroactively install updates before loading a game. I would assume it would create the necessary folders if they didn't exist. But again I'm just being hopeful.

So I've been looking at the files. They all have the original name and are pretty much in order I believe thanks to the version number. It would be hard to tell if that's true unless I go 1 by 1 and verify. There may be a small change in the name or some other technicality switching the correct order. I feel like the probability of that happening with so many files is likely and is hidden somewhere in there, unfortunately.

Extracting and making a single file seems like the safest option but also seems time-consuming. Especially after renaming all my DLC one by one by: Game Name - DLC Name - Game ID. I wouldn't mind copying them over as is, but based on what you said I think I will have to either: Verify and make corrections, Rename them all, or Create a single pkg for games with more than 1 pkg.

 

[DeViL303]

You could check in the games update xml, if only 1 update is mentioned there then you only need one, if they are all referenced then you need them all.

Just replace both content IDs with any game you wan to check. This is the xml the PS3 uses to find the updates.
https://a0.ww.np.dl.playstation.net/tpl/np/BCUS98114/BCUS98114-ver.xml

As you can see the example used here, Gran Turismo 5, has a lot of updates and need them all.

About installing updates before game data, I think it should be ok as normally the updates install when you first launch the game, before it has had chance to install game data. At least this is the way I think it works normally..

 

[Berion]

I can't see why it would hurt installing them all in the correct order either way.

Because if some file is duplicating in i.e two PKG archives, then replacing files in incorrect order will cause overwriting file from older version on newer version which can cause unexpected problems in game. NEVER do that.

I also don't know if the game needs to create the game data before patching/updating or if installing them before they are installed/loaded matters.

You don't need to load game first as every game folder has game region unique ID. On PS3, in most cases, "dev_hdd0/game/<GameID>/" have mixed patches, dlc and disc cache. In some cases, DLC have game region unique ID but different than game itself and cannot be placed in the same path as cache and patches.

Since I don't think sony would expect people to retroactively install updates before loading a game. I would assume it would create the necessary folders if they didn't exist. But again I'm just being hopeful

Developers doing that on debug kits (DEX) so it is totally expected and normal.

Extracting and making a single file seems like the safest option but also seems time-consuming. Especially after renaming all my DLC one by one by: Game Name - DLC Name - Game ID.

Making "single file" means unpacking all packages, merging their content IN ORDER, then making from it another package.

If You have in mind that You already sorted all Your packages, then well, that's true, it is additional work. Personally I prefer to keep original files (because they are properly signing) and naming them like below. Sorted fine and looks informative. ^^ With one exception: Gran Turismo 6 which is a hell to update (it have internal patching system, very slow).

<Game Title> <GameID> <Version / DLC>

i.e:
Gran Turismo 5 (BLES-12345) v1.17.pkg
Gran Turismo 5 (BLES-12345) (Nice Car DLC).pkg

 

[Yo_Yo]

You could check in the games update xml, if only 1 update is mentioned there then you only need one, if they are all referenced then you need them all.

Just replace both content IDs with any game you wan to check. This is the xml the PS3 uses to find the updates.
https://a0.ww.np.dl.playstation.net/tpl/np/BCUS98114/BCUS98114-ver.xml

As you can see the example used here, Gran Turismo 5, has a lot of updates and need them all.

About installing updates before game data, I think it should be ok as normally the updates install when you first launch the game, before it has had chance to install game data. At least this is the way I think it works normally..

Alright, that sounds good. I think I can figure out some way to make it work then.

I did just run into a big issue with my DLC that would also apply to mass installing updates. Partway in it stopped and said installation complete. However, I don't think it was complete with them all. maybe one of the DLC installs prompts a popup when done? It gave me an option to go back. then I noticed not everything is in-game data utility. I was able to track it down to Lost planet as the last timestamp. I think that is where it stopped. Unless some games don't show up in game data utility? If it was going in order it should have Drakengard on the list. It is DLC for a PKG game though. But the last of us should have more data than what it's showing so I think it only installed 2 pkgs for the game before it stopped on the list. I'm really confused with why it finished so soon and why I can't see all the games listed.

Now I'm wondering If there is a way to check if this is true. If so is there a way to prevent this on mass installations for DLC and updates?

 

[Yo_Yo]

Because if some file is duplicating in i.e two PKG archives, then replacing files in incorrect order will cause overwriting file from older version on newer version which can cause unexpected problems in game. NEVER do that.



You don't need to load game first as every game folder has game region unique ID. On PS3, in most cases, "dev_hdd0/game/<GameID>/" have mixed patches, dlc and disc cache. In some cases, DLC have game region unique ID but different than game itself and cannot be placed in the same path as cache and patches.



Developers doing that on debug kits (DEX) so it is totally expected and normal.



Making "single file" means unpacking all packages, merging their content IN ORDER, then making from it another package.

If You have in mind that You already sorted all Your packages, then well, that's true, it is additional work. Personally I prefer to keep original files (because they are properly signing) and naming them like below. Sorted fine and looks informative. ^^ With one exception: Gran Turismo 6 which is a hell to update (it have internal patching system, very slow).

<Game Title> <GameID> <Version / DLC>

i.e:
Gran Turismo 5 (BLES-12345) v1.17.pkg
Gran Turismo 5 (BLES-12345) (Nice Car DLC).pkg

That is probably the best option. I'm already planning on doing the next project before I even finish my DLC project. I'll probably go that route once I have had a breather from this one. I do have GT 5 and heard it was a pain so I was thinking of skipping that one and letting sony do all the work. lol

 

[Yo_Yo]

Because if some file is duplicating in i.e two PKG archives, then replacing files in incorrect order will cause overwriting file from older version on newer version which can cause unexpected problems in game. NEVER do that.



You don't need to load game first as every game folder has game region unique ID. On PS3, in most cases, "dev_hdd0/game/<GameID>/" have mixed patches, dlc and disc cache. In some cases, DLC have game region unique ID but different than game itself and cannot be placed in the same path as cache and patches.



Developers doing that on debug kits (DEX) so it is totally expected and normal.



Making "single file" means unpacking all packages, merging their content IN ORDER, then making from it another package.

If You have in mind that You already sorted all Your packages, then well, that's true, it is additional work. Personally I prefer to keep original files (because they are properly signing) and naming them like below. Sorted fine and looks informative. ^^ With one exception: Gran Turismo 6 which is a hell to update (it have internal patching system, very slow).

<Game Title> <GameID> <Version / DLC>

i.e:
Gran Turismo 5 (BLES-12345) v1.17.pkg
Gran Turismo 5 (BLES-12345) (Nice Car DLC).pkg

I just finnished my DLC project and started trying to follow your method for combining updates, but I'm a little lost. I'm using Aldos tools I can unpack by: right-click unpack. Then I copy the root of update 2 into the root of update 1 while replacing all files. I think that is correct? Then I go to repack it and I'm not sure what to do.

I opened Aldos make pkgs but I don't know what to do with it. I also tried using Trueancestor PKG Repacker. I'm not sure how these work. I'm assuming I place the version 1 folder that has been overwritten with all succeeding updates somewhere in these program directories then run the program? When I launch Aldos make pkg app it opens and closed real quick. So I'm not sure what to do at this point.

 

[DeViL303]

Then I copy the root of update 2 into the root of update 1 while replacing all files. I think that is correct?

Yep. start with oldest update as base, then paste in the newer updates overwriting as needed until you get to the latest one.

I opened Aldos make pkgs but I don't know what to do with it. I also tried using Trueancestor PKG Repacker. I'm not sure how these work. I'm assuming I place the version 1 folder that has been overwritten with all succeeding updates somewhere in these program directories then run the program? When I launch Aldos make pkg app it opens and closed real quick. So I'm not sure what to do at this point.

You should be able to just right click the folder and choose "make pkg"

 

[Berion]

Yes, extracted content from newer version, You need to place to older, overwriting all files. So in example: if You have 3 patches, let's say v1.00, 1.01 and 1.02, an in addition DLC which is for sure in the same GameID dir. Then You placing DLC (as first step, same for all DLCs because last patch for covering also all DLCs) contents to 1.00; then contents of 1.01 into 1.00, and then contents of 1.02 to 1.00. You should end with all DLC and patches integrated, and last APP_VER from last PARAM.SFO should be greater or equal to APP_VER from last released DLC.

When You have data prepared, You must create package from "mixed 1.00" and fulfill config file for packaging application.

package.conf example of my RetroArch repack made for "Dark Gimmick!" hack. Read on wiki about ContentType, I don't remember what You must write there, for sure not GameExec.

ContentID  = EP0002-GMCK00001_00-00000000000000R1
k_licensee  = 0x00000000000000000000000000000000
DRMType  = Free
ContentType  = GameExec
PackageVersion  = 01.10

ContentID should be real NPCommID and version should be last from last patch.

package creating:

psn_package_npdrm.exe <path to config, put it in root of "v1.00 mixed"> <GameID>

example:
psn_package_npdrm.exe package.conf GMCK00001

PS: I don't use Aldostools AIO package. I'm also not sure if You should use make_npdrm instead of psn_package_npdrm. And of course You must place needed keys in home under ".ps3" (in Windows %AppRoaming% or in %HOMEPATH%).

 

[Yo_Yo]

Yep. start with oldest update as base, then paste in the newer updates overwriting as needed until you get to the latest one.


You should be able to just right click the folder and choose "make pkg"

Hm... That would be really nice. Unfortunately, I don't have that option. Maybe the install is not complete for the program. I know Aldos tools didn't work right out of the box. I had to follow a tutorial to get it working. There were some "extra steps" in the tutorial to get everything working. I don't think I did that part. I'll go check. This is what I'm referring to:

 

[Yo_Yo]

Yes, extracted content from newer version, You need to place to older, overwriting all files. So in example: if You have 3 patches, let's say v1.00, 1.01 and 1.02, an in addition DLC which is for sure in the same GameID dir. Then You placing DLC (as first step, same for all DLCs because last patch for covering also all DLCs) contents to 1.00; then contents of 1.01 into 1.00, and then contents of 1.02 to 1.00. You should end with all DLC and patches integrated, and last APP_VER from last PARAM.SFO should be greater or equal to APP_VER from last released DLC.

When You have data prepared, You must create package from "mixed 1.00" and fulfill config file for packaging application.

package.conf example of my RetroArch repack made for "Dark Gimmick!" hack. Read on wiki about ContentType, I don't remember what You must write there, for sure not GameExec.

ContentID  = EP0002-GMCK00001_00-00000000000000R1
k_licensee  = 0x00000000000000000000000000000000
DRMType  = Free
ContentType  = GameExec
PackageVersion  = 01.10

ContentID should be real NPCommID and version should be last from last patch.

package creating:

psn_package_npdrm.exe <path to config, put it in root of "v1.00 mixed"> <GameID>

example:
psn_package_npdrm.exe package.conf GMCK00001

PS: I don't use Aldostools AIO package. I'm also not sure if You should use make_npdrm instead of psn_package_npdrm. And of course You must place needed keys in home under ".ps3" (in Windows %AppRoaming% or in %HOMEPATH%).

OK, great I got it working! I think I just made my first single PKG update. I am keeping my originals updates separate. I do like the idea of combining them with DLC and updates all in 1 pkg. I already installed all of my DLC though. Maybe I should have done them together. Would it be possible though to combine all of my updates into 1 pkg for each game and then later combine the DLC with them? Or would I have to start over again recombining everything? Since you mentioned there was a specific order.

I guess another option would be to have 1 complete update pkg and 1 complete DLC pkg for each applicable game. I guess that wouldn't be too bad either.

 

[Berion]

You can add later whatever You want. Data inside is not modified during package creating. Consider it as encrypted RAR archive (of course not literally). ;p

If version from DLC is higher than from last patch, overwrite all data found in DLC. If version is higher in patch instead to DLC, then do not overwrite files from DLC (I never experienced higher last DLC version than last patch version but keep this in mind).

As I said before, not all games keeping DLCs in the same folder as patches. For them You cannot keep all data in one PKG because *.pkg doesn't allow to keep multiples dirs and pkg creation apps for unknown reason validating them before allow to create (if they wouldn't do that, it would be possible to create package of "game" and few folders inside with patches, dlc etc. for various of games). BTW: I really hate when app authors put such functionality without a way to disabling it. Validation should be optional and lie on user, not on app. Similar situation I experienced on PSTV where VitaShell will not unpack *.zip if contains icons in 24bit. LiveArea read them fine and I don't care if in specification for devs is 8bit. I do not hacking console to installing different policeman's.

Speaking of ZIPs. You do not need create PKG. As I remember, all current file managers would unpack *.zip and update database informations. But You must put special filename like i.e "~dev_hdd0~game~npea12345.zip". But I don't remember now, sorry.

 

[DeViL303]

@Berion There is a special package creator that lmn7 made, that allows you to make a normal app or gamedata package, but it also allows you to add a "dev_hdd0" or "dev_blind" folder to the root of the package too.

So for example this means you can have a normal package installing to dev_hdd0/game/CONTENT01/ , but inside that package there can be a dev_hdd0/game/CONTENT02/ folder, and also if you wanted a dev_hdd0/exdata/ folder. So this way one package could be made that can install the updates, the DLC and the rap files all in one go.

So the top level of your package before you build it could look like this:

To use it just drag the top folder of your package onto the make.bat.

I can not find the exact post now, and I am not sure which version this is and if it is the latest, but it works and I use it regularly in my mods. It is possible this version has issues with large packages and there is a newer one somewhere that has that fixed.

Note: only the main folder will get added to the database, any other files in the extra dev_hdd0 or dev_blind folders are just copied to locations like a normal copy operation. So you can not add 2 apps to the database in one go.

 

[Yo_Yo]

You can add later whatever You want. Data inside is not modified during package creating. Consider it as encrypted RAR archive (of course not literally). ;p

If version from DLC is higher than from last patch, overwrite all data found in DLC. If version is higher in patch instead to DLC, then do not overwrite files from DLC (I never experienced higher last DLC version than last patch version but keep this in mind).

As I said before, not all games keeping DLCs in the same folder as patches. For them You cannot keep all data in one PKG because *.pkg doesn't allow to keep multiples dirs and pkg creation apps for unknown reason validating them before allow to create (if they wouldn't do that, it would be possible to create package of "game" and few folders inside with patches, dlc etc. for various of games). BTW: I really hate when app authors put such functionality without a way to disabling it. Validation should be optional and lie on user, not on app. Similar situation I experienced on PSTV where VitaShell will not unpack *.zip if contains icons in 24bit. LiveArea read them fine and I don't care if in specification for devs is 8bit. I do not hacking console to installing different policeman's.

That is very interesting! I guess for consistency and to be safe it might be better for me to keep DLC and updates separate so I don't have to worry about the fine details for each game.

LOL yes we all just want to enjoy our products without being limited by the man.

No worries about the zip process I can work with PKGs. The only problem with PKGs is the PS3 PKG installer. I found it does not like mass installations. through my testing, It usually gets tired after about 120 - 130 installs and kicks you out. I thought it was a power setting, or something else keeping them from installing, but I found it's consistently around that number no matter how fast or slow they install. I even tried pushing the buttons on the controller frequently and same thing. SOI will probably have to install my updates in 2 chunks. Unless there is a better homebrew option for a PKG installer?

 

[Yo_Yo]

So I have about 30 left and I have a few more questions since I have done this quite a few times

1. When I extract the PKG there is a txt file in the root. I just checked it and I guess it is an extraction log. So I have been copying all of those into folder 1 and then creating a package. I don't know if I was supposed to do that or if it even matters. Is that ok or do I have to start over?

2. I was planning on installing these on a fat32 drive. After combining the updates for Guilty Gear Xrd Revelator the file size is 4.41GB. Is it recommended to split this with a file splitter or to only put 1/2 of the updates in a PKG and 1/2 of the updates in a second package naming them part 1 and part 2? Or just doing them all manually instead of combining?

3. I read here that GT 6 has a special trick to get it to update correctly:
https://www.psx-place.com/threads/fix-gran-turismo-6-when-updaing-multiple-updates.889/
Does this mean I should put update 2 - 7 in a pkg and leave the rest alone? and install according to the instructions? Or should this game be left alone and install separate packages 2 - 7. then 1 at a time while loading the game between each update?

4. Is there a difference between a patch and an update? LittleBigPlanet 3 has a lot of updates. #1 is a patch and the rest are labeled as updates. I just want to be sure combining it all is correct. After creating the pkg the program never said done but I had a pkg and some other text was off. I just want to be sure that I didn't do anything wrong with it. and that it is ok to merge what is labeled as a patch with an update.

5. Since this is such a tedious and repetitive task, I'm hoping I didn't screw up anything. Like accidentally hitting skip instead of replacing all, or overwriting in the wrong order on one or two. What is the worst that can happen? What would have to be corrected if I eventually found an issue? I'm assuming just deleting the game folder, and reinstalling any DLC/Updates?

Sorry, for so many questions at once. These are just concerns I had as I was doing this process.

 

[Berion]

Ad1. If You will add or left any additional data to game package which wasn't part of the game, nothing will happen. At least this is how I understood Your question.

Ad3. You can probably merge all updates for GT6 but this is pointless. This game have it's own internal patching system, which means that GT6 updates in package form, are unpacking as normal but doesn't contains data to be replaced and/or to add, as normally. Instead unpacking data are patches which games applying. Very slow, annoying and stupid. So in case of this game, it is better to install all updates, applying them by game and after that, creating custom package from it's "dev_hdd0/game/<GT6>".

Ad4. Updates and patches are synonyms.

Ad5. Unexpected game bugs, freezes etc. You will not be able to do everything which games offering, that's for sure.
Yes, deleting game folder, and making another "mega patch" again, an installing it again.

 

[Yo_Yo]

Ad1. If You will add or left any additional data to game package which wasn't part of the game, nothing will happen. At least this is how I understood Your question.

Ad3. You can probably merge all updates for GT6 but this is pointless. This game have it's own internal patching system, which means that GT6 updates in package form, are unpacking as normal but doesn't contains data to be replaced and/or to add, as normally. Instead unpacking data are patches which games applying. Very slow, annoying and stupid. So in case of this game, it is better to install all updates, applying them by game and after that, creating custom package from it's "dev_hdd0/game/<GT6>".

Ad4. Updates and patches are synonyms.

Ad5. Unexpected game bugs, freezes etc. You will not be able to do everything which games offering, that's for sure.
Yes, deleting game folder, and making another "mega patch" again, an installing it again.

Thank You @Berion !

I only have 12 more left to go before trying this out! I'm not sure what to do with Guilty Gear Xrd Revelator, but I will probably just do it manually with GT 6. I guess my GT 5 is the XL edition so no updates there which is good. I'll keep you all posted with the progress but I'm feeling good about everything now.