PS3 webMAN MOD/prepISO - Detecting games is inconsistent

[StaleHermit]

Recently purchased and jailbroke a PS3 CECH-2501A model on 4.87 firmware with the Evilnat CFW. Running webMAN MOD v1.47.35 Full with a Seagate Portable 2TB external drive. It has been formatted with MBR to NTFS.

Here's what's happening:

1. Run prepISO and some games will be detected.
2. Run prepISO again and maybe more games will be detected, or less, or the same.
3. Repeat process until it detects all games. Might take more than a dozen runs.

Once it detects all the games it usually sticks. Games run just fine off of the HDD, but detecting them is a pain. I've reformatted the drive several times and reset/reinstalled webMAN MOD. Any ideas why this is happening?

 

[aldostools]

Try unchecking /dev_ntfs in /setup.ps3 You need to expand the section "Scan for content".
That option allows scan NTFS content (PS3ISO, PSXISO, DVDISO, BDISO) without run prepISO.
prepISO is needed for scan additional content like exFAT or PSPISO, PS2ISO, Movies, etc.

The option /dev_ntfs may cause a conflict when prepISO sends to refresh the XML and tries to re-scan the content already scanned by prepISO.

 

[StaleHermit]

Try unchecking /dev_ntfs in /setup.ps3 You need to expand the section "Scan for content".
That option allows scan NTFS content (PS3ISO, PSXISO, DVDISO, BDISO) without run prepISO.
prepISO is needed for scan additional content like exFAT or PSPISO, PS2ISO, Movies, etc.

The option /dev_ntfs may cause a conflict when prepISO sends to refresh the XML and tries to re-scan the content already scanned by prepISO.

So webMAN MOD is able to read games on an NTFS drive without prepISO? Mine doesn't detect any games when I first turn on the console until I run prepISO. Having it detect them automatically would be awesome. The /dev_ntfs option is currently checked, and Disable scan at startup is unchecked. Refreshing the XML and reloading XMB doesn't get it to detect anything either.

 

[Hanz77]

So webMAN MOD is able to read games on an NTFS drive without prepISO? Mine doesn't detect any games when I first turn on the console until I run prepISO. Having it detect them automatically would be awesome. The /dev_ntfs option is currently checked, and Disable scan at startup is unchecked. Refreshing the XML and reloading XMB doesn't get it to detect anything either.

Did you find a solution to this? I have the same issue.

 

[bguerville]

The option /dev_ntfs may cause a conflict when prepISO sends to refresh the XML and tries to re-scan the content already scanned by prepISO.

Have you checked what that "conflict" is?
Is it a thread synchronisation issue when reading usb storage space?
Or just a xml production issue?

 

[bguerville]

So webMAN MOD is able to read games on an NTFS drive without prepISO? Mine doesn't detect any games when I first turn on the console until I run prepISO. Having it detect them automatically would be awesome. The /dev_ntfs option is currently checked, and Disable scan at startup is unchecked. Refreshing the XML and reloading XMB doesn't get it to detect anything either.

Game reading from a mounted iso is done externally through rawseciso, neither wMM or prepISO are involved in that part.

What prepISO does is scan ntfs partitions & generate a mapping file in /dev_hdd0/tmp/wmtmp for every iso it finds & copy the game icon file. Those mapping files are then used later when needed for iso mounting & rawseciso processing.

However wMM can do the same thing through its own ntfs support implementation if the feature is enabled.
It can create the same mapping files on its own.

And it would appear that there is conflict when the 2 mapping files generation implementations run together, something like that.

In the old days, there was only prepNTFS as there was no ntfs library available for prx projects, wMM had no way to access ntfs file systems directly.
Then the libntfs library got ported for prx project & support was added in the wMM.
In theory it should have replaced prepISO altogether (the ntfs part at least), but then that maximum cluster limitation made that move somewhat complicated.
In any case, both ways to scan ntfs partitions were kept leading to the current situation...

 

[aldostools]

Have you checked what that "conflict" is?
Is it a thread synchronisation issue when reading usb storage space?
Or just a xml production issue?

Yes. No. No.

If the option /dev_ntfs is checked in /setup.ps3, the files .ntfs[*] stored in /dev_hdd0/tmp/wmtmp are cleared and the ISO in NTFS are re-scanned by webMAN MOD when prepISO exits (it performs a /refresh.ps3 after scan).

The issue here is that the NTFS scanning in the plugin is more limited than prepISO:
1. The .ntfs[*] files found for exFAT are removed.
2. The fake isos created for PKG, Movies, etc. stored in NTFS/exFAT are removed.
3. The files that can be found in prepISO but not by the ntfslib in the plugin are also removed.

 

[bguerville]

Yes. No. No.

If the option /dev_ntfs is checked in /setup.ps3, the files .ntfs[*] stored in /dev_hdd0/tmp/wmtmp are cleared and the ISO in NTFS are re-scanned by webMAN MOD when prepISO exits (it performs a /refresh.ps3 after scan).

The issue here is that the NTFS scanning is more limited than prepISO:
1. The .ntfs[*] files found for exFAT are removed.
2. The fake isos created for PKG, Movies, etc. stored in NTFS/exFAT are removed.
3. The files that can be found in prepISO bare also removed.

I see..

Maybe the exfat files should be processed separately, as .exfat & not as .ntfs. Same thing for the fake isos.

And we should be able to process exfat in the same way as ntfs using the ffs library, shouldn't we?

The max amount of clusters in ntfs is the only real limitation afaik & we might be able to get rid of it, we need to look more closely into the problem..

As for the rest, wMM should probably use prepISO OR its own support but not both, imho anyway.

 

[aldostools]

I see..

Maybe the exfat files should be processed separately, as .exfat & not as .ntfs. Same thing for the fake isos.

And we should be able to process exfat in the same way as ntfs using the ffs library, shouldn't we?

The max amount of clusters in ntfs is the only real limitation afaik & we might be able to get rid of it, we need to look more closely into the problem..

As for the rest, wMM should probably use prepISO OR its own support but not both, imho anyway.

IMO that segmentation that you mention for exfat & fake isos would make the code more complex without additional benefits.

I use the extension .ntfs[PS3ISO] in exFAT to allow that other backup managers mount them too.

The scanning issue could be fixed making prepISO tell the plugin that don't use the prepNTFS during the content scanning.
It should not be too difficult to fix. I simply haven't found the time to work on this known issue.

 

[bguerville]

IMO that segmentation that you mention for exfat & fake isos would make the code more complex without additional benefits.

I use the extension .ntfs[PS3ISO] in exFAT to allow that other backup managers mount them too.

I don't really appreciate the added complexity issue you are raising but you know the code better than I so I trust your judgement, using a different extension like .exfat was only one way to add some differentiation, there are other viable options.
I don't think that mixing every iso mapping into one kind without differentiation at all is a good idea, even if you would not currently use it, just like I never thought that using the same repository folder for multiman, webman, wMM, etc.. was a good idea on the long term either.. And those problems are somewhat linked, everything you do in wMM must meet the requirements of multiman etc.. The problem being, in my view, that only wMM gets updated so it's "kept back" by whatever multiman may already support.

webman used to be multiman's web companion but I think webMAN-MOD has totally outgrown multiman. Tbh, as things stand, I would probably favour a totally standalone & independent webMAN-MOD with its own repo folders, its own fs support libraries & without prepISO but wMM has become complex given the numerous upstream & downstream dependencies, I can easily understand any reluctance in rewriting things.

Anyway if we disagree about these things, that's OK. ;-)

Ultimately the best way to address all those file system related issues once & for all would be to add extra fs support at lv2 syscalls level & all usb partition types (fat/exfat/ntfs/ext) mounted as /dev_usbxxx.

You already know how to fix the scan issue, so that is the main thing..

 

[aldostools]

I don't really appreciate the added complexity issue you are raising but you know the code better than I so I trust your judgement, using a different extension like .exfat was only one way to add some differentiation, there are other viable options.
I don't think that mixing every iso mapping into one kind without differentiation at all is a good idea, even if you would not currently use it, just like I never thought that using the same repository folder for multiman, webman, wMM, etc.. was a good idea on the long term either.. And those problems are somewhat linked, everything you do in wMM must meet the requirements of multiman etc.. The problem being, in my view, that only wMM gets updated so it's "kept back" by whatever multiman may already support.

webman used to be multiman's web companion but I think webMAN-MOD has totally outgrown multiman. Tbh, as things stand, I would probably favour a totally standalone & independent webMAN-MOD with its own repo folders, its own fs support libraries & without prepISO but wMM has become complex given the numerous upstream & downstream dependencies, I can easily understand any reluctance in rewriting things.

Anyway if we disagree about these things, that's OK. ;-)

Ultimately the best way to address all those file system related issues once & for all would be to add extra fs support at lv2 syscalls level & all usb partition types (fat/exfat/ntfs/ext) mounted as /dev_usbxxx.

You already know how to fix the scan issue, so that is the main thing..

I agree with your suggestions in terms of organization and isolation of the operation.
That would have been the best decision if we did it in 2014.

However, webMAN MOD has been developed to cooperate with existing applications instead of compete with them.
It was an early decision, and change the way it works at this point would have a big impact in the large user base.

If the extension .exfat[PS3ISO] is used, it would be easier to tell that the file came from an exFAT file system. Nothing else.
Internally, the file only has a table of raw sectors independent of the file system (exFAT/NTFS) that is used by the plugin rawseciso.

I know that .ntfs[PS3ISO] for exFAT is misleading, but it was the file extension given to these cache files instead of .rawsec[PS3ISO].
To change that would require changes in prepISO and +60 lines of webMAN MOD. IRISMAN would be affected too.
All users would be forced to update to a new version breaking the backward compatibility. I don't want that.

In regards to the scan issue, I was reviewing the code today and it looks that I fixed the issue some time ago but I forgot that I did it.
Anyway I still prefer to have the setting /dev_ntfs turned off when I use prepISO.

 

[bguerville]

I agree with your suggestions in terms of organization and isolation of the operation.
That would have been the best decision if we did it in 2014.

However, webMAN MOD has been developed to cooperate with existing applications instead of compete with them.
It was an early decision, and change the way it works at this point would have a big impact in the large user base.

If the extension .exfat[PS3ISO] is used, it would be easier to tell that the file came from an exFAT file system. Nothing else.
Internally, the file only has a table of raw sectors independent of the file system (exFAT/NTFS) that is used by the plugin rawseciso.

I know that .ntfs[PS3ISO] for exFAT is misleading, but it was the file extension given to these cache files instead of .rawsec[PS3ISO].
To change that would require changes in prepISO and +60 lines of webMAN MOD. IRISMAN would be affected too.
All users would be forced to update to a new version breaking the backward compatibility. I don't want that.

In regards to the scan issue, I was reviewing the code today and it looks that I fixed the issue some time ago but I forgot that I did it.
Anyway I still prefer to have the setting /dev_ntfs turned off when I use prepISO.

So we do agree then & on all points really, it is reassuring somehow even though it would have been OK otherwise too.
And like I said I totally understand your reluctance to make such big changes with all the consequences it would have at this point.
Obviously there is a room for a wMM fork that would go its own way, cutting off all ties with other projects (except ps3netsrv of course), if someone was interested in working on such a project...

 

[RinMaru90]

Ive recently started having issues with PrepISO i have a 2TB HDD Partitioned as such 1.4TB NTFS MBR 32K Clusters and a FAT32 430GB 64K Clusters PrepISO used to read both but now it only reads FAT32. no idea what happened outsize my system crashing