First step to trace this would be remove all game rips so they don't get scanned, (rename GAMES, GAMEZ, PSXISO, PS2ISO, PS3ISO, PSPISO folders while testing), then if the backup manager works OK with no games have a look through your games for anything that stands out, corrupt filename, special characters in file name, tiny filesize etc.
If nothing stands out there, what I would do is just add 50% of the games back and see if freeze comes back, if you don't get any freeze you know the issue is in the other 50% so add half of those and so on, it won't take long to track it down like that...divide and conquer.
The last time I saw an issue like this, the user had renamed some games in their param.sfo adding "I <3".