How did s#ny, with the new non-jailbrakeble consoles, distinguish between legitimate software signed with the old key and the homebrews signed with the same compromised old key ? How did they make it secure without breaking backwards compatibility?
Am gonna try to keep this explanation as clear and technically lightweight as possible, to do so I may deliberately oversimplify things a little at times.
They revoked some app/game IDs afaik but that's about it, they pretty much gave up on existing console models (< fw 3.60) as there was nothing much they could really do, short of a full recall, instead they decided to slightly modify the chain of trust to focus on securing the new PS3 models (3xxx and 4xxx) coming out of their factories.
The new chain of trust remains unbroken to this day, partly because its implementation is more solid, but maybe also because the best hackers who worked on PS3 to date had already moved on to other pastures by the time it was released, in the end, I don't think anyone took up the challenge to defeat the new system with a view to release CFWs for metldr.2 models, keeping in mind also that S#ny didn't appreciate being ridiculed publicly (especially when their supposedly unbreakable ecdsa signature algorithm implementation was shown to be using a constant number rather than a random number to generate signatures and that led to the derivation of their private keys) and was being more than "a little growly" with hackers at the time.
Of course, there's always the possibility that private exploits have been achieved but never shared for a number of reasons.
The improved chain of trust introduces metldr.2 which is in charge of verifying the metadata of system binaries loaded after lv0ldr and lv0, while lv0ldr itself is authenticated using data stored in the cellBE processor, in a location purposefully inaccessible by design, that process is hardware based and not likely to be vulnerable, once lv0 is loaded, the loaders are immediately available as they're embedded into lv0 (appldr, isoldr, rvkldr, lv1ldr, lv2ldr...).
On older consoles made before 3.60, metldr.2 is never used, which is why when you lose jailbreak because you installed OFW, you can simply patch the Flash Memory from OFW using (modded) re-signed CoreOS files and automatically regain jailbreak allowing you to install a CFW.
The re-signing of those patched CoreOS system files is done using old keys, yet it works as you can see, that illustrates for you how the OS does not really differentiate between old and new keys when the old metldr is used.
On newer consoles (minimum version 3.60+), metldr.2 is ALWAYS used, therefore the lower level system files used in boot such as lv0, lv1, lv2 etc.. won't work when re-signed with old keys, that's why we cannot install any CFW even with a kernel (lv2) exploit like HEN or an hypervisor (lv1) exploit. Once HEN is enabled, it patches kernel functions so that the system will accept to run binaries re-signed with old keys but at boot HEN is not enabled so ONLY officially signed OFW binaries can be loaded.
In order to get CFW on 3xxx/4xxx models, we would need to find an exploitable vulnerability in the lower level system files such as lv0ldr or lv0.
It's likely that such vulnerabilities exist, and they could be triggered through specific patches in the Flash Memory (or possibly syscon) where some data those files need to process is stored.
Low level system files often use such data assuming that it's not been tampered with which is a bad assumption to make whenever the data is accessible in read/write mode from userland.
Imagine for example a low level system file processing something like say the revocation list data which keeps track of any revoked file, that data is stored in the Flash Memory (nor or nand/eMMC) and the size of the revocation list data is also stored there. The system file reads the size of the list first then it copies it to the isolated SPU memory for processing.
Now let's say that with a userland exploit, you modify the value stored as size of the revocation list in the Flash Memory in such a way that it would overflow the isolated SPU memory buffer capacity, the system file trusts the data stored in Flash memory implicitly and does not bother to check the validity of the size value it reads there, it starts copying Flash memory data, it exceeds the buffer area, it continues copying and finally begins to overwrite the low level executable code itself, if done properly with a precisely controlled overflow, you can take control of code execution of the isolated SPU this way, from there you could do a number of cool things such as leaking data or ensuring that a revoked file can run etc...
In practice, take the eid_root_key, it can be dumped with the ERK Dumper, that hack follows a very similar method.
