PS VITA / PS TV Need help extracting individual files in PSARC (NOT WHOLE), see details

[bonkmaykr]

Hello. I'm the lead developer of a mod manager for WipEout 2048/HD called Firestar. I'm still cooking up the next Stable release but running into a major problem preventing me from continuing.

Up until this point, we ran mod installations by dumping the entire game and just rebuilding it with the custom assets. However, this is very inconvenient for users with slower CPUs who have to wait several minutes to decompress the assets, and the full size of the final PSARC file ends up being large and difficult to transfer.

The solution until now was to simply remove all but the DLC. Version 1.3 of Firestar introduces a patching system though, so this won't work anymore. We are trying to implement a system to pull files from larger PSARCs on a per-file basis when they are required. But the tool we use, psp2psarc (from the official Vita SDK) does not have a way to do this without sending the commands in an XML.

When I tried to do this using an XML as instructed by the PS3 hacks wiki, it complained that it didnt understand the command. (I don't have the exact error message on hand right now but it was something like that). I follow the instructions for writing the <extract> tag as described, no typos, nothing.

We tried reimplementing part of the format in our own Java and C libraries as a workaround, something we planned on doing by next year anyway to avoid licensing restrictions, but couldn't make any progress on the zlib decompression as that part of the format was poorly documented and we had never worked with compression formats like this before.

According to the PS3 hacking wiki page for psarc, I did find one open source tool that has a CLI option to list files by ID and extract a range of ID numbers, but I could not find a copy because of link rot.

This single roadblock has delayed the next Firestar update by 2+ months. I'm desperate right now and calling on any experts here for help. Is there some black magic with the XML left out of the PS3 wiki? What's a better tool/approach if any? Many thanks in advance.

 

[Charles_n_town]

I can't really help much as I've never used that app before, but zlib compression is usually designated by 78 DA in hex iirc, on both ps3 and vita as I remember vita shell being compressed by it. it's possible it contains multiple zlib compressed segments, each starting with 78 DA, I dunno. some files on the ps3 and vita are that way or it could be compressed by it as a whole. from what I remember psarc are a bit like exe files that extract a bunch of files into specific directories. I don't think they're encrypted, just compiled files with hundreds or thousands of files. @LuanTeles or @DeViL303 might have an idea on using an xml to extract specific files. I wouldn't know how to. I vaguely remember it on the ps3, since the uncharted games use psarc.

 

[bonkmaykr]

I checked with Bless and 78DA appears numerous times. If PSARC is just a lazy wrapper for standard zlib then that helps a ton. I don't think the entire file is compressed this way but rather on a per-file basis, from what I understand the algorithm specified in the header is just a default and the developer can break this default where needed. I.e. HD Fury disables compression for RIFF/MP3.

I also remembered I have a license to a partially open source Java tool from Watto called GameExtractor, it seems to support PSARC albiet extremely poorly. It's pretty clear looking at the code comments that Watto didn't know this was a PSARC file since it's made for the game Brink on idTech4 of all things.

package org.watto.ge.plugin.archive;

...

  @Override
  public Resource[] read(File path) {
    try {

      // NOTE - Compressed files MUST know their DECOMPRESSED LENGTH
      //      - Uncompressed files MUST know their LENGTH

      addFileTypes();

      ExporterPlugin exporter = Exporter_ZLib_CompressedSizeOnly.getInstance();

      // RESETTING GLOBAL VARIABLES

      FileManipulator fm = new FileManipulator(path, false);

      long arcSize = fm.getLength();

      // 4 - Header (PSAR)
      // 2 - Version Major? (1)
      // 2 - Version Minor? (4)
      // 4 - Compression Algorithm (zlib)
      // 4 - Unknown
      // 4 - Directory Entry Size (30)
      fm.skip(20);

      // 4 - Number of Files
      int numFiles = IntConverter.changeFormat(fm.readInt());
      FieldValidator.checkNumFiles(numFiles);

      // 4 - Unknown
      // 4 - Unknown (2)
      fm.skip(8);

      // Skip the first (blank) entry
      fm.skip(30);
      numFiles--;

      Resource[] resources = new Resource[numFiles];
      TaskProgressManager.setMaximum(numFiles);

      // Loop through directory
      long[] offsets = new long[numFiles];
      for (int i = 0; i < numFiles; i++) {
        // 16 - Hash?
        fm.skip(16);

        /*
        // 4 - Decompressed Length?
        int decompLength = IntConverter.changeFormat(fm.readInt());
        FieldValidator.checkLength(decompLength);

        // 4 - Compressed Length?
        int length = IntConverter.changeFormat(fm.readInt());
        FieldValidator.checkLength(length, arcSize);
        */
        fm.skip(8);

        // 2 - Unknown
        fm.skip(2);

        // 4 - File Offset
        long offset = IntConverter.unsign(IntConverter.changeFormat(fm.readInt()));
        FieldValidator.checkOffset(offset, arcSize);
        offsets[i] = offset;

        String filename = Resource.generateFilename(i);

        //path,name,offset,length,decompLength,exporter
        resources[i] = new Resource(path, filename, offset);
        resources[i].setExporter(exporter);

        TaskProgressManager.setValue(i);
      }

      Arrays.sort(offsets);

      for (int i = 0; i < numFiles; i++) {
        Resource resource = resources[i];

        long thisOffset = resource.getOffset();
        int arrayPos = Arrays.binarySearch(offsets, thisOffset);

        if (arrayPos == numFiles - 1) {
          long length = arcSize - thisOffset;
          resource.setLength(length);
          resource.setDecompressedLength(length);
        }
        else {
          long length = offsets[arrayPos + 1] - offsets[arrayPos];
          resource.setLength(length);
          resource.setDecompressedLength(length);
        }

      }

      //calculateFileSizes(resources, arcSize);

      fm.close();

      return resources;

    }
    catch (Throwable t) {
      logError(t);
      return null;
    }
  }

The implementation in GE can't read file names, since it barbarically skips over important TOC information and ignores the manifest (first file) but it works well enough to retrieve file contents, I was able to identify multiple files and the order at which GE reads them seems to be identical to the ID of the file in the PSARC meaning I can see what file it is in GE by matching it to psp2psarc.exe list. It's good enough to learn from.

 

[Charles_n_town]

I'm not sure how you'd normally decompress one of those files or if that app does it, but I've used simplyzip in the past for zlib. Problem is knowing where the compression ends, especially if there are multiple files. There has to be a way easier than simplyzip, since that would be one file at a time. I'm not sure how much LuanTeles knows about the vita or psarc, but I think he's well versed using XML files on at least the PS3.