I had some spare time today so I had a go at reducing the memory consumption of pkg.py
Please try this build :
https://github.com/sahlberg/pop-fe/actions/runs/8677042320
When encrypting the PKG it basically copied the file contents into a plaintet buffer and then generate a cipher buffer, all in memory.
Leading to all file data being stored three times in memory.
I rewrote it to do the processing in a temporary file instead and in tests with RE2 the peak memory consumption dropped from ~3.7G to ~1.2G when running pkg.py.
This is the change to pkg.py :
https://github.com/sahlberg/PSL1GHT/commit/9a24d3986148748c4c27c06f56bef316775ca717
I don't use the upstream PSL1GHT but rather a fork of a fork as at the time upstream did not yet have converted it to python3.
We need to check if the PSL1GHT folks want this patch.
Then I should switch to use their version instead of the fork of a fork I use currently.
EDIT:
I added another commit to
https://github.com/sahlberg/PSL1GHT/tree/use-python3
This brings down the peak memory consumption of pkg.py to ~25M when generating the RE2 pkg.
If someone wants to try to get these two patches into upstream PSL1GHT that would be awesome.