Best Compression for Specific Emulators

[progets]

Just my 2 cents but a lot has changed in emulation over the past couple of years. In the old days I zipped large files to save space but this caused long load times. These days you can use compressed formats that are native to a specific emulator. This saves space and decreases load times. Below shows what I'm doing today.

PS2 - .cso (PCSX2)
PS - .chd (RetroArch mednafen_psx_libretro core)
PSP - .cso (RetroArch ppsspp_libretro core)
GameCube - .gcz (RetroArch dolphin_libretro core)
Wii - .wbfs (Dolphin)
Sega CD - .chd (RetroArch genesis_plus_gx_libretro core)
Sega Dreamcast - .chd (RetroArch redream_libretro core)
Sega Saturn - .chd (RetroArch mednafen_saturn_libretro core)
TurboGrafx-16 CD - .chd (RetroArch mednafen_pce_fast_libretro core)
Panasonic 3DO .chd (RetroArch 4do_libretro core)

If anyone is interested in this I'd be happy to discuss the tools I used to convert the games or how you feel is the best way to store large games.

[akafox]

Awesome work thank you for that progets!

You are coming from a PC side looks like..

What about a Raspberry PI?...why do you need space..well because your SD is small on a raspberry pi! ;P

Okay so here are my findings:

PSX (Psx-ReARMed)
     Crash Team Racing
ISO 605.7 MB difference of 0
IMG 605.7 MB difference of 0
BIN 605.7 MB difference of 0
CSO 421.9 MB difference of 183.8 MB
PBP 372.9 MB difference of 232.8 MB 
CHD 317.9 MB difference of 287.8 MB <--- WINNER!*

*Psx-Rearmed can now use a chd format use it!
*Psx-Rearmed can not use a chd format but that is a 55 MB difference...add it across 10 games or so.well you get my point..

Mega Drive/Genesis:
         3 Ninjas Kick Back
BIN 2.1 MB  difference of 0
ZIP 917.8 KB  difference of 1232.6 KB (1.2 MB)
7z 741.9 KB  difference of 1408.5 KB (1.3 MB)

So compression is a GOOD thing for the PI on space...but you trade that for speed (thus slower load times), as stated by progets in the original post. With smaller ROMs (Atari, NES..ect. Might not make a big noticeable difference compressed on loading. And I am not even sure how much space you would get all together..but with a small SD card..yes even a 128GB..well space is important! I don't have $1,000 for a TB SD Card..I can't wait until I can afford one though..but hard drives are cheap

Question... know any good programs that will mass compress roms..or any ideas how to do it?

[progets]

I like the Pi and it's cheap fun but it's not the best hardware for emulation. On the Pi I store small uncompressed roms locally (SD card) and access larger roms from my media center so space isn't a Pi issue for me. I honestly don't play much on the Pi because my media center and arcade cabinet are much more capable. That being said I use Windows mainly because some emulators aren't available for any other OS and it also has the largest selection of frontends.

In terms of the Pi, you are correct in saying .pbp is the best compression option for PS1 games using Psx-Rearmed. The Pi can use .chd files for some of the other systems I mentioned.

I've been gaming for a while and have a lot of roms. If I were to start today knowing what I know now I would get all my roms from a current MAME set because it has every game with checksums to know they are correct. Now that MAME is merged with MESS a complete rom set is close to 3TB but includes just about everything that can be emulated except GC, Wii, PS2 and PS3. MAME games are already compressed and ready to use in .zip, .7z or .chd formats. You can use these games with other emulators too, not just MAME and you can get the pieces you want without having to download the whole thing. If you take this path this thread might be helpful http://forum.attractmode.org/index.php?topic=2307.msg15606#msg15606.

Like most of us, I didn't get my roms like I mention above. I have uncompressed, converted and compressed most of my files more than once over the years. Here's some simple .bat files you can create to automate the process. Create a text file, copy the lines listed below into it and rename it. Place the .bat file and the corresponding .exe into the folder with the roms you want to compress and run the bat file. The .bat file will look in subdirectories and compress anything that matches the file extention. It will leave all your existing roms untouched, you can manually delete them if you're happy with the results. FWIW - All the tools (.exe) I mention below are free.


Any file extention to .zip (I'm using 7za.exe for these)

nes2zip.bat

Code: [Select]

for /r %%i in (*.nes) do 7za a -tzip "%%~ni.zip" "%%i"pce2zip.bat

Code: [Select]

for /r %%i in (*.pce) do 7za a -tzip "%%~ni.zip" "%%i"sms2zip.bat

Code: [Select]

for /r %%i in (*.sms) do 7za a -tzip "%%~ni.zip" "%%i"
Any file extention to .7z (I'm using 7za.exe for these)

nes27z.bat

Code: [Select]

for /r %%i in (*.nes) do 7za a -t7z "%%~ni.7z" "%%i"pce27z.bat

Code: [Select]

for /r %%i in (*.pce) do 7za a -t7z "%%~ni.7z" "%%i"sms27z.bat

Code: [Select]

for /r %%i in (*.sms) do 7za a -t7z "%%~ni.7z" "%%i"
Disc based systems can be converted using chdman.exe (chdman.exe is included with the MAME emulator, newer versions have the best compression). FWIW - To convert to .chd your existing audio files can't be compressed (i.e. no .mp3 audio files) they need to be .wav or possibly another uncompressed audio format.

convert2chd.bat

Code: [Select]

for /r %%i in (*.cue) do chdman createcd -i "%%i" -o "%%~ni.chd
for /r %%i in (*.gdi) do chdman createcd -i "%%i" -o "%%~ni.chd

PSP and PS2 games like the .cso format and they can be converted using maxcso.exe (get the latest version for best results)

iso2cso.bat (for performance adjust the thread count for your system, since these files are large too many threads could cause unresponsiveness or overheating (i.e. adjust or remove the thread statement all together but expect for your system to be taxed to the max if you do))

Code: [Select]

for /r %%i in (*.iso) do maxcso "%%i" --threads=2 -o "%%~ni.cso

Hopefully these will be helpful to compress your rom sets. You can obviously edit them to suit your needs but this should put you on the right path.

[akafox]

I was expecting some ideas not a tutorial! 

Thank you! 

Sadly I found msu hacks for the snes couple of days ago..so I effectively turned a few snes games into ISOs lol (2.5mb to 456mb!)
But they sound AWESOME!

Yeah don't get me wrong I would love for the pi to run everything..but it's not made for that..maybe in ten years? I am just trying to put as much as I can on the pi and get that off my PC I don't have a high end pc and I have a small HDD..that and well everybody wants it plugged into the TV... we'll see what the PI4 has.. I am hoping the new Raspberry PI zero (0.2?) will be built off the new pi 4 specs..yeah in two years maybe..but then I can get a stack of them

I have a USB drive I run everything from works great for me..just hate having to "haul an extra piece around"..that and they like to put it on top of the spare CRT I have. I saw a SNES build where they built the HHD into a snes cart and "stuck it in the machine" might do that.

I also want to point out that for 3DO using the phoenix emulator you can use it to compress the roms down (binar it's called). It is actually just a little bit smaller than a chd of the same rom. they are smaller than a bin or iso but chd is still smaller overall..but this emulator can't read/understand a chd

[progets]

The MSU hacks sound interesting but aren't you going in the wrong direction in terms of space? LOL

I have seen the Phoenix emulator get a lot of hype lately. I don't like the fact that it's not in English (I know there's a patch) and it doesn't support launching games from the command line. I know game launching can be done with other tools but it hasn't interested me enough to try it. Truth be told, I play MAME games 95% of the time on my cabinet. I mostly only play console systems when my brothers come over and want to relive rivalries of games from our childhood.


If you're trying to pack your SD card for a Pi a couple other good tools include:

NDSTokyoTrim - This will shrink GBA, NDS and 3DS roms by eliminating the white space at the end of the roms.

PSX2PSP - This you might already be using. It converts games to PS1 games to .pbp. Version 1.3 can do batching. The newer 1.4.2 can't do batching from what I remember since it calls every rom the same name.


Another option to increase space on Pi would be to use a USB stick (a nano one would barely stick out of the Pi), this could double or triple your space for under $50. Since this would be an additional drive you could even use a file system that natively supports compression (the ext4 partition (the large partition of RetroPi) doesn't natively support compress but there are 3rd party tools that could also be used there).

NUC devices are a great alternative to the Pi without the limitations. Of course they cost more but they are small and portable PCs (4.5" x 4.5" x 2") and you can usually pickup a used one for pretty cheap. Once you add up the cost of a Pi, power supply, SD card, case and USB hard drive you probably have over $100 invested. A used NUC can hold a 2TB+ drive and if you find a good deal a used would cost less than $200. I know that everyone can't afford a NUC and that's understandable. I don't understand people that spend $500+ building a cabinet and then put a $40 Pi in it.

I guess I should also mention that the SD card bus on a Pi is very slow to be compatible with most SD cards. You can overclock this aspect and you will see a significant performance gain. This is risky and you would need to use a top notch SD card for it to work (like a high end Samsung or Sandisk). Any SD card not up to the task will get corrupted on the first boot so make sure you make a backup before trying. Adding "dtparam=sd_overclock=100" to the config.txt will double the speed of the SD card bus on the Pi.

[akafox]

Well I need to figure out a linux equivalent of that command line..all it did for me was compress it into one big archive 

so I made a bash script just to test it out though..why..because I am curious that's why 

soooo..my findings

SNES roms randomly chosen:

Uncompressed              .zip                              .rar                         .7z
Count: 103                  Count: 95*                  Count: 96*              Count: 103
Size: 137.7Mb              Size:68.5Mb                 Size: 65.5               Size: 62.1Mb
Saved: N/A                   Saved: 69.2Mb             Saved: 72.2Mb        Saved: 75.6Mb

*Yes some files were missing...something to do with my script I am sure. Not going for accuracy here just information

So 7z is the "best" compression and with large sets a little can add up. However I think the space saved is not going to make a big difference. But it's good to know if you have a smaller SD card. "A couple of gigs" might be a big thing. You can not deny though that compressing iso/bin files to chd is a good idea!

As larger SD cards become affordable then space will not matter..at least until we fill them up with more and larger files..you know.."because we have room" 

But as the PI improves in hardware and capability so will the number of games that can be emulated..so will we ever "get ahead"? *lol*

[firewater]

I'm replying you here so that more people see this great post. Thank you SO much for this tip! I had no idea any type of lossless/almost lossless compression existed at such level. I've almost halved the size of my PlayStation collection, and I'm looking forward to doing the same for other systems, in a format that is also "cleaner" (at least in terms of being a single file).

[rand0m]

Reicast core in retroarch now supports chd v5 (so does Demul and Redream)

[MikeFromPA]

Any file extention to .zip (I'm using 7za.exe for these)

nes2zip.bat

Code: [Select]

for /r %%i in (*.nes) do 7za a -tzip "%%~ni.zip" "%%i"pce2zip.bat

Code: [Select]

for /r %%i in (*.pce) do 7za a -tzip "%%~ni.zip" "%%i"sms2zip.bat

Code: [Select]

for /r %%i in (*.sms) do 7za a -tzip "%%~ni.zip" "%%i"
Any file extention to .7z (I'm using 7za.exe for these)

nes27z.bat

Code: [Select]

for /r %%i in (*.nes) do 7za a -t7z "%%~ni.7z" "%%i"pce27z.bat

Code: [Select]

for /r %%i in (*.pce) do 7za a -t7z "%%~ni.7z" "%%i"sms27z.bat

Code: [Select]

for /r %%i in (*.sms) do 7za a -t7z "%%~ni.7z" "%%i"convert2chd.bat

Code: [Select]

for /r %%i in (*.cue) do chdman createcd -i "%%i" -o "%%~ni.chd
for /r %%i in (*.gdi) do chdman createcd -i "%%i" -o "%%~ni.chd
iso2cso.bat (for performance adjust the thread count for your system, since these files are large too many threads could cause unresponsiveness or overheating (i.e. adjust or remove the thread statement all together but expect for your system to be taxed to the max if you do))

Code: [Select]

for /r %%i in (*.iso) do maxcso "%%i" --threads=2 -o "%%~ni.cso

@progets Sorry to revive a dead post, but this batch script reminded me of one I saw maybe 10 years ago, and could never recreate.  Instead of recompressing the files, it used 7zip to check the compressed files for errors based on the errorlevel, then would delete the corrupted files and log the filenames into a text file of what all was deleted.

Any chance you could throw this old dog a bone and modify one of these scripts for me to do the same?

[MikeFromPA]

So I've been thinking *ALOT* about this thread, because I feel making sure you have good compressed files go hand in hand with converting their compression.

After almost a full day of searching, I've found the batch file I mentioned above, which uses 7zip to verify the file has no compression errors.  Was sitting, waiting for me on an external hd I had in storage.

cls
FOR /R %%A IN (*.zip) DO (
7za t "%%A" | FIND "Everything is Ok" || (DEL "%%A" & ECHO.%%A BAD >> error.log)
)

And this is what the error.log looks like.

G:\eXoDOS\Games\Lemonade Stand (1999).zip BAD
G:\eXoDOS\Games\Lorna (1990).zip BAD
G:\eXoDOS\Games\MicroLink Shut the Box (1989).zip BAD


Now, if someone would like to explain HOW  FOR /R %%A IN (*.zip) DO (

cycles through every file in the subdirectory, errr, folder, I'd love to know!  ( =

[progets]

Now, if someone would like to explain HOW  FOR /R %%A IN (*.zip) DO (
cycles through every file in the subdirectory, errr, folder, I'd love to know!  ( =

for = command
/r = recursive search (i.e. all subfolders)
%%a = variable that equals each file name
in = is used to specify the set of files you want to scan (*.zip in this case)
do = command/action to take on the files

[MikeFromPA]

Now, if someone would like to explain HOW  FOR /R %%A IN (*.zip) DO (
cycles through every file in the subdirectory, errr, folder, I'd love to know!  ( =

for = command
/r = recursive search (i.e. all subfolders)
%%a = variable that equals each file name
in = is used to specify the set of files you want to scan (*.zip in this case)
do = command/action to take on the files

It's nice not understanding how it works doesn't stop it from working. 😁

[tgmagician]

Hi progets, thanks for all the info you have provided along with everyone else.

I know this is an older post but I am trying to convert my Saturn games into chd's.

I have tried a few thing but without any luck. What tools did you use and what is the process?

My saturn games are 7z archived Cue, Bin files. I can extract them and get them into a chd by chdman.exe but when I try to play alot of them it just brings up a Music player in the Emu. I am not sure if I have the correct parameters for chdman or if there is a different one that I should be using.

Thanks for any help you or anyone else can give me.

[progets]

Hi progets, thanks for all the info you have provided along with everyone else.

I know this is an older post but I am trying to convert my Saturn games into chd's.

I have tried a few thing but without any luck. What tools did you use and what is the process?

My saturn games are 7z archived Cue, Bin files. I can extract them and get them into a chd by chdman.exe but when I try to play alot of them it just brings up a Music player in the Emu. I am not sure if I have the correct parameters for chdman or if there is a different one that I should be using.

Thanks for any help you or anyone else can give me.

I recommend using the latest chdman.exe which is include with every official MAME release https://www.mamedev.org/release.html. You don't need to use MAME or even get it to function, just grab the chdman.exe from it and you can delete it.

Once you have your Saturn files extracted from the 7z format, place chdman.exe into the same folder and also create "convert2chd.bat" in this folder with these contents

Code: [Select]

for /r %%i in (*.cue) do chdman createcd -i "%%i" -o "%%~ni.chd
for /r %%i in (*.gdi) do chdman createcd -i "%%i" -o "%%~ni.chd

Now you can run convert2chd.bat from the command line and if there are any errors you'll see them.

Common issues:
- the .cue file doesn't contain the correct path or name to your .bin file(s)
- you have compressed audio files (like .mp3) which aren't compatible with chdman.exe

[tgmagician]

Thank you so much for the info, I will give that a try!