Attract-Mode Support Forum

Related Stuff => Emulators => Topic started by: progets on June 12, 2018, 11:48:46 PM

Title: Best Compression for Specific Emulators
Post by: progets on June 12, 2018, 11:48:46 PM
Just my 2 cents but a lot has changed in emulation over the past couple of years. In the old days I zipped large files to save space but this caused long load times. These days you can use compressed formats that are native to a specific emulator. This saves space and decreases load times. Below shows what I'm doing today.

PS2 - .cso (PCSX2)
PS - .chd (RetroArch mednafen_psx_libretro core)
PSP - .cso (RetroArch ppsspp_libretro core)
GameCube - .gcz (RetroArch dolphin_libretro core)
Wii - .wbfs (Dolphin)
Sega CD - .chd (RetroArch genesis_plus_gx_libretro core)
Sega Dreamcast - .chd (RetroArch redream_libretro core)
Sega Saturn - .chd (RetroArch mednafen_saturn_libretro core)
TurboGrafx-16 CD - .chd (RetroArch mednafen_pce_fast_libretro core)
Panasonic 3DO .chd (RetroArch 4do_libretro core)

If anyone is interested in this I'd be happy to discuss the tools I used to convert the games or how you feel is the best way to store large games.
Title: Re: Best Compression for Specific Emulators
Post by: akafox on June 14, 2018, 02:01:20 PM
Awesome work thank you for that progets!

You are coming from a PC side looks like..

What about a Raspberry PI?...why do you need space..well because your SD is small on a raspberry pi! ;P

Okay so here are my findings:

PSX (Psx-ReARMed)
     Crash Team Racing
ISO 605.7 MB difference of 0
IMG 605.7 MB difference of 0
BIN 605.7 MB difference of 0
CSO 421.9 MB difference of 183.8 MB
PBP 372.9 MB difference of 232.8 MB  <--- WINNER!
CHD 317.9 MB difference of 287.8 MB*

*Psx-Rearmed can not use a chd format but that is a 55 MB difference...add it across 10 games or so.well you get my point..

Mega Drive/Genesis:
         3 Ninjas Kick Back
BIN 2.1 MB  difference of 0
ZIP 917.8 KB  difference of 1232.6 KB (1.2 MB)
7z 741.9 KB  difference of 1408.5 KB (1.3 MB)

So compression is a GOOD thing for the PI on space...but you trade that for speed (thus slower load times), as stated by progets in the original post. With smaller ROMs (Atari, NES..ect. Might not make a big noticeable difference compressed on loading. And I am not even sure how much space you would get all together..but with a small SD card..yes even a 128GB..well space is important! I don't have $1,000 for a TB SD Card..I can't wait until I can afford one though..but hard drives are cheap :P

Question... know any good programs that will mass compress roms..or any ideas how to do it?
Title: Re: Best Compression for Specific Emulators
Post by: progets on June 15, 2018, 10:48:18 PM
I like the Pi and it's cheap fun but it's not the best hardware for emulation. On the Pi I store small uncompressed roms locally (SD card) and access larger roms from my media center so space isn't a Pi issue for me. I honestly don't play much on the Pi because my media center and arcade cabinet are much more capable. That being said I use Windows mainly because some emulators aren't available for any other OS and it also has the largest selection of frontends.

In terms of the Pi, you are correct in saying .pbp is the best compression option for PS1 games using Psx-Rearmed. The Pi can use .chd files for some of the other systems I mentioned.

I've been gaming for a while and have a lot of roms. If I were to start today knowing what I know now I would get all my roms from a current MAME set because it has every game with checksums to know they are correct. Now that MAME is merged with MESS a complete rom set is close to 3TB but includes just about everything that can be emulated except GC, Wii, PS2 and PS3. MAME games are already compressed and ready to use in .zip, .7z or .chd formats. You can use these games with other emulators too, not just MAME and you can get the pieces you want without having to download the whole thing. If you take this path this thread might be helpful

Like most of us, I didn't get my roms like I mention above. I have uncompressed, converted and compressed most of my files more than once over the years. Here's some simple .bat files you can create to automate the process. Create a text file, copy the lines listed below into it and rename it. Place the .bat file and the corresponding .exe into the folder with the roms you want to compress and run the bat file. The .bat file will look in subdirectories and compress anything that matches the file extention. It will leave all your existing roms untouched, you can manually delete them if you're happy with the results. FWIW - All the tools (.exe) I mention below are free.

Any file extention to .zip (I'm using 7za.exe for these)

Code: [Select]
for /r %%i in (*.nes) do 7za a -tzip "" "%%i"pce2zip.bat
Code: [Select]
for /r %%i in (*.pce) do 7za a -tzip "" "%%i"sms2zip.bat
Code: [Select]
for /r %%i in (*.sms) do 7za a -tzip "" "%%i"
Any file extention to .7z (I'm using 7za.exe for these)

Code: [Select]
for /r %%i in (*.nes) do 7za a -t7z "%%~ni.7z" "%%i"pce27z.bat
Code: [Select]
for /r %%i in (*.pce) do 7za a -t7z "%%~ni.7z" "%%i"sms27z.bat
Code: [Select]
for /r %%i in (*.sms) do 7za a -t7z "%%~ni.7z" "%%i"
Disc based systems can be converted using chdman.exe (chdman.exe is included with the MAME emulator, newer versions have the best compression). FWIW - To convert to .chd your existing audio files can't be compressed (i.e. no .mp3 audio files) they need to be .wav or possibly another uncompressed audio format.

Code: [Select]
for /r %%i in (*.cue) do chdman createcd -i "%%i" -o "%%~ni.chd
for /r %%i in (*.gdi) do chdman createcd -i "%%i" -o "%%~ni.chd

PSP and PS2 games like the .cso format and they can be converted using maxcso.exe (get the latest version for best results)

iso2cso.bat (for performance adjust the thread count for your system, since these files are large too many threads could cause unresponsiveness or overheating (i.e. adjust or remove the thread statement all together but expect for your system to be taxed to the max if you do))
Code: [Select]
for /r %%i in (*.iso) do maxcso "%%i" --threads=2 -o "%%~ni.cso

Hopefully these will be helpful to compress your rom sets. You can obviously edit them to suit your needs but this should put you on the right path.

Title: Re: Best Compression for Specific Emulators
Post by: akafox on June 16, 2018, 11:23:24 AM
I was expecting some ideas not a tutorial!  :o

Thank you!  ;D

Sadly I found msu hacks for the snes couple of days I effectively turned a few snes games into ISOs lol (2.5mb to 456mb!)
But they sound AWESOME! :D

Yeah don't get me wrong I would love for the pi to run everything..but it's not made for that..maybe in ten years? I am just trying to put as much as I can on the pi and get that off my PC I don't have a high end pc and I have a small HDD..that and well everybody wants it plugged into the TV... we'll see what the PI4 has.. I am hoping the new Raspberry PI zero (0.2?) will be built off the new pi 4 specs..yeah in two years maybe..but then I can get a stack of them

I have a USB drive I run everything from works great for me..just hate having to "haul an extra piece around"..that and they like to put it on top of the spare CRT I have. I saw a SNES build where they built the HHD into a snes cart and "stuck it in the machine" might do that.

I also want to point out that for 3DO using the phoenix emulator you can use it to compress the roms down (binar it's called). It is actually just a little bit smaller than a chd of the same rom. they are smaller than a bin or iso but chd is still smaller overall..but this emulator can't read/understand a chd
Title: Re: Best Compression for Specific Emulators
Post by: progets on June 16, 2018, 11:35:34 PM
The MSU hacks sound interesting but aren't you going in the wrong direction in terms of space? LOL

I have seen the Phoenix emulator get a lot of hype lately. I don't like the fact that it's not in English (I know there's a patch) and it doesn't support launching games from the command line. I know game launching can be done with other tools but it hasn't interested me enough to try it. Truth be told, I play MAME games 95% of the time on my cabinet. I mostly only play console systems when my brothers come over and want to relive rivalries of games from our childhood.

If you're trying to pack your SD card for a Pi a couple other good tools include:

NDSTokyoTrim - This will shrink GBA, NDS and 3DS roms by eliminating the white space at the end of the roms.

PSX2PSP - This you might already be using. It converts games to PS1 games to .pbp. Version 1.3 can do batching. The newer 1.4.2 can't do batching from what I remember since it calls every rom the same name.

Another option to increase space on Pi would be to use a USB stick (a nano one would barely stick out of the Pi), this could double or triple your space for under $50. Since this would be an additional drive you could even use a file system that natively supports compression (the ext4 partition (the large partition of RetroPi) doesn't natively support compress but there are 3rd party tools that could also be used there).

NUC devices are a great alternative to the Pi without the limitations. Of course they cost more but they are small and portable PCs (4.5" x 4.5" x 2") and you can usually pickup a used one for pretty cheap. Once you add up the cost of a Pi, power supply, SD card, case and USB hard drive you probably have over $100 invested. A used NUC can hold a 2TB+ drive and if you find a good deal a used would cost less than $200. I know that everyone can't afford a NUC and that's understandable. I don't understand people that spend $500+ building a cabinet and then put a $40 Pi in it.

I guess I should also mention that the SD card bus on a Pi is very slow to be compatible with most SD cards. You can overclock this aspect and you will see a significant performance gain. This is risky and you would need to use a top notch SD card for it to work (like a high end Samsung or Sandisk). Any SD card not up to the task will get corrupted on the first boot so make sure you make a backup before trying. Adding "dtparam=sd_overclock=100" to the config.txt will double the speed of the SD card bus on the Pi.

Title: Re: Best Compression for Specific Emulators
Post by: akafox on July 21, 2018, 04:03:55 PM
Well I need to figure out a linux equivalent of that command line..all it did for me was compress it into one big archive  ::) :P

so I made a bash script just to test it out though..why..because I am curious that's why  :P ;D findings

SNES roms randomly chosen:

Uncompressed              .zip                              .rar                         .7z
Count: 103                  Count: 95*                  Count: 96*              Count: 103
Size: 137.7Mb              Size:68.5Mb                 Size: 65.5               Size: 62.1Mb
Saved: N/A                   Saved: 69.2Mb             Saved: 72.2Mb        Saved: 75.6Mb

*Yes some files were missing...something to do with my script I am sure. Not going for accuracy here just information

So 7z is the "best" compression and with large sets a little can add up. However I think the space saved is not going to make a big difference. But it's good to know if you have a smaller SD card. "A couple of gigs" might be a big thing. You can not deny though that compressing iso/bin files to chd is a good idea!

As larger SD cards become affordable then space will not least until we fill them up with more and larger know.."because we have room"  ;)

But as the PI improves in hardware and capability so will the number of games that can be will we ever "get ahead"? *lol*
Title: Re: Best Compression for Specific Emulators
Post by: firewater on November 05, 2018, 08:33:34 AM
I'm replying you here so that more people see this great post. Thank you SO much for this tip! I had no idea any type of lossless/almost lossless compression existed at such level. I've almost halved the size of my PlayStation collection, and I'm looking forward to doing the same for other systems, in a format that is also "cleaner" (at least in terms of being a single file).