Attract-Mode Support Forum

Related Stuff => Emulators => Topic started by: progets on June 12, 2018, 11:48:46 PM

Title: Best Compression for Specific Emulators
Post by: progets on June 12, 2018, 11:48:46 PM
Just my 2 cents but a lot has changed in emulation over the past couple of years. In the old days I zipped large files to save space but this caused long load times. These days you can use compressed formats that are native to a specific emulator. This saves space and decreases load times. Below shows what I'm doing today.

PS2 - .cso (PCSX2)
PS - .chd (RetroArch mednafen_psx_libretro core)
PSP - .cso (RetroArch ppsspp_libretro core)
GameCube - .gcz (RetroArch dolphin_libretro core)
Wii - .wbfs (Dolphin)
Sega CD - .chd (RetroArch genesis_plus_gx_libretro core)
Sega Dreamcast - .chd (RetroArch redream_libretro core)
Sega Saturn - .chd (RetroArch mednafen_saturn_libretro core)
TurboGrafx-16 CD - .chd (RetroArch mednafen_pce_fast_libretro core)
Panasonic 3DO .chd (RetroArch 4do_libretro core)

If anyone is interested in this I'd be happy to discuss the tools I used to convert the games or how you feel is the best way to store large games.
Title: Re: Best Compression for Specific Emulators
Post by: akafox on June 14, 2018, 02:01:20 PM
Awesome work thank you for that progets!

You are coming from a PC side looks like..

What about a Raspberry PI?...why do you need space..well because your SD is small on a raspberry pi! ;P

Okay so here are my findings:

PSX (Psx-ReARMed)
     Crash Team Racing
ISO 605.7 MB difference of 0
IMG 605.7 MB difference of 0
BIN 605.7 MB difference of 0
CSO 421.9 MB difference of 183.8 MB
PBP 372.9 MB difference of 232.8 MB 
CHD 317.9 MB difference of 287.8 MB <--- WINNER!*

*Psx-Rearmed can now use a chd format use it!
*Psx-Rearmed can not use a chd format but that is a 55 MB difference...add it across 10 games or so.well you get my point..

Mega Drive/Genesis:
         3 Ninjas Kick Back
BIN 2.1 MB  difference of 0
ZIP 917.8 KB  difference of 1232.6 KB (1.2 MB)
7z 741.9 KB  difference of 1408.5 KB (1.3 MB)

So compression is a GOOD thing for the PI on space...but you trade that for speed (thus slower load times), as stated by progets in the original post. With smaller ROMs (Atari, NES..ect. Might not make a big noticeable difference compressed on loading. And I am not even sure how much space you would get all together..but with a small SD card..yes even a 128GB..well space is important! I don't have $1,000 for a TB SD Card..I can't wait until I can afford one though..but hard drives are cheap :P

Question... know any good programs that will mass compress roms..or any ideas how to do it?
Title: Re: Best Compression for Specific Emulators
Post by: progets on June 15, 2018, 10:48:18 PM
I like the Pi and it's cheap fun but it's not the best hardware for emulation. On the Pi I store small uncompressed roms locally (SD card) and access larger roms from my media center so space isn't a Pi issue for me. I honestly don't play much on the Pi because my media center and arcade cabinet are much more capable. That being said I use Windows mainly because some emulators aren't available for any other OS and it also has the largest selection of frontends.

In terms of the Pi, you are correct in saying .pbp is the best compression option for PS1 games using Psx-Rearmed. The Pi can use .chd files for some of the other systems I mentioned.

I've been gaming for a while and have a lot of roms. If I were to start today knowing what I know now I would get all my roms from a current MAME set because it has every game with checksums to know they are correct. Now that MAME is merged with MESS a complete rom set is close to 3TB but includes just about everything that can be emulated except GC, Wii, PS2 and PS3. MAME games are already compressed and ready to use in .zip, .7z or .chd formats. You can use these games with other emulators too, not just MAME and you can get the pieces you want without having to download the whole thing. If you take this path this thread might be helpful http://forum.attractmode.org/index.php?topic=2307.msg15606#msg15606.

Like most of us, I didn't get my roms like I mention above. I have uncompressed, converted and compressed most of my files more than once over the years. Here's some simple .bat files you can create to automate the process. Create a text file, copy the lines listed below into it and rename it. Place the .bat file and the corresponding .exe into the folder with the roms you want to compress and run the bat file. The .bat file will look in subdirectories and compress anything that matches the file extention. It will leave all your existing roms untouched, you can manually delete them if you're happy with the results. FWIW - All the tools (.exe) I mention below are free.


Any file extention to .zip (I'm using 7za.exe for these)

nes2zip.bat
Code: [Select]
for /r %%i in (*.nes) do 7za a -tzip "%%~ni.zip" "%%i"pce2zip.bat
Code: [Select]
for /r %%i in (*.pce) do 7za a -tzip "%%~ni.zip" "%%i"sms2zip.bat
Code: [Select]
for /r %%i in (*.sms) do 7za a -tzip "%%~ni.zip" "%%i"
Any file extention to .7z (I'm using 7za.exe for these)

nes27z.bat
Code: [Select]
for /r %%i in (*.nes) do 7za a -t7z "%%~ni.7z" "%%i"pce27z.bat
Code: [Select]
for /r %%i in (*.pce) do 7za a -t7z "%%~ni.7z" "%%i"sms27z.bat
Code: [Select]
for /r %%i in (*.sms) do 7za a -t7z "%%~ni.7z" "%%i"
Disc based systems can be converted using chdman.exe (chdman.exe is included with the MAME emulator, newer versions have the best compression). FWIW - To convert to .chd your existing audio files can't be compressed (i.e. no .mp3 audio files) they need to be .wav or possibly another uncompressed audio format.

convert2chd.bat
Code: [Select]
for /r %%i in (*.cue) do chdman createcd -i "%%i" -o "%%~ni.chd
for /r %%i in (*.gdi) do chdman createcd -i "%%i" -o "%%~ni.chd

PSP and PS2 games like the .cso format and they can be converted using maxcso.exe (get the latest version for best results)

iso2cso.bat (for performance adjust the thread count for your system, since these files are large too many threads could cause unresponsiveness or overheating (i.e. adjust or remove the thread statement all together but expect for your system to be taxed to the max if you do))
Code: [Select]
for /r %%i in (*.iso) do maxcso "%%i" --threads=2 -o "%%~ni.cso

Hopefully these will be helpful to compress your rom sets. You can obviously edit them to suit your needs but this should put you on the right path.




Title: Re: Best Compression for Specific Emulators
Post by: akafox on June 16, 2018, 11:23:24 AM
I was expecting some ideas not a tutorial!  :o

Thank you!  ;D

Sadly I found msu hacks for the snes couple of days ago..so I effectively turned a few snes games into ISOs lol (2.5mb to 456mb!)
But they sound AWESOME! :D

Yeah don't get me wrong I would love for the pi to run everything..but it's not made for that..maybe in ten years? I am just trying to put as much as I can on the pi and get that off my PC I don't have a high end pc and I have a small HDD..that and well everybody wants it plugged into the TV... we'll see what the PI4 has.. I am hoping the new Raspberry PI zero (0.2?) will be built off the new pi 4 specs..yeah in two years maybe..but then I can get a stack of them

I have a USB drive I run everything from works great for me..just hate having to "haul an extra piece around"..that and they like to put it on top of the spare CRT I have. I saw a SNES build where they built the HHD into a snes cart and "stuck it in the machine" might do that.

I also want to point out that for 3DO using the phoenix emulator you can use it to compress the roms down (binar it's called). It is actually just a little bit smaller than a chd of the same rom. they are smaller than a bin or iso but chd is still smaller overall..but this emulator can't read/understand a chd
Title: Re: Best Compression for Specific Emulators
Post by: progets on June 16, 2018, 11:35:34 PM
The MSU hacks sound interesting but aren't you going in the wrong direction in terms of space? LOL

I have seen the Phoenix emulator get a lot of hype lately. I don't like the fact that it's not in English (I know there's a patch) and it doesn't support launching games from the command line. I know game launching can be done with other tools but it hasn't interested me enough to try it. Truth be told, I play MAME games 95% of the time on my cabinet. I mostly only play console systems when my brothers come over and want to relive rivalries of games from our childhood.


If you're trying to pack your SD card for a Pi a couple other good tools include:

NDSTokyoTrim - This will shrink GBA, NDS and 3DS roms by eliminating the white space at the end of the roms.

PSX2PSP - This you might already be using. It converts games to PS1 games to .pbp. Version 1.3 can do batching. The newer 1.4.2 can't do batching from what I remember since it calls every rom the same name.


Another option to increase space on Pi would be to use a USB stick (a nano one would barely stick out of the Pi), this could double or triple your space for under $50. Since this would be an additional drive you could even use a file system that natively supports compression (the ext4 partition (the large partition of RetroPi) doesn't natively support compress but there are 3rd party tools that could also be used there).

NUC devices are a great alternative to the Pi without the limitations. Of course they cost more but they are small and portable PCs (4.5" x 4.5" x 2") and you can usually pickup a used one for pretty cheap. Once you add up the cost of a Pi, power supply, SD card, case and USB hard drive you probably have over $100 invested. A used NUC can hold a 2TB+ drive and if you find a good deal a used would cost less than $200. I know that everyone can't afford a NUC and that's understandable. I don't understand people that spend $500+ building a cabinet and then put a $40 Pi in it.

I guess I should also mention that the SD card bus on a Pi is very slow to be compatible with most SD cards. You can overclock this aspect and you will see a significant performance gain. This is risky and you would need to use a top notch SD card for it to work (like a high end Samsung or Sandisk). Any SD card not up to the task will get corrupted on the first boot so make sure you make a backup before trying. Adding "dtparam=sd_overclock=100" to the config.txt will double the speed of the SD card bus on the Pi.

Title: Re: Best Compression for Specific Emulators
Post by: akafox on July 21, 2018, 04:03:55 PM
Well I need to figure out a linux equivalent of that command line..all it did for me was compress it into one big archive  ::) :P

so I made a bash script just to test it out though..why..because I am curious that's why  :P ;D

soooo..my findings

SNES roms randomly chosen:

Uncompressed              .zip                              .rar                         .7z
Count: 103                  Count: 95*                  Count: 96*              Count: 103
Size: 137.7Mb              Size:68.5Mb                 Size: 65.5               Size: 62.1Mb
Saved: N/A                   Saved: 69.2Mb             Saved: 72.2Mb        Saved: 75.6Mb

*Yes some files were missing...something to do with my script I am sure. Not going for accuracy here just information

So 7z is the "best" compression and with large sets a little can add up. However I think the space saved is not going to make a big difference. But it's good to know if you have a smaller SD card. "A couple of gigs" might be a big thing. You can not deny though that compressing iso/bin files to chd is a good idea!

As larger SD cards become affordable then space will not matter..at least until we fill them up with more and larger files..you know.."because we have room"  ;)

But as the PI improves in hardware and capability so will the number of games that can be emulated..so will we ever "get ahead"? *lol*
Title: Re: Best Compression for Specific Emulators
Post by: firewater on November 05, 2018, 08:33:34 AM
I'm replying you here so that more people see this great post. Thank you SO much for this tip! I had no idea any type of lossless/almost lossless compression existed at such level. I've almost halved the size of my PlayStation collection, and I'm looking forward to doing the same for other systems, in a format that is also "cleaner" (at least in terms of being a single file).
Title: Re: Best Compression for Specific Emulators
Post by: rand0m on November 27, 2018, 10:36:46 AM
Reicast core in retroarch now supports chd v5 (so does Demul and Redream)
Title: Re: Best Compression for Specific Emulators
Post by: MikeFromPA on January 04, 2019, 02:13:39 AM
Any file extention to .zip (I'm using 7za.exe for these)

nes2zip.bat
Code: [Select]
for /r %%i in (*.nes) do 7za a -tzip "%%~ni.zip" "%%i"pce2zip.bat
Code: [Select]
for /r %%i in (*.pce) do 7za a -tzip "%%~ni.zip" "%%i"sms2zip.bat
Code: [Select]
for /r %%i in (*.sms) do 7za a -tzip "%%~ni.zip" "%%i"
Any file extention to .7z (I'm using 7za.exe for these)

nes27z.bat
Code: [Select]
for /r %%i in (*.nes) do 7za a -t7z "%%~ni.7z" "%%i"pce27z.bat
Code: [Select]
for /r %%i in (*.pce) do 7za a -t7z "%%~ni.7z" "%%i"sms27z.bat
Code: [Select]
for /r %%i in (*.sms) do 7za a -t7z "%%~ni.7z" "%%i"convert2chd.bat
Code: [Select]
for /r %%i in (*.cue) do chdman createcd -i "%%i" -o "%%~ni.chd
for /r %%i in (*.gdi) do chdman createcd -i "%%i" -o "%%~ni.chd
iso2cso.bat (for performance adjust the thread count for your system, since these files are large too many threads could cause unresponsiveness or overheating (i.e. adjust or remove the thread statement all together but expect for your system to be taxed to the max if you do))
Code: [Select]
for /r %%i in (*.iso) do maxcso "%%i" --threads=2 -o "%%~ni.cso

@progets Sorry to revive a dead post, but this batch script reminded me of one I saw maybe 10 years ago, and could never recreate.  Instead of recompressing the files, it used 7zip to check the compressed files for errors based on the errorlevel, then would delete the corrupted files and log the filenames into a text file of what all was deleted.

Any chance you could throw this old dog a bone and modify one of these scripts for me to do the same?
Title: Re: Best Compression for Specific Emulators
Post by: MikeFromPA on January 05, 2019, 07:42:32 AM
So I've been thinking *ALOT* about this thread, because I feel making sure you have good compressed files go hand in hand with converting their compression.

After almost a full day of searching, I've found the batch file I mentioned above, which uses 7zip to verify the file has no compression errors.  Was sitting, waiting for me on an external hd I had in storage.

cls
FOR /R %%A IN (*.zip) DO (
7za t "%%A" | FIND "Everything is Ok" || (DEL "%%A" & ECHO.%%A BAD >> error.log)
)

And this is what the error.log looks like.

G:\eXoDOS\Games\Lemonade Stand (1999).zip BAD
G:\eXoDOS\Games\Lorna (1990).zip BAD
G:\eXoDOS\Games\MicroLink Shut the Box (1989).zip BAD


Now, if someone would like to explain HOW  FOR /R %%A IN (*.zip) DO (

cycles through every file in the subdirectory, errr, folder, I'd love to know!  ( =
Title: Re: Best Compression for Specific Emulators
Post by: progets on January 05, 2019, 08:38:32 PM
Now, if someone would like to explain HOW  FOR /R %%A IN (*.zip) DO (
cycles through every file in the subdirectory, errr, folder, I'd love to know!  ( =

for = command
/r = recursive search (i.e. all subfolders)
%%a = variable that equals each file name
in = is used to specify the set of files you want to scan (*.zip in this case)
do = command/action to take on the files
Title: Re: Best Compression for Specific Emulators
Post by: MikeFromPA on January 07, 2019, 03:06:20 PM
Now, if someone would like to explain HOW  FOR /R %%A IN (*.zip) DO (
cycles through every file in the subdirectory, errr, folder, I'd love to know!  ( =

for = command
/r = recursive search (i.e. all subfolders)
%%a = variable that equals each file name
in = is used to specify the set of files you want to scan (*.zip in this case)
do = command/action to take on the files

It's nice not understanding how it works doesn't stop it from working. 😁
Title: Re: Best Compression for Specific Emulators
Post by: tgmagician on June 26, 2019, 08:08:25 AM
Hi progets, thanks for all the info you have provided along with everyone else.

I know this is an older post but I am trying to convert my Saturn games into chd's.

I have tried a few thing but without any luck. What tools did you use and what is the process?

My saturn games are 7z archived Cue, Bin files. I can extract them and get them into a chd by chdman.exe but when I try to play alot of them it just brings up a Music player in the Emu. I am not sure if I have the correct parameters for chdman or if there is a different one that I should be using.

Thanks for any help you or anyone else can give me.
Title: Re: Best Compression for Specific Emulators
Post by: progets on June 26, 2019, 09:06:55 PM
Hi progets, thanks for all the info you have provided along with everyone else.

I know this is an older post but I am trying to convert my Saturn games into chd's.

I have tried a few thing but without any luck. What tools did you use and what is the process?

My saturn games are 7z archived Cue, Bin files. I can extract them and get them into a chd by chdman.exe but when I try to play alot of them it just brings up a Music player in the Emu. I am not sure if I have the correct parameters for chdman or if there is a different one that I should be using.

Thanks for any help you or anyone else can give me.

I recommend using the latest chdman.exe which is include with every official MAME release https://www.mamedev.org/release.html (https://www.mamedev.org/release.html). You don't need to use MAME or even get it to function, just grab the chdman.exe from it and you can delete it.

Once you have your Saturn files extracted from the 7z format, place chdman.exe into the same folder and also create "convert2chd.bat" in this folder with these contents
Code: [Select]
for /r %%i in (*.cue) do chdman createcd -i "%%i" -o "%%~ni.chd
for /r %%i in (*.gdi) do chdman createcd -i "%%i" -o "%%~ni.chd

Now you can run convert2chd.bat from the command line and if there are any errors you'll see them.

Common issues:
- the .cue file doesn't contain the correct path or name to your .bin file(s)
- you have compressed audio files (like .mp3) which aren't compatible with chdman.exe
Title: Re: Best Compression for Specific Emulators
Post by: tgmagician on June 28, 2019, 08:44:18 AM
Thank you so much for the info, I will give that a try! ;D
Title: Re: Best Compression for Specific Emulators
Post by: gamesmame on April 09, 2021, 05:28:57 AM
I see this:

Warning: this topic has not been posted in for at least 120 days.
Unless you're sure you want to reply, please consider starting a new topic.


but sorry i need this help...  :-[
Hello friends ...

I have a lot of PS2 games here, in format: 7z and Gz and I want to transform everything into CHD .. I need a bat to extract my many .gz(7z) and transform everything in CHD and delete the .gz(7z) automatically. .. Masters of bats and codes, can you help me? Thanks  ;D


I need from a .bat that I do:  extract the .Gz then excludes and transformation .ISO to CHD with the  CHDMAN
Modify message
Title: Re: Best Compression for Specific Emulators
Post by: hermine.potter on April 09, 2021, 11:01:18 AM
-mark all archvies (ctrl+a)
-extract here
-move all .iso to one single folder
-mark all archives (ctrl+a)
-delete
-search for a tool that converts iso to chd
-create there a batch-file:
Code: [Select]
@echo off
for /r %%i in (*.iso) do (
"path_to_your_conversion_Tool\your_Conversion-Tool.exe" "%%~ni.iso" "%%~ni.chd"
)
-sort for all iso-files
-mark all iso-files
-delete
Title: Re: Best Compression for Specific Emulators
Post by: gamesmame on April 09, 2021, 03:58:04 PM
Dont have anyway create one bat for extract and convert for .CHD and delete the .gz automatic? I have extract manually?  :P :'(
Title: Re: Best Compression for Specific Emulators
Post by: progets on April 09, 2021, 11:19:46 PM
Dont have anyway create one bat for extract and convert for .CHD and delete the .gz automatic? I have extract manually?  :P :'(

This can be done but it's not a smart approach. There will likely be errors and if you remove all the original files and unzipped files in one script you won't be able to correct them. This would result in broken games with no way to fix them other than a backup or redownload.
Title: Re: Best Compression for Specific Emulators
Post by: hermine.potter on April 10, 2021, 12:48:53 AM
Based on your question and laziness, I think your plan with one single file is extremely dangerous, if you don't know how and what you're doing. Sorry ^-^

Use three batch files instead:
>Extract:
Code: [Select]
@echo off
for /r %%i in (*.zip) do (
"C:\Program Files\7-Zip\7z.exe" e %%i -o"C:\myFolder" *.iso -r
)


>Convert:
see post above

>Delete:
Code: [Select]
@echo off
for /r %%i in (*.zip) do (
del /F /Q "%%i"
)
Title: Re: Best Compression for Specific Emulators
Post by: gamesmame on April 10, 2021, 10:20:14 AM
On second thought, you are right ... I will extract manually then use the .bat to transform for CHD !!! Thanks for all the help !!! ^ _ ^
Title: Re: Best Compression for Specific Emulators
Post by: akafox on April 10, 2021, 11:40:30 AM
I did convert my psx (one) games from pbp to chd and saved 150(?) GB so now they fit nicely onto a 512 GB SSD for my pi with room to breathe. I do know that versions of chdman has made tighter and tighter (better?) compression with each version.

On to the PS2 though, I don't have a full set (I can't even image the amount of space that would take!..actually 3-4TB would be my estimation.) I compressed Dark Cloud from a 1.8G iso file to a 938Gb cso format file then to a 759GB chd format file. So a whole Gigabyte from raw iso to chd format? That is a good chunk of spaced saved but not all games will compress down that far I would think... (Bloody Roar 3 in bin/cue is 671MB to cso format it is 517MB to chd it is 488Mb so only a 183MB difference..maybe i should compare another raw iso not a bin/cue?)

 From what I understand the cso format that is used for ps2 compression is lossless much like chd..I could be wrong? PCSX2 supports chd now I hear...but with drives getting bigger and cheaper..will it matter if things are compressed? Than again the Raspberry Pi 8 might run PS2 on day?...and 4Terabyte Micro SD cards are not cheap! :P

Interesting to think that we thought 650GB was huge then..now it just seems..quaint after 25 years  ::)
Title: Re: Best Compression for Specific Emulators
Post by: akafox on April 10, 2021, 11:46:06 AM
Just to note here...


For those of you who are afraid of the command line/terminal the Gui program from qmc2 still works. While i do not recommend that you use thee emulator part (though I doubt it will steal all your chickens if you do...it is just and older version of mame that has not been updated) as i said the gui works fine and yes it uses the newest version of chdmame (the version you tell it to use. So grab the newest one from mame dev).

In windows it should work "out of the box" in liunx use wine
Title: Re: Best Compression for Specific Emulators
Post by: Mark Norville on March 15, 2022, 03:24:34 AM
Ok time for a 2022 update

Firstly to AkaFox, I do not have a full PS2 file set and unpacked in bins and ISO I am around 1,000 odd short

However, I have 1,654 roms in one directory and that is just the USA roms and that equals 4.03 TB for that little lot alone, so for another 1,000 roms then you are looking at potentially 7 TB maybe at an estimate.

I have not even added any artwork as yet, that is going to increase the size on the hard drive as well. I was trying to fit the artwork, videos etc onto a 1 TB hard drive but that soon became full, so now I am having to look at my artwork being on mechanical drives alongside of the roms. I might at a later date look at a 2TB SSD and try again to put the artwork on.

File compressor used http://www.softsea.com/download/ISO-Compressor.html

PS2 Test 5 files (all the 007 games) ISO file format

15.6 GB

Ok so my findings after a quick two minute try

1) I could only do one file at a time, the program froze, so I allowed to let it carry on and eventually it completed
2) The game 007 - Agent Under Fire (USA) 1.51 GB before and 1.35 afterwards so a saving of 0.16 (level 6 compression)

* I just tried level 9 compression, it took a hell of a lot longer and once it had finished it was still 1.35 gigs so the same as level 6 and much slower

An estimated saving across the 1,654 odd files will be an very rough estimated 26 gigs.

I am just running a speed test on PCSX2 emulator as just a test bed, I might try RetroArch in the future.

However, I do not see much of a difference loading a CSO file or an ISO file they were both very fast so no clear winner in the loading department.

My findings

1) It will take far too long just to convert all of the files just to save 26 gigs estimated
2) Yes maybe if there was a batch file (which there might be) then it might be tempting, but I would be tying up my PC for days just doing this
3) If you are short on money or hard drive space, then yes this might be worth doing, but potentially you might just want the best of the best games
4) If you are a serious collector, and are not worried about money or hard drive space, then it is pointless wasting time compressing when you could be playing instead
5) I don't think that my OCD or my nerves would be able to handle thinking that I have used a program to compress these files so I delete the ISO file and then I find out at a later date, that the compressor did not work properly and my previous working game, is now not working, and that I have deleted the original ISO. However, I am going off a worse case scenario and this potentially would not happen.
6) The prices of hard drives are going down, yes sometimes they rise quickly but on the whole an 18 TB is now around £300 or below or an 8 TB £165
7) Cheap motherboards and CPU bundles on eBay sometimes you can find a bargain. My latest buy I7 3770K 8 SATA ports, 16 gigs of ram £137 including shipping and that was three auctions as he sold an 8 gig by itself. I also brought his old fan for some stupid reason. So from my previous build an I5 with 8 gigs of ram, I now have more storage and a better CPU, plus I have my old computer ready for other projects.

So if you equate all of this into two factors, time is money and time is life. If you was a business compressing all of these files, then how many man hours would it take you to complete? If you was employed at £10 an hour, that might be 12 hours or more work at £120. Again when you get to my age 12 hours just to get slightly smaller file sizes is 12 hours off my life span, when I could be doing something that I enjoy more, seems a waste of time to me.

Also looking at it like this, those twelve hours would be better off setting up Attract Mode so you are adding more time on top of how long it is going to take to get the whole lot up and running.

Regards

Mark
Title: Re: Best Compression for Specific Emulators
Post by: progets on March 16, 2022, 01:23:55 AM
Different games compress different amounts. I would recommend testing a larger batch of games.

Even in the result you listed, your compression saved over 10%. So for every 10 TB you would save 1 TB. That's not bad at all for a game that will load just as quick (and possibly faster) than an uncompressed game. I also think that other games will compress even more (and some games might compress less).

This is just the case for the PS2 games you're talking about. Compressing with .chd for other systems will likely save even more space. I have seen some systems that used CDs make a full 700 MB iso image that compresses to less than 100 MB when converted to .chd.

If you read the whole thread you will see that I used a different command line program to convert PS2 files as well as batch file to automate the process. This batch file has an adjustment for the number of threads of the CPU that get used so you can run it in the background while still doing other things, as well as see the progress in the console window.

This process isn't for everyone. Maybe you're better off with your unzipped games. I like the idea that the entire game is in one file. I like the idea of saving space. This thread was also written almost four years ago when an 8 TB drive was much, much more expensive then it is today.

The choice is yours. Everyone has different preferences. Happy gaming.

P.S. Your .16 GB savings times 1,654 games is 264 GB and not 26 GB. Like I said, other games will likely compress at a different percentage.
Title: Re: Best Compression for Specific Emulators
Post by: Mark Norville on March 16, 2022, 01:52:14 AM
Yes I will admit that I did not read the entire thread, I read the main stuff and decided to give it a go.

So my testing was very quick and also on the fly, which yes will mean that my findings are going to be reasonably more errored than someone who has done more research and spent more time finding the best compressor.

I downloaded the first software that I could find, and ran a quick test but it might not be the best software, and there might be better software out there for the job.

I done pretty much what I call the idiot test, testing it on myself means that I know what other people will try to do as usually it is the first software someone finds, they will try and do it without doing any further research on it.

I need to buy another three hard drives to fill this machine up, so I shall be doing that over the space of this year, maybe one or two every few months, when I saw the 18tb hard drive for that price, it made a lot more sense for me to start looking at 18tb rather than 2 x 8 TB as it works out slightly cheaper and I can potentially have the entire PS collection on one hard drive, as I am just downloading 1.857 TB of PSP which is all 7zip.

I don't even have the (censored) company stuff sorted yet, I have the 8-bit systems done, I have unpacked the square roms ;) so I have the full USA set on those, but nothing for the bigger system such as the swap. As the 360 stuff is becoming better emulated then that is another power house of roms to collect.

So in the end I will just repurpose the hard drives after replacing them, and reusing the old PC, as I am looking at pinball next, the Atgames legend pinball table is now available in the UK, so that will be my next major project to get started.

Maybe at some point get my jukebox back up and running, but I cannot stand tagging all the time, and as I have google home and alexa I just use my voice now.

My cinema PC needs more hard drives and I want a dedicated TV show set up as well.

However, I am rambling lol

Regards

Mark
Title: Re: Best Compression for Specific Emulators
Post by: progets on March 16, 2022, 02:11:16 AM
Do what works best for you but keep in mind that just formatting a hard drive takes over 10% of space. On a 18 TB hard drive that's almost 2 TB. Compressing the files can recover this lost space and probably even more.

Like I said, different strokes for different folks. Use what works best for you.

I will mention that the collection you talk about will take hundreds, if not, thousands of hours to obtain, curate, get artwork, etc. to perfect. Any file conversion time doesn't seem so much in the grand scheme of things. If you want to save time, only obtain games you'll really play instead of trying to accomplish (the moving target of) the holy grail, which you will likely only use 5% or less of the games.
Title: Re: Best Compression for Specific Emulators
Post by: Mark Norville on March 16, 2022, 02:51:38 AM
I am pretty much already there, I have ten odd years of zip files to go through plus all the recent stuff that I am getting.

I jumped from Hyper Spin to CoinOps due to the arcade wall as I loved the cabinets, it was new and exciting. However, I wasted two or more years on coinops with a certain person, keep changing and bringing out new themes for CoinOps, and then bringing out Retro Arcade 2, then Player Max, then Coinops Universe, and all of them have pretty much gone back to basic rubbish, now that GJsmith left, such as slapping a monitor bezel on a piece of art and then thinking that is an improvement.

So far touch wood AM seems to be able to handle full sets, but I have spent the past three odd weeks, transferring files, building a new PC, transferring more files and at the moment I am just unzipping a lot of stuff to try and get into place.

One such file I have is C64 Emulator  6000 Games.rar 07/11/2012 so for me really, it has been a long journey, but hard drive space as you know was always the main issue.

Hopefully come Monday next week, I can restart building up AM but my main issue and challenge is mainly renaming a lot of the rom sets, as the majority of them are not HyperSpin or list friendly, you have people who just download roms from a lot of different sources, whack them all together and you have file names with stuff like game (USA) (En, Fr) V1.1 and stuff like that, thankfully I have a file renamer to do the majority of the work for me, but I am going through the Wiki pages to make sure that I have every rom, and sadly the lists at HyperList are useless and incomplete, they have done a great job on the MAME list, so shooter/flying vertical but when it comes down to say Sega Model 2 plus all the rest, it is just shooter, rather than shooter/flying vertical.

So if you do a playlist for vertical shooters, then you are not going to get everything just mainly MAME stuff. As I am medically retired then I have time on my hands, so this is a crime of passion really too me, and at first, emulation to me was playing the games that I had played in the arcades as a teenager, but we had reasonably limited games in the UK, when you compare that to the states, so I have been through MAME and found all kinds of new games that look good, a lot of which is not included with coinops as an example, which is supposed to be the best of the best, but all what they do is released the same old rubbish and they have missed so many fantastic games. BP has done this for a decade now, and pretty much every release it is the same boring 1,000 odd games.

If I do get stuck in the future, E.G I have filled everything up with 8 x 18 TB and I still have more to do, then yeah compression would be my last resort, but hopefully by that time, I might actually be able to play a game.

To me, I enjoy the collecting and set up side of things, more so than I actually do playing games side. I occasionally pick up a game and have a few minutes of game play, Donkey Kong being my last but I also have the PS5 and also Xbox Series X which is gathering dust as they are usually not turned on, but I like my boys toys, and for me I got them both as due to age then this might be the last generation that I ever own.

I am determined to get all my bucket lists, ticked off before I do kick the bucket lol

Regards

Mark
Title: Re: Best Compression for Specific Emulators
Post by: progets on March 16, 2022, 03:08:47 AM
Go for it! Whatever make you happy.

I have all sorts of stuff on my homemade cabinet. I play less than 100 games, all from the 80's. LOL
Title: Re: Best Compression for Specific Emulators
Post by: Mark Norville on March 22, 2022, 02:53:08 AM
Ok, I was messing around with my PS1 Europe roms yesterday trying to get a PS1 emulator to run the games, all of the files were zipped, but I was having trouble with some emulators such as retro arch saying that it was having problems with uncompressing.

I also gave Duckstation a go and I reasonably gave that up, I think it only saw CHD files but I can't 100% remember that now as it was yesterday.

However, I unpacked 1,361 zip files which comes to a massive 610 gigs and that is 99% the complete PS1 Europe titles, I am missing four files, which I might have somewhere as I have another directory with roms in that do not have any Country marking on

E.G PS1 Rom (Europe) so I will have to look at those in the future

So here are the before and after statistics

All files zipped : Sadly I did not take a note of that before I deleted the zip files

All files uncompressed : 1,361 = 610 Gigs

All files CHD : 1,361 = 327 Gigs

Total saving of = 284 Gigs

Time taken = 28 hours + more for unzipping

Cost (I will add this just for extra nerd stats fun.) = £10 an hour if employed £280 at hourly rate not including unzipping, not even including downloading. Electric costs maybe a £1 or something for electric bill. Including downloading and unzipping maybe over the £500 mark.

Cost of buying all the games original thousands. If self employed doing this around £3.26 per rom based on the £500 mark. In real costs as you are not a self employed business £5 electric so 0.00367 per rom.

However, I have started the USA roms, only 500 for those, I will check the other folders, my PS1 Japanese roms are 3,643 files including some double disc, so I am not looking forward to that little lot.

I will also probably do this for the USA and JAP roms as well.

Just to give some time frame on how long all of this is taking I started at 09:47 this morning, and now 18:00 (6pm) and I am up to the F's so eight hours so far. The computer will be on all night so will try and work out how long and how far I am in the morning. I am guessing this might take around two days to complete, I was kind of stupid not for doing it in batches, rather than just letting the whole thing run. However, stupid is my middle name.

Well it is now 06:52 so a few hours short of 24 hours, so I am on approx 21 hours at the moment and I am up to Si at the moment so pretty much almost done, S & T are usually the worse letters to do, once they are over then the rest of them should just fly through. 12 noon so now over the 24 hour mark and the W's have now started so almost home although W is pretty big with about 61 files so I am looking at potentially another five hours maybe. All of this is potentially dependent upon your CPU some I5 systems might take longer I am now on an i7 but if I was doing it on my i5 if I was doing other stuff in the background, it would slow things down and sometimes lock things up. Finally finished 13:48 so a lot faster than my estimated five hours more.

I have just started the PS1 Japan Roms I am doing this in batches and I am not counting all the uncompressed files

2,954 zipped files = 672 Gigs
Uncompressed = Who knows maybe 1 TB or more? Far too many to unzip all at once so doing everything in batches instead this time
All files CHD = 569 Gigs (2,791 files some were multi disc or demos or something else such as gameshark kind of stuff)
Total saving = 103 Gigs (2,791 files some were multi disc or demos or something else such as gameshark kind of stuff)

I have not timed doing this little lot, the reason being so many files and after doing all the European files I am a bit bored of it all now. I am on the second day, and still have 1,477 files to do I started doing letter batches and now I am doing them 500 at a time, I removed a few demos that I found, but I have a few duplicate files but they are different version numbers of the games. I hope by the time I get to the last thousand, I might be able to do them all in one go as I might have the space to do that. Sadly if I mess anything up then reasonably I might never be able to get these files again as they all came from pleasuredome sadly shut down. So really these sets are rare now, especially the complete Japan rom set. I will add that it is a noticeable difference between letters, so as an example Europe B was about 36 odd files where Japan B was about 65. So a lot of cultural differences on how popular each letter is. 

Well today is the final day for Japan roms, I am on my last batch of 500 CHD's to make but I still have 627 Japan multi discs which I am not sure how I am going to do them, I think PBP is the file format for multiple discs but will have to do some research on that one. I still have a lot of USA roms that I need to do also including multi disc games, but apart from that I am almost at the end of completing it all. I am now on my third day for Japan roms, 934 more USA roms to do.

Regards

Mark
Title: Re: Best Compression for Specific Emulators
Post by: akafox on March 30, 2022, 10:09:51 AM
Most of my care of compression was because I was using my Rasp pi 3 and SD cards were not cheap at the time..nor are they now at least a 1TB isn't and when I mean cheap I mean less then 30-40 of what ever currency you use...

Like progrets said, it really is better to use a cheap PC as they are way more capable. If the Rasp Pis stayed at 30-40 instead off 100 or more...then well the trade off for me was worth it. As you know though it did not stay that way and you cab pick up a used PC for a little more than a rasp pi4 now with a LOT more power and storage space.

I think the Pi zero 2 can do PSX (ps1) games just fine (and thus everything under it..aside from N64 and 3DO) I would think about getting two of them for the price of one pi 4...but you have to FIND one at that price..and well the pi4 can run dreamcast (some anyway and PSP..again some!) So again for just a few more of whatever currency you can get a "monster of a PC" compared to a rasp pi4 for dedicated emulation *shrugs*

...although there is that small noiseless form factor of the PI..and well maybe the price of a PI is worth having once it is set up (IF it ever is!) to sit behind the TV and be used so that it is easy to use so that others can use it..I can have my hard drive space back and keep people off my main machine..*shrugs*  just comes down to what you want to use it for..which is why it was made in the first place....

Yet i sit back and remember the first single MEGABYTE file (compressed I might add!) and i used to think it was HUGE for my 150 Megabyte hard drive on it! and yet things have not changed all that much have they lol