Best Compression for Specific Emulators

[gamesmame]

I see this:

Warning: this topic has not been posted in for at least 120 days.
Unless you're sure you want to reply, please consider starting a new topic.

but sorry i need this help... 
Hello friends ...

I have a lot of PS2 games here, in format: 7z and Gz and I want to transform everything into CHD .. I need a bat to extract my many .gz(7z) and transform everything in CHD and delete the .gz(7z) automatically. .. Masters of bats and codes, can you help me? Thanks 


I need from a .bat that I do:  extract the .Gz then excludes and transformation .ISO to CHD with the  CHDMAN
Modify message

[hermine.potter]

-mark all archvies (ctrl+a)
-extract here
-move all .iso to one single folder
-mark all archives (ctrl+a)
-delete
-search for a tool that converts iso to chd
-create there a batch-file:

Code: [Select]

@echo off
for /r %%i in (*.iso) do (
"path_to_your_conversion_Tool\your_Conversion-Tool.exe" "%%~ni.iso" "%%~ni.chd"
)-sort for all iso-files
-mark all iso-files
-delete

[gamesmame]

Dont have anyway create one bat for extract and convert for .CHD and delete the .gz automatic? I have extract manually? 

[progets]

Dont have anyway create one bat for extract and convert for .CHD and delete the .gz automatic? I have extract manually? 

This can be done but it's not a smart approach. There will likely be errors and if you remove all the original files and unzipped files in one script you won't be able to correct them. This would result in broken games with no way to fix them other than a backup or redownload.

[hermine.potter]

Based on your question and laziness, I think your plan with one single file is extremely dangerous, if you don't know how and what you're doing. Sorry

Use three batch files instead:
>Extract:

Code: [Select]

@echo off
for /r %%i in (*.zip) do (
"C:\Program Files\7-Zip\7z.exe" e %%i -o"C:\myFolder" *.iso -r
)

>Convert:
see post above

>Delete:

Code: [Select]

@echo off
for /r %%i in (*.zip) do (
del /F /Q "%%i"
)

[gamesmame]

On second thought, you are right ... I will extract manually then use the .bat to transform for CHD !!! Thanks for all the help !!! ^ _ ^

[akafox]

I did convert my psx (one) games from pbp to chd and saved 150(?) GB so now they fit nicely onto a 512 GB SSD for my pi with room to breathe. I do know that versions of chdman has made tighter and tighter (better?) compression with each version.

On to the PS2 though, I don't have a full set (I can't even image the amount of space that would take!..actually 3-4TB would be my estimation.) I compressed Dark Cloud from a 1.8G iso file to a 938Gb cso format file then to a 759GB chd format file. So a whole Gigabyte from raw iso to chd format? That is a good chunk of spaced saved but not all games will compress down that far I would think... (Bloody Roar 3 in bin/cue is 671MB to cso format it is 517MB to chd it is 488Mb so only a 183MB difference..maybe i should compare another raw iso not a bin/cue?)

 From what I understand the cso format that is used for ps2 compression is lossless much like chd..I could be wrong? PCSX2 supports chd now I hear...but with drives getting bigger and cheaper..will it matter if things are compressed? Than again the Raspberry Pi 8 might run PS2 on day?...and 4Terabyte Micro SD cards are not cheap!

Interesting to think that we thought 650GB was huge then..now it just seems..quaint after 25 years 

[akafox]

Just to note here...


For those of you who are afraid of the command line/terminal the Gui program from qmc2 still works. While i do not recommend that you use thee emulator part (though I doubt it will steal all your chickens if you do...it is just and older version of mame that has not been updated) as i said the gui works fine and yes it uses the newest version of chdmame (the version you tell it to use. So grab the newest one from mame dev).

In windows it should work "out of the box" in liunx use wine

[Mark Norville]

Ok time for a 2022 update

Firstly to AkaFox, I do not have a full PS2 file set and unpacked in bins and ISO I am around 1,000 odd short

However, I have 1,654 roms in one directory and that is just the USA roms and that equals 4.03 TB for that little lot alone, so for another 1,000 roms then you are looking at potentially 7 TB maybe at an estimate.

I have not even added any artwork as yet, that is going to increase the size on the hard drive as well. I was trying to fit the artwork, videos etc onto a 1 TB hard drive but that soon became full, so now I am having to look at my artwork being on mechanical drives alongside of the roms. I might at a later date look at a 2TB SSD and try again to put the artwork on.

File compressor used http://www.softsea.com/download/ISO-Compressor.html

PS2 Test 5 files (all the 007 games) ISO file format

15.6 GB

Ok so my findings after a quick two minute try

1) I could only do one file at a time, the program froze, so I allowed to let it carry on and eventually it completed
2) The game 007 - Agent Under Fire (USA) 1.51 GB before and 1.35 afterwards so a saving of 0.16 (level 6 compression)

* I just tried level 9 compression, it took a hell of a lot longer and once it had finished it was still 1.35 gigs so the same as level 6 and much slower

An estimated saving across the 1,654 odd files will be an very rough estimated 26 gigs.

I am just running a speed test on PCSX2 emulator as just a test bed, I might try RetroArch in the future.

However, I do not see much of a difference loading a CSO file or an ISO file they were both very fast so no clear winner in the loading department.

My findings

1) It will take far too long just to convert all of the files just to save 26 gigs estimated
2) Yes maybe if there was a batch file (which there might be) then it might be tempting, but I would be tying up my PC for days just doing this
3) If you are short on money or hard drive space, then yes this might be worth doing, but potentially you might just want the best of the best games
4) If you are a serious collector, and are not worried about money or hard drive space, then it is pointless wasting time compressing when you could be playing instead
5) I don't think that my OCD or my nerves would be able to handle thinking that I have used a program to compress these files so I delete the ISO file and then I find out at a later date, that the compressor did not work properly and my previous working game, is now not working, and that I have deleted the original ISO. However, I am going off a worse case scenario and this potentially would not happen.
6) The prices of hard drives are going down, yes sometimes they rise quickly but on the whole an 18 TB is now around £300 or below or an 8 TB £165
7) Cheap motherboards and CPU bundles on eBay sometimes you can find a bargain. My latest buy I7 3770K 8 SATA ports, 16 gigs of ram £137 including shipping and that was three auctions as he sold an 8 gig by itself. I also brought his old fan for some stupid reason. So from my previous build an I5 with 8 gigs of ram, I now have more storage and a better CPU, plus I have my old computer ready for other projects.

So if you equate all of this into two factors, time is money and time is life. If you was a business compressing all of these files, then how many man hours would it take you to complete? If you was employed at £10 an hour, that might be 12 hours or more work at £120. Again when you get to my age 12 hours just to get slightly smaller file sizes is 12 hours off my life span, when I could be doing something that I enjoy more, seems a waste of time to me.

Also looking at it like this, those twelve hours would be better off setting up Attract Mode so you are adding more time on top of how long it is going to take to get the whole lot up and running.

Regards

Mark

[progets]

Different games compress different amounts. I would recommend testing a larger batch of games.

Even in the result you listed, your compression saved over 10%. So for every 10 TB you would save 1 TB. That's not bad at all for a game that will load just as quick (and possibly faster) than an uncompressed game. I also think that other games will compress even more (and some games might compress less).

This is just the case for the PS2 games you're talking about. Compressing with .chd for other systems will likely save even more space. I have seen some systems that used CDs make a full 700 MB iso image that compresses to less than 100 MB when converted to .chd.

If you read the whole thread you will see that I used a different command line program to convert PS2 files as well as batch file to automate the process. This batch file has an adjustment for the number of threads of the CPU that get used so you can run it in the background while still doing other things, as well as see the progress in the console window.

This process isn't for everyone. Maybe you're better off with your unzipped games. I like the idea that the entire game is in one file. I like the idea of saving space. This thread was also written almost four years ago when an 8 TB drive was much, much more expensive then it is today.

The choice is yours. Everyone has different preferences. Happy gaming.

P.S. Your .16 GB savings times 1,654 games is 264 GB and not 26 GB. Like I said, other games will likely compress at a different percentage.

[Mark Norville]

Yes I will admit that I did not read the entire thread, I read the main stuff and decided to give it a go.

So my testing was very quick and also on the fly, which yes will mean that my findings are going to be reasonably more errored than someone who has done more research and spent more time finding the best compressor.

I downloaded the first software that I could find, and ran a quick test but it might not be the best software, and there might be better software out there for the job.

I done pretty much what I call the idiot test, testing it on myself means that I know what other people will try to do as usually it is the first software someone finds, they will try and do it without doing any further research on it.

I need to buy another three hard drives to fill this machine up, so I shall be doing that over the space of this year, maybe one or two every few months, when I saw the 18tb hard drive for that price, it made a lot more sense for me to start looking at 18tb rather than 2 x 8 TB as it works out slightly cheaper and I can potentially have the entire PS collection on one hard drive, as I am just downloading 1.857 TB of PSP which is all 7zip.

I don't even have the (censored) company stuff sorted yet, I have the 8-bit systems done, I have unpacked the square roms so I have the full USA set on those, but nothing for the bigger system such as the swap. As the 360 stuff is becoming better emulated then that is another power house of roms to collect.

So in the end I will just repurpose the hard drives after replacing them, and reusing the old PC, as I am looking at pinball next, the Atgames legend pinball table is now available in the UK, so that will be my next major project to get started.

Maybe at some point get my jukebox back up and running, but I cannot stand tagging all the time, and as I have google home and alexa I just use my voice now.

My cinema PC needs more hard drives and I want a dedicated TV show set up as well.

However, I am rambling lol

Regards

Mark

[progets]

Do what works best for you but keep in mind that just formatting a hard drive takes over 10% of space. On a 18 TB hard drive that's almost 2 TB. Compressing the files can recover this lost space and probably even more.

Like I said, different strokes for different folks. Use what works best for you.

I will mention that the collection you talk about will take hundreds, if not, thousands of hours to obtain, curate, get artwork, etc. to perfect. Any file conversion time doesn't seem so much in the grand scheme of things. If you want to save time, only obtain games you'll really play instead of trying to accomplish (the moving target of) the holy grail, which you will likely only use 5% or less of the games.

[Mark Norville]

I am pretty much already there, I have ten odd years of zip files to go through plus all the recent stuff that I am getting.

I jumped from Hyper Spin to CoinOps due to the arcade wall as I loved the cabinets, it was new and exciting. However, I wasted two or more years on coinops with a certain person, keep changing and bringing out new themes for CoinOps, and then bringing out Retro Arcade 2, then Player Max, then Coinops Universe, and all of them have pretty much gone back to basic rubbish, now that GJsmith left, such as slapping a monitor bezel on a piece of art and then thinking that is an improvement.

So far touch wood AM seems to be able to handle full sets, but I have spent the past three odd weeks, transferring files, building a new PC, transferring more files and at the moment I am just unzipping a lot of stuff to try and get into place.

One such file I have is C64 Emulator  6000 Games.rar 07/11/2012 so for me really, it has been a long journey, but hard drive space as you know was always the main issue.

Hopefully come Monday next week, I can restart building up AM but my main issue and challenge is mainly renaming a lot of the rom sets, as the majority of them are not HyperSpin or list friendly, you have people who just download roms from a lot of different sources, whack them all together and you have file names with stuff like game (USA) (En, Fr) V1.1 and stuff like that, thankfully I have a file renamer to do the majority of the work for me, but I am going through the Wiki pages to make sure that I have every rom, and sadly the lists at HyperList are useless and incomplete, they have done a great job on the MAME list, so shooter/flying vertical but when it comes down to say Sega Model 2 plus all the rest, it is just shooter, rather than shooter/flying vertical.

So if you do a playlist for vertical shooters, then you are not going to get everything just mainly MAME stuff. As I am medically retired then I have time on my hands, so this is a crime of passion really too me, and at first, emulation to me was playing the games that I had played in the arcades as a teenager, but we had reasonably limited games in the UK, when you compare that to the states, so I have been through MAME and found all kinds of new games that look good, a lot of which is not included with coinops as an example, which is supposed to be the best of the best, but all what they do is released the same old rubbish and they have missed so many fantastic games. BP has done this for a decade now, and pretty much every release it is the same boring 1,000 odd games.

If I do get stuck in the future, E.G I have filled everything up with 8 x 18 TB and I still have more to do, then yeah compression would be my last resort, but hopefully by that time, I might actually be able to play a game.

To me, I enjoy the collecting and set up side of things, more so than I actually do playing games side. I occasionally pick up a game and have a few minutes of game play, Donkey Kong being my last but I also have the PS5 and also Xbox Series X which is gathering dust as they are usually not turned on, but I like my boys toys, and for me I got them both as due to age then this might be the last generation that I ever own.

I am determined to get all my bucket lists, ticked off before I do kick the bucket lol

Regards

Mark

[progets]

Go for it! Whatever make you happy.

I have all sorts of stuff on my homemade cabinet. I play less than 100 games, all from the 80's. LOL

[Mark Norville]

Ok, I was messing around with my PS1 Europe roms yesterday trying to get a PS1 emulator to run the games, all of the files were zipped, but I was having trouble with some emulators such as retro arch saying that it was having problems with uncompressing.

I also gave Duckstation a go and I reasonably gave that up, I think it only saw CHD files but I can't 100% remember that now as it was yesterday.

However, I unpacked 1,361 zip files which comes to a massive 610 gigs and that is 99% the complete PS1 Europe titles, I am missing four files, which I might have somewhere as I have another directory with roms in that do not have any Country marking on

E.G PS1 Rom (Europe) so I will have to look at those in the future

So here are the before and after statistics

All files zipped : Sadly I did not take a note of that before I deleted the zip files

All files uncompressed : 1,361 = 610 Gigs

All files CHD : 1,361 = 327 Gigs

Total saving of = 284 Gigs

Time taken = 28 hours + more for unzipping

Cost (I will add this just for extra nerd stats fun.) = £10 an hour if employed £280 at hourly rate not including unzipping, not even including downloading. Electric costs maybe a £1 or something for electric bill. Including downloading and unzipping maybe over the £500 mark.

Cost of buying all the games original thousands. If self employed doing this around £3.26 per rom based on the £500 mark. In real costs as you are not a self employed business £5 electric so 0.00367 per rom.

However, I have started the USA roms, only 500 for those, I will check the other folders, my PS1 Japanese roms are 3,643 files including some double disc, so I am not looking forward to that little lot.

I will also probably do this for the USA and JAP roms as well.

Just to give some time frame on how long all of this is taking I started at 09:47 this morning, and now 18:00 (6pm) and I am up to the F's so eight hours so far. The computer will be on all night so will try and work out how long and how far I am in the morning. I am guessing this might take around two days to complete, I was kind of stupid not for doing it in batches, rather than just letting the whole thing run. However, stupid is my middle name.

Well it is now 06:52 so a few hours short of 24 hours, so I am on approx 21 hours at the moment and I am up to Si at the moment so pretty much almost done, S & T are usually the worse letters to do, once they are over then the rest of them should just fly through. 12 noon so now over the 24 hour mark and the W's have now started so almost home although W is pretty big with about 61 files so I am looking at potentially another five hours maybe. All of this is potentially dependent upon your CPU some I5 systems might take longer I am now on an i7 but if I was doing it on my i5 if I was doing other stuff in the background, it would slow things down and sometimes lock things up. Finally finished 13:48 so a lot faster than my estimated five hours more.

I have just started the PS1 Japan Roms I am doing this in batches and I am not counting all the uncompressed files

2,954 zipped files = 672 Gigs
Uncompressed = Who knows maybe 1 TB or more? Far too many to unzip all at once so doing everything in batches instead this time
All files CHD = 569 Gigs (2,791 files some were multi disc or demos or something else such as gameshark kind of stuff)
Total saving = 103 Gigs (2,791 files some were multi disc or demos or something else such as gameshark kind of stuff)

I have not timed doing this little lot, the reason being so many files and after doing all the European files I am a bit bored of it all now. I am on the second day, and still have 1,477 files to do I started doing letter batches and now I am doing them 500 at a time, I removed a few demos that I found, but I have a few duplicate files but they are different version numbers of the games. I hope by the time I get to the last thousand, I might be able to do them all in one go as I might have the space to do that. Sadly if I mess anything up then reasonably I might never be able to get these files again as they all came from pleasuredome sadly shut down. So really these sets are rare now, especially the complete Japan rom set. I will add that it is a noticeable difference between letters, so as an example Europe B was about 36 odd files where Japan B was about 65. So a lot of cultural differences on how popular each letter is. 

Well today is the final day for Japan roms, I am on my last batch of 500 CHD's to make but I still have 627 Japan multi discs which I am not sure how I am going to do them, I think PBP is the file format for multiple discs but will have to do some research on that one. I still have a lot of USA roms that I need to do also including multi disc games, but apart from that I am almost at the end of completing it all. I am now on my third day for Japan roms, 934 more USA roms to do.

Regards

Mark