I've searched the forum and one post said you have to assign the buttons in a menu option called 'UI Cancel', but it's not listed in any of the AM menus I can see.
I just want to be able to press the Player 1 and 2 start buttons simultaneously to quit out of a Mame game back to the AM select screen. Is this possible?
Run MAME and press the "Tab" key (to enter MAME configuration mode) and then go to the menu that has the "UI Cancel" controller setting (Config settings -> input general -> user interface -> ui cancel = esc) and press the buttons you desire. This isn't an AM setting.
NOT RECOMMENDED - You could also set AM using Configure-->Emulators-->
system_name-->Exit Hotkey=
XP.S. You didn't tell us your hardware, OS or version of MAME, these thing might be a factor in fixing your issue.