From your description, it sound like some thing might be missing.
Try the following steps, hopefully it helps..
1. make sure you can run you roms from retroarch itself. Retroarch needs to have the core you want to use installed. You can do this directly in retroarch.
2. The retroarch.app like all mac apps? Has it's sub set of directories in the app itself. So after dropping retroarch in your applications directory, right click on the retroarch.app and select "open package" explore the directory structure (I would think it's the same as mine as they are the same program)
. In here this is where you can find the retroarch(exe) and the core you want to use.
In my case the retroarch(exe) is here /applications/retroarch.app/Contents/MacOS/
and the cores are located here /applications/retroarch.app/Contents/Resources/cores/
3. Using the above info you can open terminal and build your command line, in my case the following works.
/applications/retroarch.app/Contents/MacOS/RetroArch -L /applications/retroarch.app/Contents/Resources/cores/snes9x_libretro.dylib ./Documents/Aladdin.zip
The -L tells retroarch to ignore the retroarch.cfg and load the core defined in the command line.
The Aladdin.zip is the rom that is in my documents directory.
If this works then you should be able to place it in your emulator config in AM and run it successfully.
BTW my AM is also in my application directory if that makes a difference.
Sorry if this is redundant for you, I'm not an expert with OSX/Terminal myself and am not sure what level of knowledge you have already, hopefully my explanation helps and my assumptions about the operating environment are correct.
Let me know...
