Sorry Omegaman, I've given up for now. There just isn't enough info out there yet on compiling stuff specifically for the RPi3, and stuff compiled for the RPi2 is a no-go (tried the 0.171 you linked, it took 30 seconds just to load mame).
I can get Jessie on there and have the hardware work. But then get stumped at the "AM wants X but advmame doesn't"-problem and the "compiling-instructions are there, but not with hardware- and X-support"-problem.
I'm absolutely positive that it should be possible, I'm just not good enough
I can solve dependencies and do basic compilation, but I'm at a loss if I have to start editing make-scripts and configure flags.
This would be the ideal setup so far: Image for the RPi3 with Dietpi, whatever X-setup that makes sense, AM (hardware-accelerated?), Advmame 0.94 & 1.4 (compiled to work with X in order to work with AM and hardware-accelerated), wifi-setup-script (think it comes with DietPi), samba-shares setup.
For now here's a script I stumbled upon that compiles and installs AM succesfully on a clean install (works on RPi3 + Jessie):
cd ~
mkdir develop
sudo apt-get install cmake libx11-dev libx11-xcb-dev libflac-dev libogg-dev libvorbis-dev libopenal-dev libjpeg8-dev libfreetype6-dev libxcb-randr0-dev libxcb-image0-dev libxcb-util0-dev libxrandr-dev libxcb-ewmh-dev libxcb-keysyms1-dev libxcb-icccm4-dev libudev-dev libavutil-dev libavcodec-dev libavformat-dev libavfilter-dev libswscale-dev libavresample-dev libfontconfig1-dev
cd ~/develop
git clone -b rpi https://github.com/mickelson/SFML sfml-pi
mkdir sfml-pi/build;cd sfml-pi/build
cmake -DEGL_INCLUDE_DIR=/opt/vc/include -DEGL_LIBRARY=/opt/vc/lib/libEGL.so -DFREETYPE_INCLUDE_DIR_freetype2=/usr/include -DFREETYPE_INCLUDE_DIR_ft2build=/usr/include/freetype2 -DGLES_INCLUDE_DIR=/opt/vc/include -DGLES_LIBRARY=/opt/vc/lib/libGLESv1_CM.so -DSFML_BCMHOST=1 ..
sudo make install
sudo ldconfig
cd ~/develop
git clone --depth 1 https://github.com/mickelson/attract attract
cd attract
make
sudo make install