Hi, thank you for the answer.
I think I may have phrased my question wrong. But I got the solution for everyone that is also trying to achieve something similar.
I was trying to run attractmode directly from the console with no Desktop/GUI installed. This is possible with the latest attractmode thanks to DRM/KMS settings. You have to build the sources yourself with the USE_DRM flag. Check
https://github.com/mickelson/attract/blob/master/Compile.md for instructions on how to compile.
This is, however, only one part as I found out you also have to build sfml-pi with DRM enabled (instructions for compiling:
https://github.com/mickelson/sfml-pi). Instead of the regular sfml. This lets you then start attractmode with no desktop insalled.
Together with retroarch (also DRM/no X11) this gives you a console like experience for dedicated arcade machines. (kinda like retropi).