Did you go through the EmulationStation input configuration for both player1 and player2?
The N64 uses analog controls which the xin-mo doesn't support. Only some emulators support digital to analog controls. Post your /opt/retropie/configs/n64/emulators.cfg file so I can see what you're using.