Alright, I figured out what's going on and it's silly.
I've used my Xarcade stick before but I didn't load the xarcade cfg like I have now. Initially, It worked perfectly fine in a few games I tested and then I tried a Metal Slug game. That's when the controls changed and I would only test any changes with Metal Slug thinking it should control for the same game when testing.
Well, after looking in the xarcade.cfg I see it changes the controls for Neogeo games. So that's it, nothing is actually wrong its just that I didn't know it changed the default layout for Neogeo games.