I'm a veteran. I say wear what you want on the field. Playing mil-sim is the closest some people will ever get to serving in the military and if you enjoy it, I'm cool with that.
It's pretty easy to understand what is crossing the line and what isn't. For example, don't walk around town in an authentic looking uniform after or before the game just to pose as a soldier (airman, marine, sailor, etc). That could piss some folks off real quick.