When they're worn
do you notice any obvious nicks of cracks in them? Generally I replace all seals every five years, if not sooner. Obviously if they're broken it's time to replace them, but you already knew this
How loose are they? Can you pull them off the bolt with your fingers (assuming you don't have long nails)? If so, probably a good idea to replace them. A nice $0.50 fix, so it's not like it will be a big investment - in this case I'd err on the side of caution and just slack some new ones on.