Cutting caffeine is usually recommended on most serious workout regiments because of the fact that it dehydrates you.
To get the most from workouts you're supposed to be fully hydrated and fully rested.. at least that's conventional wisdom.
I switch to drinking only water and natural fruit juice when I get serious about it (which is rare).. but the worst is cutting out beer/booze.
Its all a matter of getting out what you put into it.. if you alter your diet 100% and push to build as much muscle as possible, yeah I would say cut out caffeine. If you are just average joe trying to look better, I wouldn't cut it out but I would cut down.