I've noticed this frequently (I'm on Verizon FIOS 20/5 and verified by speedtest.net) in the last year. Enough so that I've resorted to implementing 'slow connection' by default so by it buffers everything.
I've found it depends on the video for example, two videos may stream normally simultaneously, while another may stream very slowly even if it's the only video being served. I've not yet been able to determine a pattern though (i.e. videos from a year or more back taking longer to buffer, etc).