As a guy that referee's because he can't play as much anymore...
Once I am 'interfering' with a player by checking them out, if they get shot... I wipe it off them and tell them to keep playing. Not their fault I'm all up in their face.
If you called me over to a player, and shot them, I'd think less of you. A lot less. I'd also wipe it off and let the guy keep playing.
Side note; I have never reffed professional speedball. Only friendly speed-ball and woodsball.