Honestly, all I can think of is images of your work piece or the drill chuck itself working loose and FLYING out of the lathe and impaling someone. The spinning motion, combined with the pressure of the cutting tool makes me think there would be a possibility that the drill chuck could slowly begin to open. Drill chucks are most likely built for straight-on (for lack of a better term) pressure, not pressure that pushes at a sideways angle.
It's hard to say. I would try seeing how it works at a LOW speed (like, below a steel turning speed) and later on then turn it up for the brass/aluminum/delrin or whatever you're working with.