I always recommend starting out with a .22 as a first firearm. It's cheap enough to shoot A LOT to become familiar with, and they tend to be very accurate for plinking. Plus, they're just plain fun. $15 will buy you a 500-round brick of bullets, and you can shoot all day long. My recommendations:
Ruger Mk. series
Beretta U22 Neos
Ruger Single Six
Heritage Rough Rider (can be had for less than $100)
Once you get used to it and are comfortable with handguns, then you can move up to something centerfire.
(The other option is to buy a 1911 or a Glock, slap on a .22 conversion slide, and then you have both options in one package.)