The problem is just that, the focus on power. The governor isn't worried about just anyone getting in his way. Some people (like the National Guard) he could have taken into his town if that were the case. The only people he kills are people who are threats to his power (like Rick). It's not just survival to him, it's maintaining his power. If someone else has power, it's a direct threat to his position in Woodbury.