I can see wanting to remove temporary bans ahead of time, but those who were permanently banned had a good reason for it. Either they were hacking or constantly breaking the rules to the point that there was no return for them.
I see no benefit in removing the permanent bans. Sure, our population might spike up, but the server would be a cesspool of trolling and hacking which wouldn't make it any more fun than it is now.
Smidge Griffkin has suggested that we review past bans and decide on a case by case basis who we will forgive. I think this is a good compromise.