When dealing with professionals, I find that Theory Y style management is best. Theory Y assumes that employees want to do a good job. It assumes money is not a motivator, so intangible rewards are best.
Theory X style management assumes employees are lazy, need constant direction, and are motivated by fear of punishment and cold hard cash.
When dealing with people who you need to be proactive and make their own decisions, a positive, rewarding, encouraging atmosphere will foster more leaders in your team, and that's what you need as a PM.
If penalties are used, they shouldn't be anything drastic, demotivating, or something that will make the team members job incredibly difficult. In software teams, the person who breaks the build usually is responsible for having to fix it. Although it's a type of penalty, the developer will learn from the mistake and also be able to take the leadership role on coordinating getting it fixed.
Another type of penalty might be putting a penny in a jar each time another developer finds a bug in your code. Since it's more like a game where the person with the most pennies loses, it's actually quite motivating.
Maybe the loser in the group has to be the one to go get coffee for the others that week.
To use an example outside programming, when I was in the Army, if I forgot something on my uniform, the penalty was 20 push-ups. It helped me stay in shape, served as a reminder for next time, and was a sort of game that we all laughed about, even with our Sergeant.
I would avoid financial penalties at all costs, as this is someone's livelihood! Everytime Roger Goodell fines an NFL player for making a mistake, I cringe.
UPDATE:
For rewards, the reward should also be intangible. You're not dealing with rats in a cage that you can shock and feed arbitrarily based on their choice of paths. Instead, the reward could be work related, such as giving the best engineer the opportunity to lead the team presentation of a new product feature to the CEO or a client. A reward such as this is work related, so it actually encourages the behaviors that may have helped the employee get to where they are in the first place.
Another example of a great reward would be more freedom to make decisions on the project independently. If an engineer has proven him/herself, then let him/her have more of an independent role on a project. Make this clear so that everyone knows this priviledge/right of passage was earned. (This is more for junior engineers. Senior engineers should already have earned this right.)
Someone else (Craig Villacorta) said recognition, which is what I'm alluding to with these examples, so I won't beat a dead horse.
In summary, keep in mind that while you are dealing with professionals who naturally will do their best because that is who they are, that doesn't mean these professionals are dry and don't like to have fun.
As long as the reward/penalty system is a fun thing, it can be a tool used to help improve morale.
Think reputation on Stack Exchange sites! Sure, most of us are here to help people, improve our writing skills, and learn from others, but isn't it just a little fun to see your reputation score go up??