Any ideas about how to evaluate a developer's performance?

Question

I introduced Scrum in our development teams some years ago. Since then our performance has increased a lot with a shorter time-to-market and increased quality. The CEO wants me to create a system to measure the individual performance of every developer. Our company is very results and performance oriented and my department resisted these kind of benchmarking for some years. This year we should introduce such a system so that we can start using it next year. Have you got any ideas / experience with a system that could work?

Answer 1

I've seen individual evaluations for programmers done reasonably well, and horribly wrong. If leadership only ever evaluates a team as a whole, then that team has no way to reward individual initiative and identify/remove personnel who are detrimental to the team. However, if leadership treats the team members only as individuals, they stop being a team.

I can't offer you a perfect system, I'm afraid, but here are some heuristics I use when I'm creating incentives for programming teams I lead, and an example of how the technical team I'm currently on evaluates members:

• This may be difficult to sell to your CEO, but: Never, ever, EVER set long-term or recurring goals based on statistics. If you do this, your best programmers (who are by nature your best game theorists) will go down in work quality, because they will align their work to minmax on the statistical measures rather than the goals and behaviors those statistical measures were a rough proxy for. It's fine to have short-term, very specific quantitative goals, but never long-term or recurring ones.

  Bad Example: Programmers get a bonus based on the number of bugs closed per quarter. This incentivizes work on trivial fixes and disincentivizes tackling big problems, or doing refactors and testing work with long-term payoff. It also disincentivizes vital, but hard-to-measure work like cross-mentoring teammates, researching problems, or collaborating with others to solve problems on bugs they will ultimately close.

  Good example: Program $foo ships in two months, and is currently a laggy processor hog, so every programmer who comes up with a fix that provides at least a 2% performance gain (without unacceptable side effects) before the code freeze gets a bonus. This is short term, and solves a specific, immediate problem. It won't rack up technical debt over the long term or twist developer behavior. It will provide focus, and show that extra effort on a concrete problem is appreciated.

• Don't try to create a linear scale from "good programmer" to "terrible programmer". Not only does this implicitly encourage employees to compete with one another, but it minimizes the benefits of having a well-rounded team with different strengths and specialties.

• ALWAYS provide cooperative goals in addition to individual goals. If you want your team to behave like a team, it helps to treat them like a team. They should have shared goals and objectives where they all have some skin in the game.

• ALWAYS make sure that there are professional development opportunities to back up performance goals. One of the fastest ways to kill morale is to provide performance incentives but not provide a clear way for employees to increase their capacity to meet those incentives. This means that training, conferences, and so on need to be covered by the company. This includes the time to actually do the professional development.

• Performance measures should be flexible. Both of the best bosses I've had have included in each performance review the ability for the employee and supervisor to agree that a goal was inappropriate or not applicable. This may happen because the organization's needs changed after the goal was set, because in retrospect the goal was not well enough defined, or because in pursuing the goal it became apparent that a shift in direction was needed for some reason. Then that goal was not used in performance assessment. I've absolutely learned and applied this when evaluating people I manage.

• It should be clear to everyone what performance measures will and will not be used for, and with whom they will be shared. Confusion around these issues can cause a great deal of stress and drama in any workplace.

• It is important that performance measures are not used as a replacement for dealing with problems in the workplace on an immediate basis. Dealing with ANY problem six months after it begins to be a problem is far too late.

• Encourage each team member to keep a running list of what they've done (this is sometimes hard, the childhood "don't brag" lessons stuck strong in many of us) and add accomplishments to every performance review. This is useful for team morale (it shows that the manager is doing their best to ensure that everyone gets credit for their work), and for making sure that things don't slip through the cracks.

• Depending on the pace of operations, reviews should happen monthly, quarterly, or biannually. Annual reviews are FAR too infrequent to be useful or helpful, and going more frequent than monthly introduces too much jitter from the occasional bad couple of days or rockstar couple of days when one should be looking at the big picture.

Generally, a good manager who's doing regular goal-setting and professional development activity with employees knows what everyone's performance is like already. Formalizing it a bit can be a useful tool when, for example, advocating for professional development resources, making the case to promote someone or let someone go, or providing raises and other incentives. The addiction to statistics is a dangerous one. It can be a little dangerous in the best of situations, but dealing with programmers (all of whom are at least a bit good at game theory) it is disastrous--a recipe for minmaxing your team into oblivion.

Once you have performance evaluations for your team, the big question is what to do with them. If they are used in a dysfunctional way, it can and will destroy your team. I see it all the time:

• Zero-sum games break down cooperation and camaraderie by pitting team members against one another, reducing performance overall.

• Pressure induced by low salaries balanced against big performance bonuses can get the 10% or so of programmers who are uber-competitive to generate short-term bursts of productivity, but will burn out the other 90% who find such environments stressful.

• The feeling that making a mistake will be held against one forever, or that one will be labelled in an unchangeable way will thwart management's attempts to foster improvement in underperforming employees.

• Fear about how performance reviews may be over-interpreted or used for social jockeying will cause employees to reduce communication with their managers, and even self-censor among their peers. They will disengage from the review process (in self-defense) and their professional development will be hindered.

• Employees who feel they "can't compete" my simply check out and do the bare minimum they need to keep their jobs.

• Employees who see an evaluation and/or reward system they feel is unfair (even if it's unfair in their favor) will lose sight of the team's real goals and get absorbed in frustration with that system. They'll have trouble finding meaning or a sense of accomplishment in their work.

That said, if you do it right, an evaluation system can have positive effects:

• By encouraging your programmers to formalize goals and track accomplishments, you can get them to think about their growth as programmers. Encouraging this can combat the tech industry culture of "promotion by job-hopping" and help you retain employees by creating a basis for promotion and evolution of their careers within the company.

• By encouraging your programmers to compete against their previous best selves, to be ambitious in tackling specific, concrete problems that the team is facing, and to solve problems and reach goals as part of a team, you can create an environment where employees feel rewarded for their effort without feeling like they are being "ridden hard".

• The information gained in this process can be invaluable for advocating up the chain for things your team needs.

• The ability to not reward someone who's a drag on the team while still supporting them in getting better can improve the morale of the people who are picking up the slack in the short term.

• Regular reviews of individual goals can help both you and your programmers notice when you have mismatched ideas about what current priorities are.

At my current workplace, my role in evaluations (apart from being on the receiving end) is purely informal. I manage some student research assistants from time to time, but we don't have much we can do based on their performance aside from keep them on or not, write awesome letters of recommendation or not. I do make a point of telling someone's supervisor if I think they've done something noteworthy, because we have that kind of culture, and I was one of the people who pushed back against 360 (peer evaluations) due to the potential politics involved. (Our center ultimately stopped doing them.)

That said, here's a rough description of what my supervisor does, which I really like:

1) He asked each of us how we like to be praised/rewarded, and followed through on the answers.

2) Each employee writes a "career management plan" in collaboration with our supervisor upon starting with the org. This CMP includes an outline of the employee's strengths and skills, their accomplishments up to this point, and their goals (both in terms of "things to accomplish" and personal professional development) for the medium and long term.

3) The CMP is reviewed by the employee and supervisor each six months (we're experimenting with quarterly for some employees, results TBD). That review involves adding new accomplishments and their ramifications, noting which goals were accomplished and how well, and why other goals didn't happen. Goals that didn't happen aren't assumed to be failures: they may have been poorly defined; priorities may have changed; they may have stalled or stopped due to something completely outside the employee's control; they may just be longer-term projects that are in progress or deferred. If something was a failure, chances are that problems are already known (we're very good about dealing with things when they happen so we don't build up the social equivalent of technical debt) so the CMP review usually involves reviewing what has changed since then.

4) The CMP is updated with new info from step 3 and filed in the HR file; a new CMP is written for the next review period.

5) My supervisor uses the CMP process to inform his decisions regarding performance-based rewards, but I daresay there are other things he takes into account as well. Unlike most other places I've worked, this is not a for-profit company, so we don't hand out much in the way of bonuses. Reviews can impact promotions, what projects one is assigned to, and annual raises. No, there's no "get this many points or you get a terrible project and no raise" system. If there were, the two of us who are the most cynical and the best at game theory would get the best treatment no matter how hard our colleagues worked, and at least two of our best people would get mad and quit.

Yes, that's extremely subjective. It rather has to be: I'm doing information security full-time now, and our team is extremely diverse. Not counting the administrative staff, we have a lawyer, a programmer, a sysadmin, a compliance expert, and me (hacker-of-all-trades). We're all doing information security, but we all do very different things. No objective grading system could exist that wouldn't favor one specialty over another, and thus there could be no objective system that we'd find fair...yet the mission needs us to all specialize in different things.

Answer 2

What I have seen tried:

Measuring story points achieved by a developer - As soon as this was introduced the developers stopped talking to each other and focused only on the story they were working on. Output of the team dropped.

Peer review / 360 feedback - This was rapidly gamed. Team members talked privately and made deals. Often reviews were substantially better for friends of the reviewer than for people they did not get on with.

Team leader review - Once again, this was clearly biased by personal relationships. One top developer was given a bad review by his team leader because they did not go to the pub with the rest of the team.

Performance measured against objectives - When the appraisal time approached some team members dropped high priority ongoing project work to focus on their objectives as they realised it was going to contribute to their pay review.

Of these the only approach I could see being made to work is the last one. The objectives would have to be carefully designed and reviewed regularly througout the year.

Alternatively I would say that review at the team level is much easier to do and encourages the team to work together.

Answer 3

First Off:

Don't tie whatever you do to any form of compensation decisions. As soon as you do this you will get 100% gaming of the exercise.

Measuring individual performance is very difficult and probably more costly than the derived benefit. But if you must, I'd recommend some type of peer appraisal process (aka a 360) combined with having the developer evaluate their own performance and if/how they are continuously improving.

You may want to read the book "Abolishing Performance Appraisals: Why They Backfire and what to Do Instead" as a primer on alternatives and the pitfalls of appraisal. Maybe your CEO should read it as well ;).

Really what you want is an organization where everyone is good at giving and receiving feedback. Lots of healthy social connections leads to a better understanding of how we are doing, positive peer pressure, and will quickly weed out most people that aren't performing.

Answer 4

If you look at the news from some of the largest software and consulting companies you still see trend of not going individual appraisals.

Individual appraisals kills self organisation and both the performance of your team and the quality of your code.

I would have a serious conversation with the CEO about what what his goals are. If he wants high quality software quickly then he needs to stop asking for individual appraisals:

http://federalnewsradio.com/commentary/2015/08/lets-kill-the-performance-appraisal-process/

That said, if he really does not understand how people work and how detrimental appraisals are you could try:

http://scrumology.com/how-to-do-agile-performance-reviews/

1. Give all of the members of an Agile team the same performance goals.
2. Assign individual goals for individual development.
3. Make sure individual goals are aligned with team goals.
4. Frequent 360 degree performance feedback is better than many alternatives.
5. Value highly the personal traits, characteristics, and behaviors of good team members.

The above should always be considered a stop-gap to fixing the route of the problem...

Answer 5

If you want to measure performance, you will have to assign some value to tasks.

If you want to differentiate between people, you will have to assign them separate valued tasks.

The smallest evaluable task for a company is usually a project. If you have external clients, this is easy, because they will pay you X for completion of their project. Internal projects will usually have a calculated value to the business.

Projects are usually too big for a single person, so they have teams assigned.

Therefore you can easily calculate the value of teams by the value of the projects they complete.

To break this down into an individual value, you have to have enough teams and projects to allow you to switch the team members around frequently enough that you can statistically calculate their value.

Unfortunately (I need to do the maths here, but) if you have say 5 team members per team and say 3 teams, it's going to take maybe 100+? completed projects before you have anything significant.

Essentially, you are relying on people's CVs and references for info on their past performance, and your personal impressions for their current performance. Just like in sales and operations.

The only difference is they don't care whether their measure is accurate or not.