Community health: measurements and targets

Online community health is a geeky passion of mine. How do you define it? How can it be measured? How can that data assist you to run a better online community? However, one concept I keep running into is the need to establish a baseline, set goals, and ‘move the needle’ in some way to make that rating improve. I think that can be a dangerous approach.

Don’t get me wrong: targets are helpful things to have. They help to focus work, and they’re often very well-crafted simplifications of complex ideas. But please, don’t turn all of your measurements into targets.

Goodhart’s Law applies here – famously paraphrased by Mary Strathern as:

When a measure becomes a target, it ceases to be a good measure.

It’s also discussed in detail in this post from LessWrong, who give the following more lengthy version:

Once a social or economic measure is turned into a target for policy, it will lose any information content that had qualified it to play such a role in the first place.

Why is that?

Let’s say that you’re trying to measure something nebulous like ‘community health.’ You’ve distilled it down to one key aspect, and decided on a way of measuring it. Of course, you’re not directly measuring the health of the community: you’re measuring a proxy, that happens to correlate pretty closely with it. That’s fine. Or at least it is at the start of the process…

Unfortunately, well-meaning or otherwise, people like making changes that produce improvements in the thing you measure – particularly when you have some bold, succinct, easily understood goals like “increase the Health score by 10% this year.” People want to impress their managers; teams like succeeding at their goals in view of other teams. Given the chance, you’ll throw everything you have at raising that score… and if some quick wins come from exploiting flaws in the system, who really cares? You might not notice. You boss’s boss almost certainly won’t.

Problem is, the more you optimise your activities around hitting those goals, the more your data becomes a reflection of how well you can game that system. You started out wanting “a healthier community” and ended up spending all your time chasing shares/likes/retweets, or trying to close support tickets in 10min instead of the next measurement bracket up.

So, some measurements are better off as measurements. Design them in a way that allows them to reflect the other work you (or your team, or organisation) are doing to provide a healthier community for your members, but don’t force them into arbitrary targets.

Also, make sure that the data you get will actually help you to take action. Assuming that you’re involved in this process because you want to improve a community, your chosen metrics should help you to identify what you need to do, in order to create that improvement.

In my next post, I’ll look at Sense of Community as an example of a health metric that addresses that last point.