Online community health is a geeky passion of mine. How do you define it, or measure it? How can that data assist you to run a better online community? However, I keep running into a desire to establish a baseline, set goals, and ‘move the needle’ in some way to improve that rating. I think that can be a dangerous approach.
Don’t get me wrong: targets are helpful things to have. They help to focus work, and they’re often well-crafted simplifications of complex ideas. But please, don’t turn all of your measurements into targets. Goodhart’s Law applies here – famously paraphrased by Mary Strathern as:
When a measure becomes a target, it ceases to be a good measure.
It’s also discussed in detail in this post from LessWrong, who give the following more lengthy version:
Once a social or economic measure is turned into a target for policy, it will lose any information content that had qualified it to play such a role in the first place.
Why is that?
Let’s say that you’re trying to measure something nebulous like ‘community health.’ You’ve distilled it down to one key aspect, and decided on a way of measuring it. Of course, you’re not directly measuring the health of the community: you’re measuring a proxy, that happens to correlate pretty closely with it. That’s fine. Or at least it is at the start of the process…
Unfortunately, well-meaning or otherwise, people like making changes that produce improvements in the thing you measure. Particularly when you have some bold, succinct, easily-understood goals like “increase the Health score by 10%.” People want to impress their managers; teams like succeeding at their goals in view of other teams. Given the chance, you’ll throw everything you have at raising that score. If some quick wins come from exploiting flaws in the system, who really cares? You might not notice those details. You boss’s boss almost certainly won’t.
So, what’s the problem?
The more you optimise your activities around hitting those goals, the more your data reflects how well you can game that system. You started out wanting “a healthier community.” Soon, you end up spending all your time chasing likes/retweets. Or trying to close support tickets in 10min instead of the next measurement bracket up. Maybe that actually helps. Maybe you lose sight of the original aim, while chasing the numbers.
How can we fix that?
Some measurements are better off as measurements – so don’t make them be something they shouldn’t be. Design them in a way that allows them to reflect the work you (or your team, or organisation) are doing to provide a healthier community for your members. Don’t force them to become arbitrary targets.
Also, make sure that the data you get will actually help you to take action. I’m assuming that you’re involved in this process because you want to improve a community. Your chosen metrics should help you to identify what you need to do, in order to create that improvement.
In my next post, I’ll look at Sense of Community as an example of a health metric that addresses that last point.
