When you have no data, everyone agrees: need more data.
When you have a lot of data, what is happening is pretty clear.
When you have a little bit of data, people can extrapolate. “It might show X”, “It might show Y”. Often declared without the caveats. Because “we don’t really know” is a much less compelling story, even if it is more accurate.
But… we don’t really know.
If you’re measuring the performance of a layout on your menu bar, with some actions exposed and some hidden away in a submenu, and you know that people more often tap the exposed options, you might declare success.
But. A little more data might show people cancelling those actions disproportionately more.
So now what do you know? That people aren’t always finding what they are looking for first try, that those options are not necessarily the ones that should be exposed.
The answer is logging everything and (I would hope this is obvious) to the same place.
And, when you think data has backed up a conclusion… think about whether you have all the data to really be sure about that.
3 replies on “A Little Bit Of Data is a Dangerous Amount”
These numbers don’t mean what you think they mean – A Little Bit Of Data is a Dangerous Amount – http://t.co/JuglUrB7AT
[…] A Little Bit Of Data is a Dangerous Amount […]
[…] we had what I term a â€œdangerous amount of dataâ€. Which is worse, in some ways, than no data. Because whilst there isnâ€™t enough there to draw […]
[WORDPRESS HASHCASH] The comment’s server IP (18.104.22.168) doesn’t match the comment’s URL host IP (22.214.171.124) and so is spam.