A Little Bit Of Data is a Dangerous Amount

When you have no data, everyone agrees: need more data.

When you have a lot of data, what is happening is pretty clear.

When you have a little bit of data, people can extrapolate. “It might show X”, “It might show Y”. Often declared without the caveats. Because “we don’t really know” is a much less compelling story, even if it is more accurate.

But… we don’t really know.

If you’re measuring the performance of a layout on your menu bar, with some actions exposed and some hidden away in a submenu, and you know that people more often tap the exposed options, you might declare success.

But. A little more data might show people cancelling those actions disproportionately more.

So now what do you know? That people aren’t always finding what they are looking for first try, that those options are not necessarily the ones that should be exposed.

The answer is logging everything and (I would hope this is obvious) to the same place.

And, when you think data has backed up a conclusion… think about whether you have all the data to really be sure about that.