Research and measurement – how to do it authentically and avoid regret

How arts organisations collect and analyse data is increasingly important. Here's some advice on doing it right.
[This is archived content and may not display in the originally intended format.]

Image: shutterstock.

Ian David Moss and with Jackie Bailey are in the business of helping people make better decisions. 

Moss is CEO of Knowledge Empower and Founder of Createquity, and Bailey is a research, strategy and evaluation consultant at BYP Group. At the centre of what they do is a rather simple idea: to collect the right information and use it purposefully.

But like most simple ideas, it’s a hard one to get right.

‘Data plays a pretty major role at the moment, but I also think there is a difference between the role it does play and the role it should play,’ said Bailey.

Bailey and Moss discussed how data can be used effectively in the arts, particularly when it comes to driving social change at a recent Creative Exchange event — a professional development series run by Creative Victoria. 

To find out more, we asked them for advice on how arts organisations can make better, more informed decisions.

1. Avoid regret

‘Decision making is a very intuitive process. We’re used to making decisions without thinking about them that much,’ said Moss.

While we might make the majority of our decisions quickly and through intuition, there are always a few that require more time and thought. ‘The trick — the hard thing to do and the non-obvious thing to do — is to distinguish the specific types of decisions that deserve more time and a deeper level of analysis,’ he explained.

But how do you decide which decision needs more time?

Figuring this out is an important first step in making better decisions — a skill that can quickly determine the success of an arts organisation. Lucky for us, Moss shared a thought experiment he takes clients through during his consultation work to help them categorise decision making priorities, which he calls a ‘decision inventory’.

‘I help organisations do a decision inventory where I work with them to really think about what decisions the organisation might need to make over, say, the next six to twelve months. We then go through it and place these decisions into categories, along the lines of what kinds of stakes might come with those kinds of decisions.’

The higher the stakes, the more important the decision is and places it as a higher priority than the others. 

‘A single executive doing the work on their own could do that kind of thought experiment in maybe an hour. It is a very high leverage activity, but it’s something most people don’t do. They don’t think explicitly in those terms, “What are the decisions we need to make?”

‘It is very rare for organisations to have a process around categorising decisions and thinking about what the consequences of various decisions might be.’ 

2. Two by two

Sometimes, Moss introduces a 2×2 grid to help organisations prioritise decisions based on what consequences might occur if they fail. In other words, what’s the risk involved if the decision you make doesn’t work out?

‘On one axis there is the likelihood that the decision is not going to yield the outcome that you want, so what’s the risk that it’s not going to go the way you hoped. And on the other axis is if it didn’t go the way that you hoped, how bad would that be?

‘It’s not necessarily something you have to attach numbers to, but if you plot those decisions on a grid like that it will very quickly show you what kind of decisions really deserve that greater level of analysis because if you make a bad guess on a higher stakes one where there is also a significant risk that it’s not going to yield the outcome that you want, then you could end up having some real regret there,’ he said.

3. Ask the right questions at the beginning

When it comes to collecting research — and if you are receiving public funding — it is important to put the audience interests first and find out what the public needs and wants.

One thing Bailey has noticed across her professional experience is a tendency for organisations to use research to justify what they do, instead of determining what they should do.

‘Instead of saying how do I build my audience with what I’m already doing, we ask what does the public actually want and need, and how can you use that to inform what you’re doing, not the other way around,’ Bailey explained.

‘If organisations and arts funders did that, we might be getting a lot of different art,’ she added. 

4. Measure your impact in an authentic way

‘Artists and arts organisations always say there’s stuff you can’t measure. I don’t actually think that’s true,’ said Bailey. ‘You can measure everything, as long as you understand that measurement is not about numbers, but about change.’

Bailey explained that measuring the impact of the work you create comes down to how you think about measurement. Over the years, she has adopted a vocabulary that lends rigour to impact in the arts — asking how does a work impact people’s hearts, bodies, minds, spirit, and also the world.

Heart means how you change the way somebody feels about themselves and the world. Mind is about whether there is a change in attitude and cognitive impact, so have you changed a person’s knowledge, skills, self-efficacy, attitude, views? Body is the behavioural impact. Have you actually changed what people do?  And spirit is have you changed the way people experience life? The idea of aesthetic change, or how all these changes actually coalesce to change the person. And then the world – how does this all change society, culture, and community?’ 

‘We think of it in very real terms that make sense to artists and arts companies, so we can use that language and also translate between the language of the more academic end of the research spectrum —  like self-efficacy, efficacy-in-community, aesthetic change, or behavioural impact. 

‘I think artists can often be scared of measuring impact but they don’t need to be because they can do it in an authentic way – a way that makes sense to you and actually helps you do what you want to do, which is change people’s lives.’ 

Brooke Boland
About the Author
Brooke Boland is a freelance writer based on the South Coast of NSW. She has a PhD in literature from the University of NSW. You can find her on Instagram @southcoastwriter.