Quantitative Research Methods

Guest blogger, Jonathan Solomon:

There are probably three truths I have picked up about quantitative research studies over the years, especially in comparison to qualitative studies:

1. Firstly, you need to talk to a lot of people which can make quantitative studies expensive.
2. Secondly, they take longer to set up as you wrestle with the questionnaire design and routing.
3. Thirdly, they are often taken as gospel by the business due to the mathematical and statistical analysis made possible by the resulting answers.

With these three things in mind, it’s important to set your quantitative research up correctly before pushing the button and releasing it onto an unsuspecting public.

Top 10 Tips for carrying out quantitative research

Here are my top ten things to consider:

1. Does your question fit with quantitative methods?

This has to be your first consideration. What exactly are you trying to measure? Quant is great for understanding the what, the who and the how, but poor at the why. This is due to the structured nature of the questions married with a fixed list of answers. Yes, you can have open ended responses, but these are harder to analyse (increasing cost) and take longer to answer, making the questionnaire longer, which can impact completion rates and cost.

2. How easy is it to replicate?

It’s very rare that you’ll want to measure something only once, so can you afford to run this survey on a regular basis? You need to consider methodology for your research, for example telephone surveys or face to face street surveys will be more expensive than a web survey. You should also factor in how many respondents are available (will you be able to find new respondents for follow up surveys).

3. Timing is key

Not just the initial survey but any follow on surveys. Running the survey on a regular or continuous basis will enable tracking of trends, spikes and dips through time. Continuous is the easiest…with respondents constantly sought…but it’s expensive (money and resource).

When thinking about timing, consideration should be given to what you’re measuring. For example, if you’re interested solely in the impact of a new advertising campaign (in itself and in comparison to previous/competitor campaigns), then you can just run your survey once the campaign is completed.

4. Do you have enough respondents for the data cuts required?

Every data cut reduces the volume of people being measured, which in turn reduces the statistical significance of the results. If you know that you’re going to want to analyse by age bands, gender, geographic location etc… then make sure you have a large enough pool of respondents to start with.

5. The significance of significant

Linked to the point above…in basic terms, statistical significance measures the likelihood of an observed behaviour (result) being pure chance or an actual finding. You’ll usually be measuring the significance of the difference between two or more results (boys versus girls, advert A versus advert B, old versus young). Without getting into the science, the more people you ask, the smaller the difference that can be measured as statistically significant. There is nothing worse than having a ‘key finding’ that cannot be proven and could just be chance.

6. Completion and responder bias

Quant surveys can be long, taking 10-20mins to complete. This puts people off from either starting the survey or actually reaching the end. The motivation for completion can also be an issue, with only the very happy or very annoyed taking the challenge. Methodology can also cause problems, with the classic example being the belief that email and web surveys will not reach older respondents…this is probably still true for some people in their eighties and above.

To help avoid these issues…

• keep your survey as short as possible – be ruthless with questions that do not add value (shorter surveys also tend to cost less to run)
• run a prize draw -something that has mass appeal with a good chance of winning (1 in 3 chances to win £100 Amazon vouchers)
• keep the survey interesting – easier with online surveys (graphics, video, audio etc…)
• consider your audience when choosing the methodology – you can see the clipboard wielding researchers on the streets ignoring some passers-by whilst making bee lines for others. This is because they have set quotas to fill… much easier when you can see the potential respondents.

7. Data overload

One thing quant studies are great at is generating data, although this can make them quite daunting to manage. You not only have the answers to the questions, but the ability to cut the answer by age, gender, socio-demographic, through time trend, YOY (year on year) comparisons, regression and correlation analysis… the list goes on.

You need to choose your research agency wisely, making sure you fully understand the analytical toolkit they have to offer. This toolkit should not only include the people capable of organising and analysing the data, but also a tool that you can use to interrogate the data yourself, displaying the answers in easy to use tables and graphs, online portals are brilliant for this.

As Einstein once said: “if you can’t explain it simply, you don’t understand it well enough”, and your internal stakeholders will not have faith in the findings.

8. Comparability with other findings

The best decisions are made from multiple inputs. Your quantitative survey should be one of these inputs, not the only one. Where the answer to the question is fundamental to your business, ensure it’s asked in more than one place. Mirroring questions in multiple surveys will provide additional confidence for big decisions.

9. Flexibility

One of the great things about quant surveys is the ability to flex how it runs. You can add questions, boost the volume of respondents from certain groups or locations, route people down different question paths etc… This flexibility can provide answers to new questions cheaply and quickly without the need to commission a separate research piece. Work closely with your research agency to ensure you have the functionality for flexibility built into your study.

10. Sample boosting

The majority of surveys will look to mirror the make-up of the general population, with respondents pulled from national panels. If you’re looking for the views of recent customers, this might not work for you:

• If you hold a 10% market share, then you should reasonably only expect to find 10% of respondents who are customers. Consequently, you’ll need to talk to 1000 people to get 100 customers (and that is before you cut by age, gender etc…)
• If your repeat purchase cycle is long (white goods for example), then again you may struggle to find a significant number of recent customers

Boosting the sample with your own data will compensate for this issue, ensuring you have a robust number of customer responses. Your research agency will be able to compare the customer sample to the national representative panel data, flagging any quirks that should be taken into consideration before combining the results.

Quantitative studies have the potential to become the backbone of your research programme, generating ongoing data for years to come. Setting them up correctly ensures that they continue to add value and statistically measurable customer feedback to empower key business decisions.

About our guest blogger, Jonathan Solomon

Jonathan Solomon is an experienced Head of CRM and Insights. Having worked for almost 15 years within the marketing and research teams of Vision Express, Citi Bank and E.ON, Jonathan has a good understanding of how market research can guide business strategy.

To find out more about how we can help you carry out effective market research for successful business decision, get in touch with us.