Is online research compromising quality?

Helen Gawor, Connect Insight, says: “Online advertising effectiveness studies became a joke when one research supplier was asking respondents ‘Have you seen any advertising for X brand’ when the ad was clearly on the page hosting the research… when it’s bad, it’s really bad!”

In tune with how we have changed the way we communicate and share information, the research world has adapted to new and often cost-effective methodologies online.

So with 70% of the UK population online, why are online methodologies so hotly debated? The final MRG Conference debate will address the issues surrounding online methodologies and perceptions of quality. Many of us are users of online methodologies, but if you ask us honestly, we often feel we have to over justify their contribution to the research plan, and quite often, if the budget is big enough, we will default to more established methodologies that we feel safe using.

Online panels in particular have come in for their fair share of criticism over the last five years or so. There are two types of online panel. The first consists of respondents all collecting data for the same purpose on a continuous basis. The UKOM panel is a good example of this. Panel maintenance and data collection processes are key to ensuring data quality and, in an ideal world, this would be true of all online panels.

But it’s precisely these issues that have called into question the second type of online panel. These panels offer a pool of respondents who are managed by a panel provider and offer a population of opted-in respondents to derive a sample from.

Critics with experience of these online panels highlight that contact with respondents is often too frequent and that they are being invited to take part in surveys that they wouldn’t qualify for had the survey been targeted according to information already given to the recruitment screener. Many of the surveys as a result are poor in quality, leaving them wondering what role the panel provider plays in ensuring that surveys conform to industry guidelines and accepted best practice.

However, this is not true of all online panels. As researchers, we find them an efficient and cost-effective way of conducting surveys, particularly for global projects and when we need quick results. For those of us capable of writing respondent friendly questionnaires, we don’t worry too much about the output.

And the book doesn’t stop at panels when we are evaluating online methodologies. When I first started working in digital media, everyone wanted to conduct research online, whether it was to profile an audience, measure ad effectiveness or just collect information about the internet population. Thankfully, surveys online are usually infrequent enough these days to not be an annoyance to potential respondents.

I used to get quite a bit of stick for capping the volume of surveys on the Orange portal, and being strict about the length of surveys. Because of this though, we had a pool of interested potential respondents on the site and never worried about response rates. We still see a fair amount of bad practice in this area, however. One of my favourite broadcaster’s seems to have user profiling surveys on its website irritatingly frequently and it doesn’t cap how many times I’m invited to participate. Online advertising effectiveness studies became a joke when one research supplier was asking respondents “Have you seen any advertising for X brand” when the ad was clearly on the page hosting the research.

So what is going on? We get testimonies of good and bad experiences of online methodologies. It clearly is the way forward in reaching and engaging with respondents in line with current communications trends and will benefit the market research industry as whole.

But when it’s bad, it’s really bad. Have we got sloppy because we no longer have to look our respondent in the eye? Is it that we fail to grasp the basic concept that respondents are communicating with us through a very personal medium, and they should be surveyed as respectfully as if we were sitting in the front room conducting an interview?

The MRG conference debate intends to get under the skin of some of these issues and perhaps dispel some myths associated with online methodologies. Hopefully we can surmise some recommendations that will instil more confidence in users and buyers of online research. We can then also apply our learnings to emerging digital data collection tools like mobile.

Helen will be speaking at this year’s MRG conference in Malta from 3rd to 6th November. Click here for more information or to book your ticket.