Has online research killed quality?

John Carroll, Senior Director, Ipsos MediaCT, looks more closely at online panels and research ahead of the MRG Conference in November.

This question is setting the theme for one of the panel debates at the forthcoming MRG Conference in Malta. I really enjoy provocative questions for a debate and this one certainly fits the bill. Indeed, on closer inspection, the question can be broken down into yet more questions.

How do we define online research in this context? Do we mean online as a method of sampling and recruitment? Most research agencies offer an online panel, either as their sole business offering or as part of a full service repertoire. Indeed, some media owners and media agencies have their own in-house managed panels too. Most, if not all, online panel providers will declare what quality criteria are met in recruiting and maintaining their panels. This is where it can get a bit interesting, or confusing, as there is no one universal quality assurance definition for online panels. And big does not always mean best. In an effort to help with this, ESOMAR have published 26 questions to help research buyers of online samples, which include areas such as recruitment method, compliance and validation.

It may be sobering to note that not everyone in the UK population is online; not everyone who is online is invited to join a panel; not everyone invited agrees to join a panel; not everyone invited to take part in an online survey does so. Which begs the question as to who have we got left and how representative are they of the population as a whole? As Katherine Page, NRS Technical Consultant, remarked in her recent MediaTel Newsline article, “…the view is that it is not possible to obtain a ‘stand-alone’ internet panel sample, which is representative enough to provide a media currency.”

So, online panels may not yet be deemed worthy enough in the UK to sit at the same media currency table as face-to-face and telephone – for now. But they can be considered fit for purpose for many other research needs, such as, for example, census staff surveys or the collection of sensitive information.

Do we mean online as a method of data collection? With care and attention, one can embrace web technology – with graphics, interactivity, video, etc – and design really engaging online surveys to maximise the quality of the output. Without care and attention then the quality of the output is diminished, if not destroyed, but this applies to all modes of data collection not just online.

Is it the researcher that is selecting online to solve the research problem that is killing quality, rather than the mode itself? Some may say that the rise of DIY online research or ‘less reputable’ agencies forcing prices down are making the adherence to quality standards very difficult. I have never forgotten Sir Martin Sorrell’s line on this from about ten years ago: “…what the client wants is…a quick and dirty answer…”. Online may be the quickest and cheapest mode for conducting a survey, which can often be the client demand, but it may not necessarily be the correct one to solve the particular research problem.

There is plenty more to say on this topic and I have certainly not presented all sides of the argument here. I will be presenting more on this subject at the MRG Conference as a prelude to what I hope will be a lively debate on the subject.

John will be speaking at this year’s MRG conference in Malta from 3rd to 6th November. Click here for more information or to book your ticket.