New Technology Changes Online Research
Thursday 24th September 2009 | 3:18 PM« Back to Listing
The global industry is looking for answers and solutions to improve quality in this important and growing research methodology.
"There is an urgent need, recognised by all our companies and by many of our clients, to raise data quality standards in the industry and conduct more ‘research-on-research' to understand what has to be preserved to maintain the predictive power of our work." Eric Salama, CEO Kantar Group, WPP
In Australia today more than 33% of market/media research is conducted online.
In a recent report published by ESOMAR the market research turnover for 2008 represented $US727m in Australia, which means online research was worth $US232m.
Conducting research online brings lots of advantages, including speed, apparently lower cost than other methods and so on and the majority of consumers are either online themselves or have fairly easy access to or understanding of the internet.
But marketers and media owners eager to improve their bottom line by making the right decisions based on research data need to demand just as much quality and rigour as is applied to other more traditional research methodologies.
The online world is exciting and allows us to do things not formerly possible but it cannot be exempt from standards if online research is to gain the credibility it deserves and underpin critical business decisions in the future.
The problems emerging in the online world are not really new and they are based on the fundamental rules of good sample, regardless of methodology.
Many leading marketing companies have formed online panels or communities of customers, as have most ,if not all, leading research companies. And there are what is called sample aggregators who can put together qualified target samples for sale to their clients.
In Australia, in common with the rest of the world, about 0.25% of the population complete 30% of all online surveys. Right now that is a very very small pool of people responding to all surveys and informing major spending decisions. If 30% of the respondents in a sample are invalid/incorrect/fraudulent that the data are likely to be as much as 50% wrong.
"I can go on an online panel and takeeight surveys today, using a differentname for every one. That's just unacceptable" Eric Salama of Kantar isn't an eight-a-day man.
There are a couple of ways online research can be wrong. Esomar (ESOMAR is the world organisation for enabling better research into markets, consumers and societies) has advised that 62% of online panel members in Australia, and the rest of the world, belong to multiple panels.
There have been a number of studies on the impact of this on research results and there is a consistent view that these ‘professional responders' are significantly more positive in their responses to questions in the hope that they can earn more money from completing more surveys, or free product trials.
A particular problem with part time and lower income people who use survey responding as a regular source of revenue. This was found to apply to both consumer research and business to business research. But if you simply removed or de-duped respondents to any survey that does not fix the problem in fact the opposite could be true. There are other problems.
People who occasionally complete surveys responding to an online invitation generally underscore on many questions because they are more critical, bored or don't focus on the questions, some use a false identity to respond and work for the competitor company or a market research company.
It is hard to know and impossible to validate until now. The majority of online panel companies go to considerable trouble to clean up their own panels but quite often, when there is a niche or hard to reach target, they need to buy sample from another or more than one other panel, and till now there is no effective way to test the quality of that combined sample.
One of the reasons for this is that different samples are recruited differently, and generally not in the accepted method of ‘random sampling' which is critical in the more traditional methodologies in order to achieve representativeness.
Each online panel created has its own bias and online panels are not subjected to any industry standard audit. The market research industry in Australia has started the regulation process by establishing the Quality Standard for Online Access Panels, and this initiative will no doubt continue in development as ‘research-on-research' clarifies issues and solutions . But this is a world wide problem.
ESOMAR is monitoring the ethics and quality of online sampling. Some online research companies have gone some way to address the issues by using cookies to identify duplicate email addresses, however, cookies can be deleted and many people have multiple email addresses.
Some forms of digital fingerprinting have been developed in the US and these are often used in conjunction with cookies. The Advertising Research Foundation in the US are so concerned about quality that a very high profile Online Research Quality Council has been established and on 29 September 2009 will be releasing a report on its research into the ‘online research issues'. The co-Chair of this Council is Stan Sthanunathan, VP of Marketing Strategy and Insights at Coca-Cola.
Research conducted on a proprietary study of technology brands in Australia recently revealed that the online sample, when compared with a validated sample, was seriously wrong not so much because the demographics were wrong but because the online sample was significantly over represented on one psychographic (attitude) segment.
Whilst it is difficult to see how this might have been resolved for this particular study, it highlights the critical need to have a much larger universe sample pool from which research companies and marketers can draw.
Having a much larger universe will reduce this problem as well as alleviating over-use of the professional responders. A team of young Australian entrepreneurs have spent close to $1m over the last two years to develop a real alternative approach which can passively fingerprint computers (without cookies) to a higher level than other comparative services, and, to an extremely high degree of efficiency and within milliseconds, de duplicate within single and across multiple panels.
Technology is also now able to detect and manage poor responders or those who just click answers in line or a pattern which demonstrates they are not taking the task seriously. The two systems combine to significantly improve the quality of online sample.
The value of this ‘quality or fraud management' on online research and consequent business decision making is profound.
The SampleWorx business launched just over 18 months ago and has has applied a similar new technology approach to landline telephone random digit dialling and mobile phone random digit dialling. Reducing error rates in telephone sample numbers from up to 50% to virtually zero.
Online is the third platform for SampleWorx. SampleWorx will launch its Online sample marketplace at the Australian Market and Social Research Organisation Conference on 30 September 2009, with an open invitation to all buyers and sellers and will roll out globally within the next few months.
SampleWorx has created this online research sample online market place for Australia to allow the buyers and sellers of online to trade in a quality environment. The goal of SampleWorx is to improve and grow the quality of online research.
Buyers of sample can come into the market place and log their projects. Suppliers can bid for jobs on the basis of both price and quality. When sample is delivered to buyers it is automatically processed through digital fingerprinting technology. SampleWorx places quality control in the hands of buyers.
Online research is a multi million dollar business growing every year. It is vital for both the buyers and sellers of research to have quality sample given the billions of dollars of business decisions at stake.
Check it out at sampleworx.com.au
Quote from Charlie Nelson: "There is much merit for the SampleWorx approach. Millions of dollars worth of business decisions are based on data which needs to have as much integrity as possible. I am supporting the view that it is beneficial to have more than one panel supplier and even more than one approach to ensure the data are based on as representative a sample as possible. I would be expecting my research suppliers to be using this approach"
Charlie Nelson is the managing director and founder of foreseechange.
Our global industry association ESOMAR has highlighted the need to "clean up online sampling" (my words not there's) but in my view it has failed to have any impact at all to date. If this new system gets a foothold, then not only will it be supported by the market research industry, the number of respondents prepared to participate in market research will also increase. It seems to me that this approach is definitely heading in the right direction. Rick Wilson, Principal OMP Worldwide
For further information contact:
Press release published by Seeking Media. http://www.seekingmedia.com.au/