7 Ways to Avoid Spreading Bad Information —
You’ve probably heard something like this before: A junior manager walks into a meeting and, trying to impress, announces "if the United States would only reduce oil imports from Norway, we could literally the reduce deaths of drivers colliding with railway trains … per capita." "It’s true, I got the data from the "X" US Department website and the "X" monthly journal of research."
This junior manager was met with blank stares, blinking eyes, and banishment to re-read Statistics 101. The poor manager fell prey to Lies, damn lies, and statistics, and while this story is an exaggeration, being misled — intentionally or incompetently — by statistics and graphs is all too common.
InvisiLight® Solution for Deploying Fiber
April 2, 2022
Go to Market Faster. Speed up Network Deployment
April 2, 2022
Episode 10: Fiber Optic Closure Specs Explained…
April 1, 2022
Food for Thought from Our 2022 ICT Visionaries
April 1, 2022Bad Polling / Surveys
Academics spend an inordinate matter of time and resources testing questions, question order, and question effects in surveys. For example, one of the largest political science studies housed at a major mid-western university, constantly welcomes and publishes pilot studies to explore "new methods and new substantive instrumentation" against their own survey questions, and they have been conducting political surveys continuously since 1948.
Survey questions can go astray any multiple of reasons — leading / biased questions, poorly worded questions, force choice questions, and questions running head into social desirability — to name a few.
Survey questions should strive for neutrality and always offer an "I don’t know" option. Great care should be taken when approaching any controversial topics to avoid respondents offering the socially desirable answer.
Not Communicating Polling Uncertainty
All research — especially survey research — contains uncertainty, and almost never is this uncertainty communicated correctly. A poll is released with a margin of error and a confidence level. For example: This poll has a margin of error of +-5% with a 95% level of confidence. This means if this exact poll is repeated 100 times, results would be expected to match 95 times — meaning on average 5 times the results are an outlier.
The margin of error describes the true figure in the population — normally expressed as plus or minus a percentage. Meaning if the survey has a margin of error of plus or minus 5% and the question result is 42% of the people like the product, you would accept results of 37%-47% as valid.
Put both together, we would accept a result of 37% – 47%, 95% of the time. This uncertainty is the inherent danger of relying on a single poll result.
Bad Actors
While they are rare, there exists bad actors (some for hire) who intentionally commission a survey with a result in mind. In addition, there are bad actors that hack away at data without a hypothesis.
Company Culture
Most importantly, it may be your company’s culture driving poor research. Ask yourself, how open is your senior executive team to hearing "I know we spent significant time, energy, and money on this research, and we found the results to be non-conclusive. The results indicate additional research needs to be done"?
Not Telling the Entire Story
"More than 80% of veterinarians recommend brand x dog food…." What they aren’t reporting is the survey allowed veterinarians to pick multiple brands. While this may be acceptable in advertising, it is not acceptable to manipulate or wordsmith when conducting research.
Faulty Correlations
Repeat to yourself: "correlation is not causation; correlation is not causation….". While laughing at the junior manager in the intro, it is all too easy to make this mistake. Human brains are hardwired to take shortcuts, group information cues, and seek out patterns. This "skill" has allowed you to understand a rustling in the jungle brush could be a hungry tiger, so you should run.
The tendency to use heuristics and jump to conclusions is what mixes up Norway oil imports and train collisions.
Human behavior is messy and proving what comes first and causes the other is extremely difficult.
Improper Presentation of Results
One of the most common ways to purposefully or unwilling manipulate an audience is manipulation of the visual presentation of results.
A vast majority of people are visual learners, and one easy manipulation of a report or a board room is the manipulation of axis or scales.
The most common x axis manipulation is the changing of time periods. Need to downplay a downward trend in this quarter’s sales? Simple, display 40 quarters of sales information.
The most common y axis manipulation is the changing of scales.
As seen below, the exact same data with different scales tell two different tales. Are sales increasing or flat?
Conclusion
The prevalence of online tools has made surveying and the presentation of results incredibly "easy" to the point that anyone can sign up and conduct "research". While most research is conducted ethically, part of the scientific and research process is healthy skepticism.
A good manager will make any marketing or data team explicitly state their underlying assumptions about their research. A great manager understands there are few "conclusive result", and is continuously asking their team Where may your research may be incorrect or incomplete?
