Trying to understand, “How bad is it? Can I trust data from non-probability samples?”
Then read this:
The Pew Research Center generously shared results from its analysis of nine online panels, and examines response consistency across these non-probability samples. Bottom line: Pew did find some variation by question. For example, “Uses Internet Daily” varies widely across the 9 panels. Marital status, and some (not all) demographics, are far more consistent.
Take-away: REALLY know your online panel providers. And you will be best off standardizing on one or two select partners, to avoid unwanted “noise” in your data.