By Trey Grainger, Presearch CTO
Blocklisting is widespread on the internet, but not many people know what it is or how it works. A blacklist refers to a list of websites, portions of websites, or other material to be blocked from viewing. The most popular use case on the internet is ad blocking. More than 42 percent of internet users use an ad blocker worldwide now.
Most people who turn on ad blockers simply assume that the software is seamlessly “protecting” them from content they don’t want to see online or from unnecessary tracking. In reality, most of the ad blockers work mainly by checking block lists that have been compiled manually over time, listing websites or parts of websites that the person changing the block list wants to censor from the point of view.
As the chief technology officer of a search engine, I’m pretty familiar with this technology. But it was only recently that I learned how few people control these increasingly ubiquitous blocklists. Presearch, the decentralized, community-run search engine with 2.3 million users that I co-maintain, ran into serious problems in the spring of 2021 caused by a poor blocklisting list.
A blocklisting headache
As of March 31st, many Presearch users reported that they were unable to connect to the site or that the connection had broken the site’s functionality. People have flooded our internal community channels and Reddit with their concerns and say things like, “My searches are just hanging, what’s going on? Newly installed and tried several browsers. Unfortunately, I switched back to Google temporarily. ”
This was a big problem, of course, but it didn’t affect everyone and we couldn’t find a reasonable cause of the problem within the platform at first. After several days of reported problems from users of Brave, uBlock, Windscribe VPN, and similar services, we discovered that Presearch had placed the entire website on a blacklist. We scoured Reddit, Discord, and Github, downloaded and combed dozens of blocklists to see which ones Presearch added and why, and tried to reach out to each service and fix it.
We noticed that Presearch was added to a blocklist that was kept public by only four people. One of them had seen a referral banner on another unrelated website and hastily added a blanket ban on ALL requests to Presearch from across the web. Overzealous editing of that one person created a cascading effect as other unrelated services were added to the public blacklist as part of their systems and also began to block Presearch.
In the case of a prominent VPN, Presearch’s website was down and appeared to have disappeared from the internet. In the case of ad blockers and privacy browsers like uBlock and Brave, the website loaded but the search results were blocked, giving users the impression that our website was badly damaged. We lost millions of searches and a week of developer time investigating the problem and developing contingency measures. Our traffic and revenue have suffered a major slump. I’m sure some users have left and never come back and that is a blow to the brand’s reputation that we have worked on for years.
We reached out to one of the developers who maintain the blacklist to ask for help. In order to be removed, we had to prove that we are not an advertising network and that the blacklist violated our website because users could not access it properly. The developer fixed the issue by adding a more selective block that would only be applied to the referral banner it originally saw. We were warned not to bypass it or they would apply a stricter filter.
I reached out to another ad blocking service that was clearly related to the original list, but the developer in charge wouldn’t reveal any details. Even though their process was a black box, they eventually helped us fix the problem by adding Presearch to an “exception list”.
Bring transparency to the web
Presearch was soon up and running again. But after spending a week navigating my way through the blacklist rabbit hole, I started re-examining this ecosystem and examining how subjective it all is. Why are blocklists, which censor content on the Internet from hundreds of millions of users, controlled by so few people? With these blacklists now built into many services by default, do consumers even know what content they are censoring? What gives a few people the right to decide what should and should not be accessible online, according to any criteria, with little or no care?
I don’t think these developers are malicious. Blocklists serve a useful purpose. However, I believe the ad blocking industry is overzealous in its efforts to “clean up” the web and decide what to display and what not, destroying or hiding portions of many websites in the process. What if this practice was infiltrated by bad actors? It’s easy to imagine how things could go downhill quickly. Entire websites could disappear at the whims of a small group of powerful blocklisters.
One of the founding principles of Presearch is the resistance to “the control of the few over the many”. We’re trying to build a decentralized search engine where decisions about content and algorithms are open and made by the community, not a small group of companies or developers.
I hope we can start building a better future when people start to see what is behind the curtain. A future in which search engines become open and reveal their algorithms, developers share how they crowd out ads, block lists are run by the community and we all benefit from a more open, transparent and decentralized web.
Trey Grainger
About the author
Trey Grainger is the Chief Technology Officer of Presearch, a decentralized search engine. He is also the founder of Searchkernel, an AI-powered search and consulting development company. He lives in South Carolina.
Sources:
https://www.adexchanger.com/data-driven-thinking/ad-blocking-theft-or-fair-use/
https://www.marketingterms.com/dictionary/ad_blocking/
https://www.reddit.com/r/Presearch/comments/mhjtt0/is_presearch_down/
https://backlinko.com/ad-blockers-users
The views and opinions expressed herein are those of the author and do not necessarily reflect those of Nasdaq, Inc.
Recent Comments