Reminiscences of an Adtech Operator – Chapter 2

I spent years spinning some of adtech’s most myth perpetuating bullshit.

This is the second chapter of my book that will never be published. You may have read the first chapter at More than 30,000 people read it in the first 24 hours after publishing, and it was the top item on Hacker News. 

I’m going to speak with you today, about the deeply rooted expedience and outright corruption we’ve come to witness in the online advertising supply-chain. The fundamental reasons that led to us being here today in this unfortunate circumstance, covering the rather dark topic of the rapidly escalating ad fraud epidemic. A phenomena that has become the single largest cybercrime. Actually ad fraud even by the most moderate estimate at $7.2 billion per year, is larger than the next five cybercrime combined.

Having played a role in creating and perpetuating the myth of adtech, and now being perhaps the only full-time volunteer working towards the betterment of the industry, I hope I will be able to, right here today, help dispel this perplexing myth. Doing it in a way that will inevitably lead towards the demise of the corrupt and often beyond belief inefficient adtech complex of today. Help kickstart the process of bursting the bubble of adtech so to speak. Not for the sake of bursting it, or for the sake of the demise of the industry, but in order to make way for something better. Something more transparent, more accountable and efficient. A flourishing internet age advertising eco-system, that as its central focus genuinely have internet users, and its role as a driver of global economy. Instead of what it has been reduced to; a way to create money out of thin air for some, and a way for others to take their cut from it. All at the sole expense of media investors.

To find a suitable analogy as a backdrop for our topic, we will first discuss subprime mortgages. If we go back to the time before subprime mortgage crisis period pre-2008, there was no knowledge of fraud in that system. Now later we know, through extensive litigation, and other sources, that there indeed were three kinds of fraud that led to the events that allegedly caused a near collapse of the global financial system, and the actual collapse of leading investment banks Bear Sterns and Lehman Brothers with many others bailed out using what now has come to trillions of dollars of tax payer money.

The first kind of fraud was where investment banks were selling their customers financial products they knew were garbage. Second kind of fraud was how the local lenders fed the investment banks with any quantity of loans to be bundled in to these new investment products they then sold to buyers. The third kind of fraud was how ratings agencies, that had the investment banks as their primary paymasters, regardless of their poor understanding of the new products gave them triple-A ratings in hopes of more business from those same investment banks. In addition to these three kinds of fraud, as each kind has been proven in the court of law later, gullible buyers were needed to buy such products. They had to believe that their investment bank partners, and the ratings agencies had the best interest of the buyer in mind. The buyers had to, to some extent, believe that investment industry was made up of people who knew what they were doing and that regardless of conflicts of interests that come with propriety trading, where the investment bank is both buying in selling, their judgement would not be biased. It is quite important to understand, that for a buyer it does not make a difference if their partner providing an investment service is ignorant or corrupt, in both cases the outcome for the buyer is more or less the same. To be just ignorant alone, leads to similar result as being both ignorant and corrupt. 

The most important aspects, that acted as the basis allowing these three kinds of fraud to appear and flourish in the way they did, allegedly bringing the global financial system on the brink of a collapse, were also three. The first was the conflict of interest that came from propriety trading, both buying and selling at the same time. The second was the rapid electronization of the financial market, the phenomena of trading becoming programmatic and algorithmic driven. Creating a system where there were no single person who comprehensively understood it all. The third was the market capitalization associated with subprime mortgages, over a trillion dollars at the time.

Having covered the financial collapse through a monthly column in Business Today for a period that spanned across the 12 months of 2009, I came to understand that the extent to which bubbles are inflated before they inevitably implode, is closely associated with the perplexity of the myth that allowed the bubble to emerge in the first place. It is not that everyone in finance was an evil person planning and working together to cheat the world of its money by means of fraud. Actually, it seems more sensible to assume that it was a bunch of brilliant and hard working people, that were passionate about whatever it is that they were doing and were part of. Nature of myths is such that to great extent, those that create them and perpetuate them, are often also the strongest believers in them. With myths, and the bubbles that come with them, nobody really wins. Those that do make monetary or other gains, do so at a terrible cost to those that trusted them the most. In some way it seems fair to argue that everyone involved in a myth bubble is inevitably a victim of it. There are four kinds of victims in myth-bubbles; genuine victims, unintended beneficiaries, accomplishes, and actual knowing perpetrators.

As I had claimed, I’m probably the only true full-time volunteer focused on the betterment of the online advertising industry. I do not work for any organization or individual associated with the industry, regardless of what I do, I’m not getting paid for it.

I want to ensure that as much as I don’t have any commercial incentives to share with you what I’m going to spend the next 30 minutes talking about. Also I’m asking you to consider that what I will say is not an opinion but a matter of fact. I’m not asking you to believe something because I’m saying it, but I am asking you to consider everything on the basis of my claim that it is not an opinion but it is in fact, a fact. By fact I mean something that has objective evidence to support it, or can be arrived at by the power of reason. I’m making this point, because few things are more myth perpetuating than considering serious allegations as opinion when they actually are presented as facts. By the end of this, I will also provide you with a clear idea, how as the guardians of your respective companies, and ultimately their shareholder value, you can take action in a very simple way. In a way that will have the effect of dispelling the myth of adtech, while starting the systematic deflation of the bubble that comes with it. A bubble that now in its shadow dwarfs what has come to be known as the dot-com bubble of 2000. What I will share with you next, has never been shared in such detail outside the closed quarters of those who perpetuate it.

The combined market capitalization of the companies that primarily make their revenues from online advertising is today well over a trillion dollars. Just two companies Google and Facebook, with their money entirely coming from the digital economy, have a combined market cap of $840. There are give or take 10,000 startups fighting for their share of the online advertising market. Ranging from man-and-van outfits to multi-billion dollar behemoths such as AppNexus and retargeting firm Criteo. While the buyers interact with the market place through investment banks in finance, buyers of media interact with the market through demand-side-platforms, DSPs. The way investment banks assured their buyers that they should invest in the new debt products, was through having those products certified by ratings agencies. The way DSPs assure their buyers that what they are buying through the DSP is a good investment, is by giving a fraud free guarantee backed by a 3rd-party verification vendor. As is the relationship between investment banks and ratings agencies, so is the relationship DSPs have with verification vendors. DSPs are the primary paymasters of the verification vendors, and if the verification vendor would suddenly have a dramatic improvement in their capability to detect fraud, so would their DSP paymaster have a corresponding decrease in their ability to sell their product. Product, which in effect is a result of their algorithms making buying decisions on behalf of the buyer.

In adtech, outside of the duopoly of Google and Facebook, even the largest and best known companies do not make money or struggle to make very small margins. Which means that such an effect would be likely to have devastating effects. This only makes sense if indeed verification vendors have inferior capability to detect ad fraud, or are for other reasons misreporting it. Today for the first time in history, we will shed light to the actual actual capabilities of the alleged ad fraud verification companies, which have until now been a completed blackbox. I will share with you evidence, that will show how many of these companies are actually not readily able to detect even simple forms of ad fraud. But before we can go there, we have to first cover the topic of sourced traffic at a very basic level.

By now you must have heard about “sourced traffic”, which is very much like the bad loans that the investment banks based their garbage subprime products on. Sourced traffic is a result of hundreds of “local lenders”, basically ad networks going to any extent in order to answer demand in the market. This demand, in the case of adtech, is created by DSPs that widely sell their verification vendor certified products to buyers and their representatives. It is actually not the advertisers who created the demand for the kind of product DSPs specifically are offering, even though they do create demand for advertising more generally. It is important to remember that it is the DSPs and Trading Desks that initially came to the clients, with a pressure to move investments in to programmatic and not the other way around. One in order to build a business, and the other in order to increase their margins.

The way traffic sourcing works in principle is that the ad network, often with an exotic name like “performance platform” goes to the market buying anything they can get their hands on at a certain price point or below. Typically $0.01 per click, or less. Then making it available through their DSP (and exchange) partners, to create a sense of legitimacy around what actually is just garbage. Like the local banks that were giving out loans to basically anyone who came to ask, even to people with no jobs, these ad networks, more widely referred to as “monetization partners”, accept traffic from sources where they know it possibly can’t have any value what so ever. Sometimes willingly representing partners they know from past to be bad actors.

This expediency in terms of partner selection does not come as a surprise when you consider what I have come to refer as “the secret history of ad networks”. For example in 2003 the leading ad network DoubleClick, was sued for the first major tech support scam campaign. Another major ad network, formerly 24/7 Real Media and now Xaxis, was part of a series of class-action lawsuits Merrill Lynch later settled out of, where its star analyst Henry Blodget according to evidence issued positive reports on 24/7 Real Media, while internal emails portrayed it as a “piece of shit”. All the while 24/7’s stock plummeted from over $60 to under $1. A recent academic research paper proved that Google charged for youtube video ad views even when they knew it was a non-human viewer. A soon to be released research by another academic team found that Google gravely misreports ad placements, especially related to ads appearing on poor quality sites on its GDN ad network. Matomy and RocketFuel, both publicly listed ad networks, have suffered substantial losses in stock price due to exposure to ad fraud on their platforms becoming evident. A recent report by Financial Times evidenced how through Google’s ad network, major advertiser ads were being placed on a jihadist propaganda site. A site owned by a known terrorist funder on the UN, EU and State Department sanctions lists. In the light of these examples, and many others like them, it should not come as a surprise that poorly known ad networks are not too particular about the partners they work with. 

In short summary, sourced traffic is widely available fake traffic that is sold at sub-penny prices right below the glittery surface of adtech. This worthless traffic is commonly marketed as being verified by one or more of the leading verification vendors, such as IAS or DoubleVerify, to validate the quality of the inventory.

To highlight the extent of the sourced traffic problem, and the effect it has on the industry, one of our research partners conducted a mystery shopper test where he went on to setup a fake site, using not more than a few hours to do so. Next he chose a monetization partner that was giving their customers a 100% fraud free guarantee and had the traffic on the site to be verified by four ad fraud verification vendors: Moat, Datadome, Oxford Biochron and IAS. Then he went to one of many LinkedIn groups that specialize in traffic trading and made a connection with a company promising to deliver any volume of IAS verified traffic at tenth of a cent per click. In comparison to Facebook or Google where the going rate for a click to a celebrity site, such as the one he had setup, is above $1, makes this traffic at least 1000 times cheaper than the going industry rate. All of this took less than one man-day to setup, even though the researcher is not tech savvy at all. He simply used the adtech industry infrastructure as it is intended to be used. Set up a site, get traffic to the site, get ads from a monetization partner, and have verification companies authenticate the traffic as something of value to buyers. In order to have it sold to gullible buyers through the DSPs where the monetization partner would make it available.

The test ran for a month, with staggering results. While in this test majority of the sourced traffic came from data centers, visits that are very rarely a result of genuine human activity, the vendors’ capability to detect it as fraud or invalid ranged from 17% to 90%. Where as the least known and smallest of the four companies, with its roots outside of adtech had the highest performance detecting 90% as fraud, the largest and best known of the lot, Integral Ad Sciences, IAS, with a meager fraud detection rate of  17% validated almost as many as humans.

IAS is widely used by DSPs for providing fraud related guarantees to their buyers, but Oxford Biochron is not used by any. Where as Oxford Biochron in its published propriety research provides detailed description of the methodology they use, reporting the highest overall industry ad fraud exposure rates out of any company, IAS in its reports claim the lowest industry exposure rates and provide no detail to methodology at all.

One of the DSPs providing a fraud free guarantee based on IAS verification is [NAME EMITTED], which through its marketing communications claim to “take fraud of the table for the buyer with the guarantee”. In a test analyzing inventory quality on the [NAME EMITTED] DSP, even after some campaign optimization, the easy to identify fraud rate in the test campaign was substantial. Mostly in association with well known arbitrage sites, and instances where ads were allegedly shown with publishers that do not place programmatic ads on their media, such as Netflix.

In a separate instance of industry adtech facepalm, Digital Content Next, a Washington DC based publisher grassroots organization conducted a ad fraud baseline report pertaining its members. The study found that direct buys with DCN members, including some of the most prominent names in online publishing, had an average rate of exposure to ad fraud at 2.4%. Because in the study the publishers knew they were being tested, and DCN commissioning the research have the responsibility of making its members look like a safe investment, we can almost expect a favorable outcome. Further, the findings of the study are contradicting common sense. To understand this, we have to briefly discuss the widely leveraged publisher yield management technique referred to as audience extension. Audience extension is a misleading practice specifically invented by big publishers, often leading to advertiser’s direct buy investment resulting in high exposure to ad fraud. Audience extension is a practice where a publisher uses their 1st-party audience data to buy their audience on other websites, including run-of-exchanges buys in programmatic where the audience is being bought on as many as a million sites by the publisher, instead of just placing ads on their own media properties as the advertiser is let believe. This practice virtually without fail, causes a significant drop in inventory quality, while the advertiser buying directly from the publisher is under the impression their ads are being served only on the publisher’s site

In the advent of programmatic advertising, after the conception of the first advertising exchange Right Media, close to 10 years ago, I was actively involved in audience extension practice with one of the best known and most respected publisher companies in the world. It was specifically related with their highest valued media asset, a highly priced and high in demand property that had very limited supply. The plan the publisher came up with, and executed at a substantial scale, was to buy sub-penny traffic from Right Media exchange, and use the audience extension method so that in effect they were able to fulfill any level of demand. Charging in tens of dollars CPM, while paying in the range of sub-penny to few pennies CPM. Every now and then the buyer found out that ads were being shown on some very dirty sites, for example porn, and what was candidly characterized as “worse than porn”. For all the money the business unit was making, the leadership of the company could not stop singing its praise. The sourced traffic arbitraging by dodgy networks we covered earlier, in principal is not too far from this approach. I have been later able to prove that even direct programmatic TV buys may be subject to high level of fraud. Any programmatic buy actually, even inside Facebook’s walled garden. A respected scientist showed in a recent highly entertaining video demonstration how at least 75% of engagement on a Facebook test page he had created for the experiment were non-human.

A central theme in the myth of adtech, is the involvement seemingly legitimate companies have with fraud. In addition to making more money from it than anyone else, the many companies of the adtech complex are very rarely truly unintended and mostly fall in to the category of an accomplish. In addition to this, hundreds of companies, mostly ad networks, some with relatively high profiles, are actively and knowingly perpetuating fraud. We even know of one major arbitrager company that is already using it’s affiliation with TAG as a way to market its products. 

I’m confident in arguing, that nothing has been more detrimental to demystifying the ad fraud aspect of the adtech bubble, than the IAB, and its allegedly fraud fighting spin-off TAG. So far TAG in its effort to tackle ad fraud, has announced a domain blacklisting program where sites are blacklisted in a process of human verification, within an eco-system of more than 10 million sites, where a non tech savvy researcher can get started with a fake site in matter of hours. This list is not available to public, and is being used to recruit new members. IAB’s previous blacklist, the web spider blacklist, costs thousands of dollars, and is consisting of information available from public sources for free. As its second initiative, TAG announced an IP blacklist where the sole source of information is Google, the greatest single beneficiary from ad fraud. A company whose server IP ranges are commonly found topping sourced traffic lists, as a result of its developer platform being abused for traffic generation purpose.

In its latest effort to tackle ad fraud, TAG announced the much anticipated Certified by TAG for Ad Fraud program. Where in effect companies sitting in TAG’s committees  responsible for coming up with the programs, are now going to be certified by the same programs. All the while anyone can easily collect the evidence of just how ineffective TAG’s program is, by going to the traffic market and bying traffic that is guaranteed to pass filters of leading vendors. This proves beyond doubt that the propriety solutions of the verification vendors are used by traffic sellers to give legitimacy to their worthless traffic. The only well known vendor whose does not come regularly in this market places is WhiteOps.

Out of the many myth perpetuating self-regulatory ordeals we have witnessed, TAG’s Ad Fraud Certification is by far the most dangerous, and the one that is likely to prove the most detrimental to advertisers and the wider society. Basically it will help legitimize very large-scale out-in-the-daylight cybercrime activity while likely failing to delivery any substantial, or perhaps even minor improvement in the overall level of capability to counter ad fraud on behalf of the industry. In short summary, it has all the ingredients for growing the ad fraud problem, and very little in terms of reducing it.

This, and everything else we have discussed today, is a mere symptom of a systemic self-regulatory failure in adtech. The greatest myth of all in adtech, is the one where we somehow keep believing that those organizations, committees and individuals that led to where we are today with fraud, are going to lead us out of it. In the past they’ve only been able to to lead us deeper in to it. To think that IAB, under whose watch we ended up here, is somehow through the power of spinning out of itself TAG, or by some other means, make things better and not worse, falls not far from madness. Actually Albert Einstein said that to do the same thing, in our case keep believing that IAB can lead the self-regulation of adtech in respect to ad fraud, but still expect a different result, is the definition of insanity.

As a point of reference to IAB’s capability to self-regulate and have positive effect in the eco-system as a result, IAB has had ad fraud themed committees since 2006, if not earlier. Their biggest member Google’s then CFO Reyes said in 2004 that fraud was the single biggest threat to internet advertising companies, a statement promptly disputed by Google’s public relations. Later to be replaced by different sentiment from the CEO Eric Schmidt who in a public speech said that “the perfect solution to fraud, was to just let it happen and let the market place self-correct it”. Google commission on all buys through its network is over 60%. One of the iterations of the IAB’s viewability standard specification in 2014 falsely claimed that ad fraud detection capabilities were included in the viewability accreditation. In IAB’s yearly leadership summit held in January, an event titled “the next 50 billion”, ad fraud was barely mentioned. The focus was on appraisal of the important work IAB had done in attracting $50 billion of annual digital investment away from other forms of marketing in US alone. In 2007 Mike Zaneis, then a VP of IAB said that they were closely monitoring the ad blocking situation, and were going to work together with vendors to resolve it. Today ad blocking has grown in to the most significant boycott in human history.

This is not a system that has been corrupted by bad actors. It’s a system behaving exactly as it was programmed to do. IAB’s success is measured in how much money is spent in digital, and that alone. Its leadership will not be materially affected if the overall ad fraud exposure rate in the industry is 10, 30 or 50%. They will still get paid, drive the same car, live in the same home, go to the same holidays and enjoy the same career advancements. In terms of ad fraud being the biggest cybercrime by a mile, there is no real accountability at the level of the organization IAB, its many committees or the individuals that lead it. While it will be hard to argue against this, we have to remember that it is not stupid who asks, who gives. If I’m asking you to pay $1,000 for a gallon of water in central London, and you pay for it, which one of us is the stupid?

In latin there is a saying “quis custodiet, ipsos custodes”. Who watches the watchmen. In the light of what I have shared with you today, which I’m afraid is just a preview to an even more hopeless story filled with corruption and abuse of the buyer’s trust, it seems more than fair to argue that self-regulation of adtech is a failure and the only way for the current powers, namely IAB, to have a role in watching the industry is if somebody is watching them. Having spent substantial time thinking about who that could be, I did not find any answers, except one. Eventually, unless things substantially change, the government would have to step-in. Likely resulting in wide reaching effects, where I’m afraid the advertisers would be the ones to pay for it most dearly. After all, the only purpose of the adtech industry is the service it can render to the advertisers. As a point of reference for the possible ramification from government intervention, Dodd-Frank, the regulation governing the behavior of banks that was created post to the subprime mortgage crisis, is 22,000 pages long.

To drive the needed change in the industry, three pronged approach is needed.

The first is an advertiser moderated coalition that focus on creating true transparency and accountability, while ultimately reducing division between stakeholders in the industry.

The second, and this is the advice I’m giving to every single media investor I have spoken with in the past 24 months, is to take legal action in order to seek retrospective remedy from any partner that is not showing strong evidence of proactive effort towards transparency and accountability. This should be done promptly.

The third, which is the only sure way to immediately halt further escalation of the ad fraud epidemic, while also sending a strong signal to the market in terms of the responsibility the eco-system has. That is to reduce investment in digital. In my view it is not enough to simply shrink digital growth, but a reduction to digital investment in absolute terms is badly needed. It is the only action that is 100% within the control of the buyers, and it  is beyond any doubt the surest way to safeguard your media investment dollars, and in this respect, the shareholder value of your companies. While giving IAB and its members the much needed time to clean up their mess. As difficult as it may seem in the midst of this early digital age craze, it is not just the best solution, but it is the one we are obligated to pursue, for both moral and legal reasons.

Because of our shared passion for the internet, we should clearly understand that it is time for us to focus on growing the quality of aspect of the internet, even if it means that the quantity will temporarily decrease. 

How you can take action

Advertisers should consider legal action and immediately take money away from digital.

Verification vendors should resolve their conflicts of interests and stop publishing marketing as research.

DSPs, Ad Networks and Exchanges should stop deceiving their customers with meaningless guarantees.

Investors should seriously consider their long positions in adtech stocks.

Researchers should work tirelessly to expose the expedience and corruption of the adtech complex.

Trade organizations should stop protecting their vendor members and admit that they have very little clue to how to actually deal with ad fraud.