Build vs Buy Ad Intelligence: Scrapers, APIs, and Real Costs

Someone above you has looked at Meta's Ad Library and made the request: "just scrape it, the data is public." Now the build vs buy ad intelligence decision sits on your desk, and the framing arrives broken. The question was never whether your team can scrape a transparency page. Any competent engineer can. The question is what each path costs over twelve months, who carries the pager when it breaks, and what you give up by spending engineering weeks on plumbing instead of product.

TL;DR: Scraping ad transparency UIs looks free and isn't. A single-platform DIY scraper runs roughly €20,000–30,000 in year one once you price engineering time, proxies, and permanent maintenance. Managed scrapers shift the breakage risk without removing it. A commercial ad library API costs €3,948/year on AdLibrary's Business plan and covers 11 platforms with one key. Build only when ad data is your product rather than an input to it. Most teams land on a hybrid: buy the data layer, build the intelligence layer.

This post walks the honest math: a comparison of all four sourcing options, a TCO model with numbers you can defend in a budget meeting, a straight treatment of the ToS and legal exposure, and a decision framework that tells you when building actually wins.

Why "Just Scrape It" Is a Cost Question, Not a Code Question

The request sounds trivial because the demo is trivial. An engineer can point a headless browser at the Meta Ad Library, pull fifty ads for one brand, and show a Slack screenshot by lunch. That demo is also the most expensive thing your team will ever ship, because it convinces leadership the problem is solved.

What leadership actually asked for is an ad intelligence capability: every competitor, every platform, refreshed daily, queryable, feeding decks and dashboards. Between the demo and the capability sits an engineering program. Transparency surfaces were built for humans clicking, not machines harvesting. They paginate badly, render through internal GraphQL calls that change without notice, and sit behind anti-bot systems that treat your scraper exactly like the abuse traffic it resembles. If you want the background on what these surfaces expose and how programmatic access works, start with what an ad library API actually is.

So treat the request as a sourcing decision with four candidate paths. Price each one. Then choose.

Four Ways to Get Competitor Ad Data

Every team sourcing competitor ads ends up on one of four paths. Here is the comparison the rest of this post defends, using a mid-senior European engineer at €2,000 per fully loaded week as the yardstick.

Manual research deserves more respect than engineers give it. For five competitors on one platform, an analyst with a swipe file habit is genuinely the cheapest option, and the competitor analysis workflow it produces has judgment baked in. It stops working the moment research becomes recurring, multi-platform, or an input to software.

The other three paths are all forms of ad spy automation. The differences are who maintains the collection layer and who absorbs the breakage. For a vendor-by-vendor look at the paid options, the ad spy API comparison covers AdLibrary, BigSpy, Apify and the rest in detail.

What Building an Ad Library Scraper Actually Involves

Here is the work hiding inside "just scrape it," itemized so you can check it against your own backlog.

The rendering problem. The Meta Ad Library is a React application. The ad data arrives through internal GraphQL endpoints with signed payloads, not in the HTML. Your choices are a headless browser farm (Playwright or Puppeteer, plus the memory bills that come with Chromium at scale) or reverse-engineering the GraphQL calls, which is faster and breaks harder when Meta rotates the payload format.

The anti-bot problem. Sustained automated traffic gets fingerprinted. You will need residential or ISP proxies, request pacing, browser fingerprint rotation, and retry logic that distinguishes "blocked" from "down" from "empty." Proxy spend for a serious crawl runs €300–800 per month on its own.

The parsing problem. Every field you want (creative URLs, copy, date ranges, the EU reach figures) must be extracted from structures that were never a contract. When a div moves, your parser silently returns nulls, and silent is the operative word. Bad data flows downstream and nobody notices until a strategy deck says a competitor went dark when they didn't.

The multiplication problem. Each platform is a separate build. The Google Ads Transparency Center is a different application with different pagination and different blocking behavior. The LinkedIn Ad Library is a third. TikTok, a fourth. If cross-platform coverage is the goal (and for real competitive intelligence it usually is), multiply everything above. We have written separate teardowns of programmatic access to Google's transparency data and the LinkedIn ad library if you want to see how different those surfaces really are.

The storage problem. Raw scraped ads need a schema, deduplication across re-crawls, and media archiving before they're useful. That's a real data engineering task. The competitor ad database guide shows what a sane version looks like.

None of this is exotic. All of it is work, and the work never finishes.

The TCO Math, With Numbers You Can Defend

Total cost of ownership is where the build vs buy ad intelligence argument gets decided, so let's do it with real numbers instead of vibes. Assumptions: a mid-senior engineer at €100,000 fully loaded, which is €2,000 per week or €400 per day. Adjust for your market. The shape of the result won't change.

DIY scraper, one platform, year one:

• Initial build (headless cluster, proxy rotation, parsers, schema, dedup, monitoring): 3 engineer-weeks = €6,000
• Maintenance when the UI or payload changes: 1.5 engineer-days/month = €7,200/year
• Residential proxies: €400/month = €4,800/year
• Compute, storage, monitoring: €150/month = €1,800/year
• Year-one total: ~€19,800. Call it €20,000–30,000 once you include the months where maintenance runs hot or a second engineer gets pulled in.

DIY scraper, three platforms: build cost roughly 2.5x (shared infrastructure, separate parsers), maintenance close to 3x because each surface breaks on its own schedule. Realistic year one: €40,000–55,000.

Managed scraper (Apify-style): rent an actor from the Apify Store for a monthly fee plus usage billed per result or compute unit, then spend half a week wiring its output into your stack. Year one per platform lands around €2,500–6,000. The number that doesn't show up on the invoice: when the platform changes its UI, you wait for the actor's maintainer to ship a fix, and your pipeline is blind until they do. You also inherit their output schema, which drifts.

Manual analyst: 6 hours/week at €35/hour is ~€10,900/year for coverage of maybe ten competitors on one platform, producing screenshots rather than structured ad spend data.

Commercial API: AdLibrary's Business plan is €329/month, €3,948/year, with 1,000+ monthly credits where one credit equals one search across 11 platforms. Integration is an afternoon. The deeper economics of that trade are in our free vs paid ad library API breakdown.

Run those against each other the way you'd pressure-test any line item in an ad budget: the DIY path costs five to seven times the commercial API in year one for one-eleventh of the platform coverage, and the gap widens every year because maintenance never amortizes. The counterintuitive part is that the "expensive" option is the cheap one. €329/month is 0.8 engineer-days. If your scraper eats more than one engineer-day per month in upkeep (and it will), building costs more than buying before you count the opportunity cost of what that engineer didn't ship. Scraped spend figures also need sanity checks against a model like our ad spend estimator, because raw transparency pages don't hand you reliable spend data.

The ToS and Legal Question, Treated Honestly

Most vendor content waves at this topic. Here is the straight version, and the standard disclaimer applies: this is an engineering blog, not legal advice.

Scraping publicly visible pages is not automatically illegal. The hiQ v. LinkedIn litigation established in the US Ninth Circuit that accessing public data likely doesn't violate the federal anti-hacking statute. The same litigation also ended with hiQ losing on breach of contract, because LinkedIn's user agreement prohibited automated collection and the court found that agreement enforceable. "It's public" is not a business plan.

Meta's Platform Terms prohibit automated data collection without written permission. Google and LinkedIn carry equivalent language. For a business, the realistic exposure usually isn't a lawsuit. It's the operational blast radius: IP ranges blocked, scraping infrastructure burned, and in the ugly cases enforcement against the company's commercial accounts on a platform where you also run paid media. Weigh that one carefully. Your scraper saves €329 a month against an ads account that spends six figures.

Two honest qualifiers. First, Meta operates a free, official Ad Library API and deserves credit for it. It covers political and social-issue ads globally, plus all ads for the EU and UK transparency window, after an app review and identity verification. If political ads on Meta are genuinely all you need, use it and pay nobody — our Meta Ad Library API limitations guide maps exactly where its walls are so you can check your use case against them. Second, buying from a commercial provider does not make legal questions evaporate. It moves the collection operation, its engineering, and its ToS exposure off your infrastructure and out of your team's job description. That is a real and valuable transfer of operational risk. It is not an immunity certificate, and a vendor who tells you otherwise is selling something other than data.

The Maintenance Burden of Scraping Transparency UIs

Maintenance is the line item that kills the build case, so it deserves its own section rather than a bullet.

Transparency UIs are the least stable surfaces these platforms operate. They carry no API contract, no versioning, no deprecation notice. Meta ships interface changes continuously, and an internal GraphQL payload that fed your parser on Tuesday can be reshaped by Friday. Your scraper doesn't fail loudly when that happens. It fails politely: the job exits green, the row counts drop 40%, a creative URL field starts coming back null, and three weeks later a strategist asks why a competitor "stopped advertising" mid-quarter. They didn't. Your parser did.

So a production scraper needs the things production systems need: data-quality monitors comparing today's yield against baseline, alerting, an owner, and an on-call answer for the week the owner is on holiday. That is how 1.5 engineer-days a month becomes the floor, not the estimate. Teams that run their own automation infrastructure already know this curve. Teams that have never operated a crawler discover it in month two.

The multiplication is what makes it brutal. Three platforms means three independent breakage schedules. Eleven platforms means your scraping program is now a team. There is a reason the cross-platform tracking pipelines that actually survive are built on top of an API that someone else keeps alive.

The engineer-week framing cuts through the noise here. Every week spent reviving parsers is a week not spent on your actual product. The scraper doesn't only cost €20,000–30,000 a year. It costs whatever the most valuable thing your team didn't build instead was worth.

A Decision Framework for Build vs Buy Ad Intelligence

Five questions settle the build vs buy ad intelligence decision for most teams. Answer them in order and stop at the first hard answer.

1. Is ad data your product, or an input to your product? If you are building an ad-intelligence product to sell, collection is your core competency and you should own it (and staff it). If competitor ads are an input to media buying, creative work, or sales decks, collection is undifferentiated plumbing. Buy plumbing.

2. Does a free official source already cover your use case? Political and social-issue ads on Meta, or EU/UK transparency research: Meta's free API exists for exactly this. Academic research on Google's ads: the Transparency Center may be enough, browsed manually. Don't pay for what's free, and don't build what's free either.

3. How many platforms do you need? One platform keeps the build conversation alive. Three or more ends it for any team without a dedicated data-infrastructure function, because cost and breakage scale linearly while the value of each marginal scraper doesn't.

4. Who fixes it during launch week? Not "who fixes it" in the abstract. Specifically: the week your biggest campaign launches and the scraper dies, who drops what they're doing? If the honest answer is "the same engineer who owns checkout," buy.

5. What is your engineer-week worth in opportunity? At €2,000/week of salary cost, the comparison already favors buying. Priced in delayed roadmap, it's not close.

A useful smell test for question one: if the phrase "we could also sell this data later" comes up to justify the build, the answer to question one was "input," and someone is decorating it.

When Build Wins

The honest column. Building is the right call in four situations, and pretending otherwise would make everything above less credible.

Ad data is the product. You're building a spy tool, a measurement company, an ad transparency research platform. Collection is your moat. Build it, staff it, and budget for the team honestly, because at that point you're not avoiding a vendor bill, you're entering the data-vendor business.

Your need is one narrow, stable slice. One platform, one country, a fixed list of twenty advertisers, refreshed weekly, covered by an official API. A small script against Meta's free endpoint is genuinely fine, and our Python cookbook patterns transfer directly to that work. The trap is scope creep: narrow slices have a habit of becoming "and TikTok too" within two quarters.

Compliance forbids third-party processors. Some regulated environments require every byte to flow through audited internal infrastructure. The decision is made for you; budget accordingly.

You already operate scraping infrastructure at scale. If your company runs proxy fleets and anti-bot evasion as an existing competency, the marginal cost of one more crawler is real engineering hours but not a new program. Most teams reading this are not that company.

Outside those four, the math from the TCO section holds, and it points the other way.

The Hybrid: Buy the Data, Build the Intelligence

Here's where the honest math on build vs buy ad intelligence usually lands, and it isn't a pure answer.

Buying data and building nothing wastes the buy. Building everything wastes the team. The pattern that works is a split: buy the collection layer, build the intelligence layer. Let a vendor fight the anti-bot war, and spend your engineering on the parts that are actually yours — your scoring model, your competitor database, your alert rules, your AI agent integrations.

Concretely, the bought layer hands you stable JSON with performance signals attached: impressions, estimated spend, a heat score, runtime. The built layer is a week of work that would have been impossible to justify on top of a flaky scraper but compounds beautifully on top of a stable feed. Teams run this as end-to-end competitor monitoring, as n8n workflows that file new competitor creatives into Airtable, as Slack alerts that fire when a rival launches a new concept, or as a nightly job feeding an internal dashboard. The automated competitor monitoring use case is the canonical shape: schedule narrow pulls over your competitor set, dedupe on the stable ad key, alert on deltas.

Hybrid year-one math: €3,948 for the data layer plus one engineer-week (€2,000) for your custom layer on top. Under €6,000 for eleven-platform coverage plus proprietary tooling, versus €20,000–30,000 for a single-platform scraper with no tooling at all. That's the comparison that ends most build conversations.

What Buying Looks Like in Practice

So the "buy" column isn't abstract, here is the actual integration, end to end, using the AdLibrary API.

Authentication is one Bearer key (created in the dashboard on the Business plan, prefix adl_, shown once, up to 10 keys per account). No app review, no OAuth dance, no 60-day token expiry. The first call:

curl "https://adlibrary.com/api/search" \
  -H "Authorization: Bearer adl_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "keyword": "protein powder",
    "appType": "3",
    "platform": "facebook",
    "geo": ["US"],
    "sortField": "-days",
    "daysBack": 90
  }'

Sorting by -days surfaces the longest-running creatives, the closest thing transparency data has to a "this one converts" signal. The response is the JSON your scraper would have spent a quarter approximating, with fields scraping can't see at all, like the spend estimate and the 0–1000 heat score, across all 11 platforms from the same endpoint:

import requests

r = requests.post(
    "https://adlibrary.com/api/search",
    headers={"Authorization": "Bearer adl_your_api_key"},
    json={"keyword": "protein powder", "appType": "3", "sortField": "-impression"},
)
data = r.json()
for ad in data["results"]:
    print(ad["advertiser_name"], ad["days_count"], ad["impression"], ad["heat"])
print(f"{data['total']} ads, {data['_credits']['remaining']} credits left")

Operationally: one credit per search, a failed search refunds its credit automatically, and limits are 10 requests/minute and 10,000/day per key with a Retry-After header on 429s. Dedupe and store on ad_key. When a creative clears your threshold, one more credit through the AI enrichment endpoint returns a full teardown and a replication brief, which is the piece no scraper can produce from page markup at any maintenance budget.

API access sits on the Business plan at €329/month with 1,000+ monthly credits, and it ships with free integration help, meaning the vendor's engineers help wire it into your stack rather than pointing you at docs. If you want to inspect the data before committing, Starter at €29/month is enough to run real searches in the app and judge field quality against your requirements.

Frequently Asked Questions

Is it legal to scrape the Meta Ad Library?

Scraping public pages isn't automatically illegal, but it almost certainly breaches platform terms: the hiQ v. LinkedIn litigation narrowed US anti-hacking claims for public data while still ending in a breach-of-contract loss for the scraper. Practical exposure for businesses is blocked infrastructure and account enforcement on platforms where you also run paid media. This isn't legal advice; ask counsel.

How much does it cost to build an ad intelligence scraper in-house?

Budget €20,000–30,000 for year one on a single platform: roughly three engineer-weeks to build (€6,000 at €2,000/week fully loaded), 1.5 engineer-days per month of maintenance (€7,200/year), €300–800/month for residential proxies, plus compute and monitoring. Each additional platform adds €8,000–12,000 per year because every transparency UI breaks on its own schedule.

When does building ad intelligence in-house make sense?

Four cases: ad data is the product you sell, a free official source like Meta's Ad Library API already covers your narrow use case, compliance rules forbid third-party data processors, or you already operate scraping and proxy infrastructure as a core competency. Outside those, buying the data layer and building your own analysis on top is cheaper in both money and engineering time.

What is the difference between Meta's free Ad Library API and a paid ad library API?

Meta's API is free and official but covers political and social-issue ads globally, plus EU/UK transparency data, on Meta platforms only, after app review with tokens that expire every 60 days. Paid APIs like AdLibrary cover commercial ads across 11 platforms with one key and add performance signals such as spend estimates, heat scores, impressions, and runtime. They complement rather than replace Meta's transparency tool.

What does a commercial ad intelligence API cost compared to building?

AdLibrary's Business plan is €329/month (€3,948/year) with 1,000+ monthly credits, one credit per search across 11 platforms, and automatic refunds on failed searches. That's roughly 0.8 engineer-days per month in cost, against a DIY scraper's €20,000–30,000 first year for one platform. A hybrid setup, API plus one engineer-week of custom tooling, runs under €6,000 in year one.

Build vs Buy Ad Intelligence: The Bottom Line

The build vs buy ad intelligence question resolves the same way for most teams once it's priced honestly. Scraping looks free until you cost the maintenance, the proxies, and the ToS exposure against an ads account you can't afford to lose. Buying looks expensive until you price an engineer-week and notice the vendor bill is smaller than your monthly parser-repair budget. Manual research wins below five competitors, official free APIs win for political-ad and EU-transparency use cases, and a pure build only wins when collection is your business.

Everything else points to the hybrid: a bought data layer that someone else keeps alive, and a built intelligence layer that's actually yours. If that's where your math lands too, start with the API access feature, or go straight to the Business plan at €329/month and have your first scheduled competitor pull running this week. The engineer who was about to spend a quarter on Chromium memory leaks will find something better to ship.