Day to day
- Run a daily review I set up myself of how every route is performing: which provider each site goes to, whether it succeeds, and whether it costs more than it earns. Then move traffic to whatever works for less.
- Work directly with engineers at the proxy providers and scraping APIs on sites that fight back with anti-bot systems.
- Pick up customer escalations when a site’s success rate drops or a page doesn’t render fully.
- Ship and review pull requests with a small engineering team, with a standup most days.
In charts
Shares of commits across every repo in the ScrapeOps GitHub organisation, counted from git. Commits measure activity, not impact, but they show where the work went.
| January | 29% |
|---|---|
| February | 52% |
| March | 19% |
| April | 19% |
| May | 23% |
| June | 46% |
| July | 52% |
| August | 39% |
| September | 44% |
| October | 34% |
| November | 68% |
November is to 8 November only.
| Cloud Browser | 90% |
|---|---|
| Proxy Tester Service | 76% |
| Monitoring Tools | 71% |
| Proxy Routing Service | 65% |
| Backend | 38% |
| Residential Proxy Service | 34% |
| Customer Dashboard | 33% |
What the work changed
Specific pieces of work, before and after. Money appears only as rounded ballparks, and no provider or customer is named.
-
01 · outage, August
About 80% shorter than it would have been
The proxy went offline late at night when its servers’ disks filled up with logs. I caught it from a phone alert outside work hours, shipped a fix that cleared what was actually filling the disks, and had it back about an hour and a half after it went down. Otherwise it would most likely have stayed down until someone came online in the morning.
ballpark Staying up from midnight until morning kept roughly six hours of overnight traffic flowing: on the order of $500–1,000 in revenue, before counting the customers an all-night outage would have annoyed.
Outage began around 22:35 and was fixed by 00:10, at an estimated cost of ~$100–150 in revenue. Left until the next person came online around 06:20, the worst case would have been ~$1,000.
-
02 · rescuing a route
From losing money to profitable, overnight
A site behind heavy anti-bot protection was failing most requests on an expensive provider chain, so it lost money on every request that did get through. I moved it to a cheaper chain that handles that protection better. Success went from 26% to 98%, the route started making money, and below-cost traffic across the whole platform roughly halved the next day.
ballpark At the rate it ran the day after, that one route went from losing money to earning roughly $13–15k a month.
Success rate went from 26% to 98%, and the route went from a daily loss to a daily profit.
-
03 · keeping a plan alive
A provider plan that would have run out a week early didn’t
A large new stream of traffic was burning through one provider’s plan at about 3× the pace it could sustain. The plan doesn’t auto-renew, and it would have run dry a week before renewal: either buy the next plan early or watch that traffic fail. I tested the biggest routes on a provider with spare capacity that auto-renews, moved them, and brought the plan back to roughly a sustainable pace. The moved traffic also earns more than it costs on its new provider.
ballpark Buying that plan a week early would have meant roughly $6–7k spent ahead of schedule. The moved traffic now makes roughly $1–2k a month more than it costs.
Plan usage pace went from 308% to 105% of the sustainable rate.
-
04 · filling a provider properly
More capacity for the traffic that matters
One of our cheapest providers has a hard limit on concurrent requests. A capacity audit found it rejecting 59–95% of what we sent, because low-value routes were holding its slots while saving almost nothing. I moved those off and its rejections fell to 48% within hours. Across every provider, the share of requests that ran into a full provider roughly halved, from about 8% to 4%. More requests landing on the cheap provider means a better margin on every one, and fewer queued requests means customers’ own concurrency gets used, so they get through more of the plan they pay for.
Share of requests rejected went from 59 to 95% down to 48%.
-
05 · success rates
Routes that work more often after rerouting
A few of the routing changes from my daily reviews. Each one is a site where customers were getting failures, fixed by changing which providers handle it and in what order.
- Route A A heavily protected site. Putting a stronger provider first raised success several times over, and it still makes money.
- Route B Added a fallback provider for the requests the first one couldn’t handle, and removed two entries that never actually ran.
- Route C Switched to a cheaper first provider that also succeeds more often: better margin and fewer failures at once.
- Route D A high-volume site moved to a provider that works slightly better, which also took a large daily load off a plan that was close to running out.
Success rate before and after rerouting Route A 8% before 54% after Route B 50% before 74% after Route C 92% before 95% after Route D 97% before 98.5% after
Customer support
Alongside the engineering, I answer customers directly. In 2026 that’s 146 tickets closed, most of them technical.
| Sites blocking requests or returning the wrong content | about 54 |
|---|---|
| Account, billing and access | about 24 |
| Product, pricing and credit questions | about 17 |
| Getting data out of dynamic pages | about 16 |
| Structured data API | about 14 |
| Speed and outages | about 8 |
| Dashboard and tool bugs | about 5 |
| Pre-sales: can you scrape this? | about 3 |
Approximate. Some topics include repeat follow-ups from the same customer.
About 33% needed a change on our side, such as rerouting a site, fixing a bug or an account action; about 60% were solved with advice and working examples; the rest were escalated or had no reply.
Changes on our side: rerouting sites, bug fixes and restores, and account actions such as refunds.
A few of the fixes
- Found a bug that let one site ignore a customer’s per-request cost cap, fixed it, and credited back the overcharge.
- Worked out a site’s hidden data endpoint, then reused cookies from one expensive request so the rest cost a fraction as much. Sent the customer a working script.
- Changed how one site’s block pages were detected, so failures stopped being counted, and charged, as successes.
- Replaced a fragile scripted-browser approach with a single header to get store-specific prices.
- Routed very large documents on one site to a provider that accepts them, after they kept failing.
- Moved an account whose country targeting kept drifting onto a more reliable proxy pool.
- Fixed a structured-data parser that missed product variations on one regional marketplace.
- Repaired broken job-failure email alerts in the monitoring product.
On my own initiative
None of these were tickets. They’re things I picked up because I thought they were worth doing.
- Fixing an outage at midnight. The August outage above. It was outside work hours: I saw the alert on my phone, shipped the fix, and handed it over for review first thing in the morning.
- A daily routing review. My idea, not a ticket: a daily report on every route’s success and cost, written up for the team with the changes I’ve made and why.
- Knowing when not to act. When my own report said to move more traffic off a provider, I held off because it would have left capacity empty overnight. I added logging instead, so the decision could be made properly a week later.
- A capacity audit. Found a provider turning most of our requests away because it was full, and moved off the traffic that was holding slots without saving anything.
- Checking what price changes really mean. When a provider changed its prices, I worked out the effect across every route instead of taking the headline.
- Trying new providers where it matters. Tested a new provider on the most expensive routes, and kept it only where it made a real difference.
- Codebase audits with each new AI model. Suggested running a full audit of the codebase whenever a new model comes out and comparing what each one finds. The easy fixes went straight into testing.
Sending each request to the right provider
The service that decides where every request goes. I wrote 65% of its commits in 2026, and 80% since July.
- Cheapest provider that works. Routing that picks the cheapest provider and settings that still succeed for each site, with fallback chains when the first choice fails.
- Routing that stays profitable. Every route accounts for what a request may cost at most and what failed attempts cost, so traffic doesn’t quietly lose money.
- Loss-making traffic made visible. Tracking for unprofitable and failed requests, so the routes that need fixing show up on their own.
- Many scraping APIs behind one interface. Integrations with third-party scraping APIs, each with its own timeouts, credits and parameters, all hidden behind the same request.
- Country targeting. Working out the right country from a site’s domain, and caching the provider order for each site.
- Residential proxies. Country, region and city targeting, rotating and sticky sessions, and one module per provider.
Getting past sites that fight back
A request that comes back with a block page isn’t a success, even if the status code says it is.
- Spotting blocks. Detection for block pages, CAPTCHAs, challenge pages and “successful” responses that aren’t, so they’re retried instead of returned.
- Site-specific workarounds. Headers, cookies, sessions and request bodies tuned per site when the generic approach gets blocked.
- Scripted browser actions. Clicking to load more content, waiting for the right elements, and checking a page has actually finished loading before it’s returned.
Browsers in the cloud
For JavaScript-heavy sites and for customers who drive their own browser. I wrote 90% of the commits on the Cloud Browser.
- Hosted browsers for heavy pages. Hosted Chromium wired into the proxy for pages that only work with JavaScript.
- A cloud browser customers connect to. A new service that Playwright and Puppeteer connect to directly, with the provider chosen on the server side. I’m effectively its owner.
- Sessions that recover. Session retries, capacity handling, and telling a dropped connection apart from an abandoned session.
- Metering for billing. Byte-level usage per session, from page loads to WebSocket traffic, rolled up daily. Billing and the dashboard for it are in progress.
Staying up under load
Upstream providers have hard limits. Going over them means failed requests.
- Shared concurrency limits. Distributed limits with priority queues and admission rules that keep traffic inside each provider’s capacity, with a local fallback if the shared store goes down.
- Circuit breakers. Automatic pauses after repeated errors, plus tuned timeouts and deadlines.
- Safer logs. Clearer error labels, and secrets redacted before anything is logged.
Knowing before customers do
Background workers that watch the system and tell the team when something changes. I wrote 71% of the commits on the Monitoring Tools in 2026.
- Alerts on drops and spikes. Workers that flag sudden falls in success rate, traffic spikes and failed validations in the team chat.
- Re-checking how hard each site is. A monthly re-check of how difficult every site is to scrape, fed back into routing.
- Provider reports. Usage, speed and capacity reports per provider, with suggestions for which sites to optimise next.
- Testing providers properly. A benchmarking service that tests providers on speed, success and cost, rebuilt in Go, with side-by-side comparisons.
The product customers use
Not everything is proxies. A lot of it is the dashboard, accounts and billing around them.
- Moving the backend. Moved authentication, accounts, projects, jobs and stats from the old Python backend to Node, endpoint by endpoint, without breaking the dashboard.
- Prepaid credits. One-off credit blocks from start to finish, with protection against double charges and leftover credits carried onto a paid plan.
- Customer alerts. Alert thresholds, health-check history, and delivery by Slack, Discord or email.
- Passkeys. Passkey sign-in and management, with rate limiting.
- Stopping free-plan abuse. Signup checks, browser fingerprint matching and domain bans for accounts abusing the free tier.
- Internal admin tools. Admin screens for products, plans and providers, usage analytics, and a live view of the request queue.
Scraper getting blocked or costing too much?
Outside ScrapeOps, I do fixed-scope audits and short engagements for teams with that problem. Here’s how that works.
- eddie0sheehy@gmail.com