The Client Story
A European last-mile logistics company offering home and pickup point delivery for online retailers needed clear visibility into how its services were positioned at checkout. Their e-commerce clients partnered with multiple couriers, testing delivery configurations, price points, and eligibility conditions.
But the logistics provider had no structured insight into which checkout flows listed them, what price was shown, or how they ranked against competitors. To maintain negotiating power and optimize delivery strategies, the company needed automated, repeatable, and compliant scraping of checkout flows across a growing network of client stores.
| Industry: | Logistics |
|---|---|
| Cooperation: | Since 2024 |
| Location: | Europe |
It wasn’t just about scraping prices. We needed to see real-time rank, eligibility logic, and delivery thresholds—inside the cart.
We wanted Power BI dashboards showing where we stand. Now we have them, and they’re changing how we negotiate.
Why Checkout Intelligence Matters More Than Ever
Logistics providers aren’t just competing on delivery performance—they’re competing on checkout presence. The order in which couriers appear, their price, and the presence (or absence) of free shipping thresholds can shift buyer behavior and contract value.
Manual checks were:
- Incomplete: Dynamic rules changed per location, product, or day
- Infrequent: Insights arrived too late for response or renegotiation
- Inconsistent: Retailers used diverse tech stacks—Shopify, WooCommerce, custom
The logistics company needed weekly scraping of:
- Product name/ID and cart context
- Destination city (Madrid, Barcelona, etc.)
- Delivery methods and exact prices
- The courier’s name and rank in the checkout list
- Free shipping rules or conditions
The goal: real-time, product-location-specific delivery visibility, integrated into their internal BI pipeline—without overstepping legal or ethical bounds.
Rebuilding Checkout Scrapers for Delivery Pricing & Competitor Rank
1. Dynamic Browser Emulation with Cart-Aware Scraping
Standard scraping methods fail to capture delivery visibility when it depends on active cart states—especially those triggered after:
- selecting a specific product from the catalog,
- entering a city, ZIP code, or pickup location,
- initiating the dynamic checkout sequence as a user would,
- and maintaining session continuity throughout the cart interaction.
To accurately mirror this experience, the system used browser automation with Playwright. Each scraper:
- simulated add-to-cart flows using client-provided product links,
- filled in destination forms with dynamic input logic,
- navigated checkout steps while waiting for delivery modules to load,
- and preserved full cart context via cookies, localStorage, and sessionStorage.
In the first rollout, all selected websites were publicly accessible without a login.
However, the system architecture fully supports authenticated flows, including login handling and credential injection—ready for protected sources in future phases.
All scraping was performed within permitted thresholds and conformed to public-site access policies.
Our clients run A/B tests every week. If we’re not listed—or we’re third in the list—we lose volume. And we don’t know until it’s too late.
2. Rules-Driven Configuration from Client Input
Rather than hardcoding site-specific rules, the system ingested instructions directly from a lightweight client-maintained CSV file, which included:
- domain,
- product URL,
- test address (city/ZIP),
- product weight (<1kg),
- execution frequency (weekly or bi-weekly).
This made the scraping highly configurable and scalable. Each scraper read the input, executed the full checkout emulation, and exported:
- delivery method name,
- courier name (when available),
- delivery price and associated free shipping rule (if applicable)
- checkout rank/position,
- free shipping condition (textual or logic-detected),
- and destination locker/address (if shown).
Output data was sent to Azure Blob Storage in .csv format, with downstream Power BI dashboards auto-refreshing via connected pipelines.
3. Iterative Resource Expansion & Platform Grouping
The first iteration focused on five representative websites from a total of 60. These were selected based on:
- shared architecture (Shopify, Magento, etc.),
- publicly available checkout flows,
- known variation in delivery logic.
A parallel resource classification effort grouped domains by:
- vendor platform,
- authorization requirement,
- and static vs. dynamic delivery behavior.
This enabled:
- shared scraping logic for platform-aligned sources (e.g., a generic Shopify parser),
- custom scripts for edge cases,
- and accurate effort estimation for future onboarding.
Time to deploy per new source ranged from 2 hours (template reuse) to 8 hours (custom logic).
Weekly maintenance remained under 6 hours for five sites, even when executed bi-weekly.
4. Scalability Logic & Technology Stack
- The scraping infrastructure was built in TypeScript, using Playwright for headless browser automation and native Azure Blob Storage for data handling.
- Power BI integration was achieved through scheduled connectors that read from .csv outputs, enabling real-time monitoring and trend detection.
- Future rollouts will scale this framework across the remaining 55+ e-commerce sites. Platform-specific modules will be reused where applicable, while custom integrations will follow a standardized intake pipeline.
- As data volume and complexity grow, automation layers will handle dynamic source discovery, input validation, and change detection, minimizing manual oversight and maximizing business insight velocity.
Market Intelligence at Scale
- Presence Verified Across Sites
Scraping revealed that the provider appeared in 4 of 5 tested sites. One site omitted them entirely because of city-based filtering logic that was not previously disclosed.
- Delivery Rank Mapped for Every Checkout
On most checkouts, the provider ranked 2nd or 3rd, behind competitors offering lower prices or free pickup.
- Free Shipping Rules Detected Dynamically
Two retailers offered free shipping above €50, but this logic was hidden behind promotional banners and did not surface in client reports.
- Pickup Address Data Extracted Automatically
For locker delivery flows, scrapers successfully captured the nearest suggested pickup location shown in the cart summary.
- Visibility Drift Tracked Over Time
By the second collection cycle, one retailer had restructured their delivery module, dropping the provider’s service during a trial period.
Market Intelligence at Scale
- Client Negotiation Evidence Produced
With concrete data showing price, positioning, and exclusion, the commercial team re-engaged two accounts, renegotiating placement based on delivery analytics.
- Scalability Unlocked Through Platform Classification
Sites that shared checkout structures were grouped for a bulk rollout. This reduced the engineering workload and accelerated time to value.
- Ethical, Compliant, Future-Proof Architecture
No logins or sensitive data were accessed. All scraping was performed in alignment with site visibility, ensuring sustainable operation and enterprise compliance.
Checkout visibility is the silent killer of conversion. If you’re not listed—or not listed first—you’re losing business. Now, we know where we stand. And we can prove it.
- Price Sensitivity Detected in Cart Behavior
In checkouts where competitors offered lower prices or more attractive shipping thresholds, the provider’s delivery selection rate declined. This confirmed that even minor pricing variations can significantly impact conversion volume and the choice of delivery.
Want to understand how your delivery service appears in checkout flows—ranked, priced, and selected?
We build scraping systems that reveal what’s happening inside your client’s cart—so you can respond before it costs you.
Ready to discuss your idea?
Our team of experts will find and implement the best Logistics solution for your business. Drop us a line, and we will be back to you within 12 hours.
