Optimizing Competitive Intelligence for Micromobility Operators with Custom Web Scraping
Developing an effective vehicle management strategy in micro-mobility requires real-time data. Operators risk inefficient resource allocation and lost revenue without accurate insights into competitor vehicle distribution, demand fluctuations, and usage patterns.
The Client Story
A leading micromobility operator managing vehicles of electric scooters and bikes across major urban centers faced a critical challenge. They needed real-time competitor intelligence to optimize vehicle allocation and maintain a competitive edge—but the data simply wasn’t accessible. There were no public APIs to pull vehicle information, and manual tracking was slow, inconsistent, and resource-intensive.
Third-party reports lacked the depth required for strategic decision-making, while internal market estimates often missed key shifts in demand. Without accurate, real-time insights, vehicle positioning became a guessing game. Vehicles were frequently misallocated, revenue opportunities slipped away, and competitors secured prime locations before they could react.
| Industry: | Micromobility |
|---|---|
| Cooperation: | since 2024 |
| Location: | USA |
We used to rely on slow, outdated reports—manually gathering competitor pricing by physically visiting locations, logging into apps, copying rates, and entering them into spreadsheets. Now, we have real-time data flowing directly into our analytics pipeline, saving time and effort.
GroupBWT’s expertise made data extraction smooth and totally above board, fitting into our compliance policies. They built a rock-solid system that kept everything legit while pulling the insights we needed to stay ahead without worrying about breaking the rules.
The Challenge of Real-Time Competitor Intelligence in the Urban Mobility Analytics Sector
Micromobility operators depend on real-time competitor data to manage vehicle distribution, adjust pricing, and respond efficiently to market shifts. However, with no publicly available APIs and third-party reports providing only historical data, operators lack immediate insights for strategic decision-making—delays in obtaining competitor intelligence result in frequent misallocation of vehicles, underutilized assets, and revenue loss.
This client required a high-frequency data extraction system capable of:
- Tracking scooter and bike deployments across key urban regions.
- Estimating trip volumes based on movement patterns.
- Integrating real-time insights directly into their analytics pipeline.
Since direct API access was not an option, the only viable solution was a custom data extraction infrastructure that could operate reliably, efficiently, and fully comply with regulations.
Ethical Data Extraction & Reverse Engineering
The company turned to GroupBWT to solve this, seeking a custom-engineered, reliable, scalable, and compliant data extraction solution. The objective was clear: extract high-frequency, real-time micro-mobility data without detection and ensure seamless integration into their analytics pipeline.
Reverse Engineering: Understanding the Application’s API
The first step involved reverse engineering the competitor’s mobile (or web) applications to determine how they retrieve and transmit vehicle data. GroupBWT’s team conducted an in-depth analysis, which included:
- Request Interception: Examining the application’s network traffic to identify API calls, response formats (JSON, Protobuf, encrypted packets), and relevant data fields such as vehicle IDs, locations, and battery levels.
- Authentication Handling: Determining whether requests require tokens, cookies, session keys, or user logins to access data.
- Security & Anti-Bot Mechanisms: Identifying rate limits, CAPTCHA requirements, IP blocks, and other security measures restricting automated access.
This phase was critical in ensuring the scraper could interact with the app as a legitimate user. However, services often modify protocols, introduce cryptographic obfuscation, or implement dynamic keys to prevent automated access. The reverse engineering phase included developing adaptive strategies to handle such changes, minimizing disruptions over time.
At GroupBWT, we developed a solution that gives our client a major edge in vehicle allocation, reducing wasted assets and improving efficiency
Core System: Managing the Scraping Infrastructure
Once the API behavior was understood, we developed a scalable core system to control data collection across multiple locations and time intervals. The core infrastructure handled:
- City-Wide Coverage: Dividing urban areas into latitude-longitude grid zones, iterating through each to gather detailed location-based insights.
- Intelligent Scheduling: Implementing a scheduler to collect data at predefined intervals while maintaining efficiency and avoiding rate limits.
- Proxy & IP Management: Distributing requests across multiple IPs and proxies to prevent detection and maintain long-term access.
- Data Storage & Processing: Structuring the extracted data into Google BigQuery for real-time analytics, ensuring seamless integration into the client’s existing systems.
The core’s modular design allowed for easy expansion. New cities, additional applications, and evolving data structures could be integrated with minimal modifications.
Scraper: Data Extraction & Emulation
The final step was implementing a high-frequency scraper that directly interacted with the application’s backend. This involved:
- Parsing & Data Extraction: The scraper sent properly formatted requests, extracting data such as vehicle IDs, locations, battery levels, and availability status.
- Request Emulation: To bypass app security, we mimicked actual user behavior by injecting valid headers, device identifiers, and GPS coordinates.
- Movement & Trip Detection: Movement tracking enabled the detection of vehicle trips by analyzing positional changes and power consumption trends over time.
Ensuring the scraper replicated natural user interactions, we maintained uninterrupted data flow without triggering security measures or API restrictions.
Strategic Insights Drive Vehicle Efficiency and Revenue Growth
Vehicle utilization increased by 15-30% within three months, measured against historical vehicle allocation data and competitor benchmarks. The introduction of real-time intelligence enabled dynamic vehicle repositioning, reducing idle time and improving ride frequency.
Revenue grew by 10-20% due to optimized fleet distribution and higher trip completion rates. The impact was assessed through pre- and post-implementation trip volume analytics, evaluating ride frequency, trip distance, and vehicle utilization efficiency.
Operational costs decreased by 30-50% as automated data collection replaced manual tracking and third-party data purchases. The time previously allocated to fleet monitoring was significantly reduced, allowing internal teams to focus on higher-value strategic initiatives.
Access to real-time competitor intelligence allowed the company to adjust fleet distribution dynamically, improving asset utilization in high-traffic areas. With the success of real-time competitor intelligence, the company expanded its analytics capabilities, incorporating microtransit demand forecasting and deeper urban mobility insights to anticipate market shifts.
Real-time competitor data is critical in micromobility vehicle management, directly influencing efficiency, operational costs, and profitability. This operator transitioned from reactive fleet management to a proactive, data-driven strategy by integrating adaptive, high-frequency data extraction.
As micromobility networks expand, real-time structured intelligence will remain key to fleet optimization, competitive positioning, and long-term market adaptability.
Ready to discuss your idea?
Our team of experts will find and implement the best eCommerce solution for your business. Drop us a line, and we will be back to you within 12 hours.
Let’s connect
Thank You!
We’ll get back to you as soon as possible!