Tracker And Traffic Redirector

High-load easily scalable traffic management system, capable of processing hundreds of millions requests per day, effectively distinguishing bots and humans requests.

Project

What is the traffic redirection and how does it work?

Let’s say you’re selling something online, but you want your conversion rates to increase and you want that to happen fast. What you need in this case, is to let potential clients know about your goods in some way. The bigger the number of people who know the better this is for your business, right?

This is when the business holders recourse to purchasing the “traffic” from the specific services. “Traffic” is, in fact, the requests made by the human beings sitting at their computers, and what you purchase basically is the number of requests that will be redirected to your desired location. In other words, the services you buy traffic from guarantee they will send the agreed number of requests to your website.
However, you might not want to send all those people to the same page, specifically if you sell a different kind of staff. Moreover, if you have prepared several landing pages that can be shown to different groups of people, depending on their subjects of interest.

This is where you need to create a set of rules, to define where you want the traffic to be redirected to. Rulesets can be different. For example, you may want to send people from specific countries to your landing page #1. You may want to send people who browse the Internet from iPhone to the specific page #2 where you sell useful things for these devices. And so on, the idea should be clear.

There are many solutions on the Internet that provide this sort of capabilities. They are all pretty costly though, and the more requests you want to process, the faster the prices grow. One of the most popular solutions in this area is Voluum (https://voluum.com/). So, the client that contacted us one day, required an analog of Voluum, that would provide the main functionality Voluum offers. At the same time, it should be capable of handling much more requests for the lowest possible price.

Challenges

The main challenge was to ensure that the system can handle the high loads. Saying the high load, we talk about up to 100 million requests per day.

No matter the load, the execution time of each redirect should not exceed 60ms despite the part of the world where the redirect had been initiated.

To ensure the desired server capacity, we’ve spun up a cluster of 10+ servers on AWS (by the way, AWS is the most reliable solution when it comes to the huge systems like this one). Saying 10+, because the system had been designed to understand whether or not more servers needed to be able to handle the load. In short, once the load reaches a specific number, the system adds one more server to the cluster. It does this until all the requests that are being processed, have been handled. Once the load is decreased, it removes the servers that are not necessarily needed for processing power right now.

Exactly the same algorithm has been used for the databases. Basically, everything has been done just to make sure that each redirect can be processed within 60ms, despite the number of requests, as the speed of processing is a crucial factor which defines whether or not you convert a link click into the lead.

The last, but not least, was the ability to determine whether the requests come from real people or from the bots. We’ve created a module that applies specific templates to each request and decides which ones should be blocked based on the main parameters (User agent, Color density, Height, Width, Platform, App version, Plugins, Time zone, Country, Orientation). For instance, if ‘timezone’ and ‘country’ parameters have more than 3 hours difference, we consider this request as the one that had been made by a bot.

3

developers worked on the project

18

servers

2

months to complete

100k

requests per day

Solution

An analog of Voluum, that provides all needed features and possibilities for the proper management of the purchased traffic. Moreover, the system is capable of handling 10 to 100 mln requests per day and can be scaled to process even more if needed.

The system is successfully working nowadays, without any particular involvement from our side, thus, it’s a robust and reliable solution that helps you make money on your business, without paying extra for the external traffic management services.