The main challenge was to ensure that the system can handle the high loads. Saying the high load, we talk about up to 100 million requests per day.
No matter the load, the execution time of each redirect should not exceed 60ms despite the part of the world where the redirect had been initiated.
To ensure the desired server capacity, we’ve spun up a cluster of 10+ servers on AWS (by the way, AWS is the most reliable solution when it comes to the huge systems like this one). Saying 10+, because the system had been designed to understand whether or not more servers needed to be able to handle the load. In short, once the load reaches a specific number, the system adds one more server to the cluster. It does this until all the requests that are being processed, have been handled. Once the load is decreased, it removes the servers that are not necessarily needed for processing power right now.
Exactly the same algorithm has been used for the databases. Basically, everything has been done just to make sure that each redirect can be processed within 60ms, despite the number of requests, as the speed of processing is a crucial factor which defines whether or not you convert a link click into the lead.
The last, but not least, was the ability to determine whether the requests come from real people or from the bots. We've created a module that applies specific templates to each request and decides which ones should be blocked based on the main parameters (User agent, Color density, Height, Width, Platform, App version, Plugins, Time zone, Country, Orientation). For instance, if ‘timezone’ and ‘country’ parameters have more than 3 hours difference, we consider this request as the one that had been made by a bot.