Introduction
LiquidM, a Berlin-based Demand Side Platform (DSP), runs on dedicated servers and does not have any containers in production. They use Docker for developers and are introducing K8 for staging. They are not against cloud, and even have a use case for it, but it’s expensive and they want to keep production as simple as possible. Learn how they do it in this technical customer success story.
The Customer
LiquidM helps advertisers set and run their programmatic advertising campaigns with granular targeting settings.
“How I explain what we do to my mom is, basically, any time you see an ad on the web, you can be sure it was sold following the principles of an e-commerce auction. This means that somebody bids for it. The only difference is that it’s not a week - it’s 100 milliseconds for a transaction to run,” says Hagen Rother, Lead Platform Engineer at LiquidM. “At 100 milliseconds, you don’t go out looking for the auctions, you get them pushed. We receive about a million requests a second, we filter them down based on all the criteria and all the market concepts, and then decide if, and how much we bid for the customer.”
The Challenge
LiquidM’s 100 millisecond SLA is an impressive and extremely efficient benchmark, largely dependent on functionality, quick infrastructure, and connectivity. The real-time bidding (RTB) protocol was a challenge in itself when LiquidM first started. Here’s why: you get a bid request and since it’s HTTP, you can only answer with your bid and the transaction ends. You find out whether you won the auction and how much you are charged at a much later time. Budget pacing, detecting click fraud, and aggregating the millions of micro-transactions into meaningful reports is a difficult problem - and LiquidM solves it as a service.
Because of the latency requirements, LiquidM’s platforms were split between several Leaseweb data centers. Bidding had to be geographically close to the auction sites, as centralization made pacing and reporting easier. The connectivity had to be strong, or else LiquidM could potentially pay for an ad but not be able to charge a customer.
Key Requirements
Fast connectivity is crucial, and for LiquidM their standard software needed to be optimized for speed. Rother and his team worked on custom implementations, including an in-memory real time replication of the PostgreSQL database, to data centers around the world. LiquidM’s database is quite large, the application scales almost linear with the number of CPU cores, and the number of required CPUs is far exceeding any existing hardware. This is a clear use case for keeping the implementation on dedicated servers.
The Solution: Dedicated Servers as Cloud
Cloud is generally associated with easy configurability and flexibility. LiquidM made dedicated servers work for them the same way with their cloud-like automation. Rother says they have the luxury of being able to predict how much traffic and what kind of traffic their customers want. These two parameters allow LiquidM to have a low response time and adjust their hardware demand. They can predict the amount of incoming traffic and decide on additional machines to order from Leaseweb, then automate the delivery of the machines and fill them enough to achieve the desired latency and performance level. In cooperation with Leaseweb, the time from order to production is down to 48 hours. “I think it’s a pretty good track record. And our bootstrapping is completely scripted,” says Rother.
The Process
LiquidM built an infrastructure to treat dedicated servers as if virtual (cattle vs. pet). Rother explains:
- LiquidM orders particular hardware from Leaseweb and specifies the type of CPU, RAM, drive type and network connection. In less than 48 hours, Leaseweb inputs the hardware into a rack and makes it available for LiquidM to use.
- Next, Rother initializes a script to scrape all his ISPs to automatically generate a name for the machine from the information in the API - meaning it registers with LiquidM’s system.
- Rother can script and reinstall any of those machines with a one-liner to request a rescue boot to boot a small image off the Leaseweb network.
- Rother has a DSL to describe the layout of the partitions he wants from the hardware, which is specific to that dedicated server or to the role of the dedicated server.
- The DSL then takes the image that he selected, puts it onto the dedicated server and reboots the machine to boot it. It also takes care of registering it with LiquidM’s monitoring infrastructure, as well as putting the right code on it, to decide whether the image is already preinstalled, or whether it requires an installation tool to take a generic image and put something specific on it.
The above process is reduced into a single line, allowing any machine to be reinstalled with a single command. LiquidM also utilizes Leaseweb’s rescue system.
"You’re not our only ISP, and you’re definitely not our first ISP, but it looks like you’re our last ISP. I hope you keep that track record up.”
Hagen Rother, Lead Architect and Head of IT, LiquidM
Quorum and Distance Requirements
To keep up with their latency requirement, LiquidM is currently set up with three data centers in Amsterdam: one ISP (selected on latency) to Leaseweb’s two data centers (AMS-01 and AMS-02). With distributed computing, a quorum is needed (two or three, or, three or five machines). “With three machines you may lose one and you can still have a distributed agreement. With five machines you may lose two machines and you still have a distributed agreement for the other three,” explains Rother. Everything continues to run, and even if one DC goes down it would not be noticeable in production. “We picked an additional ISP in Amsterdam that is really nicely connected to Leaseweb, and we put a single machine there that has nothing to do other than to avoid split-brains.”
There is also a distance requirement: these machines should not be more than 80 kilometers apart. “Even with 80 kilometers, you can notice how long light needs to travel at the latency requirements that we have. So, the DCs need to be really close together.”
In the DCs, LiquidM has complimentary (symmetrical) copies of the same software and their orders with Leaseweb are split into two, where half of the machines are deployed in AMS-01 and the other half in AMS-02. The actual data content is scripted, and the machines themselves are stateless and can be added or removed at any time.
Server Selection Criteria
It’s important for LiquidM to have well-connected machines. However, as far as particular models go, their selection criteria has evolved over time. While initially the competitive price was most important, eventually the team had a better understanding of the quality and price ratio. With Leaseweb, the relationship developed to the point whereby an Account Manager offered Rother new hardware to test in order to decide whether the performance and the price point were suitable for him.
“Luckily for me, Leaseweb was offering the new AMD machines for testing. Those are so amazingly good that I actually managed to replace 52 large machines with 10 harmoniously big 96 vCore AMD machines. And at that scale, five to one, it doesn't really matter how much the new machines cost. That was a 50% price reduction, and our total system performance actually went up. I appreciate the ability to cut cost and the flexibility Leaseweb gives me,” says Rother. “I have been fully supported by Leaseweb on reducing costs wherever possible, and if you look at the billing prices, they have all dropped considerably.”
LiquidM has benefitted from being a dedicated server shop, staying sensitive to commodity hardware, and performing quickly - all at a competitive price. As far as the available technology goes, discussing new servers is a hot topic. “Today, I can move all my stuff within a day, pay half the price, and get better performance,” confirms Rother. He also admits he no longer looks at individual hardware for specific components but cares most about how much it will cost to run his operations. He appreciated being able to try and test the machines and would not have gotten the full picture if the hardware was virtualized away from him.
The Outcome
LiquidM has been doing high amounts of smart automation, benefitting from their dedicated server solution. If you are interested in getting started with dedicated servers, check out their Three Tips for Those Considering an Automated Dedicated Server Solution. Hagen Rother has been extremely happy with dedicated servers and production, and he sees the benefit of Docker and Kubernetes for developers and staging environments. For those interested in the tool LiquidM built to manage their hardware with Leaseweb, it is open-source and available on GitHub.
About Hagen Rother
Hagen Rother is the Lead Architect and Head of IT at LiquidM. His entire career, Hagen has worked to enable developers to focus on business requirements by providing them with the tools and platforms they need at any scale - from a smartphone to enterprise. His specialty is refactoring existing platforms in production.