"Fast Data" platform where ML models with billions of data points are run on top of 10PB+ of consumer data & 200TB+ data streams in real time (x-device advertising). Recently acquired by Twitter.
Building a predictive marketing platform that not only predicted complex behaviors but personalized consumer interactions across all devices required us to have a holistic understanding of consumer activity. In order to ingest multiple streams of data in a manner that would allow us to index and execute on multi-petabyte scale data volumes in real-time, we realized it was time to come up with a new approach.
While the Hadoop-based MapReduce method of computation we were using was quite powerful, it just wasn't meant to provide this kind of sub-second data ingestion. So, we designed a new platform that would allow us to continuously merge all of our batch-processed data with high-velocity data streams. Running this data against our machine learning models at query time, we’ve been able to generate actionable insights on billions of data points in less than 40ms in a 200K QPS production environment. After building big data technology that could now perform in real-time, we decided to call this approach "Fast Data".
We currently have 3 engineering teams: Data Platform, Data Products, and Mission Control (a little about each can be found below):
Since we're now constantly experimenting with, developing, and refining new ideas in our Fast Data environment, our Data Platform team also wanted to create a services oriented architecture that felt just as speedy to the rest of our engineers. So, we created a new infrastructure layer that could support all of the many iterations that this volume of experimentation would require. Today, we're able to manage thousands of production machines with Mesos and Aurora, having created an abstraction layer similar to Google’s Borg, that allows our services to be very lightweight and run in production clusters with minimal setup, monitoring, and maintenance.
In the Data Products team, we have been creating innovative solutions that utilize the latest advances in web technology, predictive analytics, machine learning, and BLE technology to offer highly-personalized, cross-device shopping experiences for 100s of millions of consumers in real time. You will be trusted to make good product decisions, taking products and features from conception to production and thinking about what to do as well as how to do it. And you will be encouraged to make contributions to open-source initiatives as you help us build new products, co-develop industry standards (with our partners), and enter brand-new markets with our technology.
As a member of the Mission Control team, you will be trusted to create highly-interactive, cross-device solutions from massive data sets to provide immersive experiences that are professionally executed across all browsers and devices. And you will be encouraged to make contributions to open-source initiatives as you help us build new products, co-develop industry standards (with our partners), and enter brand-new markets with our technology.
Join us and work with (and hack alongside) engineers who have worked on internet-scale enterprise and consumer solutions that have driven billions of dollars in revenue.