Lead Engineer for Hadoop Platform - Data Schema Management team Uber
San Francisco, CA

Job Description

Uber Overview

At Uber, we ignite opportunity by setting the world in motion. We take on big problems to help drivers, riders, delivery partners, and eaters get moving in more than 600 cities around the world.

We welcome people from all backgrounds who seek the opportunity to help build a future where everyone and everything can move independently. If you have the curiosity, passion, and collaborative spirit, work with us, and let's move the world forward, together.

Job Description

About the Role

Uber is currently looking for an experienced engineer to lead Uber's "Data schema & metadata Management" team. Based in San Francisco or Palo Alto, the team is responsible for building the infrastructure that supports defining, storing, and serving schema and metadata. The team's mission is to unleash the real power of data by managing the schemas and enforcing evolution rules and being the gatekeeper to prevent incompatible schema or metadata changes that would prevent data accessibility. This includes providing solutions to create/update and retrieve schemas and metadata, supporting different schema frameworks used across different teams (i.e. Avro, Parquet, Thirft, Protocol Buffers, and Json), and seamlessly converting between different frameworks to create consistency and alignment. Come help us scale our Big Data team and fundamentally influence the quality of data used to make key business decisions at Uber.

What You'll Do / What You'll Need / Bonus Points / About the Team What You'll Do

* Build and lead a diverse team of engineers with a mix of Big data, Infrastructure, and full stack engineering backgrounds
* Operate at the intersection of Infrastructure, Big Data, Storage, & Online Services and explore all aspects of a Data analytics platform that collects, stores and serves 100s petabytes of information.
* Lead, mentor and retain the best tech talent in the Bay area
* Develop cross-org partnerships with peer organizations; collaborate and address their schema and metadata requirements
* Influence and guide strategy, execution and innovation for all aspects of data analytics at Uber
* Engage with the open source community to understand existing work and influence future roadmap; Represent Uber via talks at conferences and blog posts

What You'll Need

* 2+ years of leading experience scaling or managing multi person teams with a track record of delivering results while growing/mentoring engineers on your team
* Experience going through the full software cycle of requirements, design, coding/testing best practices and operational excellence in delivering world-class software and services
* Communication and leadership skills, with the ability to initiate and drive processes and projects proactively
* Solid understanding of Schema & Metadata, Data Analytics, Storage, Compute & network infrastructures
* Be customer obsessed and have the ability to translate customer and technical requirements into detailed engineering plans, architecture, and design
* Give technical feedback and drive quality via code reviews, design reviews, and postmortems
* Dedication to moving fast in the short-term, while simultaneously building for long-term

Bonus Points If

* Under the hood experience with Apache Avro, Apache Parquet, Apache Thirft, Protocol Buffers, etc. is a strong plus
* You have a strong vision for cross-domain data integration in a fast-paced environment like Uber while staying on top of rapid developments in the industry
* Experience with highly available/fault-tolerant, distributed systems, large-scale data processing systems or enterprise/cloud analytics systems is also a strong plus

About the Team

The "Data schema & metadata Management" team is part of the Hadoop Platform and Big Data team at Uber. The Hadoop Platform and Big Data teams are based in Palo Alto and San Francisco and are responsible for building all the data ingestion and dispersal frameworks (Marmaray), the interactive and batch querying systems (based on Hive, Presto, Vertica), advanced data processing platforms (based on Spark), Hadoop security (Knox, Sentry) and the underlying storage and resource management infrastructure (with HDFS, Hudi, YARN), for the rest of the company.

We have a small tightly knit team with a diverse set of backgrounds from companies such as Facebook, Google, Cloudera, Hortonworks, Amazon, Microsoft, Vertica, LinkedIn, Twitter, Pinterest, Dropbox, other startups and college grads from the top schools. The team is proud to be a part of the open source community in innovating and shaping such exciting technologies as we move forward. Uber, as a business, is also growing rapidly, and Data at Uber is at the heart of almost all products e.g. Pricing predictions, Uber Pool route optimizations, Uber Eats restaurant recommendations, fraud detection, storage and processing of data collected from Autonomous vehicles etc.

By solving these business problems you will not only be helping Uber but also have a front-row seat to build and innovate the future Big Data systems and contribute them back to open source. This is an exciting time to be part of the Data team at Uber. Be sure to check out our engineering blogs to learn more about the team (i.e. Uber's Big Data Platform, Hudi, Marmaray).