Backend Software Engineer - Data Ingestion (Data Platform) Airbnb
San Francisco, CA

Job Description

The Data Platform team at Airbnb empowers engineers, data scientists and everyone else in the company with massive amounts of data and state-of-the-art data infrastructure. Airbnb data infrastructure drives core product features (such as Search Ranking and Smart Pricing) as well as mission-critical metrics and dashboard for data-informed decision making.

The team leverages open source technologies like Kafka, Hadoop, Spark, AWS cloud services, and other inhouse tools to provide the capability to ingest, process, analyze and store large-scale data sets. As a member of our team you would spend time designing and growing our existing infrastructure, democratizing data access at the company, and promoting the correct use of data and analytics at the company.

Projects

* Logging Infrastructure: Build, maintain and scale the Airbnb logging infrastructure used by engineers across the company to log messages from micro-services and clients for real-time consumption, offline analytics, and training/scoring machine learning models.
* Data Ingestion: Build, maintain and scale critical data ingestion pipelines and related distributed systems that ingests in near real-time massive amount of data from Kafka to the data warehouse.

Examples of problems you will be solving:

* Streaming ingestion of logging events (Kafka Summit presentation)
* Spark at Airbnb (Spark Summit presentation)

The following experience is relevant to us:

* 6+ years of full-time, relevant industry experience
* Working with data at the petabyte scale
* Experience with large-scale logging and/or data ingestion systems is a plus
* Experience with any of Kafka, AWS Kinesis, Apache Spark is a plus
* Experience with Java, Scala and/or Python is preferred
* Design and operation of large-scale distributed systems
* Working knowledge of relational databases and query authoring (SQL)
* Love to use and develop open source technologies like Kafka, Hadoop, Hive, Presto, and Spark
* Rigor in high code quality, automated testing, and other engineering best practices
* BS/MS/PhD in Computer Science or a related field (ideal)
* Excellent communication skills, both written and verbal, and a team player

Benefits

* Stock
* Competitive salaries
* Quarterly employee travel coupon
* Paid time off
* Medical, dental, & vision insurance
* Life insurance and disability benefits
* Fitness Discounts
* 401K
* Flexible Spending Accounts
* Apple equipment
* Commuter Subsidies
* Community Involvement (4 hours per month to give back to the community)
* Company sponsored tech talks and happy hours
* Much more...