Team Lead - Data Engineering
New York, NY

Job Description

Team Lead - Data Engineering

We are looking to hire a Team Lead for our Data Engineering team within Corporate Systems. Your team will be tasked with designing a custom data pipeline solution for a multi-terabyte big data platform. This platform will facilitate analysis of structured and unstructured data. We continually evaluate new ways to improve the ingestion process and expect you to foster an environment where team members participate in this evaluation and future enhancements.

Leadership Responsibilities

* Lead a small team of engineers and drive the design decisions for platform.
* Act as a mentor and thought-leader within your team.
* Foster a close partnership with the users of your data platform.

Technical Responsibilities

* Gather and process raw data at scale (including writing scripts, calling APIs, write SQL queries, etc.).
* Build solutions to process unstructured data into a form suitable for analysis and ingest into the big data platform.
* Work closely with the data science team to integrate their algorithms into our production systems.
* Support business decisions with adhoc analysis as needed.
* As we will be leveraging AWS extensively, you will be working on moving our internal code into a cloud-based environment.

Skills Required

* 3+ years of experience leading a team.
* 5+ years of experience building scalable and reliable data pipelines based on Big Data processing technologies like Hadoop, MapReduce, Spark, etc.
* 5+ years of industry experience in Python, Scala or other functional programming language.
* 3+ working with cloud computing platforms such as AWS.
* Experience processing large amounts of structured and unstructured data.
* Deep experience with ElasticSearch or similar big data platform.
* Highly analytical with good problem solving skills and creativity; able to work independently in a fast-paced environment
* Positive individual with strong communication skills; able to interface well with users and other development teams
* Self-driven individual with an ability to work under pressure

Skills Desired

* Strong knowledge of and experience with statistics; potentially other advanced math as well.
* Experience with writing Kafka producers and consumers.
* Experience with Spark Streaming and Spark SQL in a production setting.