Sr. ML Platform Engineer - Cortex Twitter
New York, NY

Job Description

Sr. ML Platform Engineer - Cortex

New York, NY

Who We Are:

Cortex empowers internal teams to efficiently leverage ML by providing a platform and by unifying, educating, and advancing the state of the art in ML technologies within Twitter. We win when our customers win by helping our users stay informed, share and discuss what matters; by serving the public conversation. We're building an AI-first company and every major initiative is increasingly dependent on the successful application of machine learning. Cortex is at the nexus of this evolution.

Our team of ML software engineers is constructing one of the strongest machine learning platforms in the world, based on the latest ML industry practices, deep learning, engineering excellence, powered by Twitter data at scale.

Twitter data is the fuel for machine learning. Our ML Platform Engineers build:

Systems to efficiently share, organize, discover features for ML models.

Easy-to-use libraries to access features in batch and serve them in real time.

Tools and dashboards to observe feature statistics over time and detect anomalies.

Integrations with our customers' feature infrastructure.

User-friendly and scalable feature generation systems, both with TensorFlow Extended modules and in-house tools.

Our tools enable ML engineers to leverage and operationalize Twitter data to improve their models. We care deeply about:

Engineering excellence such as good design abstractions, API stability, best practices and unit testing.

Staying abreast of and leveraging fast-moving ML open source innovations such as the Tensorflow Extended ecosystem.

An exceptional developer experience for our customers.

Making the feature delivery a seamless experience as models go from training (offline) to production (online).

What You'll Do:

If this sounds like a team you want to be part of, great! We are looking for engineers who love writing code, have a desire to learn new technologies, thrive on teamwork and are committed to serving their customers.

Your responsibilities may include:

Prototyping Apache Beam solutions at Twitter, resonating them with customers and understanding opportunities and obstacles.

Leveraging our Search Infrastructure for ultra-low latency and very high scale real-time feature delivery.

Facilitating the shift from a Scala/Scaling/Hadoop-based feature infrastructure to a Python and TFX-based one.

Building feature extraction, transformation and serving infrastructure.

Designing elegant abstractions, shareable libraries, and robust APIs.

Adapting, extending and contributing to open source and third party solutions to seamlessly function within our toolchain.

Actively looking for ways to improve the end-to-end experience for developers across Cortex's product portfolio.

Working closely with product managers, engineers and stakeholders across the company.

Shaping the direction of our toolchain and product portfolio.

Who You Are:

You have a passion for machine learning.

You thrive on working in concert with other smart people, including from distributed offices.

You communicate fluidly, at the level of your audience, and seek to understand and being understood.

You have the ability to take on complex problems, learn quickly, iterate, and persist towards a good solution.

You are adamant about studying customer needs and enabling their success through our products.

You take pride in polishing and supporting our products.

You welcome feedback on are constantly looking for ways to improve yourself.

You are a senior engineer and have been around the block a couple of times.

You thrive on building platform tools for developers.

You work hand-in-hand with modeling engineers and data-scientists, and your passion is to enable them with better infrastructure.

Requirements:

You have a sound grasp on OOP concepts, data structures and algorithms.

You have a disciplined approach to writing unit and integration tests.

You are rigorous in software design life cycle best practices (design docs, code reviews, support, Sprint planning, Agile methodologies).

You have working knowledge of Java or Scala and a scripting language (e.g. Python).

You have a proven understanding of distributed computing architectures.

You easily articulate complex concepts in writing and speech.

BS, MS, or PhD in Computer Science or equivalent work experience.

You have working knowledge in two or more of these or related big data technologies: Hadoop, Spark, Apache Beam, Dataflow, Presto

You have 2+ years of experience in working on ML infrastucture projects.

You have 4+ years in distributed systems.

You have been successful with geographically distributed teams.

Nice to have:

Feature delivery, feature hydration production experience

Familiarity with feature management solutions, notably feature generation, access and hydration offline/online in a production setting.

We are committed to an inclusive and diverse Twitter. Twitter is an equal opportunity employer. We do not discriminate based on race, color, ethnicity, ancestry, national origin, religion, sex, gender, gender identity, gender expression, sexual orientation, age, disability, veteran status, genetic information, marital status or any legally protected status.