Hadoop & Spark Engineer State Street
Quincy, MA

Job Description

As a Hadoop & Spark Engineer, you will be working in a fast paced environment in a team of highly skilled engineers responsible for leading platform engineering, design and implementation of next generation BigData infrastructure. You will also be expected to assist and act as final level of support for the production services supported by the team.

* Design, Engineer & Evangelize Apache Hadoop, Kafka & Spark across the enterprise. '
* Experience in implementing applications leveraging common patterns including SEDA, Kappa, Lambda and similar architectures.
* Perform optimization and tuning of Kafka & Spark Work on creating best practices, standards,
* Integration patterns and on-boarding process automation
* Design monitoring solutions and baseline statistics reporting
* Create stubs for producers, consumers and consumer groups for helping onboard applications from different languages/platforms
* Work with Enterprise Architecture team in identifying and implementing use cases best suited for Kafka
* Provide technical expertise in scoping end to end business requirements and assist in reflecting them in implementation for different Kafka connectors
* Conduct technical reviews of integration implementation for all SDLC phases to assess the quality of integration solutions built on Kafka
* Provide input into technical infrastructure design for achieving high scalability of integration solutions
* Act as liaison to vendors and align product road maps to business process & application integration road map
* Experience with platform engineering and administrations - Unix / Linux Experience with scripting and orchestration - Puppet / Ansible / Python / Shell
* Experience managing (Operational engineering & administration) Cloudera / Hortonwork Distribution of Hadoop
* Experience with Confluent Kafka platform
* Experience supporting linux in a production environment
* Work as a responsible member of engineering team in leading various infrastructure initiatives and also mentor junior staff
* Requiring a minimum of 10 years of IT experience including 5 years with Hadoop & Spark, 3 years with Kafka, and a bachelors degree in computer science