Senior Data Engineer
New York, NY

Job Description

Job Purpose

The New York Stock Exchange, a subsidiary of Intercontinental Exchange, Inc. (ICE) presents a unique opportunity to work with cutting-edge technology and address complex business challenges in the financial sector. We are searching for Senior Data Engineer that will work on the collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then implementing, maintaining and monitoring them. You will also be responsible for integrating them with the architecture used across the company. We operate a state of the art analytics environment that includes a data lake, scale out SQL platforms, and Hadoop. Our team is world-class, and offers unparalleled exposure to big data in action.

Responsibilities

* Build massive reservoirs for Big data
* Design, Develop, construct, test and maintain architectures such as large-scale data processing systems
* Tool selection and POC analysis
* Gather and process raw data at scale that meet functional / non-functional business requirements (including writing scripts, REST API calls, SQL Queries, etc.)
* Develop data set processes for data modeling, mining and production
* Integrate new data management technologies and software engineering tools into existing structures
* Create custom software components (e.g. specialized UDFs) and analytics applications
* Employ a variety of languages and tools (e.g. scripting languages) to marry systems together
* Install and update disaster recovery procedures
* Recommend ways to improve data reliability, efficiency and quality
* Collaborate with data architects, modelers and IT team members on project goals
* Support our business users with ad-hoc analysis and reports
* Build high-performance algorithms, prototypes, predictive models and proof of concepts
* Research opportunities for data acquisition and new uses for existing data

Knowledge and Experience

* Master's degree in Computer Science, software/computer Engineering, applied math, physic, statistics or related field (or relevant work experience and technical expertise)
* Hadoop based technologies (e.g. hdfs, Spark). Spark Experience is must
* Strong SQL skills on multiple platform (preferred MPP systems)
* Database Architectures
* Data Modeling tools (e.g. Erwin, Visio)
* 5+ years of Programming experience in Python, and/or Java
* Experience with Continuous integration and deployment
* Strong Unix/Linux skills
* Experience in petabyte scale data environments and integration of data from multiple diverse sources
* Kafka, Cloud computing, machine learning, text analysis, NLP & Web development experience is a plus
* NoSQL experience a plus (HBase, Cassandra)
* Finance experience, most notably in Equities and Options trading and reference data is a plus

Intercontinental Exchange, Inc. is an Equal Opportunity and Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, sexual orientation, gender identity, national origin or ancestry, age, disability or veteran status, or other protected status.

Additional Information

* Job Type: Standard
* Schedule: Full-time