Big Data Engineer Contractor Ten X
San Mateo, CA

Job Description

Auction.com is the nation's leading online real estate marketplace focused exclusively on the sale of residential bank-owned and foreclosure properties via online auctions and live trustee sale events. By offering access to exclusive properties and technology designed to seamlessly connect buyers and sellers, Auction.com empowers residential real estate investors and financial institutions to achieve optimal, mutually beneficial results - to go beyond the bid.

Responsibilities:

Responsible for design, implementation, and ongoing support of the Big Data platforms (Hadoop, HBase, HIVE, Spark), ensure high availability and reliability

Migrating large data sets between data centers and the cloud (such as AWS SQL server and Hadoop)

Design, test and implement cloud BI / DW infrastructure

Understand and support AWS native big data / analytics system

Use Streaming, Spark & Big Data technologies to enrich and transform data for real time ingestion and build low latency feeds.

Requirements:

Thorough understanding of Hadoop ecosystem (HDFS, YARN, Hive, Pig, MapReduce, Spark, Spark2, Sqoop, Solr, kafka, oozie)

Strong experience in setting up, configuraing, upgrading and managing security for Hadoop clusters, setting up Ranger policies for HDFS and Hive

Experience in managing Hadoop cluster with Ambari and developing custom tools/scripts to monitor the Hadoop Cluster health

Strong knowledge and hands on experience related to mission critical backup and recovery

Strong experience with load balancing and high volume, high availability environments

Able to automate administrative tasks using scripting languages (Python, Shell, Ansible)

Strong working experiences of implementing Big Data processing using MapReduce algorithms and Hadoop/Spark APIs

Experience building workflow to perform predictive analysis, muilti-dimensional analysis, data enrichment etc

Understanding of software development methodologies and coding standards.

A burning desire to master new technologies and apply them to real world challenges

Nice to have:

Relational design, understand business requirements and perform data design reviews

Data migration ETL concepts, open source ETL tools

Working experience at a web or internet start-up experience

To all recruitment agencies: Auction.com does not accept agency resumes unless you are part of our preferred partner network. Please do not forward resumes to our jobs alias, Auction.com employees or any other company location. Auction.com is not responsible for any fees related to unsolicited resumes.