Software Engineer, Data Collection
San Francisco, CA

Job Description

Description

Company Overview

Based in San Francisco, Qadium is an automated Internet intelligence company. We surface facts and relevant context about digital assets of interest to our customers in real time at global Internet scale. Our technology gives large organizations complete visibility into and actionable insights about their true network boundaries to mitigate information security risks beyond the firewall.

Qadium has raised over $65 million of venture capital funding from Founders Fund, NEA, IVP, and TPG. The Company was founded in 2012, after the founders met at DARPA.

Position Summary

As an engineer on the Data Collection team, you will be responsible for collecting the many and varied datasets that power Qadium's global internet intelligence platform. You will learn the ins and outs of a wide array of network protocols, familiarize yourself with the basic infrastructure of the internet, onboard 3rd-party datasets, and securely integrate client data. With greater automation and robustness, your software will integrate terabytes daily and build a coherent view of the internet. Your expertise will help define and secure the perimeters of the world's most important organizations.

What You'll Do

Evolve the Qadium Rapid Internet Sensing Platform's (QRISP) ability to more robustly, more frequently, and more thoroughly detect services across the global internet

Manage, monitor, and improve the quality and rate of refresh for internet intelligence datasets, including RDAP and BGP information, DNS records, certificate transparency logs, GeoIP, threat feeds, honeypot logs, and more

Develop a platform supporting rapid development atop Qadium data, as well as automated pipelines that cohere raw datasets into a single, high-quality, and continuous view of the internet

Bring the best technologies to bear against the most pressing issue facing the modern internet - cybersecurity

Take ownership of projects and drive them to completion

About You

3+ years of experience as a professional software engineer

Experience designing and developing stream/event-processing architectures using technologies such as Kafka, RabbitMQ, Storm, Spark, or their equivalents

Familiarity with distributed datastores, such as HBase, Elasticsearch, and S3, and their application for storing and indexing massive (several hundred terabyte) datasets

Experience optimizing data pipelines for latency, throughput, and/or economical use of cloud resources

Skilled in using "infrastructure as code" tools to manage cloud infrastructure and to execute software releases

Familiarity with Kubernetes a plus

Proficiency in Java and/or Go

Comfortable supporting your work in production

Willing and able to work in San Francisco, CA

Benefits

100% coverage of employee healthcare benefits (Medical, Dental, Vision)

50% coverage for dependent healthcare benefits

20 vacation days with 10 company holidays

Parental Leave 6 to 12 weeks

401k Plan

Pre-tax commuter & FSA benefits

Monthly contribution to phone bill

Stocked kitchen + catered lunch and dinner

New hire office productivity budget - $300

Quarterly discretionary team building stipend

On-site activities such as Lunch and Learns, Game Night, and Speaker Series