Sr. Platform Engineer, Splunk
Universal City, CA

Job Description

Responsibilities

As a Senior Cyber Security Platform Engineer for Splunk, you'll be responsible for the Splunk platform that supports the technology and cyber security needs of the NBCUniversal global enterprise. You'll be joining NBCUniversal at an exciting time where it is transforming from an operational, tool-based cyber defense program to an intelligence and threat-based organization, which means we're rapidly building and expanding the scope of what we ask our platforms to do - and Splunk is one of the biggest and most critical of those platforms.

Responsibilities

* Deliver consistent functionality and availability of our multi-site distributed Splunk environment, including standard operational processes, troubleshooting, and execution of specific project objectives
* Provide excellent support and service to other teams that rely on Splunk - help share knowledge, and assist in creation and management of Splunk dashboards, alerts, reports and other knowledge objects
* Establish and follow consistent processes to ensure health and stability of Splunk platform, while enabling reasonable self-service across the security team
* Data onboarding and normalization, including engagement with numerous teams and customers throughout the enterprise
* Access and authorization management for Splunk, including index design as appropriate to facilitate role-based access
* Install and configure Splunk Apps & Add-Ons, and support discovery of new Apps to enable existing and novel security use cases
* Facilitate knowledge sharing by creating and maintaining detailed documentation and diagrams, while also collaborating with other team members on standard processes and technology roadmaps.
* Participate in an on-call rotation for support of systems outside of normal business hours, and be available to perform maintenance and critical operations as needed
* This position will require occasional business travel

Qualifications/Requirements

* 5-10 years' experience working with Splunk in a large enterprise distributed environment.
* Substantial knowledge of managing a distributed Splunk installation - Multi-site Indexer Cluster, Search Head Cluster, Forwarders, Deployment Server, Syslog servers, etc.
* Experience installing, building and working with Splunk Apps and add-ons in a distributed cluster
* Experience with configuring HTTP Event Collector, Deployment Server, and Deployer
* Familiar with data on-boarding procedures, CIM compliance and data normalization techniques, and Splunk parsing model configuration
* Strong knowledge of Splunk search language, regular expressions, and other constructs.
* Creating dashboards, reports, and building content for various application and security teams.
* Knowledge of Splunk Best Practices, Workflows, and Processes
* Experience with automation of Splunk infrastructure configurations
* Experience configuring and managing rsyslog/syslog-ng
* Comfort managing large numbers of Linux servers in a distributed environment
* Strong sense of urgency and commitment - able to meet deadlines and ensure work matches defined objectives
* Highly collaborative; personally, and professionally self-aware; able to and interested in interacting with employees at all levels; embody integrity; and represent and inspire the highest ethical standards

Desired Characteristics

* Experience working in security engineering or operations - building, managing, or using security technologies in a business environment. Bonus points for threat-centric, intelligence-based security operations.
* AWS engineering experience (EC2, ELB, S3, Glacier, etc)
* Comfort and enthusiasm using automation to solve problems
* Experience with building and using Linux system automation tooling such as Ansible (but Chef, Puppet, etc, are also fine)
* Experience with version control systems (e.g. Git, SVN, Perforce, etc.)
* Interest in emerging technologies, passion for experimenting and pushing technology into "off label" uses in enterprise defense
* Splunk training and/or certifications would be a major plus