Principal Software Engineer, Alternate Content Engineering Operations / Linear
Denver, CO

Job Description

Comcast's Technology & Product organization works at the intersection of media and technology. Our innovative teams are continually developing and delivering products that transform the customer experience. From creating apps like TVGo to new features such as the Talking Guide on the X1 platform, we work every day to make a positive impact through innovation in the pursuit of building amazing products that are enjoyable, easy to use and accessible across all platforms. The team also develops and supports our evolving network architecture, including next-generation consumer systems and technologies, infrastructure and engineering, network integration and management tools, and technical standards.

The Alternate Content Engineering Operations / Linear team is a growing, fast-moving team of world-class experts who are innovating in end-to-end IP Video Signal Processing that is focused on Dynamic Ad Insertion, Virtual Stream Stiching, Content Content Blackout delivery to Comcast and it's Syndication Partners as well as IP Linear responsibities and relationship across these components. We are a team that thrives on big challenges, results, quality, and agility.

The Alternate Content/ Linear Operations Engineer is the technical counterpart to the on IP Video projects that involve Ad Insertion, Virtual Stream Stitching and Content Blackouts. The group is accountable for the overall deployment, delivery and operational support of all Signal Processing and IP Linear services. The AltCon/ Linear Engineer 5 role gets involved early in the project lifecycle and continues to support the technical solution beyond successful deployment, ensuring detailed software designs, infrastructure, and operational support meet the project objective.

Who does the delivery engineer work with?

The Engineer 5 is the Lead' for the teams technical solutions, capacity planning and overall operational support. Throughout the life of the project, the Principal AltCon / Linear Ops Engineer will collaboratively work with many project stakeholders, including the project manager, service delivery, architect, software development leads, infrastructure team, network engineers, system adiministrators and Comcast leadership to ensure efficient delivery of IP Video Signal Processing and Ad Insertion Projects and its relationship across the IP Linear footprint. This includes analyzing deployment requirements and physical design, defining deployment strategies and help decomposing into deliverable tasks. This Engineer 5 will drive capacity management analysis and audits to understand the IP Video Signal Processing and Ad Insertion applications data processing, throughput and storage needs, as well as growth projections to determine deployment phasing and establishing tracking metrics and robust performance models. These models will be used to validate that applications are performing as anticipated, highlight application issues, and to establish plans to scale the application infrastructure over time. We would like a senior-level engineer with experience managing large scale web sites that utilize caching, load balancing, etc. This engineer should also have automation experience and a demonstrated knowledge of how to utilize git for managing projects/source code. They should be comfortable working with developers and able to communicate via various methods to accomplish their tasks.

What are some interesting problems you'll be working on?

In this role, you will bridge many technical gaps during the life of a project. Examples include:

* Drive the implementation of an IP Video Signal Processing and Ad Insertion solution, both for Comcast and for companies external to Comcast.
* Lead troubleshooting efforts to find root causes and corrective actions thoughout the life of a project.
* Determine requirements, create, validate and audit system capacity plans.
* Identify and create advanced application monitoring (Splunk, ELK, Sysdig, Prometheus) for improved reliability.
* Establish automated application deployments to various environments (Kubernetes, Helm).
* Develop scripts and utilities to automate data collection.
* Evaluate new code releases for basic reliability and systems integration support.
* Provide guidance to QA teams who will perform functional and load testing.

Responsibilities:

* Work directly with technical systems solutions team (Delivery Engineering and Developers) and provide hands-on project support to implement advanced IP Video Signal Processing and Ad Insertion technologies and services.
* Manage the work of AltCon Ops engineers, assign, prioritize and balance project tasks.
* Create and maintain performance models for existing and new applications.
* Perform "what if" scenario analysis to support business decisions, forecast infrastructure needs, and budgeting.
* Assure systems are backed up and copies are readily available to team
* Identify process improvements and create fully documented troubleshotting procedures for offshore team
* Analyze and recommend improvements to the scalability and resiliency of applications.
* Identify and create advanced application performance metrics to monitor for improved application reliability.
* Interact with Software Architects, Service Delivery Engineers and stakeholders to analyze complex projects and break them down into detailed and functional tasks.
* Create and present analysis and interdependencies to project, partner or senior leadership stakeholders.
* Assist development teams by deploying and configuring components in various environments.
* Troubleshoot and triage services and solutions.
* Collaboratively drive deployment of scalable software solutions.
* Development of tools and processes for managing servers.
* Development of load, capacity, longevity limitations of each platform.
* Development of systems and code performance validation tools.
* Foster cross-functional knowledge sharing and mentoring amongst the various engineering teams.
* Responsible for implementation, troubleshooting, and management of customer facing systems with high potential for impact
* Leader of long-term projects that have high impact to internal and external customers.
* Writes SMOPs for implementation of changes to software, reviews for others
* Self starter, projects often require curiosity and a love of learning to complete
* Must be a problem solver and able to utilize new technologies or methods to solve complex problems

Here are some of the specific technologies we use:

* Programming Languages: Ruby, Python, Go, Javascript, Bash
* DevOps Tools: Splunk, Kubernetes, Docker, Sysdig, Prometheus, Git, Helm, Concourse, Jenkins
* Open Source Technologies: Nginx, PostgreSQL, Varnish, Apache Tomcat, HAProxy, Redis, Kafka
* General Knowledge: Linux, MPEG, HTTP Adaptive Streaming, IP Networking, VMWare, Kubernetes, OpenStack

Familiarity with the following industry specifications and standards is helpful but not required:

* CableLabs Event Signaling and Management (ESAM)
* CableLabs Event Signaling and Notification Interface (ESNI)
* Society of Cable Telecommunications Engineers (SCTE-35)
* Digital Video Ad Serving Template (VAST)
* CableLabs Encoder Boundary Point (EBP)

Skills & Requirements

* 9+ years of hands-on experience in software development and/or DevOps engineering.
* 3+ years of experience as a team lead.
* Experience with a variety of Unix/Linux automation and scripting languages such as Python, Bash, Puppet.
* Experience writing core programming languages such as Go, Java or C/ C++.
* Strong Excel skills
* Strong ability to prioritize, assign, track and shift team resources as needed for multiple projects.
* Strong experience gathering requirements and supporting advanced software development teams in an agile environment.
* Ability to plan, organize and document complex system designs.
* Understanding how to scale applications depending on load.
* Knowledge of networking concepts (VLAN, TCP/IP, Multicast, Unicast, OSI).
* Experience with developing advanced application performance monitoring.
* Ability to navigate Unix operating systems.
* Excellent presentation and communication skills to explain system designs and technologies to senior leadership.
* Strong ability to collaborate with peers and stakeholders around system designs
* High attention to detail and strong ability to problem solve systems issues.
* Experience with CI/CD methodologies.
* Willing to take ownership of problems and independently drive them to resolution.
* Must be able to work independently, be self-motivated and handle multiple priorities.
* Comfortable working in a fast paced agile environment. Requirements change quickly and our team needs to constantly adapt to moving targets.
* Container Experience
* * Docker
* * Kubernetes
* * Helm
* Experience with Git and source code management
* Proven experience in one of the following languages:
* * Python
* * Ruby
* * golang
* Proven Experience using Splunk or Logstash/Kibana/ElasticSearch for reporting
* Management of Web Services in a Virtualized Env
* * Apache/Nginx
* * Varnish/HaProxy
* * IPVS or other load balancing
* Experience troubleshooting operational issues with developers
* Experience troubleshooting networking from a sys admin level
* * Packet captures
* * Trace routes
* Experience with Software Deployment/Configuration Management utilizing
* * Puppet

Comcast is an EOE/Veterans/Disabled/LGBT employer