Performance Analysis SME
Alexandria, VA

Job Description

Performance Analysis SME

Overview:

This position provides Real User Application Monitoring (RUAM) expertise to configure, develop and maintain critical monitoring system using Splunk as the front end integration along with other OpenSource tools. Configures, troubleshoots and provides in depth analysis with AppDynamics, and develops tailored scripting, and provides in depth analysis with Splunk. Performs performance measurement integration with many different key applications (i.e. AppDynamics, Splunk, HP SiteScope, Tevron's Citratest) and provides the quarterly "Critical Systems Availability" report for management. Manages Capacity Management Database and incorporates data from data repository to create reports. Requires expertise in producing web based application and infrastructure performance reports critical to analysis process.

This is a Full-Time position and work location will be in Alexandria, VA.

Job Responsibilities:

* Provide expert analysis in the Real User Application Monitoring (RUAM) space using the current tools AppDynamics and OpenNMS JMX Collections.
* Provide expert analysis in the Application Event Collection and Analysis System (AECAS) space using the current tool Splunk.
* Provide expertise in the development and support of current application monitoring tool Synthetic Application Availability Monitoring (SAAM) which integrates Splunk, HP's Site Scope, Tevron's CitraTest, Python and Selenium and provide analysis and monitoring support for all applications.
* Provide deep dive performance analysis of application load testing and regression testing conducted by Performance Application Testing (PAT) in the software testing process.
* Advise OCIO in the development of test scenarios that will illuminate performance anti-patterns.
* Participate as a contributor to the discovery, early detection and evaluation of stability problems in critical application systems.
* Assist in the deployment, core infrastructure design and installation, data collection, and best practices framework implementation of the current solution, Splunk.
* Assist in the administration of Splunk including but not limited to building Red Hat Package Manager (RPM) files for Puppet deployment, testing all new versions, deploying to production.
* Perform necessary scripting within the solution to integrate all formats of log files.
* Provide performance analysis of application event data, specifically application and platform log data.
* Design and develop Splunk dashboards that include integration with other tools and data (not just specifically Splunk data, ie. RedHat CloudForms, Oracle Cloud Control, AppDynamics, TeamQuest, cloud hosted data etc.).
* Design performance alerts based on specific application performance and log data.
* Oversee and maintain the RUAM tool(s) (currently AppDynamics and JMX) that capture the real user experience of user community and record business transactions results: that are successful, that have failed or that affected the application performance.
* Use the RUAM tool(s) to enable the decomposition of a transaction slowdown or failure in terms of the different tier levels it traverses to completion.
* Perform in-depth performance incident analysis using the RUAM tool(s) ability to tag a transaction across all tiers to pinpoint what component is not meeting its service goals.
* Provide expert analysis in the discovery of trends of failures in components to assist management in deciding where to invest in providing a more reliable component which will result in more reliable applications for customers.
* Analyze issues involving J2EE Internals (understanding of complex memory management logic).
* Integrate new and existing JAVA applications in to Java performance application monitors.
* Set up, Testing and Implementation of new releases of AppDynamics for J2EE monitoring.
* Perform Java Performance analysis on all JVM's instrumented in SPB java performance tools.
* Integrate Java Performance data into SAS/ITRM.
* Use Open Source, AppDynamics and other various tools for Java performance analysis.
* Continuously monitor performance of JAVA applications through many different methods.
* Create custom JAVA monitoring methods for JAVA applications through JAVA web logs.
* Build alerting and data repository for JAVA instrumented applications.
* Provide hundreds of performance analyses on demand for every aspect of JAVA monitoring.
* Evaluate and implement advanced real-time monitoring of Java environments.
* Use in-depth knowledge of TQ Suite performance metrics, provide expert performance analysis and integrate this data into SAS/ITSV as well as Splunk.
* Use TQ to develop and assign application workload sets.
* Use TQ to design and apply performance alarm sets.
* Assist SPB personnel in the administration of TQ including but not limited to building RPMs and puppet code for Puppet deployment, testing all new versions, and deploying to production.
* Support software development leads through the development of monitoring scripts using Splunk, Site Scope, CitraTest, Python, Selenium and other technologies for application availability and performance monitoring. This includes integrating all technologies for application availability and performance monitoring into SAAM.
* Integrate SAAM data into SAS/ITRM.
* Deploy synthetic transaction monitoring technology to the VM/ESX virtual and RHEV platforms.
* Perform incident analysis using Riverbed's Steel Central Transaction Analyzer Plus packet analysis solution. This includes in-depth knowledge of TCP protocol and the ability to deconstruct a packet at the individual level to find the root cause of a performance problem.
* Evaluate and advise on new technologies, as requested by the Task Order Manager
* Identify tools and develop techniques to monitor and evaluate the performance and availability of cloud-hosted applications.
* Implement synthetic transaction monitoring of cloud based applications.
* Implement tools for collecting service level measurements of cloud based applications.
* Implement tools for triaging service delays and reliability issues with cloud based applications.
* Implement a procedure for evaluating and responding to service incidents related to cloud based applications.
* Manage within SAS/ITRM the Capacity Management Database for key performance information.
* Build the data repository within SAS/ITRM for use in developing reports for website.
* Develop custom reports using SAS/ITRM code from data gathered from all performance tools.
* Constantly validate accuracy of data gathered and of customized reports.
* Perform major upgrades to SAS and the data repository.
* Customize data gathering for critical applications and integrate this data into SAS/ITRM.
* Manage and enhance SAS/ITRM performance collection processing using Unix Cron scheduler.
* Requires the ability to develop the content for the SPB website using the SAS/ITRM data repository.
* Place into practical application the knowledge of ITIL best practices for performance measurement, capacity planning and availability measurement.
* Evaluate, recommend, implement, and advise management on best practices in application availability measurement and alerting.
* Evaluate, recommend, implement, and advise management on the applicability of specific measurement technologies to the monitoring of application availability including response time measurement.
* Advise management on the issues of incident tracking, and reconciliation to automated availability measurements.
* Assist in identifying critical applications candidates for availability measurement and reporting.
* Provide advice to groups on appropriate scenarios for application system tests.
* Work with application owners, project management staff, and development staff to document critical use cases, which best correspond to the user experience, and provide best indication of the availability of application components.
* Perform analysis on the selected use cases to identify the optimum mix of technologies required for synthetic transaction monitoring. These technologies and products would include and support current installation of the following: Splunk, HP's SiteScope, Tevron CitraTest, Python, Selenium, Perl scripts, Unix shell scripts or managed service monitoring from any vendor. Analysis should include and take into consideration any new technologies that may serve more effectively.
* Work with application owners to identify the relationships involved in combining multiple use case monitors into a single composite application availability metric.
* Develop multiple monthly reports to support application availability analysis using a combination of these tools: SAS, SAS/ITRM, Splunk, SiteScope, CitraTest, Perl (parsing of many different types of log files), K-Shell, Python, and Selenium etc.
* Advise management on best practices in the creation of Service Level Agreements (SLAs)
* Advise management on the selection of key SLA metrics related to availability, reliability, and service.
* Advise management on the available technologies for implementing SLA monitoring.

Required Skills:

Familiarity with as many of the following skills and technologies as possible:

* AppDynamics
* Splunk
* TeamQuest Suite of Tools
* Site Scope
* Red Hat Cloudforms Management Engine
* Red Hat Foreman OpenStack Manager
* CitraTest
* SAS
* Selenium
* Python
* Puppet Code
* Ruby Code
* Git
* Groovy code
* SQL
* Linux command languages and scripting
* Putty SSH
* Javascript
* OpenNMS JMX Collections
* Red Hat Enterprise Virtualization Software (RHEV)
* VMware vSphere client software
* Pager Duty

Education:

* Bachelor's Degree or Equivalent Experience

Clearance Requirements:

You do not need a current/active clearance to apply, but must be able to pass and hold a government Public Trust (SF-85) background investigation. You must either be a US Citizenship or Green Card Holder to be eligible.

IIa is proud to be an EEO/AA employer M/F/D/V. We maintain a drug-free workplace and perform pre-employment and random substance abuse testing.