Role : Senior Observability Engineer - Datadog & Splunk

Experience : 5+ Years

Role Focus : Observability, Application Performance Monitoring, Log Management, Infrastructure Monitoring

Key Technologies : Datadog, Splunk, APM, OpenTelemetry, Kubernetes, AWS/Azure, Python/Bash, Terraform

Job Summary :

We are looking for an experienced Senior Observability Engineer with strong hands-on expertise in Datadog, Splunk, and modern monitoring solutions. The role involves designing, implementing, and managing observability solutions to ensure application performance, infrastructure reliability, and production stability.

The ideal candidate should have practical experience working with metrics, logs, and distributed traces, along with the ability to identify performance bottlenecks, troubleshoot production issues, and improve system reliability. You will work closely with DevOps, SRE, application development, and infrastructure teams to strengthen monitoring capabilities and support effective incident management.

Key Responsibilities :

1. Observability and Monitoring :

- Design, configure, and maintain end-to-end observability solutions using Datadog and Splunk.

- Implement monitoring across applications, APIs, cloud infrastructure, and Kubernetes environments.

- Establish visibility into system health and application performance through metrics, logs, and distributed traces.

- Configure dashboards, monitors, alerts, and notifications to identify performance issues and potential service disruptions.

- Continuously improve monitoring coverage and observability practices across production environments.

2. Datadog Monitoring and APM :

- Configure and manage Datadog dashboards, monitors, alerts, and infrastructure monitoring.

- Implement and maintain Application Performance Monitoring (APM) to track application performance and service health.

- Monitor application transactions, response times, error rates, resource utilization, and service dependencies.

- Identify performance bottlenecks and work with relevant teams to improve application reliability.

- Configure relevant monitoring and alerting mechanisms to support proactive issue detection.

3. Splunk and Log Management :

- Develop and maintain Splunk Search Processing Language (SPL) queries for log analysis and troubleshooting.

- Create and manage Splunk dashboards, reports, alerts, and searches.

- Analyse application, infrastructure, and system logs to identify errors, anomalies, and operational issues.

- Improve log visibility and support effective investigation of production incidents.

- Use log analytics to identify recurring issues and provide actionable insights to technical teams.

4. Production Troubleshooting and Incident Management :

- Monitor production systems and respond to performance degradation, service interruptions, and infrastructure issues.

- Perform root cause analysis (RCA) to identify the underlying causes of incidents.

- Collaborate with DevOps, SRE, application, and infrastructure teams to resolve production issues.

- Support incident management activities through monitoring data, logs, metrics, and traces.

- Recommend improvements to reduce recurring incidents and strengthen production stability.

5. Automation and Infrastructure Monitoring :

- Automate monitoring and operational tasks using Python and Bash scripting.

- Use Terraform to support monitoring infrastructure and configuration automation.

- Monitor cloud infrastructure and application environments hosted on AWS and Azure.

- Support monitoring across Linux-based systems and Kubernetes environments.

- Contribute to improving the efficiency, consistency, and scalability of monitoring processes.

6. Cross-functional Collaboration :

- Work closely with DevOps, SRE, application development, and infrastructure teams to improve system reliability.

- Support teams in adopting effective observability practices and monitoring standards.

- Provide insights and recommendations based on application and infrastructure performance data.

- Contribute to continuous improvement of production monitoring and operational processes.

Required Skills and Qualifications :

- 5+ years of relevant experience in Observability Engineering, Monitoring, Site Reliability Engineering (SRE), or a related role.

- Strong hands-on experience with Datadog and Splunk.

- Practical experience configuring dashboards, monitors, alerts, and application performance monitoring.

- Strong knowledge of Application Performance Monitoring (APM), log management, and observability concepts.

- Experience developing Splunk SPL queries, dashboards, reports, and alerts.

- Understanding of metrics, logs, and distributed tracing.

- Experience monitoring applications, APIs, cloud infrastructure, and production environments.

- Working knowledge of Linux and cloud platforms such as AWS or Azure.

- Scripting experience with Python and/or Bash.

- Experience in production troubleshooting, incident management, and root cause analysis.

- Ability to collaborate with DevOps, SRE, application, and infrastructure teams.

Preferred Skills :

- Familiarity with Kubernetes monitoring and observability.

- Knowledge of OpenTelemetry and distributed tracing implementation.

- Hands-on exposure to Terraform for infrastructure and monitoring automation.

- Experience with monitoring cloud-native applications and distributed environments.

- Understanding of system reliability, application performance, and production operations.

Key Competencies :

- Strong analytical and troubleshooting skills.

- Ability to investigate complex application and infrastructure issues.

- Proactive approach to monitoring and incident prevention.

- Strong collaboration and communication skills.

- Focus on improving application performance, infrastructure reliability, and production stability.