Sr. Software Engineer - Splunk Admin

Location : Pune, Noida, Bangalore, Hyderabad

Experience : 6 -11 Years

Employment Type : Full-time

Job Overview :

We are seeking a highly skilled and experienced Sr. Software Engineer - Splunk Admin with 6-11 years of dedicated experience to manage and optimize our critical Splunk Enterprise infrastructure. The ideal candidate will possess deep expertise in core Splunk administration, including complex deployments, configuration management, and troubleshooting within robust Linux environments. Proficiency in managing Splunk in virtualized environments (VMware) and cloud platforms (AWS) is a significant advantage. This role requires a strong focus on ensuring the high performance, scalability, and continuous availability of Splunk, providing critical data insights for various organizational functions.

Key Responsibilities :

Splunk Platform Administration :

- Lead the end-to-end installation, configuration, and maintenance of complex Splunk Enterprise deployments, including standalone, distributed, and clustered environments (e.g., Search Head Clusters, Indexer Clusters, Distributed Management Console).

- Perform Splunk upgrades, migrations, and patching activities with minimal downtime, ensuring compatibility and data integrity.

- Manage and optimize Splunk licensing and capacity planning to support evolving data volumes and user demands.

- Implement and manage Splunk Apps and Add-ons from Splunkbase and custom-developed solutions.

Performance Optimization & Troubleshooting :

- Continuously monitor Splunk infrastructure health, performance, and resource utilization (CPU, memory, disk I/O, network).

- Implement advanced performance tuning techniques for Splunk indexes, search heads, forwarders, and data ingestion pipelines.

- Proactively diagnose and resolve complex Splunk-related issues, including search performance bottlenecks, indexing failures, data ingestion problems, and system instability.

- Utilize Splunk's internal logs and diagnostic tools to identify root causes and provide effective resolutions.

Data Ingestion & Management :

- Configure and manage various data inputs and forwarders (Universal, Heavy, and Light) to ingest diverse log and machine data.

- Develop and optimize data parsing rules, field extractions, and data transformations at various stages of the Splunk pipeline.

- Implement data retention policies, archiving strategies, and data tiering within Splunk.

Linux System Administration :

- Administer, troubleshoot, and optimize Linux operating systems (e.g., RHEL, CentOS, Ubuntu) where Splunk components reside.

- Develop and maintain shell scripts (Bash, Python) for automating Splunk operational tasks, system health checks, and data manipulation.

- Manage Linux networking configurations, file systems, user permissions, and process management relevant to Splunk operations.

Collaboration & Documentation :

- Collaborate closely with security, operations, development, and engineering teams to understand their logging, monitoring, and analytical requirements and ensure Splunk meets their needs.

- Provide expert guidance on Splunk best practices for data onboarding, search optimization, and dashboard creation.

- Create and maintain comprehensive technical documentation for Splunk architecture, configurations, operational procedures, runbooks, and troubleshooting guides.

Required Skills :

Core Splunk Administration :

- 6-11 years of hands-on experience as a Splunk Administrator/Engineer.

- In-depth expertise in Splunk Enterprise deployment models, including distributed deployments, Search Head Clusters (SHC), and Indexer Clusters (Clustering).

- Strong understanding of Splunk architecture components (Forwarders, Indexers, Search Heads, Deployment Servers, License Masters).

- Proven ability to configure and manage data inputs (files, network, scripts, APIs) and Splunk Apps/Add-ons.

- Expertise in Splunk Query Language (SPL) for troubleshooting, data validation, and basic report generation.

- Experience with Splunk configuration files (props.conf, transforms.conf, inputs.conf, outputs.conf, indexes.conf, server.conf).

- Familiarity with Splunk ITSI or Splunk ES (Enterprise Security) is a plus.

Linux Operating System :

- Advanced proficiency in Linux system administration, including command-line tools, user management, process management, and troubleshooting.

- Strong experience with shell scripting (Bash) for system automation and Splunk related tasks.

- Solid understanding of Linux networking concepts (firewalls, routing, DNS) and file system management.

Preferred Qualifications (Secondary Skills) :

- VMware : Experience with virtualized environments and managing Splunk components deployed on VMware vSphere/ESXi infrastructure.

- AWS Cloud : Hands-on experience with AWS cloud services relevant to Splunk deployments, such as EC2, S3, CloudWatch, VPC, IAM.

- Automation Tools : Experience with configuration management tools like Ansible, Chef, or Puppet for Splunk deployment automation.

- Version Control : Proficiency with Git for managing configuration files and scripts.

- Networking Fundamentals : Strong understanding of network protocols and topologies.

- Security Concepts : Basic understanding of cybersecurity principles and common attack vectors.