- Data Engineering, Kafka, Infrastructure Automation

Role Summary :

We are seeking an experienced Platform Data Engineer to join the Global Cybersecurity Data Acquisition and Engineering team. The role is responsible for designing, developing, and supporting enterprise-scale data ingestion pipelines that onboard security data into a cyber data lake and analytics platform.

The ideal candidate :

The ideal candidate will have strong expertise in Azure-based data engineering solutions, Databricks, PySpark, and cloud-native technologies. This role requires hands-on experience building reliable, scalable, and performant data pipelines while collaborating with cybersecurity, infrastructure, and analytics teams to deliver high-quality security data for advanced analytics and threat detection.

Key Responsibilities :

- Design, develop, and maintain scalable data ingestion and integration pipelines using Azure Data Factory, Databricks, and PySpark.

- Develop and support secure, reliable, and high-performance data acquisition patterns for the cyber analytics platform.

- Lead technical enhancements across existing and future analytics technologies including Databricks, Data Factory, Kafka, NiFi, and related platforms.

- Build and optimize automation and orchestration capabilities across the cybersecurity data platform.

- Support the development and deployment of security analytics and detection use cases.

- Collaborate with cybersecurity teams, infrastructure teams, and business stakeholders to implement data-driven security solutions.

- Drive continuous improvement initiatives across platform engineering, data quality, monitoring, and operational processes.

- Provide technical guidance and mentorship to team members and promote engineering best practices.

- Support platform monitoring, troubleshooting, and performance optimization activities.

- Ensure solutions follow security, governance, and operational excellence standards.

Primary Skills (Must Have) :

1. Data Engineering & Analytics :

- Azure Data Factory (ADF)

- Databricks

- PySpark

- Data Pipeline Development

- Data Ingestion & Transformation

- ETL / ELT Processes

- Data Quality and Data Cleansing

2. Cloud Platform :

- Microsoft Azure

- Azure Storage Services

- Azure Functions

- Azure Compute Services

- Azure Networking

- Azure Identity & Access Management

3. Streaming & Real-Time Processing :

- Apache Kafka

- Azure Event Hubs

- Spark Streaming

- Real-Time Data Processing

4. Infrastructure & Automation :

- Infrastructure as Code (Terraform / ARM)

- Automation Tools

- CI/CD Concepts

- Cloud Infrastructure Management

5. Operating Systems :

- Linux Administration (RHEL / Ubuntu)

Secondary Skills (Nice to Have) :

- Apache NiFi

- Fluentd

- Azure Power BI

- Azure Containers and Kubernetes

- Chef / Ansible

- Cloud-based SIEM Platforms

- SOAR Technologies

- Cybersecurity Analytics Platforms

- Security Data Management

Key Competencies :

- Strong leadership and technical problem-solving abilities.

- Excellent stakeholder management and collaboration skills.

- Experience working in large-scale enterprise environments.

- Ability to design sustainable, scalable, and secure platform solutions.

- Strong troubleshooting and performance optimization expertise.

- Continuous learning mindset and interest in cybersecurity technologies.

- Effective communication and mentoring capabilities.

Experience & Qualifications :

- 5 - 7 years of experience in Data Engineering, Platform Engineering, or Cloud Data Solutions.

- Hands-on experience with Azure Data Factory, Databricks, and PySpark.

- Proven experience building and optimizing cloud-based data pipelines.

- Strong understanding of real-time data processing and streaming technologies.

- Experience with Azure cloud services and infrastructure automation tools.

- Knowledge of cybersecurity data platforms, security analytics, or SIEM ecosystems is preferred.

- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.

Ways of Working :

- Work closely with Cybersecurity, Infrastructure, and Analytics teams to deliver strategic data engineering solutions.

- Drive continuous enhancement of the enterprise cyber data lake and analytics ecosystem.

- Participate in platform modernization, automation, and operational excellence initiatives.

- Support end-to-end ownership of data onboarding, engineering, monitoring, and optimization activities.

- Contribute to knowledge sharing, technical mentoring, and engineering best practices across the team.