Company Overview :

Magnacore Technologies Private Limited is a specialized technology services firm focused on delivering high-end data engineering, cloud infrastructure, and digital transformation solutions. Operating at the intersection of complex data ecosystems and scalable cloud architecture, the company partners with global enterprises to modernize their data stacks and drive actionable intelligence. With a culture rooted in technical excellence and engineering rigor, Magnacore provides a collaborative environment for professionals to solve intricate architectural challenges for diverse industry clients.

Role Overview :

As a Databricks and Azure Developer, you will serve as a technical lead responsible for architecting, building, and optimizing large-scale data processing pipelines within the Azure ecosystem. You will collaborate closely with cross-functional data science teams, cloud architects, and business stakeholders to translate complex requirements into robust, production-grade data solutions. Your work will directly influence the efficiency of data ingestion, transformation, and delivery, ensuring that the organization's data assets are reliable, scalable, and optimized for high-performance analytics.

Key Responsibilities :

- Design and implement scalable ETL/ELT pipelines using Databricks and Azure Data Factory to streamline data movement across enterprise systems.

- Architect high-performance data lakehouse solutions on Azure, ensuring optimal storage and compute efficiency for downstream reporting and machine learning initiatives.

- Optimize Spark-based processing jobs to reduce latency and infrastructure costs, directly improving the performance of critical business intelligence dashboards.

- Collaborate with data engineers and architects to establish best practices for CI/CD, version control, and automated testing within the Azure DevOps environment.

- Mentor junior team members and conduct technical code reviews to maintain high standards of code quality and architectural integrity across the data platform.

- Troubleshoot complex performance bottlenecks in production environments to ensure 99.9% availability of data services for global stakeholders.

Required Skillset :

- Demonstrated expertise in building and managing end-to-end data pipelines using Databricks, PySpark, and Azure Data Lake Storage.

- Advanced proficiency in SQL and Python for data manipulation, with a deep understanding of Spark optimization techniques and Delta Lake architecture.

- Strong command over Azure cloud services, including Azure Data Factory, Azure Synapse Analytics, and Azure Key Vault for security and governance.

- Proven ability to lead technical discussions and communicate complex architectural decisions to non-technical stakeholders effectively.

- Exceptional problem-solving skills with a track record of managing large-scale data migrations or platform modernization projects.

- Ability to thrive in a fully remote environment, maintaining high levels of self-discipline, proactive communication, and collaborative output.

- A Bachelor's or Master's degree in Computer Science, Information Technology, or a related quantitative field.

- A minimum of 9 - 11 years of hands-on experience in data engineering and cloud-based data platform development.