Databricks Engineer

Company Overview :

As a Databricks Engineer at Sysmind Tech, you will be instrumental in designing, developing, and maintaining robust data pipelines and solutions on the Azure cloud platform. You will collaborate closely with data scientists, analysts, and other engineers to build scalable and efficient data processing workflows using Databricks, PySpark, and other Azure services. Your expertise will ensure the seamless flow of data from various sources to enable data-driven decision-making and advanced analytics for our clients.

Key Responsibilities :

- Design and implement scalable data pipelines using Azure Data Factory, Databricks, and PySpark to ingest, process, and transform large datasets.

- Develop and optimize Databricks SQL queries for efficient data retrieval and analysis, ensuring optimal performance and cost-effectiveness.

- Build and maintain data lakes and data warehouses on Azure Data Lake Storage (ADLS) Gen2 and Azure Synapse Analytics, ensuring data quality, security, and accessibility.

- Collaborate with data scientists and analysts to understand their data requirements and translate them into technical solutions.

- Monitor and troubleshoot data pipeline performance, identifying and resolving issues to ensure data integrity and availability.

- Implement data governance and security best practices to protect sensitive data and comply with regulatory requirements.

- Automate data processing tasks and deployments using CI/CD pipelines to improve efficiency and reduce manual effort.

Required Skillset :

- Demonstrated ability to design and implement data pipelines using Azure Data Factory, Databricks, and PySpark.

- Proven expertise in developing and optimizing Databricks SQL queries for data analysis and reporting.

- Experience in building and maintaining data lakes and data warehouses on Azure Data Lake Storage (ADLS) Gen2 and Azure Synapse Analytics.

- Strong understanding of data modeling, data warehousing concepts, and data governance principles.

- Excellent problem-solving and analytical skills, with the ability to troubleshoot complex data-related issues.

- Effective communication and collaboration skills, with the ability to work effectively in a team environment.

- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

- Ability to adapt to a hybrid work environment, collaborating effectively with both on-site and remote team members.