Big Data Engineer

Position Overview :

Key Responsibilities :

- Design & Implement Data Pipelines : Build and maintain robust, scalable, and efficient data pipelines to process large volumes of data from diverse sources.

- Data Integration : Integrate data from multiple sources into a centralized data warehouse or data lake for analytical purposes.

- Data Modeling : Collaborate with stakeholders to design and implement data models that optimize storage, retrieval, and processing.

- Optimization & Tuning : Ensure the performance, scalability, and reliability of data systems by optimizing queries, processing workflows, and data storage.

- Data Processing Frameworks : Develop and manage batch and real-time data processing frameworks using big data technologies like Hadoop, Spark, or Kafka.

- Collaboration : Work with data scientists, analysts, and software engineers to define data requirements, understand business needs, and design technical solutions.

- Monitoring & Troubleshooting : Monitor the health of data pipelines, troubleshoot issues, and ensure data quality and integrity.

Skills & Qualifications :

- Experience with Big Data Technologies: Proficient in Hadoop, Spark, Kafka, Hive, Flink, and related technologies.

- Data Warehousing : Familiarity with data warehousing concepts and technologies (e.g., Redshift, Snowflake, BigQuery).

- Programming Languages : Strong proficiency in languages such as Python, Java, Scala, or SQL.

- ETL Development : Experience with designing and implementing ETL processes for data integration and transformation.

- Cloud Platforms : Experience working with cloud platforms such as AWS, GCP, or Azure, especially in data related services.

- Distributed Systems : Strong understanding of distributed systems and cloud-native architectures.

- Data Modeling & Design : Solid understanding of relational and NoSQL databases, as well as best practices in database design.

- Problem-Solving & Analytical Skills : Strong problem-solving ability and analytical mindset to identify data issues and optimize workflows.

Preferred Qualifications :

- Experience with Real-Time Data Processing : Experience with tools like Apache Flink, Kafka Streams, or Apache Pulsar.

- Machine Learning Experience : Knowledge of integrating machine learning models or AI systems into the data pipeline.

- Advanced Degree : A Masters degree or PhD in Computer Science, Engineering, Data Science, or related fields is a plus.

Benefits :

- Health, dental, and vision insurance

- Generous PTO and sick leave

- Retirement plan options (401k, etc.)

- Professional development and training opportunities

- Flexible working hours and remote work options

- A dynamic, collaborative, and inclusive work environment