ArabyAds - Data Engineer - Python Programming

Responsibilities :

- Fulfill any requirement related to data pipelines, data warehouses and databases required by the application.

- Create ingestion pipelines to collect data from different sources and push to targets.

- Design and Create SQL, tables, procedures, functions etc.

- Operate and Maintain existing Hadoop Cluster.

- Write Spark Tasks/Applications mainly in Python/Java

- Evaluate, test and update data applications according to changing requirements.

- Monitor and report data systems health and security.

- Support internal teams to use our systems, troubleshoot & identify issues, and resolve them.

- Follow best coding practices and participate in the code review process.

- Use efficiently the development environment, compiling tools, version control and bug

tracking systems for all projects.

Technical Competencies :

Must Have :

- 3 - 5years of experience in Data Engineering.

- Good Knowledge/Experience in Python/Java, C# and .NET core.

- Good Knowledge/Experience Hadoop Stack, Oozie, Spark, Kafka

- Hands-on in writing SQL queries, Stored Procedures and Database code

- Good Knowledge with concepts, design and mechanics of traditional databases, data

warehouses and data lakes

- Good Knowledge with ETL, ELT, Transformation, Processing, Computation etc.

- Experience with relational databases such as MySQL, PostgreSQL, MariaDB.

- Good Knowledge/Experience with Git, CI/CD, and writing unit tests.

- Good Knowledge/Experience working on an agile development project.

- Good Knowledge/Experience with Jira and the Atlassian tool suite.

- Good Knowledge/Experience in Linux Shell Scripting

Nice to Have :

- Good knowledge of Snowflake databases, schemas, tables, procedures, functions, views, stages, pipes etc.

- Knowledge on one of the ETL tools

- Experience with performance and scalability issues.

Non Technical Competencies :

- Positive and 'can do' attitude.

- Able to work independently as well as in a team.

- Able to take up challenges and come up with innovative solutions.

- We highly appreciate self-starters, initiators and contributors.