Responsibilities :

1. Design, Develop, Maintain and own data engineering projects built from requirements by different stakeholders, including but not limited to product teams, data scientists, data analysts and BI.

2. Fulfill any requirement related to data pipelines, data warehouses and databases required by the application.

3. Create ingestion pipelines to collect data from different sources and push to targets.

4. Design and Create SQL, tables, procedures, functions etc.

5. Operate and Maintain existing Hadoop Cluster.

6. Write Spark Tasks/Applications mainly in Python.

7. Evaluate, test and update data applications according to changing requirements.

8. Monitor and report data systems health and security.

9. Support internal teams to use our systems, troubleshoot & identify issues, and resolve them.

10. Follow best coding practices and participate in the code review process.

11. Use efficiently the development environment, compiling tools, version control and bug tracking systems for all projects.

Technical Competencies :

Must Have :

- 3 - 5years of experience in Data Engineering.

- Good Knowledge/Experience in Python. Hadoop Stack, Oozie, Spark, Kafka & Airflow

- Hands-on in writing SQL queries, Stored Procedures and Database code

- Good Knowledge with concepts, design and mechanics of traditional databases, data warehouses and data lakes

- Good Knowledge with ETL, ELT, Transformation, Processing, Computation etc.

- Experience with relational databases such as MySQL, PostgreSQL, MariaDB.

- Good Knowledge/Experience with Git, CI/CD, and writing unit tests.

- Good Knowledge/Experience working on an agile development project.

- Good Knowledge/Experience with Jira and the Atlassian tool suite.

- Good Knowledge/Experience in Linux Shell Scripting

Nice to Have :

- Experience with NoSQL databases such as Aerospike, DynamoDB, Redis, Elasticsearch, Cassandra, MongoDB.

- Good to have experience in BigQuery.

- Knowledge on one of the ETL tools

- AWS Lambda, Kinesis, SQS and SNS

- Experience with performance and scalability issues.

- Non Technical Competencies

- Good English and communication skills.

- Positive and 'can do' attitude.

- Able to work independently as well as in a team.

- Able to take up challenges and come up with innovative solutions.

- We highly appreciate self-starters, initiators and contributors.