Senior Python Data Engineer - ETL

About the Role :

We are seeking an experienced Senior Python Data Engineer with a strong background in building scalable data pipelines, cloud-based data engineering solutions, and advanced reporting systems.

The ideal candidate will be proficient in Python-based development, AWS cloud services, and handling complex time-series data workflows.

This role is hands-on, requiring expertise in data ingestion, transformation, storage, and visualization to support business-critical analytics and reporting needs.

Key Responsibilities :

- Implement and optimize ETL processes for ingesting data from web APIs, IoT devices, and enterprise systems.

- Build scalable and fault-tolerant solutions leveraging AWS IoT SiteWise, Lambda, Glue, and EC2 for real-time and batch processing.

- Manage time-series datasets, ensuring efficient storage, retrieval, aggregation, and analysis for operational and analytical purposes.

- Develop and automate large-scale Excel and CSV-based reports, ensuring accuracy, performance, and usability.

- Integrate Grafana dashboards for real-time monitoring, visualization, and insights.

- Collaborate with data scientists, product managers, and business teams to deliver data-driven solutions aligned with organizational goals.

- Ensure best practices in data governance, quality, and security across all engineering workflows.

Technical Skills Required :

Cloud & Infrastructure (AWS):

- Experience with AWS data integration and automation using Boto3 SDK.

Python Development :

- Strong experience in time-series data handling and data transformation pipelines.

- Knowledge of PVLIB for solar data modeling and performance analysis (preferred, not mandatory).

- Proven experience in ETL pipelines and integrating with multiple web APIs.

- Hands-on experience in automated Excel report generation at scale.

Visualization & Reporting :

- Expertise in generating CSV & Excel reports, including scheduling and automation.

- Experience in building hourly/daily data aggregation frameworks for analytics.

Preferred Qualifications :

- Experience working in IoT-driven data ecosystems.

- Familiarity with data warehouse concepts and distributed data systems.

- Strong understanding of performance optimization and cost-effective AWS architecture