Experience
Senior Data Engineer, NeenOpal Inc., Apr 2023 — Present
- Key contributor in architecting end-to-end AWS infrastructures, strategically selecting services to optimize data warehousing and ETL solutions for diverse client projects
- Significantly contributed to elevating the company’s AWS partnership tier by conducting AWS Well-Architected reviews for numerous clients
- ETL and BI Implementation for a US-based Employee Educational Benefits Firm (Financial Sector)
- Led data engineering initiatives, overseeing the entire project lifecycle
- Engineered a highly efficient ETL solution by constructing a data lake on Amazon S3. Migrated and integrated data from MongoDB, MySQL, and PostgreSQL for advanced analytics using AWS Glue and Spark
Data Engineer, NeenOpal Inc., Jun 2021 — Apr 2023
-
Enterprise-wide ETL + BI Implementation (AWS + PowerBI) for a multinational Logistics company
- Engineered a robust, end-to-end ETL solution for 6 distinct SBUs within the company
- Designed and implemented a resilient data model, creating a data warehouse on Amazon Redshift from the ground up.
- Built and optimized ETL pipelines using Spark Glue for seamless data migration from diverse sources, including SAP, third-party data sources via APIs, and manual Excel files
- Developed complex SQL queries to transform raw data tables into informative datasets, enabling PowerBI dashboards to deliver actionable insights
-
ETL + Forecatsing for a London based real-estate company
- Took charge of the complete data engineering responsibilities for the entire project
- Developed ETL pipelines for monthly data migration from government sources, extracting information from APIs and manual files stored in Amazon S3, and loading it into an RDS PostgreSQL database
- Developed complex SQL queries for transforming raw property data into valuable insights
- Partnered with data scientists to develop a sophisticated multivariate time-series forecasting model aimed at predicting average property price growth in various UK regions over the next decade
-
ETL+ BI Implementation (AWS + Tableau) for an Edtech Company
- Led a team of 2 Data Engineers for development of automated ETL jobs
- Orchestrated the development of ETL workflows to seamlessly migrate transactional data from AWS Aurora to Redshift, effectively utilizing Glue
- Ensured the consistent quality control and performance monitoring of daily ETL scripts, promptly addressing any issues or errors that arose
-
BI Implementation (Azure + PowerBI) for a leading provider of employee scheduling software
- Led a team of 2 Data Engineers for development of automated ETL jobs
- Actively engaged in collaborative meetings with stakeholders to gain a deep understanding of the underlying logic behind various sales KPIs
- Developed SQL queries and Python scripts to execute complex calculations for the KPIs
- Created backend data models and views in Azure SQL database to dynamically transform raw Salesforce data into insightful data for powering 8 different dashboards consisting of 30+ sales KPIs
-
Database Optimization for a ed-tech firm
- Boosted overall system efficiency, leading to optimized performance for both the database and frontend application
- Fine-tuned backend data models and SQL queries, significantly improving the performance of the frontend application
- Optimized the client’s Redis database to enhance data caching capabilities
Data Scientist Intern, NeenOpal Inc., Nov 2020 - May 2021
- Sales and Inventory Forecasting for one of the largest Tiles producers in Sri Lanka
- Assisted a Senior Data Scientist in developing a hybrid forecasting model leveraging ML algorithms and statistical models
- Automated the manual effort of importing, cleaning, and preprocessing data, scheduled ML model autoruns
- Used Tableau for presenting interesting insights to the stakeholders
Research Intern, HCI Lab, IIIT Allahabad, May 2019 - Jul 2019
- Early detection of Autism in toddlers
- Assisted Prof. Anupam Agarwal in his reserach
- Performed Literature survey on early detection of Autism in toddlers
- Worked on Autism Detection application using Image Processing and Machine Learning techniques
- Worked on POCs on various new domains of machine learning like Transfer Learning