Data Engineer at Snapnet Limited

Job Description

• We are seeking a talented Azure Data Engineer to join our organization.
• As an Azure Data Engineer, you will be responsible for developing data pipeline solutions using Azure Data Factory, as well as integrating Azure Databricks functionalities and loading data into Azure SQL Database.
• You will also develop solutions to load data from Network Shared Folders into Azure Data Lake Gen 2.

Responsibilities:

• Design, develop, and maintain data pipeline solutions using Azure Data Factory.
• Collaborate with stakeholders to understand data integration requirements and translate them into scalable and efficient data pipelines.
• Implement data ingestion processes to load data from Network Shared Folders into Azure Data Lake Gen 2.
• Utilize Azure Data Factory components, such as linked services, datasets, pipelines, and activities, to orchestrate and automate data movement and processing workflows.
• Develop and maintain integration solutions that enable starting, pausing, and stopping Jupyter notebooks in Azure Databricks.
• Implement data loading processes to load transformed data into Azure SQL Database efficiently and accurately.
• Ensure data pipeline solutions are robust, scalable, and optimized for performance.
• Collaborate with data architects, data scientists, and other cross-functional teams to ensure data integration solutions align with business requirements and data strategies.
• Perform data validation, testing, and troubleshooting to ensure data integrity and quality throughout the pipeline.
• Monitor and optimize data pipelines to ensure efficient and reliable data movement and processing.
• Develop and maintain documentation for data pipeline solutions, including design documents, deployment instructions, and operational guidelines.
• Stay updated with the latest advancements in Azure Data Factory, Azure Databricks, Azure Data Lake, and Azure SQL Database to continuously improve data integration capabilities.
• Provide technical guidance and support to junior team members, fostering a collaborative and learning-oriented environment.

Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field. Equivalent work experience may be considered.
• Proven experience as an Azure Data Engineer or similar role, with expertise in developing data pipeline solutions using Azure Data Factory.
• Strong understanding of data integration concepts and best practices.
• Experience in loading data from Network Shared Folders into Azure Data Lake Gen 2.
• Experience in initiating/running Jupyter Notebooks in Azure Databricks from Azure Data Factory
• Proficiency in SQL and experience with Azure SQL Database or other relational databases.
• Strong knowledge of Azure services, including Azure Data Factory, Azure Data Lake, and Azure SQL Database.
• Experience with data ingestion, data modeling, transformation, and loading processes.
• Strong analytical and problem-solving skills, with the ability to troubleshoot and resolve complex data integration issues.
• Excellent communication and collaboration skills, with the ability to work effectively in a team environment.
• Self-motivated and proactive, with the ability to prioritize tasks and manage time effectively.
• Relevant certifications as an Azure Data Engineer (DP-203), Azure Data Factory, Azure Databricks, or related technologies are highly desirable.