Senior Data Engineer at Credit Direct Limited

Job Summary

• The role of a Senior Data Engineer is critical in designing, developing, and maintaining robust data infrastructure and systems that enable efficient and accurate data processing, storage, and retrieval.
• Senior Data Engineers play a pivotal role in ensuring the availability, integrity, and security of data to support data-driven decision-making and enable advanced analytics and machine learning initiatives.

Responsibilities

• Support leadership with research on key business initiatives and challenges.
• Perform analyses on large sets of data to extract actionable insights that will drive decisions, with a particular focus on our marketing and sales strategy. Engineer novel data sets and features that shed new light on the dynamics of consumer choices.
• Define success measures and build tools that predict and track performance of strategic projects.
• Communicate data-driven insights and recommendations to non-technical audiences, through clear visualizations and presentations.
• Perform data loads and optimize data for extraction and reporting use.
• Design and implement ETL jobs and transformations to populate a data warehouse.
• Create and maintain data warehouse and database architectures to support data storage and retrieval.
• Optimize data workflows for performance, reliability, and scalability.
• Perform data profiling, cleansing, and transformation to ensure data quality and integrity.
• Monitor and troubleshoot data pipelines, identifying and resolving any data-related issues.
• Implement data governance and security measures to protect sensitive information.

Job Requirement

• Degree in Computer Science/Engineering, Information Technology, Mathematics, Statistics, Economics, or another quantitative field; advanced degree is desirable.
• 5+ years) of relevant experience analyzing complex data with SQL, Python, and/or R.
• Ownership of end-to-end integration and ingestion data ecosystem
• Strong knowledge of data warehouse concepts, ETL frameworks with strong usage of Apache Airflow, DBT, and CI/CD solutions.
• Strong understanding of database concepts and experience with SQL and NoSQL databases. Experience with big data technologies and frameworks such as Hadoop, Spark, or Apache Kafka.
• Sound understanding of Microsoft SQL Server Integration Services and SQL Server Analysis Services
• Understanding of cloud technologies such as AWS and GCP.

Experience:

• Experience in using Python for building and redefining data pipelines is required.
• Solid understanding of analytic data architecture and data modelling concepts and principles, including data lakes, warehouses and marts.
• Capable of tackling very loosely defined problems, leading and owning diagnostic data engineering frameworks
• Comfortable building and maintaining relationships with senior management, as well as team members.

Required Knowledge, Skills & Competencies:

• Familiarity with Google Cloud Platform (e.g GCS, Google Cloud Composer and BigQuery) is a plus.
• Must have expert-level experience working with Google's batch or streaming data processing solutions (such as BigQuery, Dataform, and BI Engine)
• Working knowledge with enterprise-class Business Intelligence tools such as PowerBI, Tableau, etc.
• Experience Designing Tables and Writing Queries within a SQL Environment.
• Experience with Docker and cloud deployment technologies.

Person Specification

• Effective verbal and written communication skills to present complex data findings in a clear and concise manner.
• Strong interpersonal skills and the ability to work collaboratively in cross-functional teams.
• Willingness to share knowledge, support colleagues, and contribute to a positive team environment.
• Commitment to maintaining data confidentiality, integrity, and ethical standards.
• Professionalism, reliability, and the ability to handle sensitive or confidential information with discretion.