Senior Data Engineer at Credit Direct Limited

Job Summary

Responsible for building and maintaining the data infrastructure necessary for the organization's data processing and analytics needs. This role involves developing data pipelines, ensuring data quality, and supporting data integration efforts.

RESPONSIBILITIES:

Data Pipeline Development:

• Design, develop, and maintain robust and scalable data pipelines.
• Implement ETL (Extract, Transform, Load) processes to integrate data from various sources.
• Ensure data pipelines are optimized for performance and reliability.
• Work closely with Machine Learning System analysts, and other stakeholders to understand data needs.
• Perform analyses on large sets of data to extract actionable insights that will drive decisions, with a particular focus on our marketing and sales strategy. Engineer novel data sets and features that shed new light on the dynamics of consumer choices.
• Provide technical support for data-related issues and challenges.
• Collaborate with the Lead, Data Engineer to implement data solutions and improvements.

Data Integration and Storage:

• Integrate data from different sources, including databases, APIs, and external data providers.
• Ensure data is clean, consistent, and available for analysis.
• Collaborate with other teams to understand data requirements and deliver integrated solutions.
• Manage and maintain data storage solutions, including databases, data lakes, and data warehouses.
• Ensure data storage systems are efficient, secure, and scalable.
• Implement best practices, data governance and security measures to protect sensitive information as well as data storage and management.

Data Quality Assurance:

• Implement processes to ensure data quality, accuracy, and consistency.
• Monitor data pipelines for errors and address issues promptly.
• Conduct regular data validation and verification.
• Optimize data workflows for performance, reliability, and scalability.
• Perform data profiling, cleansing, and transformation to ensure data quality and integrity.
• Optimize data processing and storage systems for efficiency and performance.
• Continuously evaluate and improve data engineering processes and tools.
• Stay updated with the latest trends and technologies in data engineering.

Requirements

• Bachelor’s degree in Computer Science, Information Technology, Data Science, or a related field.
• At least 3 - 5 years’ experience in data engineering, data architecture, or a related role.
• Experience with ETL processes, data integration, and data pipeline development.
• Experience in a financial institution or similar environment is preferred. 
• Strong programming skills in languages such as Python, Java, or Scala.
• Proficiency in SQL and database management.
• Experience with big data technologies (e.g., Hadoop, Spark) and cloud data platforms (e.g., AWS, Azure, GCP).
• Knowledge of data warehousing solutions (e.g., Redshift, BigQuery, Snowflake).
• Familiarity with data visualization tools (e.g., Tableau, Power BI) is an added advantage.

COMPETENCIES REQUIREMENTS:

Technical:

• Cloud Computing
• Cybersecurity Management 
• Data Analysis
• Database Management
• Digital Transformation
• Information Security Management​
• IT Application Support ​
• IT Governance and Compliance
• IT Infrastructure Management
• IT Service Management (ITSM)
• Mobile Device Management 
• Network Administration
• Programming and Scripting
• Project Management 
• Quality Assurance 
• Regulatory Compliance
• Software Development

Behavioural:

• Entrepreneurial Mindset
• Excellence
• Execution
• Energy
• Empathy
• Evolution
• Emotional Intelligence
• Business Acumen
• Decision-Making
• Result Oriented
• Communication written & verbal
• Stakeholder Management
• Analytical Thinking
• Managing Risk