Data Engineer at OnePyramid

About the Project:

• You will work on a system that processes sensitive partner data locally on partner infrastructure (not in the cloud) before any transfer occurs. Security-first design is mandatory.

Key Responsibilities:

• Build a Python-based application that reads structured exports (CSV, Excel) from partner databases
• Generates consistent pseudonymous identifiers using HMAC-SHA256
• Removes direct identifiers and generalizes quasi-identifiers
• Detects residual sensitive information in free-text fields using NLP (SpaCy NER)
• Handles real-world data quality issues safely and gracefully
• Ensures no sensitive data leaks via logs, temp files, or memory
• Test pseudonymous identifier consistency across multiple runs
• Measure record linkage success and failure rates
• Evaluate re-identification risk using statistical methods
• Produce quantitative evidence suitable for research publication
• Write clear technical specifications for anonymization methods
• Deliver readable, well-commented code for external review
• Create a methodology document enabling independent replication

Required Skills & Experience:

• Python: Strong pandas skills; experience with large, messy datasets; performance optimization
• Cryptography: Hashing algorithms, HMAC, secure key management
• Databases: PostgreSQL or MongoDB
• Cloud & Big Data: AWS, Azure, or GCP; Spark or Hive
• NLP: SpaCy or similar for Named Entity Recognition (using existing models)
• Security Mindset (Critical): Defensive coding practices, No logging of sensitive values and Awareness of data leakage risks (logs, memory, temp files)
• Data Analysis & Communication
• Statistical analysis and metrics
• Jupyter notebooks and visualizations
• Ability to explain technical concepts to non-technical stakeholders
• Comfortable coordinating remotely with partner IT teams