Senior Data Engineer (Databricks + PySpark + ADF + Unity Catalog)
Full-Time | Remote / Mexico City, Bogotá, Costa Rica, Buenos Aires, Lima, São Paulo
About the Role:
We are seeking a highly skilled Senior Data Engineer with hands-on experience in Databricks, Unity Catalog, and Azure Data Factory (ADF) to support a large-scale enterprise data modernization initiative. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, enabling secure data governance, and supporting cloud-based analytics solutions in Azure.
This role will collaborate closely with architects, business stakeholders, and cross-functional engineering teams to build reliable, high-performance data platforms and accelerate data-driven decision-making.
Key Responsibilities:
- Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
- Build and maintain scalable data ingestion frameworks for structured and unstructured data sources.
- Develop data transformation processes using PySpark, Spark SQL, and Databricks notebooks.
- Implement and manage Unity Catalog for enterprise data governance, security, and access controls.
- Create and optimize Delta Lake architectures for analytics and reporting workloads.
- Monitor pipeline performance and troubleshoot production issues.
- Collaborate with data architects to implement enterprise data models and best practices.
- Ensure data quality, integrity, security, and compliance standards are met.
- Support CI/CD deployment processes and DevOps practices for data engineering solutions.
- Work closely with business and analytics teams to understand requirements and deliver scalable solutions.
Your Skils & Experience:
- 4-7+ years of experience in Data Engineering or Big Data development.
- Strong hands-on experience with Azure Databricks.
- Proficiency in PySpark, Spark SQL, and distributed data processing.
- Experience implementing and managing Unity Catalog.
- Strong experience with Azure Data Factory (ADF).
- Experience working with Delta Lake and modern Lakehouse architectures.
- Solid understanding of data modeling, ETL/ELT, and data warehousing concepts.
- Experience with Azure cloud services and ecosystem.
- Proficiency with Git, version control, and CI/CD processes.
- Strong analytical and problem-solving skills.
- Excellent communication and stakeholder management abilities.
Preferred Qualifications:
- Experience with Azure Synapse Analytics.
- Familiarity with Collibra or other data governance platforms.
- Experience with Infrastructure as Code (Terraform, ARM Templates).
- Databricks Certification (Data Engineer Associate/Professional).
- Microsoft Azure Data Engineer Associate Certification.
Nice to Have:
- Experience supporting enterprise-scale data migration projects.
- Knowledge of healthcare, retail, or consumer products data environments.
- Exposure to real-time data processing and streaming technologies.
The pay rate for this position is
- Senior Associate L1 (4+ YoE) $31 - $36/hour
The estimated pay rate is based on the candidate’s demonstrated experience, English proficiency, technical assessment results, and overall alignment with the role requirements.