Databricks Migration Engineer (Remote)

<h2>Overview</h2> <ul> <li>Tier One Technologies is seeking a Databricks Migration Engineer to support our US Government client.</li> <li>This remote Contract-to-Hire position will be originated in Rosslyn, VA. Washington DC area and East Coast time zone candidates preferred.</li> <li>Must be a US Citizen.</li> <li>SELECTED CANDIDATES WITHOUT REQUIRED CLEARANCE WILL BE SUBJECT TO A FEDERAL GOVERNMENT BACKGROUND INVESTIGATION TO RECEIVE IT.</li> </ul> <h2>Responsibilities</h2> <ul> <li>Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake), ensuring best practice Lakehouse design.</li> <li>Translate traditional relational data warehousing paradigms into scalable, distributed Lakehouse frameworks (Bronze, Silver, Gold).</li> <li>Design robust, reusable ETL/ELT frameworks using PySpark, Delta Live Tables (DLT), and Databricks Workflows.</li> <li>Architect and refine the Gold Layer (dimensional models, star schemas) specifically to maximize Power BI performance.</li> <li>Optimize Databricks SQL Warehouses to support high-concurrency, low-latency Power BI queries (DirectQuery and Import modes).</li> <li>Implement advanced optimization techniques, including Z-Ordering, data skipping, liquid clustering, and materialized views.</li> <li>Define and enforce governance standards for cluster sizing, auto-scaling policies, and serverless SQL compute to balance performance with cost.</li> <li>Implement proactive monitoring dashboards to track Databricks Unit (DBU) consumption and identify cost-saving opportunities.</li> <li>Establish best practices for partition strategies and file size management within Delta Lake.</li> <li>Design and implement a robust data security model using Unity Catalog for centralized governance.</li> <li>Enforce row-level and column-level security policies to ensure compliant data access for Power BI consumers and internal analysts.</li> <li>Align the Lakehouse security architecture with existing enterprise Azure Active Directory (Microsoft Entra ID) and RBAC standards.</li> <li>Act as the primary technical lead, conducting dedicated pair-programming sessions, workshops, and code reviews to transition the team from SQL-centric to Spark-centric thinking.</li> <li>Create comprehensive technical documentation, including architecture diagrams, design patterns, and optimization playbooks.</li> <li>Build a foundational knowledge transfer framework to ensure the internal team is fully self-sufficient post-migration.</li> <li>Communicate effectively verbally and in written form to both technical and non-technical audience</li> <li>Work in an organized fashion, completing tasks timely while paying close attention to details</li> </ul> <h2>Qualifications</h2> <ul> <li>Bachelor's degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical field.</li> <li>5+ years of experience in Data Engineering, Data System Development or related roles.</li> <li>5+ years of experience with Cloud platforms (e.g. Azure, AWS, GCP).</li> <li>1+ year leading complex, cross-functional data projects and technical teams.</li> <li>Experience with Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms.</li> <li>Experience with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark.</li> <li>Excellent communication skills.</li> <li>Must be a US Citizen and be able to obtain a Position of Public Trust Clearance.</li> <li>Must have resided in the US for the last 5 years and not have traveled outside the US for a combined total of 6 months or more in last 5 years.</li> </ul>