Data Team Lead Job at Incedo Inc., San Rafael, CA

V3RBQjZmUUJJcjRKVjVRcnJWaFJJYVE9
  • Incedo Inc.
  • San Rafael, CA

Job Description

Job Title Databricks Data Lead

Location

  • San Rafael, CA - Bay Area, California (Local candidates only)
  • Hybrid: 3 days/week onsite at client office in San Rafael, CA

Experience Level

  • 8-12 years in data engineering, analytics engineering, or Distributed data systems

Role Overview

We are seeking a Databricks Data Lead to support the design, implementation, and optimization of cloud-native data platforms built on the Databricks Lakehouse Architecture . This is a hands-on, engineering-driven role requiring deep experience with Apache Spark, Delta Lake, and scalable data pipeline development , combined with early-stage architectural responsibilities.

The role involves close onsite collaboration with client stakeholders , translating analytical and operational requirements into robust, high-performance data architectures , while adhering to best practices for data modeling, governance, reliability, and cost efficiency .

Key Responsibilities

  • Design, develop, and maintain batch and near-real-time data pipelines using Databricks, PySpark, and Spark SQL
  • Implement Medallion (Bronze/Silver/Gold) Lakehouse architectures , ensuring proper data quality, lineage, and transformation logic across layers
  • Build and manage Delta Lake tables , including schema evolution, ACID transactions, time travel, and optimized data layouts
  • Apply performance optimization techniques such as partitioning strategies, Z-Ordering, caching, broadcast joins, and Spark execution tuning
  • Support dimensional and analytical data modeling for downstream consumption by BI tools and analytics applications
  • Assist in defining data ingestion patterns (batch, incremental loads, CDC, and streaming where applicable)
  • Troubleshoot and resolve pipeline failures, data quality issues, and Spark job performance bottlenecks
  • Collaborate onsite with client data engineers, analysts, and business stakeholders to:
  • Gather technical requirements
  • Review architecture designs
  • Validate implementation approaches
  • Maintain technical documentation covering data flows, transformation logic, table designs, and architectural decisions
  • Contribute to code reviews, CI/CD practices, and version control workflows to ensure maintainable and production-grade solutions

Required Skills & Qualifications

  • Strong hands-on experience with Databricks Lakehouse Platform
  • Deep working knowledge of Apache Spark internals , including:
  • Spark SQL
  • DataFrames/Datasets
  • Shuffle behavior and execution plans
  • Advanced Python (PySpark) and SQL development skills
  • Solid understanding of data warehousing concepts , including:
  • Star and snowflake schemas
  • Fact/dimension modeling
  • Analytical vs operational workloads
  • Experience working with cloud data platforms on AWS, Azure, or GCP
  • Practical experience with Delta Lake , including:
  • Merge/upsert patterns
  • Schema enforcement and evolution
  • Data compaction and optimization
  • Proficiency with Git-based version control and collaborative development workflows
  • Strong verbal and written communication skills for client-facing technical discussions
  • Ability and willingness to work onsite 3 days/week in San Rafael, CA

Nice-to-Have Skills

  • Exposure to Databricks Unity Catalog , data governance, and access control models
  • Experience with Databricks Workflows , Apache Airflow , or Azure Data Factory for orchestration
  • Familiarity with streaming frameworks (Spark Structured Streaming, Kafka) and/or CDC patterns
  • Understanding of data quality frameworks , validation checks, and observability concepts
  • Experience integrating Databricks with BI tools such as Power BI, Tableau, or Looker
  • Awareness of cost optimization strategies in cloud-based data platforms
  • Prior Lifesciences Domain Experience

Education

  • Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience)

Why This Role

  • Hands-on ownership of Databricks Lakehouse implementations in a real-world enterprise environment
  • Direct client-facing exposure with a leading Bay Area organization
  • Opportunity to evolve from senior data engineering into formal data architecture responsibilities
  • Strong growth path toward Senior Databricks Architect / Lead Data Platform Engineer

Job Tags

Work at office, Local area, 3 days per week,

Similar Jobs

AIM World Services

Senior Consultant Job at AIM World Services

 ...Duty Location: CONUS with frequent travel to Middle East locations Position Type: Consultant (initial engagement) with potential conversion to Full-Time Work Environment: Remote support during planning and design phases; in-country deployment as required... 

MANGO

Keyholder Job at MANGO

 ...shopfloor are spotless: product well replenished and that the store clean, tidy and merchandised in an attractive way in order to maximize sales and offer the experience that our customers deserve. In terms of security standards, either in fitting rooms, stockrooms, shopfloor,... 

GLOBAL BUILDING SERVICES, INC.

HOUSEKEEPING - LIMPIEZA - (DAY PORTER - PORTERO DE DIA) - SAN MATEO Job at GLOBAL BUILDING SERVICES, INC.

 ...Buscamos nuevos miembros para nuestro equipo para limpiar tiendas de ventas durante el turno del dia. Tareas tipicas incluyen aspirar, limpieza de banos, barrer, trapear, limpiar ventanas, recolectar basuras y otras tareas tipicas de limpieza. El trabajo es de medio o... 

Online Marketing Gurus

Remote SEO Copywriter Job at Online Marketing Gurus

 ..., helping over 1,000 businesses thrive since our establishment in 2012. With a team of 200+ certified Growth Gurus, we specialize in SEO, PPC, and social advertising, delivering data-driven strategies that transform online presences. Our commitment to research and development... 

AYS Inc

Social Media Moderator Job at AYS Inc

 ...Position: Social Media Moderator Location: REMOTE Company Overview: AYS Inc is a leading provider of innovative solutions in the field of digital marketing and social media management. Our team of experts is dedicated to helping businesses of all sizes reach...