Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Foundations

• Overview of the Big Data ecosystem and Spark's role in contemporary data platforms
• Insights into Spark architecture: driver, executors, cluster manager, lazy evaluation, DAG, and execution planning
• Distinguishing between RDD and DataFrame APIs and identifying appropriate use cases for each
• Creating and configuring SparkSession, with a focus on fundamental application configuration

Module 2: PySpark DataFrames

• Reading and writing data from enterprise sources and formats, including CSV, JSON, Parquet, and Delta
• Manipulating PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced operations such as window functions, timestamp handling, and working with nested data structures
• Applying data quality checks and writing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Fundamentals for Spark Users

• Conceptual view for cluster consumers: how Docker images and Kubernetes pods relate to Spark driver and executor processes
• Examining the lifecycle of a Spark job on Kubernetes: pod scheduling, resource requests/limits, and mapping session settings to cluster resources
• Defining the boundary between Spark session parameters controlled by users and cluster provisioning managed by platform teams, and why this distinction is crucial for tuning
• Hands-on exercise: Inspect a live Spark-on-Kubernetes job within the sandbox cluster to identify driver and executor pods

Module 4: Spark Session Configuration on Kubernetes — Deep Dive

• Analyzing executor count versus executor size: navigating memory-versus-cores trade-offs and reasoning strategies
• Driver and executor memory allocation, memory overhead, and their translation into pod resource requirements
• Dynamic allocation: exploring its behaviour on Kubernetes, identifying scenarios where it conserves resources and where it does not
• Hands-on exercise: Execute identical jobs under varying executor and core configurations to compare runtime and resource consumption

Module 5: Scaling Behaviour & Cost Optimisation

• Understanding how node addition or removal impacts job behaviour, completion time, and resource consumption
• Evaluating the performance and cost implications of many small executors versus fewer large ones
• Shuffle behaviour and partition tuning, including estimating the financial impact of configuration choices
• Hands-on exercise: Scale the sandbox cluster from five to ten nodes and monitor the effects on job completion time and resource usage

Module 6: Efficient Data Ingestion & Partitioning

• Analyzing the small-files problem: why sources fragmented across numerous 1–5 MB Parquet files degrade performance and distort partitioning
• Strategies for repartitioning and coalescing data
• Controlling partition sizes during both read and write operations
• Writing Parquet files efficiently to prevent the recurrence of small-file issues downstream
• Hands-on exercise: Load a dataset consisting of many small Parquet partitions, apply various repartitioning strategies, and assess performance improvements

Module 7: Pandas Memory Management & Failure Diagnosis

• Identifying root causes of out-of-memory errors in Pandas and recognizing warning signs
• Applying memory-efficient conversion patterns between Spark and Pandas environments
• Preventing memory exhaustion when exporting large datasets to CSV
• Utilizing chunked processing and dtype optimization in resource-constrained environments
• Hands-on exercise: Reproduce a typical Pandas out-of-memory scenario and resolve it using chunking and dtype optimization

Module 8: Polars as a Complementary Tool

• Positioning Polars relative to Pandas: analyzing performance characteristics, lazy evaluation, and memory behaviour
• Understanding Polars' role alongside PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Hands-on exercise: Rewrite a Pandas-heavy transformation in Polars and benchmark memory usage and processing speed

Module 9: Applying Optimisation to ETL & ML Workloads

• Integrating configuration, partitioning, and memory-management principles into a realistic ETL pipeline
• Understanding optimization considerations unique to machine learning workloads sharing the same cluster resources
• Applying a systematic tuning workflow to diagnose cost and performance issues
• Hands-on exercise: Complete an end-to-end mini-project involving data loading, transformation, and a simple model-training step, with participants independently tuning Spark configurations