Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Course Outline
Day 1 – Real-Time Analytics Architecture and ClickHouse Fundamentals
Modern real-time analytics architecture
- Batch analytics versus streaming analytics
- Operational databases versus analytical databases
- OLTP and OLAP workload characteristics
- The respective responsibilities of Kafka, Spark, and ClickHouse
- Reference architecture for on-premises deployment
- Considerations for data flow, latency, throughput, and durability
- Planning for datasets exceeding 100 million records
ClickHouse architecture
- Concepts of column-oriented databases
- ClickHouse server, client, and interfaces
- Data parts, granules, background merges, and sparse indexes
- Single-node and distributed deployment models
- Storage, memory, and CPU considerations
- Native protocol, HTTP interface, and JDBC connectivity
Self-managed installation and configuration
- Linux installation options
- Package and repository management
- Directory structure and configuration hierarchy
- Server and user configuration files
- Starting, stopping, and validating services
- Network ports and remote connections
- Users, roles, and access permissions
- Configuration overrides and environment separation
Hands-on laboratory
- Install the specified ClickHouse LTS release
- Configure server and client connectivity
- Create users, roles, and databases
- Load an initial analytical dataset
- Execute baseline analytical queries
Day 2 – ClickHouse Data Modelling, Engines, and Performance
Data types and schema design
- Numeric, decimal, string, date, and time types
- Arrays, tuples, maps, nested structures, and JSON
- Considerations for LowCardinality, Enum, and Nullable types
- Selecting efficient types for storage and performance
- Schema design for event and transactional data
ClickHouse table engines
- MergeTree family architecture
- MergeTree and ReplicatedMergeTree
- ReplacingMergeTree
- SummingMergeTree and AggregatingMergeTree
- CollapsingMergeTree and VersionedCollapsingMergeTree
- Distributed tables
- Selecting the appropriate engine for a specific workload
Ordering, partitioning, and indexing
- Primary keys and ordering keys in ClickHouse
- Sparse primary indexes
- Partitioning strategies
- Data-skipping indexes
- Projections
- Materialized views
- TTL-based retention and movement
- Avoiding excessive partitions and small parts
Large-volume ingestion and optimisation
- Bulk inserts and asynchronous inserts
- Insert batch sizing
- Compression codecs
- Query execution and
EXPLAIN - Identifying excessive reads
- Optimising filters, aggregations, and joins
- Monitoring parts and merges
- Benchmarking methodology
Hands-on laboratory
- Generate and load over 100 million synthetic records
- Compare alternative ordering and partitioning strategies
- Measure compression and query performance
- Implement a materialized aggregation
- Diagnose and optimise inefficient queries
Day 3 – ClickHouse Administration, Backup, Recovery, and Migration
Production administration
- Configuration management
- Storage policies and volumes
- User and role administration
- Resource limits and quotas
- Workload isolation
- Logging and system tables
- Query monitoring and cancellation
- Capacity and disk-space monitoring
- Managing background merges and mutations
Security
- Authentication and role-based access control
- Network restrictions
- TLS considerations
- Protecting credentials and configuration
- Least-privilege service accounts
- Audit and query logging
Backup and recovery
- Backup objectives, RPO, and RTO
- Full and incremental backups
- Synchronous and asynchronous backups
- Local-disk and object-storage destinations
- Database, table, and access-management backups
- Restore procedures and validation
- Automating backup execution
- Recovery testing and operational runbooks
Migration
- Schema and data migration approaches
- Native and file-based migration methods
- Parallel loading strategies
- Validating row counts, checksums, and query results
- Minimising downtime
- Migration rehearsal and rollback planning
- Upgrade and patch-management considerations
Hands-on laboratory
- Configure a backup destination
- Execute full and incremental backups
- Simulate data loss
- Restore into a separate environment
- Perform a controlled database migration
- Validate data completeness and recovery time
Day 4 – ClickHouse High Availability and Distributed Operations
High-availability architecture
- Replication versus sharding
- Designing shards and replicas
- ClickHouse Keeper architecture
- Keeper quorum and failure tolerance
- ReplicatedMergeTree engines
- Distributed table architecture
- Distributed DDL
- Read and write routing
- Load balancing and failover
Cluster deployment and operation
- Cluster topology configuration
- Creating replicated databases and tables
- Adding and removing replicas
- Replica synchronisation
- Monitoring replication queues
- Detecting delayed or inactive replicas
- Handling broken and detached parts
- Network partitions and quorum behaviour
- Rolling maintenance and upgrades
- Disaster-recovery architecture
Hands-on laboratory
- Deploy a multi-node ClickHouse cluster
- Configure ClickHouse Keeper
- Create replicated and distributed tables
- Load a large dataset across shards
- Simulate a failed replica
- Restore availability and validate consistency
- Test routing, failover, and replica recovery
Day 5 – Apache Kafka Setup, Producers, Consumers, and Operations
Kafka fundamentals
- Event-streaming concepts
- Brokers, controllers, topics, and partitions
- Records, keys, offsets, and ordering
- Replication and in-sync replicas
- Producers, consumers, and consumer groups
- Kafka durability and delivery semantics
- Kafka as an event log versus a traditional message queue
Kafka 3.9 installation and configuration
- Java and operating-system requirements
- KRaft architecture
- Controller and broker roles
- Storage formatting and cluster IDs
- Broker and listener configuration
- Creating and managing topics
- Partition counts and replication factors
- Retention by time and size
- Log compaction
- Producer and consumer configuration
- Acknowledgements, retries, batching, and compression
- Consumer offsets, rebalancing, and lag
Availability, security, and monitoring
- Broker and partition failure scenarios
- Minimum in-sync replicas
- Unclean leader-election considerations
- TLS and SASL concepts
- Access-control lists
- Monitoring throughput, lag, and broker health
- Capacity and retention planning
- Troubleshooting producers and consumers
Hands-on laboratory
- Deploy a multi-broker Kafka 3.9 cluster
- Create replicated topics
- Configure command-line and application producers
- Configure consumer groups
- Test partitioning and message ordering
- Simulate a broker failure
- Measure and troubleshoot consumer lag
Day 6 – Kafka-to-ClickHouse Ingestion and Spark Foundations
Kafka-to-ClickHouse ingestion
- Direct ingestion options
- ClickHouse Kafka table engine
- Kafka engine configuration
- Materialized views for persistent ingestion
- Parsing JSON, Avro, and other event formats
- Consumer-group configuration
- Controlling ingestion parallelism
- Delivery semantics and duplicate handling
- Replay and offset management
- Schema evolution
- Malformed messages and dead-letter patterns
- Monitoring Kafka ingestion from ClickHouse
- Overview of the Kafka Connect ClickHouse sink
Hands-on ingestion laboratory
- Publish high-volume events to Kafka
- Create a Kafka engine table
- Create destination MergeTree tables
- Implement materialized-view ingestion
- Stop and restart consumers
- Replay events safely
- Diagnose malformed messages and ingestion lag
Apache Spark foundations
- Spark driver, executors, workers, and applications
- Spark standalone, YARN, and Kubernetes deployment options
- Installing Spark 4.0.4
- Java 17 and Scala 2.13 requirements
- SparkSession and Spark configuration
- Spark SQL and DataFrames
- Transformations, actions, and lazy evaluation
- Partitions, stages, tasks, and shuffles
- Reading and writing large datasets
Day 7 – Spark Streaming, Workload Management, and Capstone
Spark Structured Streaming
- Structured Streaming processing model
- Kafka source configuration
- Parsing and transforming Kafka events
- Event time and processing time
- Windows and aggregations
- Watermarks and late-arriving data
- Output modes
- Checkpointing and recovery
- Trigger intervals and micro-batching
- Failure and restart behaviour
Spark and ClickHouse integration
- Official ClickHouse Spark connector
- Connector and JDBC approaches
- Spark DataSource V2
- Reading ClickHouse tables
- Writing transformed data to ClickHouse
- Batch sizing and parallel writes
- Avoiding excessive small inserts
- Schema and type mapping
- Performance considerations
Workload and resource management
spark-submit- Driver and executor memory
- Executor cores and parallelism
- Static and dynamic resource allocation
- FIFO and FAIR scheduling
- Fair-scheduler pools
- Shuffle management
- Handling data skew
- Spark UI, event logs, and metrics
- Diagnosing failed and slow jobs
End-to-end capstone
Participants will:
- Generate or receive a high-volume event stream
- Publish the events to Kafka
- Consume and transform selected events using Spark
- Ingest events into ClickHouse
- Create an optimised analytical model
- Process a dataset exceeding 100 million rows
- Execute real-time analytical queries
- Monitor Kafka lag, Spark workloads, and ClickHouse performance
- Simulate a component failure
- Recover the pipeline without losing analytical availability
- Present the final architecture and operational runbook
Requirements
Participants should possess:
- Working knowledge of SQL
- Basic Linux command-line and system administration skills
- A fundamental understanding of databases and data modelling
- Familiarity with application development concepts
- Basic knowledge of Python, Java, or Scala
For the practical Spark exercises, PySpark is recommended.
Prior experience with ClickHouse, Kafka, or Spark is not required.
35 Hours
Testimonials (3)
Learning that the QGIS and a tool that can used by other different professionals such land survey
Bame Duncan Koko - Bentel Technologies (Pty) Ltd
Course - QGIS for Geographic Information System
How to use open satellites data for real applications
Tshering Dorji - Druk Holding and Investments
Course - Advanced Geographic Information Systems (GIS)
Hands-on examples allowed us to get an actual feel for how the program works. Good explanations and integration of theoretical concepts and how they relate to practical applications.
