Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 35 hours
Course Outline
Introduction and Diagnostic Foundations
- Overview of LLM system failure modes and specific Ollama-related issues.
- Establishing reproducible experiments within controlled environments.
- Debugging toolkit: local logs, request/response captures, and sandboxing techniques.
Reproducing and Isolating Failures
- Techniques for crafting minimal failing examples and seeds.
- Distinguishing between stateful and stateless interactions to isolate context-related bugs.
- Managing determinism, randomness, and controlling non-deterministic behaviour.
Behavioural Evaluation and Metrics
- Quantitative metrics: accuracy, ROUGE/BLEU variants, calibration, and perplexity proxies.
- Qualitative evaluations: human-in-the-loop scoring and rubric design.
- Task-specific fidelity checks and defining acceptance criteria.
Automated Testing and Regression
- Unit testing for prompts and components, alongside scenario and end-to-end tests.
- Developing regression suites and establishing golden example baselines.
- Integrating Ollama model updates into CI/CD pipelines with automated validation gates.
Observability and Monitoring
- Implementing structured logging, distributed tracing, and correlation IDs.
- Key operational metrics: latency, token usage, error rates, and quality signals.
- Configuring alerting, dashboards, and SLIs/SLOs for model-backed services.
Advanced Root Cause Analysis
- Tracing through graphed prompts, tool calls, and multi-turn conversation flows.
- Conducting comparative A/B diagnosis and ablation studies.
- Investigating data provenance, debugging datasets, and resolving dataset-induced failures.
Safety, Robustness, and Remediation Strategies
- Implementing mitigations: filtering, grounding, retrieval augmentation, and prompt scaffolding.
- Applying rollback, canary, and phased rollout patterns for model updates.
- Conducting post-mortems, extracting lessons learned, and fostering continuous improvement loops.
Summary and Next Steps
Requirements
- Substantial experience in building and deploying LLM applications.
- A solid understanding of Ollama workflows and model hosting practices.
- Proficiency in Python, Docker, and fundamental observability tools.
Target Audience
- AI Engineers.
- MLOps Professionals.
- QA Teams managing production LLM systems.