Self-Hosting Decision Models: Serving, Quantisation, Latency and Operations Training Course

Deployment Strategies

• Comparing hosted API versus self-hosted advantages and trade-offs.
• Patterns for on-premises, edge, and air-gapped environments.
• Single model versus managing a model portfolio.
• Comparative analysis of cost per decision.

Hardware Sizing and Capacity Planning

• Assessing memory footprint relative to parameter count.
• CPU-only execution versus GPU acceleration.
• Planning for concurrency and throughput.
• Allowing headroom for increasing task lengths.

Serving Stacks

• Evaluating vLLM and SGLang options.
• Utilising llama.cpp and GGUF execution.
• Using Ollama for local development workflows.
• Selecting tools based on the specific shape of the decision workload.

Quantisation While Preserving Calibration

• Precision formats and associated memory savings.
• Measuring quality degradation and calibration loss.
• Per-tensor and importance-aware quantisation variants.
• Conducting acceptance testing post-quantisation.

Latency Engineering

• Analysing median versus tail latency.
• Understanding latency growth as context length increases.
• Implementing warm starts and model caching.
• Batching multiple decision queries.

Edge and Constrained Environments

• Running small decision models on CPU resources.
• Deployment on Apple Silicon and embedded systems.
• Offline operation strategies and update protocols.
• Distributing weights in air-gapped scenarios.

Containerisation and Scaling

• Designing images optimized for model serving.
• Managing volumes and caches effectively.
• Implementing horizontal scaling and routing.
• Setting resource limits and sharing GPUs.

Observability

• Tracking request, latency, and error metrics.
• Monitoring confidence distributions.
• Tracking decision outcomes and escalations.
• Setting alerts for calibration drift.

Versioning and Rollout

• Versioning models and adapters.
• Deploying via shadow and canary methods.
• Using pinned model files for audit purposes.
• Establishing rollback procedures.

Security and Multi-Tenancy

• Implementing authentication and rate limiting.
• Ensuring tenant isolation for prompts and logs.
• Managing network isolation and egress controls.
• Maintaining audit logs for decisions.