Machine Learning Pipeline - Multi-Agent MLOps Orchestration

Design and implement a complete ML pipeline for: $ARGUMENTS

Use this skill when

• Working on machine learning pipeline - multi-agent mlops orchestration tasks or workflows
• Needing guidance, best practices, or checklists for machine learning pipeline - multi-agent mlops orchestration

Do not use this skill when

• The task is unrelated to machine learning pipeline - multi-agent mlops orchestration
• You need a different domain or tool outside this scope

Instructions

• Clarify goals, constraints, and required inputs.
• Apply relevant best practices and validate outcomes.
• Provide actionable steps and verification.
• If detailed examples are required, open resources/implementation-playbook.md.

Thinking

This workflow orchestrates multiple specialized agents to build a production-ready ML pipeline following modern MLOps best practices. The approach emphasizes:

• Phase-based coordination: Each phase builds upon previous outputs, with clear handoffs between agents
• Modern tooling integration: MLflow/W&B for experiments, Feast/Tecton for features, KServe/Seldon for serving
• Production-first mindset: Every component designed for scale, monitoring, and reliability
• Reproducibility: Version control for data, models, and infrastructure
• Continuous improvement: Automated retraining, A/B testing, and drift detection

The multi-agent approach ensures each aspect is handled by domain experts: - Data engineers handle ingestion and quality - Data scientists design features and experiments - ML engineers implement training pipelines - MLOps engineers handle production deployment - Observability engineers ensure monitoring

Phase 1: Data & Requirements Analysis

subagent_type: data-engineer prompt: | Analyze and design data pipeline for ML system with requirements: $ARGUMENTS

Deliverables: 1. Data source audit and ingestion strategy: - Source systems and connection patterns - Schema validation using Pydantic/Great Expectations - Data versioning with DVC or lakeFS - Incremental loading and CDC strategies

1. Data quality framework:

   • Profiling and statistics generation
   • Anomaly detection rules
   • Data lineage tracking
   • Quality gates and SLAs

2. Storage architecture:

   • Raw/processed/feature layers
   • Partitioning strategy
   • Retention policies
   • Cost optimization

Provide implementation code for critical components and integration patterns.

subagent_type: data-scientist prompt: | Design feature engineering and model requirements for: $ARGUMENTS Using data architecture from: {phase1.data-engineer.output}

Deliverables: 1. Feature engineering pipeline: - Transformation specifications - Feature store schema (Feast/Tecton) - Statistical validation rules - Handling strategies for missing data/outliers

1. Model requirements:

   • Algorithm selection rationale
   • Performance metrics and baselines
   • Training data requirements
   • Evaluation criteria and thresholds

2. Experiment design:

   • Hypothesis and success metrics
   • A/B testing methodology
   • Sample size calculations
   • Bias detection approach

Include feature transformation code and statistical validation logic.

Phase 2: Model Development & Training

subagent_type: ml-engineer prompt: | Implement training pipeline based on requirements: {phase1.data-scientist.output} Using data pipeline: {phase1.data-engineer.output}

Build comprehensive training system: 1. Training pipeline implementation: - Modular training code with clear interfaces - Hyperparameter optimization (Optuna/Ray Tune) - Distributed training support (Horovod/PyTorch DDP) - Cross-validation and ensemble strategies

1. Experiment tracking setup:

   • MLflow/Weights & Biases integration
   • Metric logging and visualization
   • Artifact management (models, plots, data samples)
   • Experiment comparison and analysis tools

2. Model registry integration:

   • Version control and tagging strategy
   • Model metadata and lineage
   • Promotion workflows (dev -> staging -> prod)
   • Rollback procedures

Provide complete training code with configuration management.

subagent_type: python-pro prompt: | Optimize and productionize ML code from: {phase2.ml-engineer.output}

Focus areas: 1. Code quality and structure: - Refactor for production standards - Add comprehensive error handling - Implement proper logging with structured formats - Create reusable components and utilities

1. Performance optimization:

   • Profile and optimize bottlenecks
   • Implement caching strategies
   • Optimize data loading and preprocessing
   • Memory management for large-scale training

2. Testing framework:

   • Unit tests for data transformations
   • Integration tests for pipeline components
   • Model quality tests (invariance, directional)
   • Performance regression tests

Deliver production-ready, maintainable code with full test coverage.

Phase 3: Production Deployment & Serving

subagent_type: mlops-engineer prompt: | Design production deployment for models from: {phase2.ml-engineer.output} With optimized code from: {phase2.python-pro.output}

Implementation requirements: 1. Model serving infrastructure: - REST/gRPC APIs with FastAPI/TorchServe - Batch prediction pipelines (Airflow/Kubeflow) - Stream processing (Kafka/Kinesis integration) - Model serving platforms (KServe/Seldon Core)

1. Deployment strategies:

   • Blue-green deployments for zero downtime
   • Canary releases with traffic splitting
   • Shadow deployments for validation
   • A/B testing infrastructure

2. CI/CD pipeline:

   • GitHub Actions/GitLab CI workflows
   • Automated testing gates
   • Model validation before deployment
   • ArgoCD for GitOps deployment

3. Infrastructure as Code:

   • Terraform modules for cloud resources
   • Helm charts for Kubernetes deployments
   • Docker multi-stage builds for optimization
   • Secret management with Vault/Secrets Manager

Provide complete deployment configuration and automation scripts.

subagent_type: kubernetes-architect prompt: | Design Kubernetes infrastructure for ML workloads from: {phase3.mlops-engineer.output}

Kubernetes-specific requirements: 1. Workload orchestration: - Training job scheduling with Kubeflow - GPU resource allocation and sharing - Spot/preemptible instance integration - Priority classes and resource quotas

1. Serving infrastructure:

   • HPA/VPA for autoscaling
   • KEDA for event-driven scaling
   • Istio service mesh for traffic management
   • Model caching and warm-up strategies

2. Storage and data access:

   • PVC strategies for training data
   • Model artifact storage with CSI drivers
   • Distributed storage for feature stores
   • Cache layers for inference optimization

Provide Kubernetes manifests and Helm charts for entire ML platform.

Phase 4: Monitoring & Continuous Improvement

subagent_type: observability-engineer prompt: | Implement comprehensive monitoring for ML system deployed in: {phase3.mlops-engineer.output} Using Kubernetes infrastructure: {phase3.kubernetes-architect.output}

Monitoring framework: 1. Model performance monitoring: - Prediction accuracy tracking - Latency and throughput metrics - Feature importance shifts - Business KPI correlation

1. Data and model drift detection:

   • Statistical drift detection (KS test, PSI)
   • Concept drift monitoring
   • Feature distribution tracking
   • Automated drift alerts and reports

2. System observability:

   • Prometheus metrics for all components
   • Grafana dashboards for visualization
   • Distributed tracing with Jaeger/Zipkin
   • Log aggregation with ELK/Loki

3. Alerting and automation:

   • PagerDuty/Opsgenie integration
   • Automated retraining triggers
   • Performance degradation workflows
   • Incident response runbooks

4. Cost tracking:

   • Resource utilization metrics
   • Cost allocation by model/experiment
   • Optimization recommendations
   • Budget alerts and controls

Deliver monitoring configuration, dashboards, and alert rules.

Configuration Options

• experiment_tracking: mlflow | wandb | neptune | clearml
• feature_store: feast | tecton | databricks | custom
• serving_platform: kserve | seldon | torchserve | triton
• orchestration: kubeflow | airflow | prefect | dagster
• cloud_provider: aws | azure | gcp | multi-cloud
• deployment_mode: realtime | batch | streaming | hybrid
• monitoring_stack: prometheus | datadog | newrelic | custom

Success Criteria

1. Data Pipeline Success:
2. < 0.1% data quality issues in production
3. Automated data validation passing 99.9% of time
4. Complete data lineage tracking

5. Sub-second feature serving latency

6. Model Performance:

7. Meeting or exceeding baseline metrics
8. < 5% performance degradation before retraining
9. Successful A/B tests with statistical significance

10. No undetected model drift > 24 hours

11. Operational Excellence:

12. 99.9% uptime for model serving
13. < 200ms p99 inference latency
14. Automated rollback within 5 minutes

15. Complete observability with < 1 minute alert time

16. Development Velocity:

17. < 1 hour from commit to production
18. Parallel experiment execution
19. Reproducible training runs

20. Self-service model deployment

21. Cost Efficiency:

22. < 20% infrastructure waste
23. Optimized resource allocation
24. Automatic scaling based on load
25. Spot instance utilization > 60%

Final Deliverables

Upon completion, the orchestrated pipeline will provide: - End-to-end ML pipeline with full automation - Comprehensive documentation and runbooks - Production-ready infrastructure as code - Complete monitoring and alerting system - CI/CD pipelines for continuous improvement - Cost optimization and scaling strategies - Disaster recovery and rollback procedures

Limitations

• Use this skill only when the task clearly matches the scope described above.
• Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
• Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.