Azure AI Operations (AIOps)
Practice Exam Questions & Study Guide
Design and implement an MLOps infrastructure (15–20%)Free
Covers creating and managing Machine Learning workspace resources and assets, and implementing infrastructure as code for Machine Learning. (15–20% of scored content)
1.1: Create and manage resources in a Machine Learning workspace
Covers creating and managing workspaces, datastores, compute targets, and identity/access management.
- Create and manage a workspace
- Create and manage datastores
- Create and manage compute targets
- Configure identity and access management for workspaces
1.2: Create and manage assets in a Machine Learning workspace
Covers creating and managing data assets, environments, components, and sharing assets across workspaces using registries.
- Create and manage data assets
- Create and manage environments
- Create and manage components
- Share assets across workspaces by using registries
1.3: Implement IaC for Machine Learning
Covers configuring GitHub integration, deploying resources with Bicep and Azure CLI, automating provisioning with GitHub Actions, restricting network access, and managing source control.
- Configure GitHub integration with Machine Learning to enable secure access
- Deploy Machine Learning workspaces and resources by using Bicep and Azure CLI
- Automate resource provisioning by using GitHub Actions workflows
- Restrict network access to Machine Learning workspaces
- Manage source control for machine learning projects by using Git
Implement machine learning model lifecycle and operations (25–30%)
Covers orchestrating model training, model registration and versioning, deploying models for production, and monitoring and maintaining models in production. (25–30% of scored content)
2.1: Orchestrate model training
Covers experiment tracking, automated machine learning, notebooks, hyperparameter tuning, training scripts, distributed training, pipelines, and comparing model performance.
- Configure experiment tracking with MLflow
- Use automated machine learning to explore optimal models
- Use notebooks for experimentation and exploration
- Automate hyperparameter tuning
- Run model training scripts
- Manage distributed training for large and deep learning models
- Implement training pipelines
- Compare model performance across jobs
2.2: Implement model registration and versioning
Covers packaging feature retrieval specifications, registering MLflow models, evaluating models using responsible AI principles, and managing model lifecycle.
- Package a feature retrieval specification with the model artifact
- Register an MLflow model
- Evaluate a model by using responsible AI principles
- Manage model lifecycle, including archiving models
2.3: Deploy machine learning models for production environments
Covers deploying models as real-time or batch endpoints, testing and troubleshooting endpoints, and implementing progressive rollout and safe rollback strategies.
- Deploy models as real-time or batch endpoints with managed inference options
- Test and troubleshoot model endpoints
- Implement progressive rollout and safe rollback strategies
2.4: Monitor and maintain machine learning models in production
Covers detecting and analyzing data drift, monitoring performance metrics, and configuring retraining or alert triggers.
- Detect and analyze data drift
- Monitor performance metrics of models deployed to production
- Configure retraining or alert triggers when thresholds are exceeded
Design and implement a GenAIOps infrastructure (20–25%)
Covers implementing Foundry environments and platform configuration, deploying and managing foundation models, and implementing prompt versioning and management. (20–25% of scored content)
3.1: Implement Foundry environments and platform configuration
Covers creating and configuring Foundry resources, identity and access management, network security, and infrastructure deployment using Bicep and Azure CLI.
- Create and configure Foundry resources and project environments
- Configure identity and access management with managed identities and role-based access control (RBAC)
- Implement network security and private networking configurations
- Deploy infrastructure using Bicep templates and Azure CLI
3.2: Deploy and manage foundation models for production workloads
Covers deploying foundation models via serverless API and managed compute, selecting appropriate models, versioning and deployment strategies, and configuring provisioned throughput.
- Deploy foundation models by using serverless API endpoints and managed compute options
- Select appropriate models for specific use cases
- Implement model versioning and production deployment strategies
- Configure provisioned throughput units for high-volume workloads
3.3: Implement prompt versioning and management with source control
Covers designing and developing prompts, creating prompt variants to compare performance, and implementing version control for prompts using Git repositories.
- Design and develop prompts
- Create prompt variants and compare performance across different prompts
- Implement version control for prompts by using Git repositories
Implement generative AI quality assurance and observability (10–15%)
Covers configuring evaluation and validation for generative AI applications and agents and implementing observability capabilities. (10–15% of scored content)
4.1: Configure evaluation and validation for generative AI applications and agents
Covers creating test datasets, implementing AI quality metrics, configuring risk and safety evaluations, and setting up automated evaluation workflows.
- Create test datasets and data mapping for comprehensive model evaluation
- Implement AI quality metrics, including groundedness, relevance, coherence, and fluency
- Configure risk and safety evaluations for harmful content detection
- Set up automated evaluation workflows by using built-in and custom evaluation metrics
4.2: Implement observability for generative AI applications and agents
Covers continuous monitoring in Foundry, performance and cost metrics, and configuring logging, tracing, and debugging for production troubleshooting.
- Examine continuous monitoring in Foundry
- Monitor performance metrics, including latency, throughput, and response times
- Track and optimize cost metrics, including token consumption and resource usage
- Configure detailed logging, tracing, and debugging capabilities for production troubleshooting
Optimize generative AI systems and model performance (10–15%)
Covers optimizing RAG performance and accuracy and implementing advanced fine-tuning and model customization. (10–15% of scored content)
5.1: Optimize retrieval-augmented generation (RAG) performance and accuracy
Covers tuning retrieval parameters, selecting and fine-tuning embedding models, implementing hybrid search, and evaluating RAG system performance.
- Optimize retrieval performance by tuning similarity thresholds, chunk sizes, and retrieval strategies
- Select and fine-tune embedding models for domain-specific use cases and accuracy improvements
- Implement and optimize hybrid search approaches combining semantic and keyword-based retrieval
- Evaluate and improve RAG system performance by using relevance metrics and A/B testing frameworks
5.2: Implement advanced fine-tuning and model customization
Covers designing advanced fine-tuning methods, creating synthetic data, monitoring fine-tuned model performance, and managing fine-tuned models from development to production.
- Design and implement advanced fine-tuning methods
- Create and manage synthetic data for fine-tuning
- Monitor and optimize fine-tuned model performance
- Manage a fine-tuned model from development through production deployment