Azure AI Operations (AIOps)

Practice Exam Questions & Study Guide

Section 1 of this course is free Notes for every focus area and practice questions. Create a free account, no card needed.
No matching topics. Try a different term.
1

Design and implement an MLOps infrastructure (15–20%)Free

Covers creating and managing Machine Learning workspace resources and assets, and implementing infrastructure as code for Machine Learning. (15–20% of scored content)

1.1: Create and manage resources in a Machine Learning workspace

1 Question Sets Available

Covers creating and managing workspaces, datastores, compute targets, and identity/access management.

Focus Areas:

1.2: Create and manage assets in a Machine Learning workspace

1 Question Sets Available

Covers creating and managing data assets, environments, components, and sharing assets across workspaces using registries.

Focus Areas:

1.3: Implement IaC for Machine Learning

1 Question Sets Available

Covers configuring GitHub integration, deploying resources with Bicep and Azure CLI, automating provisioning with GitHub Actions, restricting network access, and managing source control.

Focus Areas:
  • Configure GitHub integration with Machine Learning to enable secure access
  • Deploy Machine Learning workspaces and resources by using Bicep and Azure CLI
  • Automate resource provisioning by using GitHub Actions workflows
  • Restrict network access to Machine Learning workspaces
  • Manage source control for machine learning projects by using Git
2

Implement machine learning model lifecycle and operations (25–30%)

Covers orchestrating model training, model registration and versioning, deploying models for production, and monitoring and maintaining models in production. (25–30% of scored content)

2.1: Orchestrate model training

1 Question Sets Available

Covers experiment tracking, automated machine learning, notebooks, hyperparameter tuning, training scripts, distributed training, pipelines, and comparing model performance.

Focus Areas:
  • Configure experiment tracking with MLflow
  • Use automated machine learning to explore optimal models
  • Use notebooks for experimentation and exploration
  • Automate hyperparameter tuning
  • Run model training scripts
  • Manage distributed training for large and deep learning models
  • Implement training pipelines
  • Compare model performance across jobs

2.2: Implement model registration and versioning

1 Question Sets Available

Covers packaging feature retrieval specifications, registering MLflow models, evaluating models using responsible AI principles, and managing model lifecycle.

Focus Areas:
  • Package a feature retrieval specification with the model artifact
  • Register an MLflow model
  • Evaluate a model by using responsible AI principles
  • Manage model lifecycle, including archiving models

2.3: Deploy machine learning models for production environments

1 Question Sets Available

Covers deploying models as real-time or batch endpoints, testing and troubleshooting endpoints, and implementing progressive rollout and safe rollback strategies.

Focus Areas:
  • Deploy models as real-time or batch endpoints with managed inference options
  • Test and troubleshoot model endpoints
  • Implement progressive rollout and safe rollback strategies

2.4: Monitor and maintain machine learning models in production

1 Question Sets Available

Covers detecting and analyzing data drift, monitoring performance metrics, and configuring retraining or alert triggers.

Focus Areas:
  • Detect and analyze data drift
  • Monitor performance metrics of models deployed to production
  • Configure retraining or alert triggers when thresholds are exceeded
3

Design and implement a GenAIOps infrastructure (20–25%)

Covers implementing Foundry environments and platform configuration, deploying and managing foundation models, and implementing prompt versioning and management. (20–25% of scored content)

3.1: Implement Foundry environments and platform configuration

1 Question Sets Available

Covers creating and configuring Foundry resources, identity and access management, network security, and infrastructure deployment using Bicep and Azure CLI.

Focus Areas:
  • Create and configure Foundry resources and project environments
  • Configure identity and access management with managed identities and role-based access control (RBAC)
  • Implement network security and private networking configurations
  • Deploy infrastructure using Bicep templates and Azure CLI

3.2: Deploy and manage foundation models for production workloads

1 Question Sets Available

Covers deploying foundation models via serverless API and managed compute, selecting appropriate models, versioning and deployment strategies, and configuring provisioned throughput.

Focus Areas:
  • Deploy foundation models by using serverless API endpoints and managed compute options
  • Select appropriate models for specific use cases
  • Implement model versioning and production deployment strategies
  • Configure provisioned throughput units for high-volume workloads

3.3: Implement prompt versioning and management with source control

1 Question Sets Available

Covers designing and developing prompts, creating prompt variants to compare performance, and implementing version control for prompts using Git repositories.

Focus Areas:
  • Design and develop prompts
  • Create prompt variants and compare performance across different prompts
  • Implement version control for prompts by using Git repositories
4

Implement generative AI quality assurance and observability (10–15%)

Covers configuring evaluation and validation for generative AI applications and agents and implementing observability capabilities. (10–15% of scored content)

4.1: Configure evaluation and validation for generative AI applications and agents

1 Question Sets Available

Covers creating test datasets, implementing AI quality metrics, configuring risk and safety evaluations, and setting up automated evaluation workflows.

Focus Areas:
  • Create test datasets and data mapping for comprehensive model evaluation
  • Implement AI quality metrics, including groundedness, relevance, coherence, and fluency
  • Configure risk and safety evaluations for harmful content detection
  • Set up automated evaluation workflows by using built-in and custom evaluation metrics

4.2: Implement observability for generative AI applications and agents

1 Question Sets Available

Covers continuous monitoring in Foundry, performance and cost metrics, and configuring logging, tracing, and debugging for production troubleshooting.

Focus Areas:
  • Examine continuous monitoring in Foundry
  • Monitor performance metrics, including latency, throughput, and response times
  • Track and optimize cost metrics, including token consumption and resource usage
  • Configure detailed logging, tracing, and debugging capabilities for production troubleshooting
5

Optimize generative AI systems and model performance (10–15%)

Covers optimizing RAG performance and accuracy and implementing advanced fine-tuning and model customization. (10–15% of scored content)

5.1: Optimize retrieval-augmented generation (RAG) performance and accuracy

1 Question Sets Available

Covers tuning retrieval parameters, selecting and fine-tuning embedding models, implementing hybrid search, and evaluating RAG system performance.

Focus Areas:
  • Optimize retrieval performance by tuning similarity thresholds, chunk sizes, and retrieval strategies
  • Select and fine-tune embedding models for domain-specific use cases and accuracy improvements
  • Implement and optimize hybrid search approaches combining semantic and keyword-based retrieval
  • Evaluate and improve RAG system performance by using relevance metrics and A/B testing frameworks

5.2: Implement advanced fine-tuning and model customization

1 Question Sets Available

Covers designing advanced fine-tuning methods, creating synthetic data, monitoring fine-tuned model performance, and managing fine-tuned models from development to production.

Focus Areas:
  • Design and implement advanced fine-tuning methods
  • Create and manage synthetic data for fine-tuning
  • Monitor and optimize fine-tuned model performance
  • Manage a fine-tuned model from development through production deployment