By Jorge Suarez — Jul 13, 2025

Supercharge Microsoft Graph API Data Retrieval with PowerShell Batch Requests

The Challenge That Started It

Picture this: You're tasked with exporting all sign-in logs from your organization's Microsoft 365 tenant for a security audit. You fire up PowerShell, run Get-MgBetaAuditLogSignIn -All, and then... you wait. And wait. And wait some more.

What should be a simple data export turns into an hours-long ordeal. Sound familiar?

This is the story of how I transformed a painfully slow process into a lightning-fast data retrieval system, reducing wait times by up to 88% using Microsoft Graph API's batch processing capabilities.

The Numbers Don't Lie

Before diving into the solution, let me share some real-world performance metrics from my testing with actual production data:

Endpoint	Objects	Traditional Method	Optimized Batch Method	Improvement
Devices	68,677	4m 7s	1m 34s (Memory Managed)	62% faster
Groups	13,585	43.7s	18.1s (Memory Managed)	59% faster
Users	24,284	3m 24s	2m 2s (Sequential)	40% faster
Applications	347	1.4s	0.8s (Parallel)	45% faster

These aren't theoretical improvements – they're actual results from production environments with real enterprise datasets.

Understanding the Root Problem

Microsoft Graph API uses pagination to prevent overwhelming clients with massive datasets. When you request data, you typically get:

A page of results (default: 100 items, max: 999)
A "nextLink" URL pointing to the next page
Repeat until all data is retrieved

The traditional approach looks like this:

# The slow way - sequential pagination
$allItems = @()
$uri = "https://graph.microsoft.com/beta/users"

do {
    $response = Invoke-MgGraphRequest -Uri $uri
    $allItems += $response.value
    $uri = $response.'@odata.nextLink'
} while ($uri)

For 10,000 users across 100 pages, that's 100 sequential HTTP requests. Each with its own:

Network latency
API processing time
Authentication overhead

There had to be a better way.

The Breakthrough: Batch Requests

Microsoft Graph supports batch requests – the ability to bundle multiple API calls into a single HTTP request. Instead of 100 sequential calls, we can make 5 batch requests, each containing 20 individual requests.

Here's the game-changer: those 5 batch requests can run in parallel.

Evolution to a Streamlined Solution

Initially, I explored a modular approach with separate helper functions for environment detection, skip token extraction, and batch processing. However, real-world testing revealed that a consolidated approach was more maintainable and performed better.

Core Capabilities:

Multi-cloud environment detection - Automatically supports Global, USGov, China, and Germany clouds
Intelligent nextLink handling - Uses complete URLs instead of token extraction for reliability
Parallel and sequential processing - Configurable based on dataset size and performance needs
Memory monitoring and warnings - Built-in thresholds to prevent memory exhaustion
Comprehensive error handling - Robust HTTP status monitoring and debugging support

Key Technical Improvements:

NextLink URL preservation - Stores complete @odata.nextLink URLs instead of extracting skip tokens
API version handling - Automatically strips version prefixes for batch request compatibility
Dynamic job management - Configurable concurrent jobs (1-20) for optimal performance
Memory-conscious design - Configurable memory thresholds with automatic warnings

The Final Implementation

The current production-ready implementation consolidates all functionality into a single, robust function. Here are the key features of the streamlined approach:

function Invoke-mgBatchRequest {
    [CmdletBinding()]
    param(
        [int]$PageSize = 999,
        [Parameter(Mandatory = $true)]
        [string]$Endpoint,
        [string]$Filter,
        [switch]$UseParallelProcessing,
        [ValidateRange(1, 20)]
        [int]$MaxConcurrentJobs = 8,
        [int]$MemoryThreshold = 100,
        [string]$ExpandProperty
    )

    # Multi-cloud environment detection
    $mgContext = Get-MgContext
    $uri = switch ($mgContext.Environment) {
        "Global" { "https://graph.microsoft.com/beta" }
        "USGov" { "https://graph.microsoft.us/beta" }
        "China" { "https://graph.chinacloudapi.cn/beta" }
        "Germany" { "https://graph.microsoft.de/beta" }
        default { "https://graph.microsoft.com/beta" }
    }

    # Initial request with optional filtering
    $firstUri = "$uri/$($Endpoint)?`$top=$PageSize"
    if ($Filter) {
        $encodedFilter = [uri]::EscapeDataString($Filter)
        $firstUri += "&`$filter=$encodedFilter"
    }
    if ($ExpandProperty) {
        $encodedExpand = [uri]::EscapeDataString($ExpandProperty)
        $firstUri += "&`$expand=$encodedExpand"
    }
    
    $first = Invoke-MgGraphRequest -Method GET -Uri $firstUri
    $allGraphObjects = @($first.value)
    
    # Store complete nextLink URLs (key improvement over token extraction)
    $nextLinks = @()
    if ($first.'@odata.nextLink') {
        $nextLinks += $first.'@odata.nextLink'
    }
    
    # Process remaining pages with intelligent batching
    while ($nextLinks.Count -gt 0) {
        if ($UseParallelProcessing) {
            # Parallel processing with configurable job count
            # Creates thread jobs for maximum performance
        } else {
            # Sequential batch processing (often optimal for medium datasets)
            # Batches up to 20 requests per API call
        }
        
        # Memory monitoring and threshold warnings
        if ($MemoryThreshold -gt 0) {
            $estimatedMemoryMB = ($totalObjectCount * 2048) / 1MB
            if ($estimatedMemoryMB -gt $MemoryThreshold) {
                Write-Warning "Memory usage estimated at $([math]::Round($estimatedMemoryMB, 1))MB"
            }
        }
    }
    
    return $allGraphObjects
}

The complete implementation includes sophisticated error handling, HTTP status monitoring, and dynamic nextLink processing that adapts to Graph API response patterns.

Key Enhancements in the Latest Version

The current implementation includes several major improvements over the original concept:

1. Multi-Cloud Environment Support

Automatically detects and supports all Microsoft Graph environments:

Global (graph.microsoft.com)
US Government (graph.microsoft.us)
China (graph.chinacloudapi.cn)
Germany (graph.microsoft.de)

2. Improved NextLink Handling

Instead of manually extracting skip tokens (which can break with complex URLs), the function now:

Stores complete @odata.nextLink URLs
Uses proper URI parsing to extract relative paths
Handles complex query parameters correctly

3. Comprehensive Testing Framework

The Test-MgBatchRequest.ps1 script provides:

Performance comparison across different methods
Memory usage monitoring
Object count verification
Production-ready command generation

4. Intelligent Processing Mode Selection

Based on real-world testing, the function now provides guidance on optimal processing modes:

Sequential batching: Best for datasets under 50K objects
Parallel processing: Beneficial for very large datasets (100K+)
Memory management: Automatic warnings and optimizations

5. Enhanced Data Retrieval with ExpandProperty

The latest version introduces the ExpandProperty parameter, enabling retrieval of related entities in a single request:

Reduce API calls: Get related data without additional requests
Improved performance: Fetch users with their managers, apps with assignments, etc.
Flexible expansion: Supports multiple properties (comma-separated)

6. Expanded Endpoint Support

The function now supports additional critical endpoints:

Service Principals: Enterprise app management
Conditional Access Policies: Security configuration retrieval
Directory Roles: RBAC management
Organization Details: Tenant configuration
Domains: Domain management and verification

Updated Usage Examples

Basic Usage

# Simple batch retrieval
$users = Invoke-mgBatchRequest -Endpoint "users"

# With filtering
$windowsDevices = Invoke-mgBatchRequest -Endpoint "deviceManagement/managedDevices" -Filter "operatingSystem eq 'Windows'"

# With expanded properties
$mobileAppsWithAssignments = Invoke-mgBatchRequest -Endpoint "deviceAppManagement/mobileApps" -ExpandProperty "assignments"

Optimized for Large Datasets

# For enterprise-scale datasets (based on test results)
$allDevices = Invoke-mgBatchRequest -Endpoint "devices" -MemoryThreshold 100
$allUsers = Invoke-mgBatchRequest -Endpoint "users"  # Sequential is optimal for 33K+ users

# Only use parallel processing for specific scenarios
$auditLogs = Invoke-mgBatchRequest -Endpoint "auditLogs/signIns" -UseParallelProcessing -MaxConcurrentJobs 5

# Get service principals with app role assignments
$servicePrincipals = Invoke-mgBatchRequest -Endpoint "servicePrincipals" -ExpandProperty "appRoleAssignments" -UseParallelProcessing

# Get conditional access policies
$enabledPolicies = Invoke-mgBatchRequest -Endpoint "identity/conditionalAccess/policies" -Filter "state eq 'enabled'"

Advanced Scenarios with ExpandProperty

# Get mobile apps with their assignments and categories
$mobileAppsDetailed = Invoke-mgBatchRequest -Endpoint "deviceAppManagement/mobileApps" `
    -ExpandProperty "assignments,categories" `
    -Filter "isAssigned eq true" `
    -UseParallelProcessing

# Export detailed app deployment report
$mobileAppsDetailed | Select-Object displayName, publisher, @{
    Name='AssignmentCount'; Expression={$_.assignments.Count}
}, @{
    Name='Categories'; Expression={$_.categories.displayName -join ', '}
} | Export-Csv "AppDeploymentReport.csv" -NoTypeInformation

# Get users with their manager information
$usersWithManagers = Invoke-mgBatchRequest -Endpoint "users" `
    -ExpandProperty "manager" `
    -Filter "accountEnabled eq true"

# Get groups with owners for governance reporting
$groupsWithOwners = Invoke-mgBatchRequest -Endpoint "groups" `
    -ExpandProperty "owners" `
    -Filter "groupTypes/any(c:c eq 'Unified')"

Testing Your Environment

# Find optimal settings for your tenant
.\Test-MgBatchRequest.ps1 -TestMode Custom -Endpoints @("users")
.\Test-MgBatchRequest.ps1 -TestMode Optimize -Endpoints @("devices") -OptimizeFor Speed

# Test new endpoints
.\Test-MgBatchRequest.ps1 -TestMode Custom -Endpoints @("servicePrincipals", "identity/conditionalAccess/policies")

Real-World Impact

The transformation from a simple concept to a production-ready tool has been remarkable. Here's what the latest version delivers:

Performance Gains by Dataset Size:

Small datasets (< 1K objects): 30-50% improvement
Medium datasets (1K-50K objects): 40-60% improvement
Large datasets (50K+ objects): Up to 62% improvement

Key Learnings:

Sequential batching often outperforms parallel processing for medium datasets due to reduced overhead
Memory management is crucial for datasets over 10K objects
Multi-cloud support is essential for enterprise environments
Proper URL handling prevents HTTP 400 errors that plagued earlier versions

Conclusion

What started as a frustration with slow Graph API calls has evolved into a robust, enterprise-ready solution. The Invoke-mgBatchRequest function now handles:

✅ Multiple cloud environments (Global, Government, China, Germany)
✅ Intelligent processing mode selection based on dataset characteristics
✅ Memory monitoring and management for large datasets
✅ Comprehensive testing framework for optimization
✅ Production-ready reliability with proper error handling
✅ ExpandProperty support for retrieving related entities efficiently
✅ Extended endpoint coverage including service principals, conditional access, and more

The journey from hours to minutes isn't just about faster code—it's about building tools that scale with your organization's needs while maintaining reliability and performance.

Get Started

Ready to transform your Microsoft Graph data retrieval?

Download the latest version from GitHub
Test with your datasets using Test-MgBatchRequest.ps1
Optimize your specific endpoints for maximum performance
Deploy with confidence using the production-ready commands

The era of waiting hours for Graph API data is over. Welcome to the age of optimized, intelligent data retrieval.

The Results Speak for Themselves

What started as a frustrating wait for data exports has become a streamlined, efficient process. Tasks that once took hours now complete in minutes. The modular architecture ensures the code is maintainable and extensible for future enhancements.

Whether you're managing a small organization or a massive enterprise, this approach to Microsoft Graph API data retrieval will save you time, reduce resource consumption, and make your PowerShell scripts significantly more efficient.

Getting Started

Load the function and start retrieving data faster than ever:

. .\Invoke-mgBatchRequest.ps1
$users = Invoke-mgBatchRequest -Endpoint "users" -UseParallelProcessing

# Or get users with their manager information
$usersWithManagers = Invoke-mgBatchRequest -Endpoint "users" -ExpandProperty "manager"

# Get mobile apps with assignment details
$appsWithAssignments = Invoke-mgBatchRequest -Endpoint "deviceAppManagement/mobileApps" -ExpandProperty "assignments"

Connect to Microsoft Graph:

Connect-MgGraph -Scopes "User.Read.All", "Group.Read.All"

Install the Microsoft Graph PowerShell SDK:

Install-Module Microsoft.Graph.Authentication -Scope CurrentUser

The complete source code is available on GitHub, and I encourage you to adapt it for your specific needs. Your future self will thank you the next time you need to export large datasets from Microsoft Graph.

Happy scripting, and may your data exports be ever swift!