A Deep Dive into the Microsoft Foundry Document Intelligence SDK: From PDF to Structured Data

Most "RAG over PDFs" pipelines have a step nobody talks about much: something has to turn a scanned invoice, a multi-column contract, or a photographed receipt into text a model can actually reason over. On Microsoft's stack, that something is usually the Document Intelligence SDK, formerly Form Recognizer, and it's worth understanding on its own terms rather than treating it as a black box that happens before the interesting part starts.

This is a hands-on deep dive into that SDK specifically. Not a tour of every Foundry Tools SDK, Vision and Speech and Content Safety each deserve their own treatment, but a real build using Document Intelligence: extracting layout as clean markdown, pulling structured fields out of a known document type, classifying documents before routing them, and training a custom extraction model on your own labeled data.

The mental model first

Two clients, and three kinds of model, cover almost everything this SDK does:

DocumentIntelligenceClient runs analysis. Every call goes through one method, begin_analyze_document, and a model_id parameter decides what kind of analysis happens. It's a long-running operation, so every call returns a poller.
DocumentIntelligenceAdministrationClient manages models. This is where you build custom extraction models and classifiers, list what's already been trained, and delete what you don't need anymore.
Prebuilt models (prebuilt-layout, prebuilt-invoice, prebuilt-receipt, prebuilt-idDocument, prebuilt-read, and others) handle common, well-known document shapes out of the box. No training required.
Custom extraction models, trained on your own labeled documents, handle document types nobody prebuilt a model for, your specific contract template, your specific intake form.
Classifiers solve a different problem entirely: given a document of unknown type, which model should even look at it? This matters more than it sounds like it should, since most real document pipelines receive a mix of types, not one known shape.

Prerequisites

A Document Intelligence resource (or a multi-service Foundry resource, which includes it), giving you an endpoint and either an API key or Entra ID access.
Python 3.9+ with the SDK installed.

pip install azure-ai-documentintelligence azure-identity

from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.core.credentials import AzureKeyCredential

endpoint = "https://YOUR-RESOURCE.cognitiveservices.azure.com"
client = DocumentIntelligenceClient(endpoint=endpoint, credential=AzureKeyCredential("YOUR-KEY"))

For anything past local experimentation, swap the key for DefaultAzureCredential and an RBAC role scoped to the resource, the same pattern every other Foundry-adjacent SDK in this series has used.

Step 1: layout extraction, straight to markdown

This is the single most useful call in the whole SDK if your end goal is feeding documents into a RAG pipeline. prebuilt-layout doesn't just extract text, it understands headings, tables, and section structure, and it can hand all of that back as GitHub-flavored markdown instead of a flat text blob.

from azure.ai.documentintelligence.models import AnalyzeDocumentRequest, DocumentContentFormat

with open("contract.pdf", "rb") as f:
    poller = client.begin_analyze_document(
        "prebuilt-layout",
        AnalyzeDocumentRequest(bytes_source=f.read()),
        output_content_format=DocumentContentFormat.MARKDOWN,
    )
result = poller.result()
print(result.content[:500])

result.content is now a markdown string, headings as #, tables as GFM pipe tables, page structure preserved. That matters more than it sounds like it should: a table flattened into plain text loses its row and column relationships, and a model reasoning over that text has to reconstruct structure it was never actually given. Markdown output keeps the structure intact.

Step 2: pulling structured fields from a known document type

For document types Document Intelligence already knows, invoices are the clearest example, you get named fields back with a confidence score per field, not just raw text.

with open("invoice.pdf", "rb") as f:
    poller = client.begin_analyze_document("prebuilt-invoice", AnalyzeDocumentRequest(bytes_source=f.read()))
result = poller.result()

for doc in result.documents:
    vendor = doc.fields.get("VendorName")
    total = doc.fields.get("InvoiceTotal")
    if vendor:
        print(f"Vendor: {vendor.value_string} (confidence: {vendor.confidence:.2f})")
    if total:
        print(f"Total: {total.value_currency.amount} (confidence: {total.confidence:.2f})")

That confidence score isn't decoration. It's the field you should actually branch on in production code, more on that in the production section below.

Step 3: add-on capabilities you'll want more often than the docs suggest

A few optional capabilities aren't on by default, since they add processing cost, but are worth turning on deliberately rather than discovering you needed them after the fact:

from azure.ai.documentintelligence.models import AnalyzeDocumentRequest, DocumentAnalysisFeature

with open("shipping-label.pdf", "rb") as f:
    poller = client.begin_analyze_document(
        "prebuilt-layout",
        AnalyzeDocumentRequest(bytes_source=f.read()),
        features=[DocumentAnalysisFeature.BARCODES, DocumentAnalysisFeature.FORMULAS],
    )

BARCODES extracts barcode and QR code payloads directly, useful for shipping labels and inventory documents where the barcode carries the actual identifier the text doesn't repeat. FORMULAS pulls out mathematical expressions as LaTeX, relevant if you're processing scientific or financial documents where a formula matters more than the surrounding prose. There's also a high-resolution mode for documents where small print matters, at the cost of slower processing.

Step 4: build a classifier to route mixed document types

Real intake pipelines rarely receive one document type. A classifier solves the "what am I even looking at" problem before you commit to an extraction model.

from azure.ai.documentintelligence import DocumentIntelligenceAdministrationClient
from azure.ai.documentintelligence.models import (
    BuildDocumentClassifierRequest,
    ClassifierDocumentTypeDetails,
    AzureBlobContentSource,
)

admin_client = DocumentIntelligenceAdministrationClient(endpoint=endpoint, credential=AzureKeyCredential("YOUR-KEY"))

poller = admin_client.begin_build_classifier(
    BuildDocumentClassifierRequest(
        classifier_id="support-doc-classifier",
        doc_types={
            "invoice": ClassifierDocumentTypeDetails(
                azure_blob_source=AzureBlobContentSource(container_url="<SAS-url-to-invoices-container>")
            ),
            "contract": ClassifierDocumentTypeDetails(
                azure_blob_source=AzureBlobContentSource(container_url="<SAS-url-to-contracts-container>")
            ),
        },
    )
)
classifier = poller.result()

You need at least five sample documents per category to train a classifier at all, and more than that for anything you'd trust in production. Once it's built, classifying an incoming document is a single call:

with open("unknown.pdf", "rb") as f:
    poller = client.begin_classify_document("support-doc-classifier", AnalyzeDocumentRequest(bytes_source=f.read()))
result = poller.result()
for doc in result.documents:
    print(f"Classified as: {doc.doc_type} (confidence: {doc.confidence:.2f})")

Step 5: build a custom extraction model for your own document type

When a document type isn't invoices, receipts, or any of the other prebuilt shapes, train your own. This needs a set of labeled training documents in Blob Storage, produced through the labeling tool in Foundry's document intelligence studio or programmatically.

from azure.ai.documentintelligence.models import (
    BuildDocumentModelRequest,
    AzureBlobContentSource,
    DocumentBuildMode,
)

poller = admin_client.begin_build_document_model(
    BuildDocumentModelRequest(
        model_id="acme-service-agreement-v1",
        build_mode=DocumentBuildMode.TEMPLATE,
        azure_blob_source=AzureBlobContentSource(container_url="<SAS-url-to-training-container>"),
        description="Extraction model for Acme's standard service agreement template.",
    )
)
model = poller.result()

Two build modes matter here, and they're not interchangeable. TEMPLATE mode is faster to train and works well when your documents follow a consistent visual layout, the same form filled out differently each time. NEURAL mode handles structural variation better, different layouts that still represent the same document type, at the cost of needing more training examples and longer build time. Start with TEMPLATE unless your documents genuinely vary in structure, not just content.

One naming constraint worth knowing before you hit it: a custom model ID can't start with prebuilt-, since that prefix is reserved for Microsoft's own models across every resource.

Where this fits in the bigger picture

This is the detail that trips people up once they've also worked with the Foundry SDK or Agent Framework elsewhere in this series: Document Intelligence doesn't go through your Foundry project endpoint at all. It has its own resource, its own endpoint (resource.cognitiveservices.azure.com), and its own authentication scope. That's what "Foundry Tools SDK" actually means as a category, prebuilt AI services with tool-specific endpoints, distinct from the Foundry SDK's unified project endpoint that Agent Framework and the Responses API build on.

The practical upshot is the pipeline most teams actually want: run prebuilt-layout over incoming documents, get markdown back, and hand that markdown to a Foundry IQ Knowledge Base as a File Knowledge Source. Document Intelligence handles turning the PDF into clean, structured text. Foundry IQ handles chunking, embedding, and retrieval on top of it. Neither service needs to know the other exists, they just happen to compose well because markdown is a reasonable interchange format for both.

Production considerations before you commit

Don't trust a field just because it came back. A field with a confidence score of 0.41 should not silently flow into a downstream system as if it were as reliable as one scored 0.98. Set a threshold, route low-confidence extractions to human review, and log the confidence distribution over time so a model quietly degrading on a document template change doesn't go unnoticed.
Classifier training minimums are a floor, not a target. Five documents per category is what the service requires to build at all. It is not enough to trust a classifier's accuracy in production. Budget for real evaluation data, held out from training, before routing real documents based on classifier output.
TEMPLATE vs NEURAL is a real tradeoff, not a default to leave unexamined. Picking NEURAL by default because it sounds more capable means slower training and a higher training-data bar for a benefit you may not need if your documents are already visually consistent.
Preview API versions and regional availability move independently of the SDK version. A given SDK release doesn't guarantee every feature is available in every region. Check current regional availability for newer capabilities (certain add-ons, newer prebuilt models) before designing around them.
Markdown output is currently scoped to prebuilt-layout. Don't assume other prebuilt or custom models will hand back the same content format, check per-model support before building a pipeline that assumes markdown everywhere.
Cost scales with pages and capability, not just call count. Add-on features like high-resolution mode and custom model training both carry their own cost beyond the base per-page analysis price. Model this before committing to a design that turns on every add-on by default.

Where this leaves you

The Document Intelligence SDK is easy to undersell because the interesting part of most AI applications feels like it's happening somewhere else, in the model, in the retrieval layer, in the agent's reasoning. But the quality ceiling of everything downstream is set right here, at the point where a physical or scanned document either does or doesn't become text a model can actually use well. Layout extraction to markdown, confidence-aware field extraction, classifiers for mixed intake, and custom models for your own document shapes cover the large majority of real document-processing needs, and all four are a few lines of SDK code once you know which one you need. The judgment call was never really about the API. It's about matching the right one of these four tools to what's actually in your inbound documents.