Custom Text Classification and Custom NER

Quick Answer: Custom text classification categorizes documents into your classes (single-label or multi-label). Custom NER extracts domain-specific entities. Both require labeled training data in Azure Blob Storage, training via Language Studio or REST API, and deployment to a prediction endpoint.

Custom Text Classification

Single-Label vs. Multi-Label

Training Data Requirements

Data Format

{
    "projectFileVersion": "2022-05-01",
    "stringIndexType": "Utf16CodeUnit",
    "metadata": {
        "projectKind": "CustomSingleLabelClassification",
        "projectName": "SupportTicketClassifier",
        "language": "en"
    },
    "assets": {
        "projectKind": "CustomSingleLabelClassification",
        "classes": [
            {"category": "Billing"},
            {"category": "Technical"},
            {"category": "Account"}
        ],
        "documents": [
            {
                "location": "ticket001.txt",
                "language": "en",
                "class": {"category": "Billing"}
            },
            {
                "location": "ticket002.txt",
                "language": "en",
                "class": {"category": "Technical"}
            }
        ]
    }
}

Calling the Custom Classification Endpoint

from azure.ai.textanalytics import TextAnalyticsClient
from azure.core.credentials import AzureKeyCredential

client = TextAnalyticsClient(
    endpoint="https://my-language.cognitiveservices.azure.com/",
    credential=AzureKeyCredential("<your-key>")
)

# Single-label classification
poller = client.begin_single_label_classify(
    documents=["I was charged twice for my subscription renewal"],
    project_name="SupportTicketClassifier",
    deployment_name="production"
)

for result in poller.result():
    for classification in result.classifications:
        print(f"Class: {classification.category}")
        print(f"Confidence: {classification.confidence_score:.2f}")

Custom Named Entity Recognition

Custom NER trains a model to extract entities specific to your domain:

Example Use Cases

Training Data Format for Custom NER

{
    "projectFileVersion": "2022-05-01",
    "metadata": {
        "projectKind": "CustomEntityRecognition",
        "projectName": "LegalEntityExtractor",
        "language": "en"
    },
    "assets": {
        "projectKind": "CustomEntityRecognition",
        "entities": [
            {"category": "CaseNumber"},
            {"category": "JudgeName"},
            {"category": "StatuteReference"}
        ],
        "documents": [
            {
                "location": "legal_doc_001.txt",
                "language": "en",
                "entities": [
                    {
                        "regionOffset": 45,
                        "regionLength": 12,
                        "labels": [
                            {
                                "category": "CaseNumber",
                                "offset": 45,
                                "length": 12
                            }
                        ]
                    }
                ]
            }
        ]
    }
}

Training and Evaluation

# Train a custom NER model
train_url = f"{endpoint}/language/authoring/analyze-text/projects/{project_name}/:train?api-version=2023-04-01"

train_body = {
    "modelLabel": "v1",
    "trainingConfigVersion": "latest",
    "evaluationOptions": {
        "kind": "percentage",
        "trainingSplitPercentage": 80,
        "testingSplitPercentage": 20
    }
}

response = requests.post(train_url, headers=headers, json=train_body)

Model Evaluation Metrics

On the Exam: Know how to interpret a confusion matrix. High precision but low recall means the model is conservative (misses some entities). Low precision but high recall means the model over-predicts (many false positives). F1 balances both.

Data Splitting Strategies

Best Practices for Training Data

• Label consistency: Ensure the same type of text is always labeled the same way
• Boundary precision: Entity labels must have exact character offsets
• Negative examples: Include documents WITHOUT the target entities
• Class balance: Roughly equal examples per entity type
• Real-world data: Use data that represents actual production inputs