3.7 ML Lifecycle Case Lab

Case: support ticket triage for a growing retailer

A retailer receives thousands of customer support tickets each week. The service director wants AI to route tickets by topic, detect urgent complaints, summarize long messages, and reduce manual triage time. The company uses AWS for application hosting and stores ticket history in Amazon S3. Some tickets include names, order numbers, addresses, and refund details. The team asks whether it should build a custom ML model in SageMaker AI.

Start with business fit. The proposed use case has repeatable tasks: classify tickets, prioritize urgent items, summarize messages, and route work. AI may be useful because volume is high and human triage takes time. The project should not automatically approve refunds, deny service, or make final policy decisions without review. A safer first release supports agents and supervisors rather than replacing them.

The target outcomes should be concrete. Examples include reducing average triage time by 30 percent, increasing correct first-route assignment, reducing urgent-ticket response delay, and maintaining or improving customer satisfaction. If the team cannot name success measures, the project is not ready. If the team only wants to use AI because competitors mention AI, the practitioner should push for a measurable business objective.

Data readiness review

The historical tickets include useful signals, but the team must check data quality. Are topics labeled consistently? Did routing categories change over time? Are urgent tickets marked in a reliable way? Are some languages underrepresented? Are refund details and addresses necessary for classification, or can they be redacted or minimized? Does the data owner approve this use? Are retention rules documented?

EDA should inspect message lengths, missing fields, category balance, duplicate tickets, language distribution, seasonal spikes, and urgent-ticket rates. If urgent tickets are rare, accuracy alone will be misleading. The team should review recall for urgent tickets and the workload created by false positives. If the model flags too many ordinary tickets as urgent, supervisors may ignore the alerts.

Model source decision

The team should not jump directly to custom SageMaker training. The use case includes several tasks that may fit managed or foundation model services. Amazon Comprehend can be evaluated for text analysis and custom classification use cases. Amazon Bedrock can be evaluated for summarization and generative assistant patterns, especially with grounding, guardrails, and human review. SageMaker Canvas may help analysts test a no-code classification baseline from approved tabular or text-derived features.

A custom SageMaker model may be justified later if managed options cannot meet quality, language, cost, or domain requirements.

A reasonable first architecture might use managed text analysis or a foundation model to draft a category, urgency score, and summary, then send low-confidence or high-risk cases to human triage. The system should not expose sensitive ticket content to users who lack permission. IAM, encryption, CloudTrail, CloudWatch, and data retention controls should be part of the design. If prompts or model responses are logged, those logs must be protected because they may contain customer data.

Metrics and deployment plan

The evaluation plan needs both model metrics and business metrics. For category routing, measure precision, recall, F1, and confusion between similar categories. For urgent detection, focus on recall and false-positive workload because missed urgent tickets can harm customers, while too many false positives can overload supervisors. For summarization, use human review, factuality checks, and agent usefulness ratings. For business value, track triage time, route correction rate, time to first response, escalation delay, cost per ticket, and customer feedback.

Deployment should be staged. Start with a pilot for one ticket channel and a limited set of categories. Show suggestions to human agents rather than auto-routing every ticket. Add confidence thresholds. For example, high-confidence routine tickets can be suggested for direct routing, medium-confidence tickets go to a triage queue, and low-confidence or sensitive tickets require human review. The team should keep a manual routing fallback.

Pilot workflow
1. Ingest approved ticket data from S3 under governed access.
2. Evaluate managed AI, Bedrock, Canvas, or SageMaker options against a baseline.
3. Pilot with human-in-the-loop suggestions for one channel.
4. Monitor route accuracy, urgent recall, latency, cost, and agent feedback.
5. Expand only after quality, privacy, and support workload are acceptable.

MLOps and operating controls

Before production, the team needs an owner for the model or service configuration, an owner for business metrics, and an owner for incident response. It needs version records for prompts, model configurations, datasets, evaluation reports, and thresholds. It needs monitoring through CloudWatch or service-specific tools. If SageMaker-hosted models are used, Model Monitor and Model Registry may support drift monitoring and version approval. If Bedrock is used, the team should consider model evaluation, Guardrails, logging controls, and prompt injection defenses.

Retraining or reconfiguration triggers should be explicit. Examples include a drop in route accuracy, increased urgent-ticket misses, new product categories, a new support channel, major policy changes, new language coverage, or a spike in customer complaints. The team should also define rollback. If AI suggestions degrade, agents can return to manual routing while the issue is investigated.

Approval decision

A practitioner can approve a narrow pilot if the team has data owner approval, measurable targets, representative evaluation data, privacy controls, human review, monitoring, and rollback. The practitioner should pause full automation until the pilot proves value and safety. The practitioner should reject any plan that routes sensitive or urgent tickets with no monitoring, no feedback, no access controls, and no human fallback.

This case demonstrates the full chapter pattern. Ask whether AI fits. Check data readiness. Choose the least complex model source that can work. Connect metrics to business outcomes. Deploy in a controlled way. Monitor after launch. Retrain, reconfigure, rollback, or retire based on evidence.