4.2 Process Groups, Process Instances & Custom Process Group Detection Rules

Process Groups, Process Instances & Custom Process Group Detection Rules

Exam Focus: Understanding the distinction between an operating system process, a Process Group Instance (PGI), and a Process Group (PG) is fundamental to mastering Dynatrace topology. The Dynatrace Certified Associate exam heavily tests automatic grouping mechanisms, the impact of Host Groups on process boundaries, how to solve grouping anti-patterns using Custom Process Group Detection (PGD) rules, and the mandatory operational requirements for applying detection rules.

In modern enterprise architectures, applications are composed of horizontally scaled, highly dynamic instances. A payment processing tier might run on eight virtual machines or expand dynamically into fifty Kubernetes pods. Treating each operating system process (PID) as an independent monitored entity would flood monitoring dashboards with transient entities, destroy historical baselines, and overwhelm root-cause analysis engines.

Dynatrace solves this fundamental challenge through the Process Group abstraction model within Smartscape.

---

The Core Entity Hierarchy: OS Process to Process Group to Service

To navigate Dynatrace topology, candidates must understand the four-tier architectural hierarchy:

1. Operating System Process (PID): The low-level operating system execution thread identified by a transient Process Identifier (e.g., PID 49122 on Linux). PIDs change whenever a process crashes, restarts, or scales.
2. Process Group Instance (PGI): The representation of a specific Process Group executing on a single host. If an application runs three worker processes on Host-A, those three processes collectively form a single PGI on Host-A.
3. Process Group (PG): A logical cluster of Process Group Instances running across one or multiple hosts that execute the same software code, perform the same business function, and share identical configuration (e.g., the PaymentService cluster across all production nodes).
4. Service: The functional entry points (web services, REST APIs, messaging queues) discovered and hosted by a Process Group. Services run on top of Process Groups, not directly on top of individual hosts.

+-------------------------------------------------------------+
|                      Service Layer                          |
|           (e.g., /api/v1/checkout, /payment/process)        |
+-------------------------------------------------------------+
                               | (runs on)
+-------------------------------------------------------------+
|                   Process Group (PG)                        |
|             (e.g., Payment-Backend-Cluster)                 |
+-------------------------------------------------------------+
          |                                         |
          v (spans across)                          v
+-----------------------+                 +-----------------------+
| Process Group Instance|                 | Process Group Instance|
|     (PGI on Host-A)   |                 |     (PGI on Host-B)   |
+-----------------------+                 +-----------------------+
    |               |                         |               |
    v (PIDs)        v (PIDs)                  v (PIDs)        v (PIDs)
 [PID: 1042]     [PID: 1043]               [PID: 8812]     [PID: 8813]

Why Process Groups Are Essential for Davis AI

Dynatrace Davis AI calculates automated performance baselines (response time, throughput, failure rate) and evaluates anomalies at the Process Group and Service layers, never at the individual PID layer. When a container restarts and acquires a new PID, the Process Group retains its continuous historical baseline, preventing false-positive anomaly storms during rolling updates.

---

Automatic Process Group Detection Mechanics

When OneAgent discovers a new process, it evaluates built-in detection heuristics specific to the underlying runtime technology to assign the process to a Process Group.

Technology-Specific Grouping Criteria

• Java Applications: OneAgent inspects the main class (e.g., com.corp.billing.BillingService) or the executable JAR filename (e.g., order-service.jar). For web application servers, it inspects the server home directory (CATALINA_BASE for Apache Tomcat, domain path for WebLogic, cell/node/server for WebSphere).
• .NET Applications: Grouped according to the executable assembly name or Microsoft IIS Application Pool name.
• Node.js: Grouped by the main entry script filename (e.g., server.js, app.js) or the name attribute defined in package.json.
• Web Servers (Apache, NGINX, IIS): Grouped by web server executable name, listening port, and server configuration path.
• Containerized Environments (Docker & Kubernetes): In Kubernetes, Dynatrace prioritizes container metadata. Processes are grouped by Kubernetes Deployment Name, Pod Namespace, and Container Image Name rather than underlying host PID attributes.

The Host Group Boundary Rule (Critical Exam Concept)

One of the most heavily tested architectural rules in Dynatrace certification is the Host Group Boundary:

Rule: If two identical processes run the exact same code, execute identical command-line arguments, and share identical runtime parameters, but are hosted on servers belonging to different Host Groups, Dynatrace will automatically separate them into distinct Process Groups.

For example, if an identical Tomcat application runs on Host-01 (assigned to Host Group Production) and Host-02 (assigned to Host Group Staging), Dynatrace automatically creates two separate Process Groups: Tomcat [Production] and Tomcat [Staging]. This prevents staging transactions and errors from contaminating production baselines in Davis AI.

---

Common Grouping Anti-Patterns in Modern Architectures

While automatic detection functions seamlessly for standard deployments, modern cloud-native architectures frequently introduce two major grouping anti-patterns:

1. The Monolithic Merging Anti-Pattern (Generic Java Grouping)

Many microservice frameworks utilize a generic launcher script or corporate Java wrapper. For example, forty distinct microservices might all be launched via:

/usr/bin/java -jar /opt/corp/wrapper.jar --config=/etc/apps/inventory.json

Because OneAgent sees /opt/corp/wrapper.jar as the main JAR across all forty processes, default detection merges all forty distinct microservices into a single, massive Process Group named wrapper. As a result, metrics from completely unrelated business services are blended together, rendering Davis AI root-cause analysis ineffective.

2. The Entity Proliferation Anti-Pattern (Fragmentation)

Conversely, some development frameworks append dynamic, ephemeral arguments to the process startup command line, such as timestamps, temporary directories, or session IDs:

/usr/bin/python worker.py --temp-dir=/tmp/run-9841284 --job-id=88319

Because the command-line string changes on every process restart, default detection may treat every single process launch as a brand-new Process Group. This leads to entity proliferation (thousands of short-lived Process Groups), degrading Dynatrace UI performance and destroying baseline continuity.

---

Custom Process Group Detection (PGD) Rules

To resolve detection anti-patterns, Dynatrace provides Custom Process Group Detection (PGD) Rules (Settings > Processes and containers > Process group detection).

PGD Rule Types

Dynatrace offers two distinct operational rule structures:

1. Process Group Detection Rules (Simple / Clustered): Define an overarching rule that groups processes matching specific criteria into a fixed, named Process Group.
2. Process Group Splitting Rules: Inspect an existing pool of processes and dynamically split them into multiple distinct Process Groups based on an extracted parameter or property.

Matching Conditions & Extraction Sources

PGD rules evaluate a rich set of process attributes:

• Executable Path & Name: Absolute path to the running binary (e.g., /usr/bin/node).
• Command-Line Arguments: Tokens, flags, or configuration paths passed to the process.
• Environment Variables: Custom environment variables injected into the process (e.g., APP_NAME=payments).
• Java System Properties (-D flags): Runtime properties defined in JVM arguments (e.g., -Dspring.application.name=catalog).
• Kubernetes Metadata: Pod labels, annotations, and namespace properties.

Dynamic Name Extraction via Regular Expressions

Rather than creating fifty separate static rules for fifty microservices, architects configure a single dynamic rule that uses a property extractor:

• Matcher: Java system property -Dapp.name exists.
• Extraction Regex: -Dapp.name=([a-zA-Z0-9_-]+)
• Process Group Name Template: {Process:Extraction:1}

When evaluated, Dynatrace extracts the unique application name from the property and automatically generates distinct, cleanly named Process Groups for each service.

---

Rule Evaluation Precedence & The Process Restart Requirement

Mastering how and when PGD rules execute is vital for operational management and exam success.

Top-to-Bottom Precedence

PGD rules are organized in a prioritized list evaluated from top to bottom:

• The first rule whose match conditions evaluate to true claims the process and assigns its Process Group identity.
• Once a rule matches, evaluation terminates; subsequent rules lower in the priority list are ignored.
• If no custom rule matches, OneAgent falls back to built-in default detection logic.

The Mandatory Process Restart Rule (Crucial Exam Concept)

Exam Rule: Custom Process Group Detection rules STRICTLY REQUIRE AN APPLICATION PROCESS RESTART to take effect.

Why is a restart mandatory?

• OneAgent evaluates process group detection rules during the initial process startup sequence (execve), when the bootstrap loader first hooks into the operating system process and inspects its environment tables, ELF headers, and command-line arguments.
• Once a process has initialized and registered its entity identity (PROCESS_GROUP_INSTANCE-xxxxxxxxxxxx) with the Dynatrace cluster, that identity is immutable for the lifetime of that running OS process.
• Modifying or adding a PGD rule in the Dynatrace web UI saves the rule to the Dynatrace cluster configuration. However, currently running application processes will remain in their old Process Group.
• Only when the application process terminates and is restarted will OneAgent re-evaluate the command line and environment against the new PGD rule, assigning the newly spawned process to the new Process Group.

Historical Data Preservation

When a PGD rule splits a process into a new Process Group, historical telemetry (metrics and PurePaths recorded prior to the restart) remains associated with the old Process Group entity ID. Historical data is not retroactively migrated to the newly created Process Group entity.