13.1 Scheduling Tasks for Data Loading and Processing
Key Takeaways
Aggregate authoritative sources first, then other applications and account groups, and then run Identity Refresh.
Aggregation marks changed identities as needing refresh by default, and the refresh option "Refresh only identities marked as needing refresh during aggregation" (off by default) uses that flag.
The Sequential Task Launcher runs tasks in a fixed order, with an optional stop on error, instead of guessing start times.
Detect deleted accounts has a Maximum deleted accounts safety limit; above it, no accounts are deleted.
Delta aggregation reads only changes and needs connector support; a periodic full aggregation and full refresh are still recommended.
Scheduling Tasks for Data Loading and Processing
Objective 5.4 asks you to schedule various tasks to achieve desired data loading and processing outcomes. Tasks are defined on Setup > Tasks from task types (templates). Their results appear on Task Results, and schedules on Scheduled Tasks.
The Data-Loading Pipeline
- Account Aggregation of authoritative sources (HR, contractor systems) creates and updates identities.
- Account Aggregation of other applications creates and updates Links and correlates them to identities.
- Account Group Aggregation loads groups and application objects as ManagedAttributes, for the catalog, group certifications, and descriptions and owners.
- Identity Refresh promotes attributes, assigns and detects roles, checks policies, processes lifecycle events, synchronizes attributes, and more.
- Follow-on tasks as needed, such as Effective Access Indexing, Refresh Role Indexes, and Full Text Index Refresh.
Authoritative first matters because non-authoritative accounts correlate to identities that must already exist.
Account Aggregation: Options to Know
| Option | Why it matters |
|---|---|
| Applications to scan | One task can aggregate several applications. Sequential Execution – Terminate on Error forces the listed order and stops at the first error. |
| Creation rule for new identities | An IdentityCreation-type rule applied when an account creates an identity |
| Only create links if they can be correlated | Update existing identities but never create new ones |
| Refresh assigned and detected roles, Check active policies, Refresh the identity risk scorecards, Maintain identity histories | Refresh-type work during aggregation. SailPoint does not recommend role refresh here if Identity Refresh manages correlation or provisioning, because it can cause unintended or incomplete provisioning. |
| Enable Delta Aggregation | Read only changes; needs connector support |
| Detect deleted accounts + Maximum deleted accounts | Removes Links for accounts that disappeared. If more than the maximum would be deleted, none are. This is a safety net against an empty or truncated feed. |
| Disable optimization of unchanged accounts | Forces full processing of unchanged accounts |
| Promote managed attributes | Adds newly seen entitlement values to the catalog |
| Disable marking the identity as needing refresh | Leave it off to support delta refresh |
| Enable Partitioning, Objects per partition (default 1,000), Loss Limit | Scale out, and bound the rework after a sudden termination |
| Terminate when maximum number of errors is exceeded | Stops a runaway task |
| Actions to include in the task result | Detailed actions such as Correlate New Account, Create New Identity, Correlate Reassign, and Remove Account |
If activity targets are used, run Target Aggregation after account aggregation, because account aggregation removes targets.
Account Group Aggregation
It aggregates groups and application objects (only where a group schema exists), with options for delta aggregation, detect deleted account groups, the description attribute and locale, a Group Aggregation Refresh rule (for example, to set owners), classification promotion, partitioning, and error limits. When scanning a single application, you can filter object types.
Delta Identity Refresh
- Aggregation sets
needsRefreshon identities it changes. This is on by default. - In Identity Refresh, choose Refresh only identities marked as needing refresh during aggregation, which is off by default. The task clears the flag unless Do not reset the needing refresh marker is selected. Use that option when several segmented refresh tasks each need the flag.
- Only aggregation sets the flag. LCM requests and other changes do not, so SailPoint recommends periodic full refreshes, such as an hourly delta with a daily full, or a daily delta with a weekly full.
- Timed triggers, such as a pre-offboarding workflow the day before a termination date, need a full refresh with Process Events, because the identity may not have changed in any aggregation.
Sequencing and Scheduling
- Sequential Task Launcher runs a list of tasks in order. It has a timeout per task, optional log statements, and Cease execution if one of the executing tasks encounters an error. It removes the guesswork of scheduling by estimated run time.
- Schedules: right-click a task and choose Schedule. Give it a unique name, a first execution time (or Run Now), and a frequency. Tasks run at the scheduled time in the time zone where they were scheduled.
- Hosts: a task's Host field can pin it to a host, or a comma-separated list where the first host with an active heartbeat runs it (section 2.1).
- Predefined tasks are not templates. Editing them changes the shipped task. Create new tasks with New Task.
Other Tasks Worth Knowing
- Run Rule runs an arbitrary rule with name/value arguments. The rule should return a status string, and anything other than "Success" produces a failed task result. It is useful for scheduled clean-up or reconciliation scripts.
- Refresh Role Indexes must run before role searching works.
- Full Text Index Refresh builds the LCM full-text indexes (section 4.1).
- Perform Identity Request Maintenance prunes old identity request objects and scans unverified access requests to check whether provisioning completed.
- Check Expired Work Items and Check Expired Mitigations handle escalations and expired exceptions.
- Prune Identity Cubes deletes identities with no accounts and no important references. Managers, identities with capabilities, owners, and protected identities are kept.
A Typical Schedule
| When | Sequence |
|---|---|
| Every 2 hours (business days) | HR delta aggregation, then AD delta aggregation, then delta Identity Refresh (attributes, roles, Process Events) |
| Nightly | Full aggregations of all applications, then Account Group Aggregation, then full Identity Refresh (policies, risk, histories), then Perform Identity Request Maintenance |
| Weekly | Perform Maintenance pruning, role index refresh, and a full refresh with scorecards |
Pair these with email task alerts on failure (section 3.4) and the Administrator Console (section 15.2) for monitoring.
A new contractor source is aggregated before the HR source each night, and many contractor accounts stay uncorrelated until the next day. What change fixes the order problem most reliably?
Run the authoritative aggregations first in a Sequential Task Launcher, followed by the non-authoritative aggregations and Identity Refresh.
Enable Promote managed attributes on the contractor task.
Mark the contractor application as a Proxy Application.
Disable marking identities as needing refresh.
An HR feed arrives truncated, and the aggregation with Detect deleted accounts would remove 9,000 accounts. Maximum deleted accounts is set to 500. What happens?
The first 500 accounts are deleted, and the rest are kept.
All 9,000 accounts are deleted.
The task deletes the accounts but marks them for undo.
No accounts are deleted, because the number exceeds the configured maximum.
A team switches Identity Refresh to process only identities marked as needing refresh. Which statement is true?
LCM requests also set the needsRefresh flag, so a full refresh is never needed.
The refresh option is enabled by default.
The flag is set by the Identity Refresh task itself.
Only aggregation sets the flag, so periodic full refreshes are still recommended.
What does the Sequential Task Launcher's "Cease execution if one of the executing tasks encounters an error" option do?
It stops the remaining tasks in the sequence when one task fails, instead of continuing in order.
It retries the failed task until it succeeds.
It skips the failed task and runs it last.
It sends the failure to the Syslog table only.
Sections you finish are checked off in the contents.