Processed Records
Each record processed by TIBCO Scribe® Online impacts performance. To successfully manage the number of records your Organization processes, it is important to know how TIBCO Scribe® Online counts processed records and how to reduce the number of records processed.
Processed Records
TIBCO Scribe® Online includes a record in the processed records total if it meets any of the following criteria:
- Replication Solutions — Records that are returned from the source datastore by the Solution, including failed records.
- Integration and Migration Solutions — Records returned by the source Query, as well as any records returned by each Fetch Block. For example, if your Query retrieves 5 records, and each pass through the Map calls a Fetch Block that returns 100 records, the total records processed is 505. The Fetch Block runs 5 times, once for each Query record.
Reducing Processed Records
Note: Batch processing does not reduce the number of processed records. See Batch Processing.
To reduce the number of processed records try some of the following options:
RS Solutions
RS Solutions are filtered by Last Modified Date. If a Source entity does not have a datetime field to use as the Last Modified Date, all records in that entity are processed each time the RS Solution executes.
- Modify the Solution to use only Recommended Entities or Selected Entities. This helps eliminate entities without a datetime field or entities that are unnecessary. See Selecting Source Entities For Replication Solutions.
IS And MS Solutions
- Limit the query to new and updated records by enabling the Process only records created or updated since last run option in on the Net Change tab in Query Blocks. See Most Recent Record Processed, Block Properties Net Change Tab, , and Query Block.
- Add filters in the Query to exclude records updated by the Ignore user. In situations where you have two datastores that synchronize in both directions, this prevents an endless loop of unnecessary updates for the same record between the two datastores. See Ignore User.
- Add more complex filters to Query and Fetch Blocks to reduce the number of records returned. See Block Properties Filter Tab.
- Merge Maps that retrieve the same data so that the Query is run only once.
- Update multiple targets with a single Map. See Multiple Targets.
- Update Source data with data from the target using an Update Block and the result set from the original Query Block within a single Map. See Update Block.
- Use an Integration Event Solution containing a Message Map. This type of Solution executes only when a message is received from an external application, therefore, it executes and processes records only when needed, instead of on a schedule. See Defining An Event Message Map.
- Use Parent/Child Relationships to process data. Joined records are not included in the processed record count. For example, query the parent entity Accounts and also get child records for Contacts and Addresses. See Query Block, CreateWith Block, UpsertWith Block, and Fetch Block.
Parent/Child Relationship Record Counts
Joined records do not count towards total records processed. Records are counted differently based on whether you use a Child Join or a Parent Join.
Child Join
- Primary Entity = Account
- Joined Entity = Contact
Assume that for each Account record, there are 10 Contact records.
If you process 100 Account records, each with 10 Contact records, total records processed for the job is 100. However, total records added to the target is 1100, or 100 Account records and 1000 Contact records.
Parent Join
- Primary Entity = Contact
- Joined Entity = Account
Assume that for each Contact record, there is 1 Account record.
If you process 100 Contact records, each with 1 Account record, your total record count for the job is 100. However, total records added to the target is 200, or 100 Contact records and 100 Account records.
