Most Recent Record Processed
In TIBCO Scribe® Online, Net Change refers to new and updated records in a source datastore. Most Recent Record Processed is the datetime value used by TIBCO Scribe® Online to determine which records in a source datastore are new or have been updated.
When the Net Change option is enabled, each time a Solution executes, TIBCO Scribe® Online compares the Most Recent Record Processed value to a datetime value in each Source record. If the datetime value in the Source is newer than the Most Recent Record Processed value, the source query selects the Source record and the Solution processes that record. Processing only new or updated records reduces the total number of records processed by TIBCO Scribe® Online.
Note: The Most Recent Record Processed feature is not supported for TIBCO Scribe® Online Migration Services (MS) or the Text Connector.
When a datetime is configured on the Query Block on the Block Properties Net Change Tab to query for new and updated records, that configuration is treated as an additional filter. The Net Change datetime filter is applied as an AND after any other filters specified on the Block Properties Filter Tab. TIBCO Scribe® Online builds a query combining both the Net Change filter and the filters on the Filter tab. See Net Change And Filters for an example.
Some Connectors for TIBCO Scribe® Online only support one filter. For those Connectors you can use either Net Change or one filter on the Filter tab, not both.
Note: The Net Change date is ignored when previewing data on the Preview tab. Filters on the Block Properties Filters tab are used to filter the data on the Preview tab.
Date Vs. DateTime
If an entity only has a date field with no time being used for Net Change, some records might be reprocessed depending on when the Solution executes. Using the date only field to query for new and updated records returns all records starting at 12:00 am of the date in the Most Recent Record Processed field through the time that the Solution executes.
For example, if the value in Most Recent Record Processed is June 10, 2017 and you run the Solution at 1:00 pm on June 12, 2017, the query returns all records created or updated from 12:00 am on June 10, 2017 through June 12, 2017 at 1:00 pm. If the last new or modified record had a date value of June 12, 2017 and you run the Solution again at 4:00 pm on June 12, 2017, the query returns all records created or updated from 12:00 on June 12, 2017 through 4:00 pm. Records created between 12:00 am and 1:00 pm on June 12, 2017 are reprocessed.
TIBCO recommends executing Solutions querying entities with date only fields at 12:00 am to minimize the number of records that are reprocessed.
Last Modified Date Field
TIBCO Scribe® Online has a Last Modified Date concept that refers to the last time a Source record was changed. The Last Modified Date value is compared to the Most Recent Record Processed value by the query to locate only new and updated Source records. The value of the Last Modified Date field is determined as follows:
- Replication Solutions - Connectors that support Replication Solutions specify the Source field to be used as the Last Modified Date field for each entity included in the Solution. When the Replication Solution executes, it retrieves the datetime value for each Source record from the field specified by the Connector. If the entity being processed does not have a datetime field that can be used as the Last Modified Date field, then all of the records for that entity are processed each time the Solution runs. To reduce the number of records processed, configure the Replication Solution to use the Recommended Entities or Selected Entities instead of All Entities.
- Integration Solutions - When the Process only records created or updated since last run option is enabled in a Map, you are required to select a datetime field in the Source entity to serve as the Last Modified Date field. The value of the selected field is compared against the Most Recent Record Processed value each time the Solution is executed. Be sure to select a datetime field that is updated anytime a Source record is modified.
Most Recent Record Processed Value
The Most Recent Record Processed value is determined as follows:
- Replication Solutions - When TIBCO Scribe® Online replicates data into a target datastore, it filters source data for each table based on the Last Modified Date field specified by the Connector. TIBCO Scribe® Online determines which field is the Last Modified Date field for a particular table, and reviews the dates in that field in the target datastore to locate the most recent datetime. That value becomes the Most Recent Record Processed value for the selected table in the current execution of the Replication Solution. If there is a Fatal Exception during replication, the next time the Replication Solution executes TIBCO Scribe® Online subtracts 500 milliseconds from the date/time field found in the target table where the failure occurred. This ensures that no records are missed, however, a few records may be reprocessed unnecessarily.
- Integration Solutions - When TIBCO Scribe® Online finishes executing an Integration Solution, it stores the datetime value of the Source record with the most recent Last Modified Date datetime. The value of the Last Modified Date field for that record becomes the Most Recent Record Processed value for the next execution of the Solution. The Most Recent Record Processed value is stored with the Solution and is updated each time the Solution executes.
A Source record triggers an update to the Most Recent Record Processed date, if it was used as the source in an operation Block that modifies records in the target, including: Create, Delete, Update, Update/Insert, or Upsert.
A Source record does not trigger an update to the Most recent record processed date, if:
- The operation does not write any information from the source record into the target system.
- It was used by an operation Block that does not modify the target, such as Fetch or Lookup.
Note: For Integration Solutions, the Most Recent Record Processed value can be manually reset to retrieve older records, if necessary. See Query Block and Block Properties Net Change Tab.
Using Most Recent Record Processed
To take advantage of the Most Recent Record Processed value, the Process only records created or updated since last run option must be enabled.
- Replication Solutions - The Process only records created or updated since last run option is enabled internally and cannot be disabled.
- Integration Solutions - The Process only records created or updated since last run option is enabled in the Query Block in a Map. See Block Properties Net Change Tab.