Most Recent Record Processed

In TIBCO Scribe® Online, Net Change refers to new and updated records in a source datastore. Most Recent Record Processed is the datetime value used by TIBCO Scribe® Online to determine which records in a source datastore are new or have been updated.

If this option is enabled, each time a Solution executes, TIBCO Scribe® Online compares the Most Recent Record Processed value to a datetime value in each Source record. If the datetime value in the Source is newer than the Most Recent Record Processed value, the Source record is selected by the source query and processed by the Solution. Processing only new or updated records reduces the total number of records processed by TIBCO Scribe® Online.

Note: The Most Recent Record Processed feature is not supported for TIBCO Scribe® Online Migration Services (MS) or the Text Connector.

When a datetime is configured on the Query Block on the Block Properties Net Change Tab to query for new and updated records, that configuration is treated as an additional filter. The Net Change datetime filter is applied as an AND after any other filters specified on the Block Properties Filter Tab. TIBCO Scribe® Online builds a query combining both the Net Change filter and the filters on the Filter tab. See Net Change And Filters for an example.

Some Connectors for TIBCO Scribe® Online only support one filter. For those Connectors you can use either Net Change or one filter on the Filter tab, not both.

Note: The Net Change date is ignored when previewing data on the Preview tab. Filters on the Block Properties Filters tab are used to filter the data on the Preview tab.

Date Vs. DateTime

If an entity only has a date field with no time being used for Net Change, some records might be reprocessed depending on when the Solution executes. In that case, using the date only field to query for new and updated records returns all records starting at 12:00 am of the date in the Most Recent Record Processed field through the time that the Solution executes.

For example, if the value in Most Recent Record Processed is June 10, 2017 and you run the Solution at 1:00 pm on June 12, 2017, the query returns all records created or updated from 12:00 am on June 10, 2017 through June 12, 2017 at 1:00 pm. If the last new or modified record had a date value of June 12, 2017 and you run the Solution again at 4:00 pm on June 12, 2017, the query returns all records created or updated from 12:00 on June 12, 2017 through 4:00 pm. Records created between 12:00 am and 1:00 pm on June 12, 2017 are reprocessed.

TIBCO recommends executing Solutions querying entities with date only fields at 12:00 am to minimize the number of records that are reprocessed.

Last Modified Date Field

TIBCO Scribe® Online has a Last Modified Date concept that refers to the last time a Source record was changed. The Last Modified Date value is compared to the Most Recent Record Processed value by the query to locate only new and updated Source records. The value of the Last Modified Date field is determined as follows:

Most Recent Record Processed Value

The Most Recent Record Processed value is determined as follows: 

Using Most Recent Record Processed

To take advantage of the Most Recent Record Processed value, the Process only records created or updated since last run option must be enabled.

Related Topics

Query Block

Managing Failed RS Execution History Records