You are here:

Managing Execution Log Table Sizes

Insight performs execution and transaction error logging while integrations run, storing the information in the EXECUTIONLOG and TRANSACTIONERRORS log tables. Because these tables continue to grow, you may want to determine how long to retain log information and when to perform a log cleanup. When you schedule a log cleanup process, the process:

Sleeps for a defined number of hours.
Awakens and reads the log tables to identify the records that are older than the defined retention period.
Deletes a defined number of old records from the logs, repeating these deletions until no more old records remain.
Note: While old records are being deleted, the log is locked temporarily and no new records can be added, such as during regular job execution. To manage access to the log and avoid performance degradation of running integrations, records are deleted in small batches. These deletions are repeated until no more old records remain.
Sleeps for a defined number of hours before beginning the cycle again.

Note: The Scribe AdminServer service performs the log cleanup. You must start this service to manage the size of log tables.

You can use the ksEdit utility to prune the Execution Log tables. Changes made to the log tables are saved in the same folder where the trace file is saved. For more information about using the ksEdit utility, contact TIBCO Support.

IDBCLEANUP Keys

The number of records to delete with each pass through the log, the number of days to retain log records, and the number of hours to sleep are defined in the KSYNC table in the Scribe Internal Database:

Key Name	Description	Default Value
IDBCLEANUP_CHUNKRECORDSTODELETE	Number of old records to delete from a log.	100
IDBCLEANUP_DAYSTOPRESERVE_ALERTLOG	Days to retain records in the Alert log.	90
IDBCLEANUP_DAYSTOPRESERVE_COMPARELOG	Days to retain records in the Compare log.	90
IDBCLEANUP_DAYSTOPRESERVE_EXECLOG	Days to retain records in the Execution log.	90
IDBCLEANUP_FREQUENCYINHOURS	Number of hours to sleep before running the log trimming processes. If this value is 0, no records are deleted from the log.	4

You can configure any of these key values. For example, to retain Alert log records for 120 days, set IDBCLEANUP_DAYSTOPRESERVE_ALERTLOG to 120, and to continue retaining Execution log records for 90 days, keep IDBCLEANUP_DAYSTOPRESERVE_EXECLOG set to 90.

Note: If you upgraded from Insight 7.6.2 or earlier, the value of the IDBCLEANUP_FREQUENCYINHOURS key is set to 0. To enable the execution log cleanup, you must:
- Modify this key to a value greater than 0.
- Start or restart the Scribe AdminServer service (Control Panel > Administrative Tools > Services).

The IDBCLEANUP_FREQUENCYINHOURS key value determines the time that the log cleaning process sleeps after deleting all old records from the log. For example, if this key value is 2, the log cleaning process first runs at noon. If the process takes 20 minutes to delete all old records, the process sleeps for 2 hours and next runs at 2:20 PM.

Manage Processes

To manage storage and processing resources, you can modify the IDBCLEANUP_CHUNKRECORDSTODELETE and IDBCLEANUP_FREQUENCYINHOURS key values:

To reduce the length of time a log is locked during each deletion pass — Reduce the IDBCLEANUP_CHUNKRECORDSTODELETE key value. This may increase processing time to complete all of the old log record deletions.
To reduce the processing time to delete all old records from logs — Increase the IDBCLEANUP_CHUNKRECORDSTODELETE key value. This may increase the length of time that the log is locked.
Note: If you are receiving query timeout errors, reduce the IDBCLEANUP_CHUNKRECORDSTODELETE key value.

If you continue to receive query timeout errors using the Scribe AdminServer service, you can use the ScribeMaintenance.sql script. To use this script, you must stop the Scribe AdminServer service.
To reduce the number of old log records that need to be processed — Decrease the IDBCLEANUP_FREQUENCYINHOURS key value. This may increase the number of times the cleanup processes run.