Setting bulk operations on a target data source allows Insight to commit a large block (up to 1000 rows) of data in a single transaction. If you are integrating a large amount of data into Dynamics CRM, Salesforce, or another data source that support bulk operations, this feature can save time.
Warning: There are some limitations to how Insight can process bulk data using bulk operations in a step. TIBCO suggests that you read this section carefully before configuring a DTS file with bulk mode steps.
In Insight, bulk operations are set on a target step as part of the Configure Steps dialog box. Bulk mode processing can be set for each step in a DTS file, and a DTS can have both bulk (block-based) and non-bulk (row-based) steps. For example:
When configuring steps for bulk operations, there are a number of differences in Insight behavior with bulk operations.
On the Operation tab, you can turn bulk mode on or off for a single step:
The value set in Number of rows per operation determines the maximum number of operations sent in a single bulk block. However, a block may actually be sent with fewer operations, depending on a number of factors, including the type of connection or the mix of features (especially Cross Reference Keys, Update Source, or Rejected Rows) used in the DTS file.
Note: Salesforce processes up to 200 rows in a single block. For example, if Number of rows per operation is set to 1000, each operation contains five API calls of 200 rows.
The Flow Control tab allows you to make changes to how a DTS file processes one step based on the results of a previous step. When configuring the Flow Control Tab, keep in mind that:
Note: Once the bulk block is committed, any operations that return Success (0) or Success (>1) are reported and managed as errors since these operations are not supported with bulk mode.
Note: When a job is stopped abruptly using an ENDJOB command (or Cancel) all of the rows in the current block are committed. For example, if a block already has 150 rows when a row encounters an ENDJOB, the 150 rows already in the block are committed before the job ends.
The All Steps tab allows you to set features across the entire DTS, whether or not there are bulk mode steps.
Note: These features can, however, be used in non-bulk steps within the same DTS file.
A number of Insight features depend on calculated or target variables, either directly or indirectly. Because Insight processes all of the rows in a bulk step in a single block, the data in the variable is not available when the row is processed. Therefore, the following features do not work with bulk mode steps if the variable is filled by the target:
For all of these features that depend on calculated or target variables, TIBCO recommends using separate chained DTS files between non-bulk and bulk steps. Alternatively, you could consider using a non-bulk Seek or Insert step to determine target values before any bulk mode steps.
Be aware that, in most cases, when working with DTS files with multiple bulk mode steps, the block is committed independently. This may cause blocks to be committed out of order. For example, if you have a DTS file with three bulk steps, if the block in Step 2 fills before the block in Step 1, the Step 2 block is committed (before the block in Step 1).
The purpose of bulk operations is to improve performance and limit API hits by submitting blocks of operations in a single call. However, this method of integration limits the flexibility of the DTS file. When designing a DTS file with bulk operations, you can easily get into a situation in which one step depends on the values of a previous step.
To avoid this situation, TIBCO recommends that when using bulk operations, simplify your step design as much as possible. Keeping your bulk steps simple improves your results. Use multiple chained DTS files rather than trying to create complex step logic within a single DTS that contains bulk operations.
Prior to Insight Version 7.6, the Bulk Insert feature was set for the entire DTS file, rather than for each step. When these DTS files are converted, all Bulk Insert mode settings are now configured on each step of the DTS file.
If there are issues with your bulk mode logic, data can become corrupted. Be sure to thoroughly test your DTS files in a development environment before deploying it to your live production environment.
See also