TIBCO Scribe® Online Connector For Salesforce

Use the TIBCO Scribe® Online Connector for Salesforce to replicate or integrate Salesforce Contact, Account, and Order data into other database or reporting systems.

Possible use cases for the Connector for Salesforce include:

Connector Specifications

This Connector supports Salesforce API Versions 37 through 51.

  Supported

Agent Types

On Premise X
Cloud X

Replication Services

Source X
Target  

Integration Services

Source X
Target X

Migration Services

Source X
Target X

Maps

Integration X
Request-Reply X
Message X

Note:  This Connector is available from the TIBCO Scribe® Online Marketplace. See Marketplace TIBCO Scribe® Certified Connectors for more information.

Setup Considerations

Salesforce Organization

To have the Expense, ExpenseReport, ExpenseReportEntry entities enabled, Field Service Lightning functionality must be activated in the Salesforce organization.

Salesforce User Account

Use an existing Salesforce account or request a trial account from www.salesforce.com. For information about configuration, see your Salesforce administrator.

Configure the account to use either:

Configure API settings:

Make sure that Bulk API is set correctly for your needs. See Bulk API vs Web Services API.

API Usage Limits

Salesforce limits the number of API calls in any 24-hour period for every Salesforce organization.

Note: Refreshing metadata for the Connector consumes API calls.

If you are an administrative user, you can view the number of API requests your organization has used in the last 24 hours, as follows:

  1. From the Salesforce User menu, select Setup. The Setup menu displays on the left side of the page.
  2. Select Administer > Company Profile > Company Information.
  3. The API Requests, Last 24 Hours field displays the current number of API requests along with the maximum available to your organization.

Note: These steps may vary in the Salesforce Lightning configuration.

Bulk API vs Web Services API

There are two methods for integrating data into or replicating data from Salesforce: Bulk API and Web Services API. While the Bulk API is slower, this method does use fewer API calls than the Web Services API.

For example, if you select the default batch size of 2000 records, for each batch:

Selecting An Agent Type For Salesforce

Refer to TIBCO Scribe® Online Agents for information on available Agent types and how to select the best Agent for your Solution.

Authentication

This connector supports the following authentication methods:

OAuth Authentication

To use OAuth Authentication, a Connected App in Salesforce is required. You can use the TIBCO OAuth App or you can create a new Connected OAuth App in Salesforce. To create a new Connected OAuth App, be sure to enable the following:

Once the app is created, you can find Client Id and Client Secret under the API>Enable OAuth Settings section.

Important: A maximum of four connections can be configured to a single Salesforce Connected app because each Connected app only supports four active Access/Refresh token pairs. If you exceed four separate connections, the Access/Refresh Token pairs are revoked as the active connections go beyond four even though the Refresh Token Policy for the app is set to Refresh token is valid until revoked.

Testing the connection generates the following error:

"Failed to re-authenticate user. Please open your connection, authenticate and save the connection."

Do not exceed the limit or configure another Connected App in Salesforce.

 

Connecting To Salesforce

Note: Best practice is to create Connections with credentials that limit permissions in the target system, following the principle of least privilege. Using Administrator level credentials in a Connection provides Administrator level access to the target system for TIBCO Scribe® Online users. Depending on the entities supported, a TIBCO Scribe® Online user could alter user accounts in the target system.

  1. Select More > Connections from the menu.
  2. From the Connections page select Add to open the Add a New Connection dialog.
  3. Select the Connector from the drop-down list in the Connection Type field, and then enter the following information for this Connection:
    • Name — This can be any meaningful name, up to 25 characters.
    • Alias — An alias for this Connection name. The alias is generated from the Connection name, and can be up to 25 characters. The Connection alias can include letters, numbers, and underscores. Spaces and special characters are not accepted. You can change the alias. For more information, see Connection Alias.
  4. Select and configure an Authentication Type for this Connection.

      Note: If you are connecting to a Salesforce sandbox, replace login.salesforce.com with your sandbox URL in the Salesforce URL field. Be sure to use the appropriate User name for the sandbox. For more information, see your Salesforce administrator.

    • For User Credentials
      • SalesforceURL — The web address that your site uses to reach the Salesforce web services API.
      • User/Password — The Salesforce user and password. The user name must be in the form of an email address.
      • Security Token — If your site has configured Salesforce to use a Security Token, enter it here. See your Salesforce administrator for more information.
    • For OAuth
      • SalesforceURL — The web address that your site uses to reach the Salesforce web services API.
    • For Connected OAuth App
      • SalesforceURL — The web address that your site uses to reach the Salesforce web services API.
      • Client Id/Client Secret — The Salesforce Consumer Key and Consumer Secret.

        Note: Use the TIBCO connect app or create a new app to connect to Salesforce. Client Id and Client Secret are required only when using your own Salesforce Connect App. See the OAuth Authentication section for information on where to find Client Id and Client Secret.

  5. Authenticate — Applies to OAuth and Connected OAuth authentication. When the Connection configuration is complete, use the Authenticate button to authenticate with Salesforce. If you are not logged into Salesforce, you are asked to log in. A Salesforce authentication dialog displays. To continue the authentication process enter login credentials, which allows TIBCO Cloud ™ Integration - Connect to access your Salesforce data. Select Close on the confirmation dialog.

    Tip: Do not test the Connection until the message at the top of the Connection dialog reads Authentication complete. Note that it may take a few minutes for this message to display.

  6.  Query Page Size — Specifies the number of query results to return per page. Enter a value between 200 and 2000. The default value is 2000.

    Note: You may want to lower this value if you are querying large records or if you are getting timeout errors.

  7. Select one or more options for Solutions using this Connection:
    • Integration or Migration Solutions:
      • IS/MS - Include Deleted Items In Query Results
        • Includes records from the query that are currently in the Salesforce Recycle Bin.
        • Salesforce sets the IsDeleted field to True (1) to indicate deleted records.
        • If a record is deleted in Salesforce, it cannot be updated.

          Note: Records in the Salesforce Recycle Bin are retained for a period dictated by Salesforce and this period may change in time. For more information on this retention period, see TIBCO's Salesforce Recycle Bin And Scribe Solutions KB article.

          TIBCO recommends running your Salesforce Solutions more frequently than this retention period.

      • IS/MS - Use Salesforce Bulk API For Batch Operations
        • Uses the Bulk API for Salesforce target operations.

          Note: You can choose whether to use Batch Processing for each Block in a Map, but to make the Bulk API available, you must enable Bulk API in Salesforce and select IS/MS - Use Salesforce Bulk API For Batch Operations.

        • For more information about integration options for Salesforce Connections, see Salesforce Connector As IS/MS Source.
    • Replication Solutions:
      • RS - Use Salesforce Bulk API For Initial Replication
        • Requires fewer Salesforce API calls.
        • Bulk API must also be enabled in Salesforce.
        • The replication is likely to be slower than if performed using the Salesforce Web Services API.
      • RS - Automatically Refresh Metadata On Every Run
        • Requires more Salesforce API calls, because the metadata must be retrieved from Salesforce.
        • The replication is slower because TIBCO Scribe® Online is waiting for the metadata. Best practice is to leave this option disabled unless you frequently modify your schema in Salesforce, such as, by adding custom fields. Enable it temporarily whenever you edit your Salesforce schema.

          Note: You must manually refresh metadata to see the updated metadata reflected in the TIBCO Scribe® Online user interface.

  8. Select Test to ensure that the Agent can connect to your database. Be sure to test the Connection against all Agents that use this Connection. See Testing Connections.

    Important: When using OAuth Authentication, testing the connection immediately saves the values on the Connection dialog. Selecting Cancel does not restore the prior configuration values.

  9. Select OK/Save to save the Connection.

Retry Logic

When a timeout occurs, the Salesforce Connector retries three times. Each time a retry fails, the time between retries is increased by 1 second.

Metadata Notes

Ignored Entities

The following entities are not available through the Salesforce Connector because they must be requested using a specific ID number and cannot be queried using any other method.

Ignored Salesforce Entities

IdeaComment

CollaborationGroupRecord

Vote

Naming

Connection metadata must have unique entity, relationship, and field names. If your Connection metadata has duplicate names, review the source system to determine if the duplicates can be renamed.

Relationships

Note: For this Connector, only one level of relationship, Parent/Child, is supported for the entity selected in the Query or Fetch Block. For records with multiple levels of relationships, such as Parent, Child, Grandchild, only the selected Parent entity and related Child entities can be retrieved. Additional levels beneath Child entities, such as Grandchildren, cannot be accessed for this Connector using the Include tab. See Block Properties Include Tab — Relational Data.

Salesforce Connector As RS Source

Salesforce can be used as a data source for Replication Solutions. The Salesforce Bulk API and the Web Services API are the available methods for replicating data from Salesforce. See Managing TIBCO Scribe® Online RS Solutions.

Recommended Salesforce Entities

When you select the Recommended entities option in a Replication Solution, TIBCO Scribe® Online replicates all entities for which you have the appropriate permissions, including custom entities, that meet the following criteria:

Enabling Salesforce Bulk API

When you select the RS - Use Salesforce Bulk API For Initial Replication check box on the Salesforce Connection dialog, the Salesforce Connection uses the Salesforce Bulk API, rather than the Salesforce Web Services API. The Bulk API can be especially useful for initial replication. It may be slower than the Web Services API, but the Bulk API requires far fewer Salesforce API calls.

Note: If you select the RS - Use Salesforce Bulk API For Initial Replication check box on the Salesforce Connection dialog without also enabling the Salesforce Bulk API in your Salesforce organization, no records are processed on the first execution of the Replication Solution.

Replicating Data Within Salesforce API Usage Limits

Depending on your edition of Salesforce, your Replication Solution may exceed the Salesforce API call usage limit for your Salesforce account.

Managing the Salesforce API call usage limit:

Preventing your site from reaching the API call limit:

For more information, see API usage metering under Implementation Considerations in the Salesforce.com Web Services Development Guide.

Net Change

After the initial replication that creates the tables and adds all the records for the selected or recommended entities in the target datastore, subsequent replications attempt to process only new and updated records. TIBCO Scribe® Online checks the SystemModStamp date field in the target datastore and locates the newest date for each replicated table. That date is then used as filter when querying Salesforce to select only those records that have a date higher than the date in the target datastore. In cases where there are entities that do not have a SystemModStamp date field, all records for those entities are processed each time a Replication Solution is executed.

To ensure that all records are replicated, the Connector subtracts one second from the most recent SystemModStamp date field, which could cause a small number of records to be reprocessed.

For example, if the last record written had a Last Modified Date of 09/04/2018 10:10:20, the next Replication starts with data that has a Last Modified Date of 09/04/2018 10:10:19. Any records that were already processed between 09/04/2018 10:10:19 and 09/04/2018 10:10:20 are processed again.

Deleted Records In Salesforce RS Solutions

Custom Entities

During replication, TIBCO Scribe® Online automatically creates any corresponding tables for custom entities in the target datastore. Depending on your subscription, you can select the entities you want to replicate from the Entities area of the Replication Solution page. See Selecting Source Entities For Replication Solutions.

If your Salesforce data contains custom entities:

Salesforce Connector As IS/MS Source

Note: With Salesforce Outbound Messaging, you can use Salesforce as the source Connection for an Event Message Map. See Defining An Event Message Map.

Filtering

When a datetime is configured on the Query Block on the Block Properties Net Change Tab to query for new and updated records, that configuration is treated as an additional filter. The Net Change datetime filter is applied as an AND after any other filters specified on the Block Properties Filter Tab. TIBCO Scribe® Online builds a query combining both the Net Change filter and the filters on the Filter tab. See Net Change And Filters for an example.

Some Connectors for TIBCO Scribe® Online only support one filter. For those Connectors you can use either Net Change or one filter on the Filter tab, not both.

Note: The Net Change date is ignored when previewing data on the Preview tab. Filters on the Block Properties Filters tab are used to filter the data on the Preview tab.

Native Query

MALFORMED_QUERY Fatal Error

Some relationships may cause a MALFORMED_QUERY to occur:

Batch Processing For Salesforce IS/MS

See Batch Processing for additional information.

Using The Salesforce Web Services API With Batch Processing

To use the Web Services API for Batch Processing, make sure that the IS - Use Salesforce Bulk API option is not selected from the TIBCO Scribe® Online Connections dialog for Salesforce.

Using The Salesforce Bulk API With Batch Processing

Make sure that the IS - Use Salesforce Bulk API option is selected from the TIBCO Scribe® Online Connection dialog for Salesforce and that Batch Processing is enabled in the appropriate operation Blocks in the Map. See Enabling Batch Processing for additional information.

Note: If you select the Bulk API, but do not select Batch Processing in the operation Blocks in your Map, the Bulk API is ignored and records are processed one at a time using the Salesforce Web Services API.

Enabling Batch Processing

After you have specified whether to use the Bulk API, you need to enable Batch Processing and specify the batch size for each Block within the Map.

  1. From the Connection dialog for Salesforce select IS/MS - Use Salesforce Bulk API For Batch Operations.
  2. Begin designing your Integration Map.
  3. For any target Blocks where Salesforce is your Connection, select the Process this operation in batches of check box and specify the batch size.

Salesforce Batch Size

The size of a batch can be between 1 and 10,000 records. The default size is 2000. Larger batches use fewer Salesforce API calls, but, especially if the individual records are very large, Salesforce processing may time out. To prevent timeouts the Connector uses the following Salesforce Bulk API limits to control the batch size:

If all of the data cannot be sent in one batch Bulk API call because of these limits, the Connector submits the data in multiple batch Bulk API calls that do not exceed these limits.

Best practice is to use the Salesforce Bulk API to integrate data that contains large fields and attachments. See the Bulk API and Web Services documentation on the Salesforce.com website for details about how Salesforce processes large amounts of data.

Salesforce Connector As IS/MS Target

Insert, Update, And Upsert With Relationships

TIBCO Scribe® Online provides virtual fields used to link external IDs during Insert, Update, or Upsert operations for child or parent records. Virtual fields eliminate the need for a Lookup when adding or updating records with relationships to the target, thus reducing the number of Salesforce API calls.

External ID fields are custom fields in Salesforce with the External ID attribute enabled. These fields contain unique record identifiers from an external system, such as an email address or an ID number. Any entity can have one or more of these fields to designate relationships with other entities.

In TIBCO Scribe® Online, if you are working with entities that have relationships established by external IDs:

Use Case

If you want to move marketing Leads and Activities from Salesforce Marketing Cloud into Salesforce and create the relationship between each Lead and its associated Activities, the process is as follows:

  1. In Salesforce create the following custom fields with the external ID attribute enabled:
    1. On the Leads entity create a custom field named ET_Lead_ID to contain the original Lead ID from Salesforce Marketing Cloud.
    2. On the Activities entity create a custom field named ET_Activity_ID to contain the original Activity ID from Salesforce Marketing Cloud. This allows you to update the Activity record again after it has been migrated to the Salesforce database.
  2. Using TIBCO Scribe® Online migrate Leads from Salesforce Marketing Cloud to Salesforce making sure that the ET_Lead_ID custom field is populated with the Lead ID from the Salesforce Marketing Cloud database. Salesforce creates a unique Lead ID for each lead record, therefore, you cannot use that field for the Salesforce Marketing Cloud Lead ID.
  3. Using TIBCO Scribe® Online migrate Activities from Salesforce Marketing Cloud to Salesforce. TIBCO Scribe® Online retrieves the relationship between Leads and Activities with the metadata from Salesforce. Any custom field in Salesforce with an Attribute of External ID on a parent or child entity displays in the Fields tab of the operation Block as a virtual field similar to the following:

    Activities_ChildActivities_ET_Lead_ID

  4. When setting up the Insert Block in the Map to migrate Activities, link the source Lead ID on the Salesforce Marketing Cloud Activity entity to the target virtual field that represents the ET_Lead_ID custom field you created on the Leads entity in Salesforce. In this use case, the field name would be:

    Activities_ChildActivities_ET_Lead_ID

    TIBCO Scribe® Online uses the Salesforce Marketing Cloud ID contained in this virtual field to look up the Salesforce Lead ID and put that Salesforce ID in the Activity record in the Salesforce database. This process creates the relationship between each activity and its parent lead.

  5. Link the source Activity ID on the Salesforce Marketing Cloud Activity entity to the target virtual field that represents the ET_Activity_ID custom field you created on the Activities entity in Salesforce. This field is used to update the Activity record in Salesforce from Salesforce Marketing Cloud at a later date using the Activity record's former Salesforce Marketing Cloud ID.

Batch Processing

Upsert Operation

For Salesforce Connections, where you can map to an external ID. a Lookup field, or another ID field, select the Upsert operation to perform the following single-step operation:

Note: The ID field must be unique to the entity and correspond to that Salesforce system.

An external ID field is a custom field created in Salesforce. If you map to multiple external ID fields, the Connector needs to know which ID takes priority when attempting to match records. To indicate which external ID field takes priority, enter the name of the external ID field in the ScribeExternalIDFieldName virtual field. When using an Upsert Block, note the following:

Case Assignments

Send Operation

Notes On Standard Entities

Order

OrderItem

Site

Miscellaneous

Performance

Under some circumstances, using Salesforce as a source can result in extremely slow processing. To improve performance:

  1. On the computer where the Agent is installed, update the Microsoft .NET Framework to v4.7.2.
  2. Install any associated Microsoft Windows updates.
  3. Restart the Agent service.

Errors

You may receive one of the following record errors when using Salesforce:

Status Code: REQUIRED_FIELD_MISSING || Error: Required fields are missing: XYZ

Resolution: There is a required field in your data that has not been mapped. Check the field mappings in your Map to make sure that all target fields in bold are properly mapped.

Status Code: DUPLICATE_VALUE || Error: Duplicate external id specified: XYZ field = ABC field

Resolution: You are trying to upsert data to a target entity with records that use the same external ID. If you are using the Upsert operation to a Salesforce target, make sure that all external ID fields are unique. This error is also generated if you are using Batch Processing and your source data contains more than one record with the same external ID within a single batch.

INVALID_TYPE_ON_FIELD_IN_RECORD

Resolution: On at least one field, your source and target data are of incompatible data types, such as if you attempt to map an ID of type GUID in the source to an Integer in the target. Make sure that data types are compatible or use one of the Data Conversion Functions.

TIBCO Scribe® Online API Considerations

To create Connections with the TIBCO Scribe® Online API, the Salesforce Connector requires the following information:

Connector Name

Salesforce

Connector ID

8ADD76FC-525F-4B4B-B79E-945A6A762792

TIBCO Scribe® Online Connection Properties

In addition, this Connector uses the Connection properties shown in the following table.

Note: Connection property names are case-sensitive.

Note: Configuring a connection that authenticates using OAuth is not supported by the API at this time.

Name Data Type Required Secured Usage
DeploymentType string Yes No User
Credentials

Url

string

Yes

No

 

UserId

string

Yes

No

 

Password

string

Yes

Yes

 

SecurityToken

string

No

No

 

QueryPageSize

integer

No

No

Range between 200 and 2000. If not provided, defaults to 2000.

UseBulkApiRS

string

Yes

No

true or false

UseBulkApiSYS

string

Yes

No

true or false

IncludeDeletedItemsSYS

string

Yes

No

true or false

RefreshMetaDataUponReconnect

string

Yes

No

true or false

License Agreement

The TIBCO Scribe® Online End User License Agreement for the Salesforce Connector describes TIBCO and your legal obligations and requirements. TIBCO suggests that you read the End User License Agreement.

More Information

For additional information on this Connector, refer to the Knowledge Base and Discussions in the TIBCO Community.