TIBCO Scribe® Online Connector For Microsoft Common Data Service (CDS)

Use the TIBCO Scribe® Online Connector for Microsoft Common Data Service (CDS) to work with the Microsoft Common Data Service (CDS) database. It supports custom fields and custom entities, relationships between entities, and supports Microsoft Dynamics 365 API’s bulk load method for higher performance.

Note: Some features are available only if you had a Microsoft Dynamics 365 / CRM organization when you created your Microsoft Common Data Service (CDS) database, such as a subset of Microsoft Dynamics 365 entities, Merge functionality, and alternate keys.

Connector Specifications

  Supported

Agent Types

On Premise X
Cloud X

Replication Services

Source X
Target  

Integration Services

Source X
Target X

Migration Services

Source X
Target X

Maps

Integration X
Request-Reply X
Message  

Note: This Connector is available from the TIBCO Scribe® Online Marketplace. See Marketplace TIBCO Scribe® Certified Connectors for more information.

Setup Considerations

Requires either a Microsoft PowerApps account and a Microsoft Common Data Service (CDS) database.

Permissions

Your Microsoft PowerApps account must have appropriate permissions to allow TIBCO Scribe® Online to perform Create, Read, Update, Delete, and Batch or non-Batch operations. Contact your system administrator for assistance.

Run As User

The Microsoft Common Data Service (CDS) Connector provides the ability for a user to perform all operations as if a different user were performing them. In the Connection, this alternate user is the Run As User. The user named in the User ID field is the "login user".

Assume that as the login user, you have administrator credentials, but you set the Run As User to be a salesperson. You then run a Solution that imports contacts. In this case, all the contacts appear to have been entered by the Run As User. In addition, all operations run in the context of the roles and privileges held by the Run As User.

Selecting An Agent Type For Microsoft Common Data Service (CDS)

Refer to TIBCO Scribe® Online Agents for information on available Agent types and how to select the best Agent for your Solution.

Connecting To Microsoft Common Data Service (CDS)

Note: Best practice is to create Connections with credentials that limit permissions in the target system, following the principle of least privilege. Using Administrator level credentials in a Connection provides Administrator level access to the target system for TIBCO Scribe® Online users. Depending on the entities supported, a TIBCO Scribe® Online user could alter user accounts in the target system.

There are three deployment types for the Microsoft Common Data Service (CDS) Connector that provide different authentication methods. Follow the link for the authentication method you plan to use for step-by-step configuration instructions: 

Online Authentication

  1. Select More > Connections from the menu.
  2. From the Connections page select Add to open the Add a New Connection dialog.
  3. Select the Connector from the drop-down list in the Connection Type field, and then enter the following information for this Connection:
    • Name — This can be any meaningful name, up to 25 characters.
    • Alias — An alias for this Connection name. The alias is generated from the Connection name, and can be up to 25 characters. The Connection alias can include letters, numbers, and underscores. Spaces and special characters are not accepted. You can change the alias. For more information, see Connection Alias.
    • Deployment — Select Online if your site is using Microsoft PowerApps and you do not want to authenticate using Active Directory.

      When you select the deployment, TIBCO Scribe® Online displays the appropriate Connection parameters fields.

  4. Discovery Service URL — The URL (web address) for Microsoft Dynamics 365. For Microsoft Common Data Service (CDS) you still connect to a Microsoft Dynamics 365 URL. This URL varies depending on the type of deployment selected.

    Microsoft URLs

    • Microsoft Office 365
    • Location

      URL

      North America

      https://disco.crm.dynamics.com

      North America 2

      https://disco.crm9.dynamics.com

      Europe, Middle East, and Africa (EMEA)

      https://disco.crm4.dynamics.com

      Asia Pacific (APAC)

      https://disco.crm5.dynamics.com

      Oceania

      https://disco.crm6.dynamics.com

      Japan (JPN)

      https://disco.crm7.dynamics.com

      South America

      https://disco.crm2.dynamics.com

      India

      https://disco.crm8.dynamics.com

      Canada

      https://disco.crm3.dynamics.com

    • Microsoft Account
    • Location

      URL

      North America

      https://dev.crm.dynamics.com

      Europe, Middle East, and Africa (EMEA)

      https://dev.crm4.dynamics.com

      Asia Pacific (APAC)

      https://dev.crm5.dynamics.com

  5. User ID/Password — A valid Microsoft PowerApps user ID and password.
  6. Organization — The Microsoft Organization you want to access. Select Browse to have TIBCO Scribe® Online determine the Organization associated with these Microsoft credentials. Browse only displays for Online Deployments.

    Note: The Microsoft-friendly Organization name is the Organization name that appears in the upper right hand corner of your Microsoft account. This name is case-sensitive.

  7. Run As User ID — Optionally, specify the valid User ID of a user to serve as the Run As User.
  8. Maximum Batch Size — Maximum number of records to include in a single call to the Microsoft API when Batch Processing is enabled for a target operation. Default is 500, minimum is 1, and maximum is 1000. Setting to 0 disables the Test button. Setting to a value higher than 1000 defaults to 1000, but continues to display the higher number.

    Note: Receiving Timeout errors is an indication that the number of records per call has exceeded the default setting for your Microsoft Organization and you should decrease this number.

    Modifying or requesting modification of the default setting by Microsoft or your hosting partner may allow for a higher value. See the following Microsoft Knowledge Base Article: Service Protection API Limits and the Microsoft D365 CRM/CE request limits/allocations and API Limits article in the TIBCO Knowledge Base.

  9. Maximum Batch Threads — Number of simultaneous calls to the Microsoft API when Batch Processing is enabled for a target operation, such as Delete. Default is 2, minimum is 1, and maximum is 32. Setting to 0 disables the Test button. Setting to a value higher than 32 defaults to 32, but continues to display the higher number.

    Note: Receiving any of the following errors, is an indication that the number of calls or threads has exceeded the default setting for your Microsoft Organization and you should decrease this number.

    • Server busy
    • The following error has occurred in the Dynamics 365 Connector: System.ObjectDisposedException: Cannot access a disposed object.
    • Generic SQL errors

    Modifying or requesting modification of the default setting by Microsoft or your hosting partner may allow for a higher value. See the following Microsoft Knowledge Base Article: Service Protection API Limits and the Microsoft D365 CRM/CE request limits/allocations and API Limits article in the TIBCO Knowledge Base.

  10. Include Picklist Display Names select this option if you want to include the display names for any picklist, status, or state fields that you are replicating. When the entity is recreated, TIBCO Scribe® Online adds a new field in the format fieldname _displayname for each qualifying field, where _displayname is the text description rather than the numeric value.

    See Picklists below for more information.

  11. Select Test to ensure that the Agent can connect to your database. Be sure to test the Connection against all Agents that use this Connection. See Testing Connections. If the Test fails, an error similar to the following is displayed. Correct the issue outlined in the error and try the test again.

  12. Select OK/Save to save the Connection.

Azure AD Authentication

The Microsoft Common Data Service (CDS) Connector supports connecting to your Microsoft Common Data Service (CDS) instance using Azure AD authentication. This authentication method requires a separate configuration to establish a connection from TIBCO Scribe® Online.

  1. In Microsoft PowerApps make sure that you are an Admin user.
  2. In TIBCO Scribe® Online, select More > Connections and then select t Add to open the Add a New Connection dialog.
  3. In the Connector Type field select Microsoft Common Data Service (CDS).
  4. Enter a Connection Name and an Alias.
  5. In the Deployment field select Azure AD.
  6. In the Discovery Service URL field, enter the URL of your discovery service. This is used to validate your Microsoft PowerApps Organization name. For Microsoft Common Data Service (CDS) you still validate using a Microsoft Dynamics 365 URL. See the Microsoft URLs table in the Connecting To Microsoft Common Data Service (CDS) section for a list of URLs. Example: https://disco.crm.dynamics.com

    Note: Only the first part of the full service URL displayed in your Microsoft PowerApps instance is used. If you enter the full service URL an error is generated. For example, if your full service URL is https://disco.crm.dynamics.com/XRMServices/2011/Discovery.svc, only enter https://disco.crm.dynamics.com/.

  7. In the D365 Instance URL field, enter the URL for your Microsoft PowerApps instance. Example:  https://<yourcompany>.crm.dynamics.com

    Note: The D365 Instance URL is case sensitive when testing the Connection, but not when authenticating. Best practice is to copy the URL from your Microsoft PowerApps instance and paste it into the Connection dialog.

  8. Select the Authenticate button. This launches a new browser tab with a Sign in page. Enter your Sign in information and select Next.
  9. On the Password dialog, enter your password and select Sign In.
  10. If you have enabled multi-factor authentication in your Microsoft PowerApps Organization, the Enter code dialog displays. Enter the code you received and select Verify.

  11. A permissions page is displayed.

  12. Select Accept to grant permission for TIBCO Scribe® Online to access this Microsoft PowerApps instance. The Success page is displayed.

  13. On the Success page select Close to return to TIBCO Scribe® Online. Next to the Grant Consent button the message Consent granted. is displayed. This message indicates that permission has been granted correctly and you can proceed with the remaining configuration steps.
  14. Organization — The Microsoft Organization you want to access. You can use either the friendly name or the unique name. Names are case-sensitive, therefore a best practice is to copy the name from your Microsoft PowerApps instance and paste it into the Connection dialog.

    Note: The Microsoft-friendly Organization name is the Organization name that appears in the upper right hand corner of your Microsoft account.

  15. Maximum Batch Size — Maximum number of records to include in a single call to the Microsoft API when Batch Processing is enabled for a target operation. Default is 500, minimum is 1, and maximum is 1000. Setting to 0 disables the Test button. Setting to a value higher than 1000 defaults to 1000, but continues to display the higher number.

    Note: Receiving Timeout errors is an indication that the number of records per call has exceeded the default setting for your Microsoft Organization and you should decrease this number.

    Modifying or requesting modification of the default setting by Microsoft or your hosting partner may allow for a higher value. See the following Microsoft Knowledge Base Article: Service Protection API Limits and the Microsoft D365 CRM/CE request limits/allocations and API Limits article in the TIBCO Knowledge Base.

  16. Maximum Batch Threads — Number of simultaneous calls to the Microsoft API when Batch Processing is enabled for a target operation, such as Delete. Default is 2, minimum is 1, and maximum is 32. Setting to 0 disables the Test button. Setting to a value higher than 32 defaults to 32, but continues to display the higher number.

    Note: Receiving any of the following errors, is an indication that the number of calls or threads has exceeded the default setting for your Microsoft Organization and you should decrease this number.

    • Server busy
    • The following error has occurred in the Dynamics 365 Connector: System.ObjectDisposedException: Cannot access a disposed object.
    • Generic SQL errors

    Modifying or requesting modification of the default setting by Microsoft or your hosting partner may allow for a higher value. See the following Microsoft Knowledge Base Article: Service Protection API Limits and the Microsoft D365 CRM/CE request limits/allocations and API Limits article in the TIBCO Knowledge Base.

  17. Include Picklist Display Names select this option if you want to include the display names for any picklist, status, or state fields that you are replicating. When the entity is recreated, TIBCO Scribe® Online adds a new field in the format fieldname _displayname for each qualifying field, where _displayname is the text description rather than the numeric value.

    See Picklists below for more information.

  18. Select Test to ensure that the Agent can connect to your database. Be sure to test the Connection against all Agents that use this Connection. See Testing Connections. If the Test fails, you must reauthenticate. Correct the issue outlined in the error and try the test again.

    Note: The D365 Instance URL is case sensitive when testing the Connection, but not when authenticating. If the test fails, check the case on the Dynamics Instance URL to make sure it is correct.

  19. Select OK/Save to save the Connection.

Azure AD Best Practices

If you authenticate using Azure AD, there are some timing issues to take into consideration to prevent errors.

Client Secret Authentication

The Microsoft Common Data Service (CDS) Connector supports connecting to your Microsoft Common Data Service (CDS) instance using Client Secret authentication. This authentication method requires a separate configuration to establish a connection from TIBCO Scribe® Online.

  1. In Microsoft PowerApps make sure that you are an Admin user.
  2. In TIBCO Scribe® Online, select More > Connections and then select t Add to open the Add a New Connection dialog.
  3. In the Connector Type field select Microsoft Common Data Service (CDS).
  4. Enter a Connection Name and an Alias.
  5. In the Deployment field select Client Secret.
  6. In the URL field, enter the URL for your Microsoft PowerApps instance. Example:  https://<yourcompany>.crm.dynamics.com

    Note: The URL is case sensitive when testing the Connection. Best practice is to copy the URL from your Microsoft PowerApps instance and paste it into the Connection dialog.

  7. Client ID — Enter the Client ID, also know as the Application ID, generated when you registered the Azure Active Directory (AAD) App.
  8. Client Secret — Enter the Client Secret generated when you registered the Azure Active Directory (AAD) App.
  9. Maximum Batch Size — Maximum number of records to include in a single call to the Microsoft API when Batch Processing is enabled for a target operation. Default is 500, minimum is 1, and maximum is 1000. Setting to 0 disables the Test button. Setting to a value higher than 1000 defaults to 1000, but continues to display the higher number.

    Note: Receiving Timeout errors is an indication that the number of records per call has exceeded the default setting for your Microsoft Organization and you should decrease this number.

    Modifying or requesting modification of the default setting by Microsoft or your hosting partner may allow for a higher value. See the Runtime Limitations section of the following Microsoft Knowledge Base Article: Use ExecuteMultiple to improve performance for bulk data load.

  10. Maximum Batch Threads — Number of simultaneous calls to the Microsoft API when Batch Processing is enabled for a target operation, such as Delete. Default is 2, minimum is 1, and maximum is 32. Setting to 0 disables the Test button. Setting to a value higher than 32 defaults to 32, but continues to display the higher number.

    Note: Receiving any of the following errors, is an indication that the number of calls or threads has exceeded the default setting for your Microsoft Organization and you should decrease this number.

    • Server busy
    • The following error has occurred in the Dynamics 365 Connector: System.ObjectDisposedException: Cannot access a disposed object.
    • Generic SQL errors

    Modifying or requesting modification of the default setting by Microsoft or your hosting partner may allow for a higher value. See the Runtime Limitations section of the following Microsoft Knowledge Base Article: Use ExecuteMultiple to improve performance for bulk data load.

  11. Include Picklist Display Names select this option if you want to include the display names for any picklist, status, or state fields that you are replicating. When the entity is recreated, TIBCO Scribe® Online adds a new field in the format fieldname _displayname for each qualifying field, where _displayname is the text description rather than the numeric value.

    See Picklists below for more information.

  12. Select Test to ensure that the Agent can connect to your database. Be sure to test the Connection against all Agents that use this Connection. See Testing Connections. If the Test fails, you must reauthenticate. Correct the issue outlined in the error and try the test again.

    Note: The URL is case sensitive when testing the Connection. If the test fails, check the case on the Dynamics Instance URL to make sure it is correct.

  13. Select OK/Save to save the Connection.

Metadata Notes

Filter, Match Criteria, Lookup, And Target Operations

Leading and trailing spaces in string field values are trimmed when used in a filter, matching criteria, or lookup criteria, or when writing those values during target operations, such as Create or Update.

Naming

Connection metadata must have unique entity, relationship, and field names. If your Connection metadata has duplicate names, review the source system to determine if the duplicates can be renamed.

Custom Decimal Fields

When adding custom decimal fields to entities in Microsoft Common Data Service (CDS) with a precision of 0, the precision is not being honored.

Picklists

In Microsoft Common Data Service (CDS) databases, some fields are generated via a picklist. Picklists include both a text value that displays in Microsoft Common Data Service (CDS) and an internal code. By default, TIBCO Scribe® Online uses the text value when running a Solution. However, you can choose to exclude the text value that displays in the Microsoft Common Data Service (CDS) interface by disabling the Include Picklist Display Names option in the Microsoft Common Data Service (CDS) Connection configuration. When this option is enabled, an additional field is created in your Microsoft Common Data Service (CDS) datastore to contain the text value.

For example, if you have a picklist field, shippingmethod, where the picklist values include available shipping methods of DHL, FedEx, UPS, or USPS, and the Include Picklist Display Names option is enabled, both the picklist value and the picklist_displayname columns display for that entity in the database, as shown:

Picklist value shippingmethod _displayname

1

DHL

2

FedEx

3

UPS

4

USPS

Including Picklist Display Names

When you create a new Microsoft Common Data Service (CDS) Connection, the Include Picklist Display Names option is selected by default. Picklist, state, and status fields include a _displayname column. You can choose to include or exclude the _displayname by editing the Connection.

  1. From the Connections page, edit the Microsoft Common Data Service (CDS) Connection you want to change.
  2. Select Include Picklist Display Names.

    To exclude the _displayname, make sure this option is not selected.

  3. Select Test and then select OK/Save to save the changes to the Connection.

Note: TIBCO Scribe® Online may be considerably slower the first time you run the Solution after adding or removing picklist names.

Changing Picklist Entities

When you make changes in Microsoft Common Data Service (CDS) to entities with picklists, the source and target are affected as follows:

Note: If you change the value of a picklist field for any records within Microsoft Common Data Service (CDS) after a replication runs, the display name changes are not reflected in previously replicated records. To properly replicate changed picklist values, drop the table in the target database. The next time the replication runs, the table is recreated with the new display names.

Using Picklists With Integration Solutions

When using picklist values with Integration Solutions, because the display names are not actual fields in the source or target entity, the following caveats apply:

Multi-Select Option Sets

Multi-select option sets are picklists that allow a Microsoft Common Data Service (CDS) user to select more than one value for a picklist for a single record. For example, a multi-select option set named Countries Visited where the user could select one or more countries instead of a single country.

Note: The Formula Editor used to map fields on the Block Properties Fields tab does not support picklist option names that contain commas or quotes, such as Yell,ow or Yell"ow. When mapping these types of names, use the numeric picklist value instead.

Ignored Entities

The following entities may exist in your Microsoft Common Data Service (CDS) database if you had a Microsoft Dynamics 365 / CRM instance when you created the CDS database. However they are not available through the Microsoft Common Data Service (CDS) Connector.

Ignored Microsoft Dynamics 365 Entities

  applicationfile

  asyncoperation

  attributemap

  bulkoperation

  clientupdate

  constraintbasedgroup

  displaystring

  duplicaterule

  entitymap

  import

  importdata

  importfile

  importjob

  importlog

  importmap

  integrationstatus

  isvconfig

  mailmergetemplate

  metadatadifference

  new_scribechangehistory

  new_scribepublisherqueue

  pluginassembly

  plugintype

  plugintypestatistic

  relationshiprole

  report

  resourcespec

  resourcegroupexpansion

  savedquery

  savedqueryvisualization

  sdkmessage

  sdkmessagefilter

  sdkmessagepair

  sdkmessageprocessingstep

  sdkmessageprocessingstepimage

  sdkmessageprocessingstepsecureconfig

  sdkmessagerequest

  sdkmessagerequestfield

  vsdkmessageresponse

  sdkmessageresponsefield

  serviceendpoint

  solutioncomponent

  usersettings

  webwizard

  wizardaccessprivilege

  wizardpage

  workflow

  workflowlog

Microsoft Common Data Service (CDS) Connection As RS Source

Microsoft Common Data Service (CDS) can be used as a data source for Replication Services Solutions. See Managing TIBCO Scribe® Online RS Solutions.

Note: For information about using Microsoft Common Data Service (CDS) with TIBCO Scribe® Online Replication Solutions, see Selecting Source Entities For Replication Solutions

Recommended Entities

When you select the Recommended entities option in the RS Solution, TIBCO Scribe® Online replicates entities for which you have the appropriate permissions, including custom entities.

Note: Custom entities with the prefixes msdyn_, msdynsm_, and msfp._ are not replicated.

Entities In Recommended List

Microsoft Common Data Service (CDS) Entities Included In Recommended List

  account

  annotation

  appointment

  bookableresource

  bookableresourcebooking

  bookableresourcebookingheader

  bookableresourcecategory

  bookableresourcecategoryassn

  bookableresourcecharacteristic

  bookableresourcegroup

  bookingstatus

  businessunit

  businessunitnewsarticle

  calendar

  campaign

  campaignactivity

  campaignresponse

  cardtype

  category

  characteristic

  competitor

  competitoraddress

  connection

  connectionrole

  contact

  contract

  contractdetail

  customeraddress

  customeropportunityrole

  customerrelationship

  discount

  discounttype

  email

  entitlement

  entitlementchannel

  entitlementcontacts

  entitlemententityallocationtypemapping

  entitlementproducts

  equipment

  fax

  feedback

  goal

  incident

  incidentresolution

  interactionforemail

  internaladdress

  invoice

  invoicedetail

  kbarticle

  knowledgearticle

  knowledgearticleviews

  lead

  leadaddress

  letter

  list

  mailbox

  metric

  opportunity

  opportunityclose

  opportunityproduct

  opportunitysalesprocess

  orderclose

  phonecall

  phonetocaseprocess

  position

  post

  pricelevel

  processsession

  product

  productpricelevel

  publisher

  publisheraddress

  quote

  quoteclose

  quotedetail

  recurringappointmentmaster

  reportcategory

  reportentity

  reportlink

  reportvisibility

  role

  salesliterature

  salesliteratureitem

  salesorder

  salesorderdetail

  service

  serviceappointment

  sharepointdocumentlocation

  sharepointsite

  site

  sla

  slaitem

  socialactivity

  socialprofile

  subject

  systemuser

  task

  territory

  topichistory

  topicmodel

  transactioncurrency

  uom

  uomschedule

When you select either the Recommended entities or the Selected entities option in the RS Solution, the internal entities below are not included, but can be replicated if you use the Select All option. If you were not using Microsoft Dynamics 365 / CRM when you created your Microsoft Common Data Service (CDS) database, these entities may not exist in your database.

Microsoft Dynamics 365 Entities Not Included In Recommended Or Selected Lists

  appconfigmaster

  authorizationserver

  businessprocessflowinstance

  childincidentcount

  cloneproduct

  delveactionhub

  documentindex

  lookupmapping

  merge

  msdyn_odatav4ds

  multientitysearch

  officedocument

  officegraphdocument

  orginsightsmetric

  orginsightsnotification

  partnerapplication

  principalobjectaccessreadsnapshot

  publishproducthierarchy

  queueitemcount

  queuemembercount

  recommendeddocument

  revertproducthierarchy

  rollupjob

  savedorginsightsconfiguration

  sharedobjectsforread

  sharepointdata

  subscriptionstatisticsoffline

  subscriptionstatisticsoutlook

  subscriptionsyncentryoutlook

  syncattributemappingprofile

  systemapplicationmetadata

  userapplicationmetadata

Net Change Filters

Replication Solutions have Net Change filters enabled by default to replicate only new and updated records. The source Connector automatically chooses the datetime field to be used for the Last Modified Date for each table. When the Replication Solution runs it selects new or updated source records based on that datetime field, and only replicates those records. In some cases, the datetime field does not contain milliseconds. To ensure that all records are replicated, the Connector subtracts one second from the most recent Last Modified Date, which could cause a small number of records to be reprocessed.

For example, if the last record written had a Last Modified Date of 09/04/2018 10:10:20, the next Replication starts with data that has a Last Modified Date of 09/04/2018 10:10:19. Any records that were already processed between 09/04/2018 10:10:19 and 09/04/2018 10:10:20 are processed again.

Deleted Records In Microsoft Common Data Service (CDS)

Replicated Entities

Custom Objects

During replication, TIBCO Scribe® Online automatically creates any corresponding tables for custom entities in the target datastore. Select the entities you want to replicate from the Select Source Entities option on the Replication Solution page.

If your Microsoft Common Data Service (CDS) data contains custom entities:

Note: When you create a custom entity in Microsoft PowerApps, a unique default prefix is added to the entity name. However, you can change the name of the default prefix within Microsoft PowerApps. If you are using custom entities or fields, be sure you know the custom prefix name and use that name to search for your custom entities. For more information, see your Microsoft PowerApps Administrator.

Microsoft Common Data Service (CDS) Connection As IS/MS Source

Relationships

Note: For this Connector, only one level of relationship, Parent/Child, is supported for the entity selected in the Query or Fetch Block. For records with multiple levels of relationships, such as Parent, Child, Grandchild, only the selected Parent entity and related Child entities can be retrieved. Additional levels beneath Child entities, such as Grandchildren, cannot be accessed for this Connector using the Include tab. See Block Properties Include Tab — Relational Data.

Hierarchical data is not supported. See Hierarchical Data for additional information.

Query And Native Query

When using a Native Query Block with FetchXML or a standard Query Block to return related entities, only the first 1000 child entities can be returned for a single parent record. If more than 1000 child entities are returned, an error similar to the following is generated:

Unable to get the next page of data. Dynamics 365 has not advanced the page cookie for Entity: <entity name>, PagingCookie: paging cookie".

To work around this issue, modify your Query to exclude child entities, then add a Fetch Block for each result to retrieve all children for the Entity selected in the Query.

When using a Native Query Block with FetchXML or a standard Query Block to return multiple records, if those records include milliseconds in the modifiedon field, the Connector may return the same page of data over and over again without advancing to the next set and generate an error similar to the following: 

Unable to get the next page of data. Dynamics 365 has not advanced the page cookie for Entity: <entity name>, PagingCookie: <paging cookie>

To workaround this issue contact Microsoft for assistance in determining the cause.

Using an Order by function for a child entity in Fetch XML in a Native Query Block limits the number of records returned to 12000. If the query exceeds 12000 records, an error similar to the following is generated:

Paging cookie is required when trying to retrieve a set of records on any high pages.

To work around this issue, modify your Query to exclude child entities, then add a Fetch Block for each result to retrieve all children for the Entity selected in the Query.

When testing a Native Query in a Map, if the source datastore does not return any data, TIBCO Scribe® Online cannot build the schema for the underlying metadata and the Map cannot be saved. To allow TIBCO Scribe® Online to build the schema, do the following:

  1. Create a single temporary record in the source datastore that matches the Native Query.
  2. Test the Native Query and ensure that it is successful.
  3. Save the Map.
  4. Remove the temporary record from the source datastore.

FetchXML

The Microsoft Common Data Service (CDS) Connector supports using the FetchXML query language in a Native Query Block to create customized queries for Microsoft Common Data Service (CDS) entity records and associated child records. The query can be as simple or complex as you need it to be; however, it should return a single result set.

After entering the FetchXML query, you must select Test to validate the query. Invalid XML is not accepted by the Connector.

You should understand the FetchXML query language before attempting to use the Native Query Block to access your Microsoft Common Data Service (CDS) data. For assistance with FetchXML, see Build queries with FetchXML on the Microsoft developer site.

Using FetchXML in a Native Query Block includes support for sorting, grouping, filtering, and relationships. See the examples below for each option.

Simple Query

The example below returns all accounts and associated accountid and name values.

<fetch mapping='logical'>
<entity name='account'>
<attribute name='accountid'/>
<attribute name='name'/>
</entity>
</fetch>

Sorting

The example below sorts returned records by the value of the creditlimit field.

<fetch mapping="logical" version="1.0">
<entity name="Account">
<all-attributes />
<order attribute="creditlimit" />
</entity>
</fetch>

Grouping

The example below groups returned records by the value of the address1_city field.

<fetch mapping="logical" aggregate="true" version="1.0">
<entity name="account">
<attribute name="accountid" alias="COUNT_accountid" aggregate="countcolumn" />
<attribute name="address1_city" groupby="true" />
</entity>
</fetch>

Filtering

The example below returns records where the name field contains Scribe.

<fetch mapping="logical" version="1.0">
<entity name="account">
<attribute name="name" />
<attribute name="accountid" />
<filter>
<condition attribute="name" operator="like" value="%Scribe%" />
</filter>
</entity>

Alias

If you have more than one entity with the same name, you must alias the entity names to distinguish the entities from each other.

Relationships

When parent and child records are returned, they are flattened and treated as a single set of records with the parent information repeated for each child. In this case, you cannot use a For Each Child Block. When mapping parent and child records, all fields for both the parent and the child records display as a single result set in the source side of the operation Block Properties Fields tab. If the parent and child entities have the same name and/or the same field names, such as a parent Account with multiple child Accounts, TIBCO Scribe® differentiates them by adding an alias to the entity or field name in the format entityalias_fieldname.

The example below performs an inner join on the account and SystemUser entities and returns System Users where the ownerid of the account matches the systemUserID of the SystemUser.

<fetch mapping="logical" version="1.0">
<entity name="account">
<attribute name="name" />
<link-entity name="SystemUser" from="systemUserId" to="ownerid" alias="s" link-type="inner" />
</entity>
</fetch>

Query Block Filtering

When a datetime is configured on the Query Block on the Block Properties Net Change Tab to query for new and updated records, that configuration is treated as an additional filter. The Net Change datetime filter is applied as an AND after any other filters specified on the Block Properties Filter Tab. TIBCO Scribe® Online builds a query combining both the Net Change filter and the filters on the Filter tab. See Net Change And Filters for an example.

Some Connectors for TIBCO Scribe® Online only support one filter. For those Connectors you can use either Net Change or one filter on the Filter tab, not both.

Note: The Net Change date is ignored when previewing data on the Preview tab. Filters on the Block Properties Filters tab are used to filter the data on the Preview tab.

Microsoft Common Data Service (CDS) Connection As IS/MS Target

Batch Processing

For information about using Batch processing, see Batch Processing, Create Block, Update Block, or Delete Block.

Id And Idtype

For Microsoft Common Data Service (CDS) target Connections, if you map an id field, such as ownerid, you may also need to set the corresponding owneridtype to the value of the id base field.

To determine if you need to map the owneridtype, see the Description of the field displayed by selecting the Info icon for the Entity. See Block Properties Fields Tab.

Ownerid And Owneridtype

For Microsoft Common Data Service (CDS) target Connections, both the ownerid and owneridtype fields may need to be mapped:

Example

For example, in the image below, the Description indicates that if you map the ownerid field, you must also set a value for the ownertypeid field.

In the image below, the Description for the owneridtype field specifies the possible base entities and values: systemuser, team. To set this value, enter the name of the base field type in quotation marks into the Formula field.

Merge Block Functionality

The Merge Block and the virtual merge entity support the Microsoft merge functionality to merge account, contact, lead, or incident records.

SkipSeekUnder

Seek Under is a lookup process where the Microsoft Common Data Service (CDS) Connector uses the Matching Criteria on a target operation Block to locate a record. The Connector then returns the entire record from the data store and builds the updated record with any fields that were mapped on the Fields tab. This process provides a greater degree of accuracy in locating the correct record and provides unmapped data for use in subsequent operations. For example, if telephone number is not one of the mapped fields on the Fields tab, it is still available for use in an operation later in the Map. However, because the target datastore is accessed twice, once to return the record and again to perform the target operation, there is an impact to performance.

The skipseekunder virtual field on the Fields tab of the target operation Block is used to enable or disable the Seek Under process. Use skipseekunder when you do not need access to additional fields for the selected entity and you are sure that the fields you provided in Matching Criteria are sufficient to locate the correct record. When using skipseekunder, note the following:

Notes On Standard Entities

If you had a Microsoft Dynamics 365 / CRM Organization when you created your Microsoft Common Data Service (CDS) database, a subset of these entities may be represented in the Microsoft Common Data Service (CDS) database.

ActivityParty

AccountLeads

CalendarRule

CloneProduct

This virtual entity isused to support complex product catalog integration requirements for products.

PublishProductHierarchy, RevertProductHierarchy

These virtual entities are used to support complex product catalog integration requirements for products, product families, and product bundles.

Salesorder

Systemuser

Miscellaneous

Errors

You may receive one of the following errors when using the Microsoft Common Data Service (CDS) Connector.

Fatal Error

If you receive a fatal error that contains either of the following messages, make sure the credentials for your target Microsoft URL and Organization are correct. In addition, make sure the system time for the computer where the TIBCO Scribe® Online Agent resides is correct.

Metadata contains a reference that cannot be resolved: 'https://example.com'

Verify that the Connection URL is correct and that you can connect to the same URL from the system running the Scribe Online Agent.

Security Time Stamp Error

This error is the result of an issue in the way Microsoft PowerApps tracks time. The Microsoft PowerApps Server, which is set to UTC/GMT, cannot generate a timestamp if the minutes value on the computer where the TIBCO Scribe® Online On-Premise Agent is installed differs by more than 5 minutes from the Microsoft PowerApps Server time.

The Microsoft PowerApps Server uses your local time zone. Therefore, this problem only occurs if the minutes, not hours, are off by more than 5 minutes known as the skew time. To ensure that you do not get this message, make sure the minutes on the computer where the TIBCO Scribe® Online On-Premise Agent is installed reflect the same minutes value as UTC/GMT time.

Unable To Insert Error

If you create a Map using a Microsoft Common Data Service (CDS) database as a target and see the error shown below, it may be caused by populating a field in the target that requires an associated field to also be populated. If the second field upon which the first one depends is either not linked or is being populated by a null or invalid value, the error can occur.

Unable to insert data into the XYZ entity. An unexpected error occurred.

For example, populating the businesscustomerid field and not populating the businesscustomeridtype in the Contract entity results in this error. To resolve this issue, map both fields with valid non-null values.

TIBCO Scribe® Online API Considerations

To create Connections with the TIBCO Scribe® Online API, the Microsoft Common Data Service (CDS) Connector requires the following information:

Connector Name

Microsoft Common Data Service

Connector ID

CBEB774F-B5E6-4ED7-8ED8-58D19466A1F5

TIBCO Scribe® Online Connection Properties

In addition, this Connector uses the Connection properties shown in the following tables.

Note: Connection property names are case-sensitive.

Note: TIBCO Scribe® Online does not support creating Connections via the TIBCO Scribe® Online API when using Azure AD for authentication.

Online Deployment Type

If you are creating a Connection to Microsoft Common Data Service (CDS) using an Online Deployment Type the following fields are used in the model sent to the TIBCO Scribe® Online API.

Name Data Type Required Secured Usage

DeploymentType

String

Yes

No

Enter "Online".

DiscoveryServiceURL

String

Yes

No

 

UserId

String

Yes

No

 

Password

String

Yes

Yes

 

HomeRealmUri

String

No

No

Not supported for this Connector, but the field is required on the Connection model.

Organization

String

Yes

No

 

Domain

String

No

No

Not supported for this Connector, but the field is required on the Connection model.

DisplayPickListNames

Boolean

No

No

Defaults to false if empty

RunAsUserName

String

No

No

Username of the user making API requests

BulkBatchSize

String

No

No

Defaults to 500 if empty

BulkConcurrentCalls

String

No

No

Defaults to 2 if empty

ClientSecret Deployment Type

If you are creating a Connection to Microsoft Common Data Service (CDS) using a ClientSecret Deployment Type, the following fields are used in the model sent to the TIBCO Scribe® Online API.

Name Data Type Required Secured Usage

DeploymentType

String

Yes

No

Enter "ClientSecret"

Url

String

Yes

No

The value of this field is case sensitive when testing the Connection.

ClientId

String

Yes

Yes

 

ClientSecret

String

Yes

Yes

 

HomeRealmUri

String

No

No

Not supported for this Connector, but the field is required on the Connection model.

Domain

String

No

No

Not supported for this Connector, but the field is required on the Connection model.

DisplayPickListNames

Boolean

No

No

Defaults to false if empty

BulkBatchSize

String

No

No

Defaults to 500 if empty

BulkConcurrentCalls

String

No

No

Defaults to 2 if empty

License Agreement

The TIBCO Scribe® Online End User License Agreement for the Microsoft Common Data Service (CDS) Connector describes TIBCO and your legal obligations and requirements. TIBCO suggests that you read the End User License Agreement.

More Information

For additional information on this Connector, refer to the Knowledge Base and Discussions in the TIBCO Community.