TIBCO Scribe® Online Connector For Twitter

Use the TIBCO Scribe® Online Connector for Twitter to extract data from Twitter and use it in a different datastore or external system by querying and filtering tweets for specific hashtags, keywords, or handles. The Twitter Connector can only be used to retrieve Source data, but cannot integrate data back into the Twitter database.

Possible use cases for data extracted using the Twitter Connector include: 

• Updating your CRM system with lead data.
• Replicating Twitter Status data for reporting and analysis.
• Responding to customer tweets more quickly and effectively by integrating customer tweet information with a CRM system. Depending on the CRM system being used, you can generate emails, press releases, product updates, or changes in pricing as a response.
• Structuring a marketing campaign based on leads that you have filtered from tweets and integrated into your marketing campaign software.
• Creating support cases for customers that ask for help.
• Responding to negative feedback on Twitter discovered using the TIBCO Cloud™ Integration - Connect Sentiment Scoring Values feature.
• Charting positive feedback on Twitter discovered using the TIBCO Cloud™ Integration - Connect Sentiment Scoring Values feature. See Sentiment Scoring Values for additional details.

Connector Specifications

Supports version 1.1 of the Twitter REST API.

	Supported
Agent Types
Connect on-premise	X
Connect cloud	X
Data Replication Apps
Source	X
Target
On Schedule Apps
Source
Target
On Event Apps
Source
Target
Flows
Integration	X
Request-Reply
Message

Supported Entities

The Twitter Connector supports the following entities. Select a linked entity name for additional information when using that entity in TIBCO Cloud™ Integration - Connect.

Entity	Query	Create	Update	Delete	Upsert
Place	X
Status	X
User	X

Setup Considerations

To use the Twitter Connector you need a Twitter User account.

Selecting An Agent Type For Twitter

Refer to TIBCO Cloud™ Integration - Connect Agents for information on available agent types and how to select the best agent for your app.

API Usage Limits

Twitter limits the number of API calls in a sliding 15 minute window for the user communicating with its API. Users that exceed those limits may be blacklisted and will not be able to access the Twitter API. TIBCO Cloud™ Integration - Connect monitors API usage to prevent you from exceeding the API limit. TIBCO Cloud™ Integration - Connect tracks the number of remaining API calls in the current 15 minute window and stops the query before the limit is exceeded.

If you are running an app and it is close to exceeding the API usage limits, the app stops running and TIBCO Cloud™ Integration - Connect generates the following error message:

API limit reached for method <name of entity>
Where <name of entity> is the name of the metadata entity being queried when the maximum was reached.

Refer to the Twitter documentation for additional information on per user usage limits. See API Rate Limits.

Managing API Limits

Twitter API rate limits can cause TIBCO Cloud™ Integration - Connect apps to fail depending on the number of records requested. Below are some options to help you work within these limits.

• Use filters to limit the number of records returned and API calls used.
• If you are using a Data replication app, enter filters on the TIBCO Cloud™ Integration - Connect Connection for Twitter.
• Create a single flow per app and schedule the apps 20 minutes apart. This provides a window of time during which your API count is reset.
• Due to API usage limits, you cannot retrieve an exhaustive number of records. Therefore, if you need to retrieve all records on a particular topic and there are more records than are allowed in a single 15 minute window, you may need to break the job up into several apps. Use the created_at field in the records that are returned to determine where to begin querying for the next set of records. Records are retrieved from the most recent to the oldest. For example, assume that you need to retrieve 30,000 tweets. To retrieve them all you would do the following:
  
  1. Run your first query and the tweets returned are stored in the target datastore.
  2. Review the data in the target and locate the record with the oldest date in the created_at field.
  3. Modify the query to filter for records that have dates older than the oldest date in your target datastore.
  4. Run the query again.
  5. If you think that you have not retrieved all of the records, repeat the process until you have all of your records.
  6. Then modify the query to only retrieve new records by enabling the Process only records created or updated since last run option and selecting the created_at field.

Connecting To Twitter

1. Select Connections from the menu.
2. From the Connections page select Create to open the Create a connection dialog.
3. Select the Connector from the list to open the Connection dialog, and then enter the following information for this Connection:
   • Name — This can be any meaningful name, up to 25 characters.
   • Alias — An alias for this Connection name. The alias is generated from the Connection name, and can be up to 25 characters. The Connection alias can include letters, numbers, and underscores. Spaces and special characters are not accepted. You can change the alias. For more information, see Connection Alias.
4. Select Authenticate to open the Twitter authorization page:
   • If you are not already logged into Twitter, the Twitter login page displays. Enter your email address and password, and select Log in.
   • If this is the first time you are creating a Twitter connection, the Twitter Authorization page displays. Click Authorize to allow Twitter to use the data specified.

     A page with the following message displays:

     

   • Click Close, return to the TIBCO Cloud™ Integration - Connect page, and then select Test to ensure that the agent can connect to Twitter. Be sure to test the connection against all agents that use this connection. See Testing Connections.
   Tip: Do not test the Connection until the message at the top of the Connection dialog reads Authentication complete. Note that it may take a few minutes for this message to display.
5. If you are using Replication Services, enter filter information in the RS Filter field. This field is required.
6. In the Sentiment Scoring Values table select Add to enter a specific word or phrase and either a positive or negative number. Maximum of 50 entries. This field is optional. See Sentiment Scoring Values for additional information.
7. Select OK/Save to save the Connection.

Sentiment Scoring Values

Use the Sentiment Scoring Values table in the TIBCO Cloud™ Integration - Connect Manage Connections dialog for Twitter to search tweets for keywords or phrases and give each occurrence a positive or negative score. The cumulative sentiment score indicates whether the tweet is negative or positive, which helps you determine how to act on that tweet. The score given to each keyword is user specified and should be based on your own calculation for what constitutes a negative or positive total score.

TIBCO Cloud™ Integration - Connect uses the data in the sentiment filter as follows: 

• Searches tweets for the exact word or phrase entered in the connection dialog
• Ignores punctuation
• Does not require quotation marks when entering a phrase
• Search is case insensitive
• Hashtag (#) and at (@) symbols are used as part of the phrase if included
• Does not allow wild cards
• Does not return partial matches or words that contain the keyword. For example if you enter Scribe, describe and subscribe are not included in the results.
• Maximum of 50 keywords of up to 140 characters each

Scores are cumulative and are stored in the target datastore in the sentiment field. For example, if you have entered good with a score of 100 and bad with a score of -25, a tweet of this phrase, "He is neither good nor bad.", generates a sentiment score of 75 or 100 + (-25)= 75.

Sentiment Scoring Values can be used with both Data replication and On schedule apps. If the Sentiment Scoring Values table is populated, a sentiment score is always generated for tweets that match the criteria. Create additional Twitter Connections to store different sets of sentiments or to remove all sentiments for instances when you do not want any sentiment scores.

Metadata Notes

Consider the following for Twitter data fields and entity types.

Naming

Connection metadata must have unique entity, relationship, and field names. If your Connection metadata has duplicate names, review the source system to determine if the duplicates can be renamed.

Entity Types

Twitter entities, as used in TIBCO Cloud™ Integration - Connect, are classified as either main entities or sublists.

• Main entities include: Place, User, and Status. Some of these entities may have relationships with the others, but are still considered main entities.
• Sublists are entities exclusively related to their parent entities. These include:
  • contributor
  • hashtag
  • media
  • placeattribute
  • symbol
  • url
  • usermention

Twitter Connector As A Data Replication App Source

Consider the following when using the Twitter Connector as Data replication app source.

• Status is the only entity that can be replicated. See Status for additional information.
• An RS Filter is required on the Manage Connections dialog to retrieve any data when running a Data replication app.
• Multiple filters can be entered in the RS Filter field using the same syntax used in the Twitter Search bar. Refer to Using Advanced Search in the Twitter help for details on syntax.

Twitter Connector As An App Source

Consider the following when using the Twitter Connector as an App source.

Filter, Match Criteria, Or Lookup

• To prevent Twitter API limitations from being exceeded, at least one text filter is required.
• Not equals is a valid filter operator only for the Status entity sentiment field. Not equals can be used within a text search, such as text = "scribe -software".
• For Boolean filter criteria, only True and False values are supported. These values should not be enclosed in quotes.
• Two or more filters using the same field are not supported, except in the case of the Status entity text field or sentiment filter. For all other entities and filters, only the first filter is used and the rest are ignored. For example, the following is not supported: granularity = "poi" AND granularity = "admin"
• OR is only supported for the Status entity text and sentiment fields. If OR is used in any other instance it is treated as AND.
• Specifying the id_str field as a search field causes all other search criteria to be ignored.
• Perspectival fields, such as retweeted may not be returned accurately unless the id_str field is used as the search field. Perspectival fields are fields that pertain to the perspective of the authenticating user. For example, if I search for my own tweets, I can see if I retweeted someone else's tweet. If I search for Ford Motor Company's tweets, I cannot see if a particular tweet has been retweeted.
• The Twitter search mechanism is similar to a Google search in that it includes items that are similar to the requested search parameters. Search results may contain unexpected records.
• If new tweets are received during a query, you may see duplicate tweets in your results because the query is rerun after each page of results is received.

Notes On Standard Entities

Place

• Does not support the Process only records created or updated since last run option because this entity does not have an associated date field.
• The data type for the following fields does not match the data type in Twitter:

Field	Twitter Data Type	TIBCO Cloud™ Integration - Connect Data Type
attributes	Object	String
bounding_box	Object	Well Known Text Formatted String
coordinates	Array	Well Known Text Formatted String

• Filter, Match Criteria, Or Lookup
  • When configured with a value, the contained_within and attribute filter fields are ignored and do not affect the results returned.
  • The following filter fields in the Place entity are search only fields and do not display on the Preview screen:
    • accuracy — Radius in meters in which to search. For feet, append ft to the number. Example: 5ft
    • attribute_query — Searches for a Place attribute, such as street_address. Syntax for the formula in the Value field is: "attribute=value"

      Example: "street_address=795 Folsom St"

    • contained_within — Limits results to only places within the specified place_id. Example: For San Francisco specify a place_id of 5a110d312052166f
    • granularity — Granularity of place types to return. Must be one of: poi, neighborhood, city, admin, or country. Default is neighborhood. The list is hierarchical and returns records for the type selected and those below it in the list. For example, setting granularity to city finds Places which have a type of city, admin, or country. In the results, granularity is displayed in the place_type field.
    • ip — User's IP address. Example: 74.125.19.104
    • lat — Latitude to search around. Valid range is -90.0 to +90.0 range. Requires a corresponding long parameter. Example: 37.7821120598956
    • long — Longitude to search around. Valid range is -180.0 to +180.0. Requires a corresponding lat parameter and geo_enabled field on the authenticated user must set to True.
    • query — Freeform text to match when executing a query against Place. Example: Germany

Placeattribute

• Placeattribute is a child of the Place entity.
• All Placeattribute data is concatenated and stored as a string in the attributes field of the Place entity.

Status

• The Twitter API entity labeled Status is labeled tweet in the Twitter user interface.
• The text field in the TIBCO Query Preview tab for the Status entity contains the text of the tweet. This differs from the text field in the TIBCO Filter screen which is used to search across multiple fields as described below in the Filter, Match Criteria, Or Lookup section.
• The coordinates field is populated using the information in the geocode field. If there is no data in geocode, coordinates is NULL.
• The sentiment field is created by TIBCO Cloud™ Integration - Connect when the tweet is stored in the target datastore. This field is used to contain the sentiment score for this tweet. If you have not configured the Sentiment Scoring Values table in the TIBCO Cloud™ Integration - Connect Twitter Connection dialog, this field is blank. See Sentiment Scoring Values for additional details.
• The data type for the following fields does not match the data type in Twitter:

Field	Twitter Data Type	TIBCO Cloud™ Integration - Connect Data Type
coordinates	Coordinates	String
created_at	String	DateTime
current_user_retweet	Object	String
withheld_ in countries	Array Of String	String

• Filter, Match Criteria, Or Lookup
  • Only indexed tweets are returned for Query or Lookup operations, therefore, some tweets expected in the filter results may not be included. For example, tweets that contain a URL are not always indexed and, therefore, are not returned.
  • There is no mechanism for automatically filtering out retweets, which can cause duplicate Statuses to be returned by a Query. Use a filter on the text field with operator is like and value -RT on the query to return only original tweets.
  • The Search API is not a complete index of all tweets, but instead an index of recent tweets. Use the created_at field to retrieve tweets that are more than 7 days old.
  • The text field used in filters corresponds to the text field used in the Search bar on the Twitter website. This is used to search across many fields. Text search rules used on the Twitter website also apply to the filter text field. Refer to Using Advanced Search in the Twitter help for details on search rules.
  • Some characters, such as asterisk (*) and underscore (_), are ignored by the Twitter API when searching in a text field. These are the same characters that are ignored when using the Search bar on the Twitter website.
  • Equals is the only valid operator for the id_str field.
  • The sentiment filter can be used more than once in a query. When used more than once, the filters must be consecutive and the logical operator between all instances of this filter must be the same, either OR or AND but not both. For example, you might filter by sentiment is greater than 500 OR sentiment is less than -500 to address customers that are either very satisfied or very dissatisfied.
    Note: If you mix the AND and OR logical operators, or insert other filters between the sentiment filters, TIBCO Cloud™ Integration - Connect displays an error message indicating that the query is not valid.
  • The following filter fields in the Status entity are search only fields and do not display on the Preview tab:
    • geocode — Searches for tweets by users within a specified latitude/longitude and radius. Example: 37.781157,-122.398720,1mi
    • locale — Specify the language of the query being sent. Only accepts ja.

Contributor

• Contributor is a child of the Status entity.
• The contributor sub-entity is part of a beta version of Twitter and is not available to all Twitter users. Due to its beta status this entity could not be tested with TIBCO Cloud™ Integration - Connect. Refer to Contributor information on the Twitter website for a definition of its functionality.

Media

• Media is a child of the Status entity.
• The media sub-entity does not allow searches for specific attachments to a tweet. The URL of the attachment is included when the media sub-entity is part of a join.

User

• Does not support the Process only records created or updated since last run option.
• Twitter API currently only returns the first 1000 Users when Users are queried.
• The data type for the following fields does not match the data type in Twitter:

Field	Twitter Data Type	TIBCO Cloud™ Integration - Connect Data Type
created_at	String	DateTime
follow_request_sent	Type	Boolean

• Filter, Match Criteria, Or Lookup
  • When filtering by screen_name, do not include the "@" symbol as part of the value because it is not part of the username.

TIBCO Cloud™ Integration - Connect API Considerations

More Information

For additional information on this Connector, refer to the Knowledge Base and Discussions in the TIBCO Community.