As a transitional step, this site will temporarily be made Read-Only from July 8th until the new community launch. During this time, you can still search and read articles and discussions.

While the community is read-only, if you have questions or issues requiring TIBCO review/response, please access the new TIBCO Community and select "Ask A Question."

You will need to register or log in or register to engage in the new community.

Guidelines on how to introduce the improved trust mechanism in Spotfire 10.3 (or higher) when upgrading

Last updated:
10:36am May 14, 2019

TIBCO Spotfire version 10.3 introduces significantly improved  governance and security for data functions. This is the same trust mechanism previously used for JavaScript and IronPython scripts as well as data connection custom queries. Spotfire now also uses the SHA-512 algorithm (instead of as previously, SHA-1) to generate the checksum for ensuring that scripts and data functions are identical to when they were trusted. Finally there are also changes to how TERR inline expression are executed. This page assumes that you are already familiar with the new trust mechanism in Spotfire 10.3  - if not, please make sure to read the Community page Script and Data function trust in Spotfire 10.3 and later before proceeding.

When upgrading any existing, pre-version-10.3 environment, special care should be taken to ensure minimum impact on the end users and environment, until all trust operations have been completed. This page provides guidance on aspects to consider when determining the right way to handle this for a particular environment. 

 

Planning the Upgrade

Immediately after upgrading an earlier version of Spotfire to Spotfire 10.3 or later, any data functions will be untrusted, meaning that they may not execute as intended. Should any end-users access analyses with data functions at this time they are likely to see error messages indicating that data functions cannot be executed. Therefore it is important to trust the data functions before users are given access to the system, or temporarily disable the data function trust check.

How to best go about trusting the data functions will depend on your specific circumstances - read more about this below.

 

Upgrade Scenarios

We will now look at a few different scenarios, and find out how we can use the available options to ensure a good end-user experience. For the first scenario, we assume no use of data functions or Inline TERR expressions, for the others we do assume that this is used. You can use these scenarios as a starting point when planning your upgrade, but make sure to consider aspects that are specific for your environment as well.

Scenario 1: An environment with no use of data functions or Inline TERR expressions.

For this scenario, there is no need for any urgent action after the upgrade. Existing and previously trusted JavaScript and IronPython scripts as well as data connection custom queries will continue to work after the upgrade, as their existing (SHA-1) trust stamps are still considered valid. It is still recommended to update that trust with the new SHA-512 trust stamps at some point, but that could e.g. be scheduled for a time with little system use, such as during a weekend.

To update the trust stamps, you can run the server command find-analysis-scripts 

>config find-analysis-scripts

In addition to its main purpose of locating scripts, data functions and custom queries in the Library so they can be trusted, it also updates the trust stamp of any already trusted items found, which have an old SHA-1 trust stamp, with a new SHA-512 trust stamp.



You can read more about the TIBCO Spotfire Servers Command Line Interface (CLI) used to execute this and the find-analysis-scripts command in the TIBCO Spotfire Server Installation and Administration Help, or by starting the command with the help parameter:

>config help find-analysis-scripts



Scenario 2: A single Production environment, where changes are done directly in the environment  

For this scenario, the assumption is that there is a single Production environment, and all changes are done directly in this environment. In such an environment, it's particularly important to consider how to avoid disruptions - from analyses not working as their data functions or inline TERR expressions have not yet been trusted/updated, and from increased load/resource usage on the TIBCO Spotfire Server(s) during the process. Consider doing the following in this scenario:  

1. Decide on how you want to deal with scripts and data functions during the upgrade. The following are three possible approaches:

  • Approach A – Update trust for existing scripts and initially leave data functions untrusted, then ask users to review existing data functions and decide whether to trust them or not.
  • Approach B – Automatically trust all existing data functions. If you feel that all existing data functions should be trusted.
  • Approach C – Automatically trust data functions in selected folders

Before making your decision, you can read more about these in the section Deciding how to deal with scripts and data functions during the upgrade below.

2. Decide on how you want to deal with TERR inline expressions, which cannot be trusted and will therefore always be executed in TERRs restricted execution mode. This means that if an inline TERR expression uses functionality not available in TERRs restricted execution mode, the inline TERR Expression must be rewritten to a TERR Expression function.

3. Upgrade the environment to Spotfire version 10.3 (or higher)

4. After the upgrade, ensure that all users that will be expected to manually set data functions as trusted for others when saved to the Library are members of Script Author group. You can manage group memberships either from TIBCO Spotfire Analyst > Tools menu > Administration Manager > Groups and Licenses, or (starting from version 10.3) from the TIBCO Spotfire Server web administration UI > Users & Groups.

5. If you need to avoid any disruptions on the server, from data functions or Inline TERR expressions no longer working, consider turning off the data function trust temporarily. This is done as follows:

  • For TIBCO Spotfire Analyst/Business Author/Consumer: Set the IgnoreTrustCheckpreference to true 
  • For TERR Service (if used): Set the disable.spotfire.trust.checks configuration property on the TERR service to true.

For more instructions on how to do this, refer to the section How to disable data function trust below.

6. Run the server command find-analysis-scripts to obtain a list of scripts and data functions in the Spotfire library, and if you choose too, automatically trust existing data functions directly, and also to update the trust stamp of already trusted scripts. Below we'll go through a few different ways you could to do this - you can combine the various options to best suit your needs.

Run on the entire Library, as quickly as possible

>config find-analysis-scripts

This will start the find-analysis-scripts command on the entire library (which may take significant time, as well as CPU, memory and disk resources on the TIBCO Spotfire Server - please consider the below alternatives before deciding how to start). The trust stamp of all existing and trusted IronPython, JavaScripts and data connection custom queries will be updated. A .csv file called report_yyyy-mm-dd.csv will be generated. The csv file contains an entry for each script in each file of the library (including scripts and data functions within analysis files, custom queries in saved data connections and separate data functions), and the file will be generated and stored in the folder find-analysis-scripts. 

 

Run single threaded, to reduce resource use at the expense of longer time to complete

>config find-analysis-scripts -i

The -i switch tells the tool to run single-threaded, meaning that the command will use less CPU and memory resources on the machine running the command but it will take longer time to complete. Without the switch the command will use multiple threads - as many as there are CPU cores available - working on multiple analyses at once. 

 

Run on a selected folder

>config find-analysis-scripts -p /library-path

If you do not wish to run the tool on the entire library at once, for example running it on a specific folder containing analyses of particular importance. The command will also include all sub-folders of the specified folder.

Example:   

>config find-analysis-scripts -p "/myfolder/data function trust"

Running the command on a specified folder with just a few DXPs in will take less time, and thus allows you to try and practice using the CLI without taking extensive time or CPU resources. It is also an option if you want to use a folder per folder approach rather than running on the entire library at once.

 

Run on small files only

>config find-analysis-scripts --library-search-expression "content_size:<2000MB"

The -library-search-expression allows you to specify an expression to run the command on the set of files that match the expression. This example specifies that the find-analysis-script command will only run on files that are less than 2000MB in size, which is handy if you have files that are very large and you want to avoid loading them with the command. The syntax and available options in library search expressions are described in the Spotfire Analyst help.





Run on files accessed after/before a certain date

>config find-analysis-scripts --library-search-expression "accessed:>""2019-01-13T18:27:59CEST"""
>config find-analysis-scripts --library-search-expression "accessed:>""1 weeks ago"""

The above two examples of using the -library-search-expression shows two different options for selecting files that were last accessed before or after a certain date or time.

 

Automatically trust any existing data functions

>config find-analysis-scripts -d true

If you decided in step 3 above that you want to automatically trust any existing data functions as part of running the command, run it using -d true.    

 

You can read more about the TIBCO Spotfire Servers Command Line Interface (CLI) used to execute this and the find-analysis-scripts command in the TIBCO Spotfire Server Installation and Administration Help, or by starting the command with the help parameter:

>config help find-analysis-scripts

 

7. Complete the work to trust data functions 

  • If you chose the option to automatically trust all existing data functions in the previous steps, you are done with this step. 
  • If you chose to not automatically trust all existing data functions in the previous step, but plan to manually analyze and trust data functions, you should now start by reviewing the find-analysis-scripts report. Using that report, you can identify what analyses should be inspected and their data functions trusted, either manually, or in bulk as described below.

Trusting data functions manually 

Setting the trust could now be handled manually, by users (as stated in step 4, members of the Script Author group) opening each of the identified analyses in TIBCO Spotfire Analyst, inspecting them and trusting the data functions, either when prompted or using the File menu > Manage trust and scripts. Make sure to save each analysis when done.

Trusting data functions in bulk, using the trust script from find-analysis-scripts

When the find-analysis-scripts command was executed in the previous step, one of its outputs should have been a script file named trust_[date].script (e.g.  trust_2019-05-07.script). This file is by default located in the [TIBCO Spotfire Server installation directory]\tomcat\spotfire-bin\find-analysis-scripts directory. The script file contains the trust commands to trust each individual untrusted data function (but also untrusted scripts, and custom queries) that was found when running the find-analysis-scripts command. If you now choose to apply trust in bulk, you can execute the entire script using a server command like this:

config run trust_2019-05-07.script

For more information about the trust command, please refer to the trust page in the TIBCO Spotfire Server and Environment - Installation and Administration manual.  

8. Complete the work on inline TERR expressions. As mentioned in step 2, inline TERR expressions cannot be trusted. See this Community article for a description of how to convert inline TERR expressions (if needed) to TERR expression functions.

9. Turn on the data function trust again (if you turned it off in step 5), in the following way 

  • For TIBCO Spotfire Analyst/Business Author/Consumer: Set the IgnoreTrustCheckpreference to false 
  • For TERR Service (if used): Set the disable.spotfire.trust.checks configuration property on the TERR service to false.

For more instructions on how to do this, refer to the section How to disable data function trust below.

 

 

Scenario 3: A Production and a Development environment, where changes are done in Development, then moved to Production 

For this scenario, the assumption is that there is a Development environment, where all changes are done to analyses, data functions and other objects in the Library, before they are moved to the Production environment. Consider doing the following in this scenario:

1. Decide on how you want to deal with scripts and data functions during the upgrade. The following are three possible approaches:

  • Approach A – Update trust for existing scripts and initially leave data functions untrusted, then ask users to review existing data functions and decide whether to trust them or not.
  • Approach B – Automatically trust all existing data functions. If you feel that all existing data functions should be trusted.
  • Approach C – Automatically trust data functions in selected folders

Before making your decision, you can read more about these in the section Deciding how to deal with scripts and data functions during the upgrade  below.

2. Decide on how you want to deal with TERR inline expressions, which cannot be trusted and will therefore always be executed in TERRs restricted execution mode. This means that if an inline TERR expression uses functionality not available in TERRs restricted execution mode, the inline TERR Expression must be rewritten to a TERR Expression function.

3. Upgrade the Development environment to Spotfire version 10.3 (or higher)

4. After the upgrade, ensure that all users that will be expected to manually set data functions as trusted for others when saved to the Library are members of Script Author group. You can manage group memberships either from TIBCO Spotfire Analyst > Tools menu > Administration Manager > Groups and Licenses, or (starting from version 10.3) from the TIBCO Spotfire Server web administration UI > Users & Groups.

5. If you need to avoid any disruptions on the Development server, from data functions or Inline TERR expressions no longer working, consider turning off the data function trust temporarily. This is done as follows:

  • For TIBCO Spotfire Analyst/Business Author/Consumer: Set the IgnoreTrustCheckpreference to true 
  • For TERR Service (if used): Set the disable.spotfire.trust.checks configuration property on the TERR service to true.

For more instructions on how to do this, refer to the section How to disable data function trust below.

6. Run the server command find-analysis-scripts to obtain a list of scripts and data functions in the Spotfire library, and if you choose too, automatically trust existing data functions directly, and also to update the trust stamp of already trusted scripts. Below we'll go through a few different ways you could to do this - you can combine the various options to best suit your needs.

Run on the entire Library, as quickly as possible

>config find-analysis-scripts

This will start the find-analysis-scripts command on the entire library (which may take significant time, as well as CPU, memory and disk resources on the TIBCO Spotfire Server - please consider the below alternatives before deciding how to start). The trust stamp of all existing and trusted IronPython, JavaScripts and data connection custom queries will be updated. A .csv file called report_yyyy-mm-dd.csv will be generated. The csv file contains an entry for each script in each file of the library (including scripts and data functions within analysis files, custom queries in saved data connections and separate data functions), and the file will be generated and stored in the folder find-analysis-scripts. 

 

Run single threaded, to reduce resource use at the expense of longer time to complete

>config find-analysis-scripts -i

The -i switch tells the tool to run single-threaded, meaning that the command will use less CPU and memory resources on the machine running the command but it will take longer time to complete. Without the switch the command will use multiple threads - as many as there are CPU cores available - working on multiple analyses at once. 

 

Run on a selected folder

>config find-analysis-scripts -p /library-path

If you do not wish to run the tool on the entire library at once, for example running it on a specific folder containing analyses of particular importance. The command will also include all sub-folders of the specified folder.

Example:   

>config find-analysis-scripts -p "/myfolder/data function trust"

Running the command on a specified folder with just a few DXPs in will take less time, and thus allows you to try and practice using the CLI without taking extensive time or CPU resources. It is also an option if you want to use a folder per folder approach rather than running on the entire library at once.

 

Run on small files only

>config find-analysis-scripts --library-search-expression "content_size:<2000MB"

The -library-search-expression allows you to specify an expression to run the command on the set of files that match the expression. This example specifies that the find-analysis-script command will only run on files that are less than 2000MB in size, which is handy if you have files that are very large and you want to avoid loading them with the command. The syntax and available options in library search expressions are described in the Spotfire Analyst help.





Run on files accessed after/before a certain date

>config find-analysis-scripts --library-search-expression "accessed:>""2019-01-13T18:27:59CEST"""
>config find-analysis-scripts --library-search-expression "accessed:>""1 weeks ago"""

The above two examples of using the -library-search-expression shows two different options for selecting files that were last accessed before or after a certain date or time.

 

Automatically trust any existing data functions

>config find-analysis-scripts -d true

If you decided in step 3 above that you want to automatically trust any existing data functions as part of running the command, run it using -d true.    

 

You can read more about the TIBCO Spotfire Servers Command Line Interface (CLI) used to execute this and the find-analysis-scripts command in the TIBCO Spotfire Server Installation and Administration Help, or by starting the command with the help parameter:

>config help find-analysis-scripts

 

7. Complete the work to trust data functions 

  • If you chose the option to automatically trust all existing data functions in the previous steps, you are done with this step. 
  • If you chose to not automatically trust all existing data functions in the previous step, but plan to manually analyze and trust data functions, you should now start by reviewing the find-analysis-scripts report. Using that report, you can identify what analyses should be inspected and their data functions trusted, either manually, or in bulk as described below.

Trusting data functions manually 

Setting the trust could now be handled manually, by users (as stated in step 4, members of the Script Author group) opening each of the identified analyses in TIBCO Spotfire Analyst, inspecting them and trusting the data functions, either when prompted or using the File menu > Manage trust and scripts. Make sure to save each analysis when done.

Trusting data functions in bulk, using the trust script from find-analysis-scripts

When the find-analysis-scripts command was executed in the previous step, one of its outputs should have been a script file named trust_[date].script (e.g.  trust_2019-05-07.script). This file is by default located in the [TIBCO Spotfire Server installation directory]\tomcat\spotfire-bin\find-analysis-scripts directory. The script file contains the trust commands to trust each individual untrusted data function (but also untrusted scripts, and custom queries) that was found when running the find-analysis-scripts command. If you now choose to apply trust in bulk, you can execute the entire script using a server command like this:

config run trust_2019-05-07.script

For more information about the trust command, please refer to the trust page in the TIBCO Spotfire Server and Environment - Installation and Administration manual.  

 

8. Complete the work on inline TERR expressions. As mentioned in step 2, inline TERR expressions cannot be trusted See this Community article for a description of how to convert inline expressions to expression functions. 

9. Turn on the data function trust again (if you turned it off in step 5), in the following way 

  • For TIBCO Spotfire Analyst/Business Author/Consumer: Set the IgnoreTrustCheckpreference to false 
  • For TERR Service (if used): Set the disable.spotfire.trust.checks configuration property on the TERR service to false.

For more instructions on how to do this, refer to the section How to disable data function trust below.

10. From your Development environment, do Library Export(s) containing all the items - analyses and data functions - that have been trusted or updated in the previous steps.

11. Upgrade the Production environment to version 10.3 (or higher)

12. If users are going to start working on the Production environment before the next step (import of trusts) has time to complete, you may choose to turn off data function trust temporarily in the Production environment, in the same way as in step 5.

13. In the Production environment, do Library Import(s) of the files that were exported from Development in step 11.

14. Turn on the data function trust again (if you turned it off in step 12).

 

 

Deciding how to deal with scripts and data functions during the upgrade

As a Spotfire Server administrator, how to deal with trust for data functions will depend on the characteristics of your Spotfire deployment. Below, three approaches that may give you guidance are described. 

A: Leave all data functions untrusted and ask users to review and trust the data functions as needed.

B: Automatically trust all existing data functions.

C: Automatically trust data functions in selected folders.

You may choose one of these, or combine the approaches to fit your unique circumstances.

Note: In all cases, you must have access to the server config tool.

Approach A: Just update trust for existing scripts and leave data functions untrusted

One possible approach is to do nothing except updating the trust stamp to SHA-512 for IronPython scripts, JavaScripts and data connection custom queries that are already trusted. Note that this means that data functions and TERR expression functions will no longer work as expected, and users will be notified that these are not trusted when they try to use them. Users that are authorized (member of the group "script authors") will be able to review and trust the data functions in the library using the installed TIBCO Spotfire Analyst client.

Procedure

1. Using the Spotfire Servers config utility, run the CLI command find-analysis-scripts at the root of the library. The command may take a long time to finish, since it traverses the entire Spotfire library and identifies scripts in the DXP files. Consider running it on specific folders one at a time as shown in the examples above, or in another way breaking the job into smaller pieces.

>config find-analysis-scripts
Tool password:

Warning. This operation requires each analysis to be downloaded and analyzed.

This may take a very long time depending on the network and the size of the library.

There are currently 3750 analysis files with an average size of 2,342 MB in the library. Do you wish to continue (Y/N)?

As shown in the above text snippet, you must provide the password for the configuration tool and confirm that you want to initiate the operation.

During this process, the command will update the trust stamp of any previously trusted IronPython, JavaScript or custom query to the new version based on SHA-512. In addition, the command will create a report listing all scripts and data functions it finds in the library, and whether they are trusted or not. The report will by default be written to a directory called find-analysis-scripts.

2. You may want to provide the list of files with untrusted data functions to the users that created or last modified the file, to give them information about which files they must review and approve. Use Spotfire or Excel to open the generated file and to filter to untrusted data functions.

Approach B: Automatically trust all existing data functions

Use this approach if you decide that all data functions and expression functions that currently are in the library should be considered trusted. Data functions created in the future will need to be trusted by authorized users, just as for JavaScript and IronPython scripts.

Procedure

1. Run the CLI command find-analysis-scripts with the switch to automatically trust data functions at the root of the library. The command may take a long time to finish, since it traverses the entire Spotfire library and identifies scripts in the DXP files.

>config find-analysis-scripts -d true

Note that as previously mentioned, running the command on the entire library may take significant time, depending on the size of your library.

During this process, the command will update the trust stamp of any previously trusted IronPython, JavaScript or custom query to the new version based on SHA-512. The command will also create a trust stamp for all data functions and TERR expression functions it finds. In addition, the command will create a report listing all scripts and data functions it finds in the library. After the command finishes, all data functions and TERR expression functions are trusted and should work as before.

Approach C: Automatically trust data functions in selected folders

A compromise between the two previous approaches could be to decide that data functions in some specific library folder should be considered trusted, but for other DXP files you may want to ask the author or the user that last modified the file to review and decide for each individual script whether they should be trusted.

Procedure

1. Run the find-analysis-scripts command with the switches to trust all scripts, and also the switch to run on the folder that you select.

>config find-analysis-scripts -d true -p "/folder you select”

During this process, the command will update the trust stamp of any previously trusted IronPython, JavaScript or custom query in the library folder you selected to the new version based on SHA-512. The command will also create a trust stamp for all data functions and TERR expression functions it finds in the specified folder. In addition, the command will create a report listing all scripts and data functions it finds in the specified folder.

If you want to trust data functions in several library folders that are not within the same hierarchical folder structure, you must repeat this step for each branch. Once the command is finished, the data functions in the folders you selected should be trusted.

2. We now want to update the trust stamp of all already trusted IronPython scripts, JavaScripts and data connection custom queries to SHA-512.

>config find-analysis-scripts

As previously mentioned, this command may take a long time to execute depending on the size of your library. When the command finishes you will also have a report of all scripts and data functions in the library, which you can use to inform script authors or users about the data functions that will need to be trusted by them

 

 

How to disable data function trust 

In the installed client (TIBCO Spotfire Analyst), TIBCO Spotfire Enterprise Runtime for R (TERR) data functions, TERR predictive analytics, and TERR custom expressions can be executed using the embedded TERR engine. However, for use in the web clients (TIBCO Spotfire Business Author and Consumer), TIBCO Spotfire Statistics Services or TIBCO Enterprise Runtime for R - Server Edition (aka the TERR Service) must be installed.  

The instructions below describe how you can disable the data function trust check for the Spotfire clients - TIBCO Spotfire Analyst,  Business Author and Consumer - using the IgnoreTrustCheck preference and for the TERR Service (if used), using the  disable.spotfire.trust.checks configuration property.

If you disable one of the trust checks, you should also disable the other.    

For more information about the various ways data functions can be executed, please refer to the page TERR, TERR Service, and TIBCO Spotfire Statistics Services in the TIBCO® Enterprise Runtime for R Service Installation and Administration manual.  

 

How to disable data function trust for TIBCO Spotfire Analyst

Data function trust in the installed client - TIBCO Spotfire Analyst - is controlled by the preference IgnoreTrustCheck. To work with preferences, first start TIBCO Spotfire Analyst, go to Tools menu > Administration Manager, and finally open the Preferences tab. You will find the IgnoreTrustCheck preference under DataFunctions > DataFunctionPreferences.

The IgnoreTrustCheck preference introduced in TIBCO Spotfire version 10.3

 

Note: this preference is only available once you have deployed a TIBCO Spotfire 10.3 deployment (or higher) to the TIBCO Spotfire Server. 

By default the trust check is enabled, i.e. IgnoreTrustCheck is set to false (or blank). Setting it to true disables the trust check. Like any other preference, this preference is set on groups, so you can e.g. disable the check for all users (using the Everyone group), or only for certain groups. Changes to the preference value will take effect the next time the TIBCO Spotfire Analyst is started.  

Note: A data function written by a malevolent person could potentially perform unexpected or undesired actions. Therefore, Spotfire uses a trust mechanism, where users called Script Authors, verified by licenses and group belonging, are the only ones that can make a data function trusted for anyone in the organization. For normal operations, it’s recommended to have this trust check enabled - it is only recommended to disable it temporarily (and if needed) during the upgrade.  

 

How to disable data function trust for the TERR Service

The TIBCO Enterprise Runtime for R (TERR) service distribution must be installed on a node for the Spotfire Server. By default, when such an analysis is shared, the TERR service provides the functionality for the Spotfire Business Author and Consumer users

When you use a TERR service to run your data functions on a Spotfire Server, in addition to setting the IgnoreTrustCheckpreference, affecting the installed (TIBCO Spotfire Analyst) or web (TIBCO Spotfire Business Author or Consumer) clients, to TRUE as described above,  you must also change the disable.spotfire.trust.checks configuration property on the TERR service to TRUE. Setting either of these options without setting the other can cause unexpected results.

For more information about the disable.spotfire.trust.checks configuration property, please refer to the page Safeguarding your environment in the TIBCO® Enterprise Runtime for R Service Installation and Administration manual.    

For more information about how to configure the TERR Service, please refer to the page Configuring the TERR service in the TIBCO® Enterprise Runtime for R Service Installation and Administration manual.