Fault Analysis

The Fault Analysis application module provides an integrated solution that allows you to:

Fault Analysis provides an interface that allows you to view software fault trends and to see specific core files. Since all the core files, binaries, and symbols are stored in the cloud, they are accessible to you and your developers whenever and wherever they're required.

Fault Analysis Module

The Fault Analysis module runs on the BlackBerry IoT Platform. Fault Analysis implements functionality in the cloud to analyze those core files and provides you useful information about those faults. You can use that information to:

The Fault Analysis module provides the following components:

  1. The Fault Analysis Agent (faAgent), which detects a fault and then uploads the core file from a device to the BlackBerry IoT Platform. This aspect of Fault Analysis simplifies the need for a person to have to log in to the device, or worst, physically go to the device to retrieve the core file.
  2. An Uploader tool, which you use to upload symbols and binaries to the BlackBerry IoT Platform. The uploaded symbols eliminate the need for developers to find the matching symbols for the fault.
  3. A Fault Analysis web application, which provides a dashboard to see the analyzed core files. The Fault Analysis web application also has an Admin Console (not to be confused with the administration console for the BlackBerry IoT Platform) for enabling applications for fault analysis and for granting others access to the analyzed data.

About the Fault Analysis module

Whenever a fault or crash occurs on a device, the installed Fault Analysis Agent (faAgent) detects it, uploads the core file to the cloud, and then removes the core file from the device. Core files contain tracebacks of the fault that occurred on the physical device. The automatic uploading of the core file greatly simplifies the process for gathering information for any faults that occur on a device. For information about how to configure your devices with the faAgent, see Configure your devices for Fault Analysis.

Provided with the Fault Analysis module is an Uploader tool, which you use to upload your symbols and associated binaries to the BlackBerry IoT Platform. The Fault Analysis module requires the matching symbols and binaries from the application software to properly analyze core files stored in the cloud.

The Fault Analysis module generates files, which summarize the analysis of faults that are uploaded to the cloud. These files are stored in the Fault Analysis organization on the BlackBerry IoT Platform. Though this organization is transparent to you, you should be aware that your organization grants capabilities that permit the Fault Analysis organization to access the core files, symbols, executables, and binaries that you upload to your organization. Likewise, your organization is granted capabilities to access the analyzed fault data in the Fault Analysis organization. The grant allows your organization and your users access to see the fault analysis information.

The Fault Analysis module also provides a Fault Analysis web application. The web application has an Admin Console and Dashboard. You can use the Admin Console to enable your application to use the Fault Analysis module; you can also grant users access to see fault data in the Dashboard. You can use the Dashboard to see the fault analysis information from multiple core files and multiple devices.

You can access the Fault Analysis application from a web browser. Alternatively, you can use the QNX Momentics IDE to access the Fault Analysis application. The advantage of using the IDE is that the core files, binaries, and associated symbols are automatically downloaded to your project. When the metadata for a binary or symbol file is provided, the source code is downloaded for you. This functionality allows you to not only view the fault analysis data, but start a postmortem debug session. Easy!

Summary of configuration tasks to use Fault Analysis

Here's a summary of the task you need to do to start using the Fault Analysis module:

  1. Create entities in your organization to use the Fault Analysis module. For more information, see Configure your organization entity for Fault Analysis.
  2. Log in to the Fault Analysis Admin Console and add your organization and applications for Fault Analysis. For more information, see Add an organization and application for Fault Analysis. You can also manage which users have access to the Fault Analysis Dashboard (e.g., developers in your organization). For more information, see Grant a user access to your fault information.
  3. Configure your software builds to use the Uploader tool. Alternatively, you can run the tool manually as required. For more information about using the Uploader tool, see Configure your builds to use Uploader tool.
  4. Install the Fault Analysis Agent on your physical devices. For more information, see Configure your devices for Fault Analysis.

Note: Before you can use the Fault Analysis module, contact a sales representative from BlackBerry to get the access to the Fault Analysis Agent and Uploader tool.

Configure your organization entity for Fault Analysis

You must configure your existing organization entity as follows:

  1. Create an application entity (for example, MyFieldApp) to represent your software application that may generate faults. Instances of the software running on physical devices are represented as device entities belonging to that application and use device-based authentication. Typically, this application entity and device entities should exist already in your organization.
  2. Create an application entity (for example, MyUploaderApp) to represent the Uploader tool.
  3. Grant (or give) the search.file capability from your organization entity to the application entity for the Uploader application that you created in the previous step.
  4. Create a device entity for the application entity you created in step 2 (the myUploaderApp) and generate a certificate. The certificate is required for the Uploader tool to authenticate, so make sure that you download it after you create the device entity. One device entity for the Uploader tool is required for each organization.

For information about how to do the tasks described in the steps above, see:

Configure your devices for Fault Analysis

To configure a device to upload files for Fault Analysis, you must install the Fault Analysis Agent (faAgent) executable on your device. The faAgent detects, creates, and uploads a core file to the BlackBerry IoT Platform. You must have the QNX Agent (bipd) configured and running on your device before you install the faAgent. The faAgent depends on functionality from the QNX Agent therefore, if you put faAgent into the startup script, faAgent must run after the QNX Agent starts.

Before you install the faAgent on your device, it's presumed that:

Here are the steps to install the Fault Analysis Agent (faAgent). The executable is called faAgent.

Note: Before you can transfer or create files and directories on your QNX Neutrino system, you may need to execute the following command:

mount -uw /base/

  1. Put the faAgent on your QNX-based device. For example, you can use scp to put the faAgent executable in the same location as the QNX Agent (bipd executable), which is the /etc/bipd/ directory.
  2. Connect to the device using ssh or console connection and create the /var/dumps/ directory. This is the default directory used by the tool. If you want a different location, create a different directory and use the -D option to specify that location when you run faAgent in the following step.
  3. The faAgent requires these directories to be manually created on the device:

    • /jam/bipd/data
    • /pps/bipd/files?server
  4. Run faAgent as a daemon from command line (e.g. faAgent &) or start it from a startup script.

Configure your builds to use Uploader tool

Provided with the Fault Analysis module is an Uploader tool. You use this tool to upload binaries and symbols to the BlackBerry IoT Platform. The binary and symbol files are required to compute the stack traces for the fault that occurred on your devices. Each file is stored as a separate file entity that's associated with the device entity that represents the physical device of where the fault occurred.

Each binary or symbol file can be tagged with metadata to give useful information about it, such as the Git repo of where the source code came from. For information about the format of the metadata, see Add Metadata to files.

In a production environment, typically there's a build server for your software. If so, it's a good practice to integrate the Uploader tool within your build process. This practice ensures that binaries and symbols (artifacts) are uploaded automatically to the BlackBerry IoT Platform and available for Fault Analysis.

Before you can use the Uploader tool, you must have an application entity and device entity in your organization to represent it. For more information, see Configure your organization entity for Fault Analysis. The following information for the application and device entities is required to use the Uploader tool:

Use the Uploader tool

The Uploader tool is an executable JAR file. To use the tool, you require the following software installed on your computer:

Here's the syntax for using the Uploader tool:

java -jar uploader.jar -binary <name of binary or symbol file>
                      [-metadata <name of metadata file>]
                       -deviceId <deviceId>
                       -apiKey <apiKey>
                       -apiSecret <apiSecret>
                       -host <URI to the BlackBerry IoT Platform>
                       -keyStore <pathToKeyStore>
                       -keyStorePassword <keyStorePassword>
                       [-overwrite]

where:

Upload multiple files

To generate accurate stack traces, you must upload all the symbols of the libraries, executables, and binaries for your software application. If you require debug symbols for the version of QNX Neutrino that's running on your device, you can obtain them from the QNX website.

Provided with the Fault Analysis module is a script called qnxUploader.sh that you can use to upload multiple files. The script creates multiple file objects that are associated with device entity.

Create a KeyStore

The Uploader tool uses a certificate and private key to authenticate with the BlackBerry IoT Platform. The tool requires that the private key and certificate to be contained within a Java KeyStore. The certificate is downloaded when you create a device entity using a certificate signing request (CSR) on the BlackBerry IoT Platform. For more information, see Create a device.

To generate a KeyStore for the Uploader tool, use these steps:

  1. Generate a PKCS #12 database consisting of your private key and certificate. For example, you can use OpenSSL (open source) to generate a Java KeyStore with your certificate and private key as follows:
    openssl pkcs12 -export -in <path to the certificate generated from the CSR>
                   -inkey <private_key generated with your CSR>
                   -out cert.p12 -name BBIoT
        
  2. Create a KeyStore using the PKCS #12 format (the cert.p12 file from the previous example) and run the keytool utility:
    keytool -importkeystore -deststorepass <password> -destkeypass <password>
            -destkeystore device.jks -srckeystore cert.p12 -srcstoretype PKCS12
            -srcstorepass <password>
        

In this example, the KeyStore is called device.jks. The path and filename of device.jks and the password you specify are used for the options pathToKeyStore and keyStorePassword of the Uploader tool, respectively.

Add Metadata to files

It's useful to add metadata to specify the location of the source code that was used to generate the binary or symbol files. That metadata is useful for letting the developer know where to get the source code. If a developer uses the QNX Momentics IDE, that metadata is used to automatically download the source code into an IDE project.

To add metadata, you simply specify JSON in a separate file. Here's the format of the JSON you must use:

{
 "scm" : "git",
 "commit_id" : "3e14e3987d23f7f178a2d23bbc00ed59199350fb",
 "project_loc" : "app",
 "compile_path" : "/build/workspace/buildjob/myapp",
 "url" : "ssh://gitserver/myrepo"
}
where:

Use the Fault Analysis Dashboard

After an administrator grants a user access to view faults in an organization, that user can access the Fault Analysis Dashboard using the QNX Momentics IDE or a web browser at http://bbryiot.com/fa.

After you log in to the Fault Analysis Dashboard, in the Select Organizations list, choose the organizations you want to see the analyzed information for. You can type the first letters of your organization in the Search box to refine the list. Here's what the Dashboard looks like:

Fault Analysis Dashboard

View faults in the QNX Momentics IDE

Instead of a web browser, you can use the QNX Momentics IDE to access the Fault Analysis Dashboard. Like the web-based version, you can use the Fault Analysis Dashboard to navigate the fault data that's been analyzed.

Note: On Linux, ensure that the WebKit is installed on your system to see the Dashboard in the IDE. To install the WebKit, run the following command: sudo apt-get install libwebkitgtk-1.0-0.

The benefit to using the IDE is that you can click Launch Debug Session, which does the following for you:

Use the Fault Analysis Admin Console

You can use the Fault Analysis Admin Console to configure your organization and one application from your organization to use Fault Analysis. After you add the organization and application, you can grant other users access to the Fault Analysis Dashboard so that they can access the fault analysis data.

Add an organization and application for Fault Analysis

When you add the organization, there's an initialization process performed on your organization to function with Fault Analysis. After this process is done, user, application, and device entities in your organization are granted more capabilities and also grant capabilities to the Fault Analysis organization.

Note: The user that you use to log in to the Fault Analysis Admin Console must have the Admin tag (or the equivalent capabilities) granted on the organization entity. If you don't, you won't be able to initialize the organization to use Fault Analysis.

Here are the steps to add a new organization and application to Fault Analysis:

  1. Log in to Fault Analysis Admin Console.
  2. In the Select Organization to manage list, select the organization that you want to initialize to use the Fault Analysis module. If you don't have the proper capabilities granted to your user, you won't see your organization.
  3. In the Device Application, select the application entity representing the software application of where faults occur.
  4. In the Uploader Application list, select the application entity representing the Uploader tool.
  5. Click Initialize.

Fault Analysis Admin Console

Grant a user access to your fault information

You can give access to users so that they can see the analyzed data for your device's software faults. Typically, you would give your developers this access. When a user is given access, the following occur:

Here's how to grant a user access to the Fault Analysis Dashboard:

  1. Log in to the Fault Analysis Admin Console at http://bbryiot.com/fa/admin.
  2. In the Select Organization to Manage list, choose the organization you want to grant a user access to.
  3. In the User's email address box, type the email address of the user and then click Add User.

Note: The email address of the user must be the same one they use to sign in to the BlackBerry IoT Platform.

Add user to Fault Analysis