Business Activity Monitoring (BAM) - BizTalk Server (2023)

The following diagram illustrates the architecture for Business Activity Monitoring (BAM).

The following tools enable you to design, develop, and deploy BizTalk solutions that integrate with BAM.

• Microsoft Excel: The BAM Add-in for Excel provides a user interface that guides business analysts through the process of creating views and activities. Excel serves as both a design tool for business analysts and a data usage tool for business users. For more information about the BAM Add-in for Excel, seeRequirements for using the BAM Add-in for Excel.

• BAM administration program. The BAM Management Utility is a deployment tool. It allows the creation of BAM definitions in Excel that can be deployed across the enterprise. The BAM management utility creates the required SQL Server databases, Analysis Services cubes, databases for SQL Notification Services, and DTS or SSIS tasks (depending on the version of SQL Server installed). For more information about BAM Manager, see theBAM administration program.

• Track Profile Editor. The Monitoring Profile Editor allows BizTalk developers to map the data elements defined by business analysts to the BizTalk implementation, including orchestrations and messaging. More information aboutTracking Profile Editor can be found in Tracking Profile Editor.

• Utility for providing tracking profiles. The Monitoring Profile Deployment utility enables IT professionals to deploy new and updated monitoring profiles to the BAM infrastructure. More information aboutFor a tracking profile deployment tool, see the Profile Deployment Tracking Utility.

presentation

The presentation layer consists of the following elements:

• BAM-Portal. The BAM Portal in Microsoft BizTalk Server provides real-time, end-to-end visibility into a business process. It is a web-based feature composed of a collection of ASP.NET 2.0 pages. You can customize BAM to increase performance and improve user experience. For more information on the BAM portal, see theBAM-Portal.

• Microsoft Excel: The BAM Add-in for Excel provides a user interface that guides business analysts through the process of creating views and activities. Excel serves as both a design tool for business analysts and a data usage tool for business users. For more information about the BAM Add-in for Excel, seeRequirements for using the BAM Add-in for Excel.

• Custom user interface. ISVs (Independent Software Vendors) and developers can create custom applications that display BAM data.

processing

The processing layer consists of the following elements:

• BAM administration web service. This web service is used by the BAM Portal application to communicate with the BAM Primary Import Table (PIT). Communication with the database is done using credentials that impersonate. These credentials are stored in the registry, which is created during configuration. Methods exposed by this web service can be used by custom clients to retrieve views and their details, related activities, and pivot table layouts for each user. It can also be used to manage warnings in the database.

• event bus. The BAM event bus service processes monitoring data (streams) stored in a source database and persists that data in a query table format in the target database.

• SQL Notification Services. SQL Notification Services evaluates the instance and aggregation notifications for BAM that are defined by the business user.

  The following diagram illustrates the physical processes underlying the BAM architecture.

  

design-time environment

The following diagram illustrates the design-time environment.

The following steps describe the process shown in the figure above.

1. The Monitoring Profile Editor assumes that BizTalk Server has been configured, at least one BAM definition has already been deployed (using BM.exe), and the infrastructure (including the primary import database) has been created. The Monitoring Profile Editor uses a registry key set during BizTalk configuration to determine where the management database is located.

   1. BAM enumerates activity in the primary import database after it is discovered. BizTalk developers select the deployed activities, which are pulled from the primary BAM import database.

   2. The BizTalk developer maps this to a physical implementation by browsing the provided assemblies, which are retrieved from the BizTalk Management database.

   3. After the visual mapping to the BizTalk elements is complete, metadata is passed to the BizTalk Server runtime engine to capture and store the data. This is done via annotations (for messaging information to decode the collected data) and the monitoring profile (which is used to get the runtime data).

2. Microsoft Excel is used as a design-time tool on the one hand and as a data utilization or presentation tool on the other. In design-time mode in Excel, users can design a BAM definition by creating BAM activities and views. Also, users have the ability to create pivot controls and charts that will eventually be displayed in the BAM portal.

deployment

There are two deployment categories.

• Expansion of the dynamic infrastructure

• Instrumenting the runtime to collect data

  The following diagram illustrates BAM deployment.

  

  The following steps describe the process shown in the figure above.

1. The dynamic infrastructure can be expanded using the BAM administration utility. Using the BAM definition or an Excel design-time workbook and the BAM XML configuration file, the BAM administration utility creates all the necessary databases and the corresponding DTS or SSIS tasks needed for the system to function.

   1. The primary BAM import database and any supporting stored procedures, triggers, and DTS or SSIS tasks are created.

   2. The BAM archive database is defined but not created until the archiving DTS or SSIS task is run.

   3. The BAM Star Schema database is created when BAM OLAP aggregations have been defined. You can determine whether aggregations have been defined when the "real-time aggregation" is disabled in the Excel spreadsheet or theCreate OlapCubein the BAM definition"TRUE" is set and the CONFIGURATION XML file has a deployment unit foranalysis database.

   4. In order for a BAM OLAP cube to be created, the BAM definition and the XML configuration file must have an aggregation defined that is not real-time aggregation.

   5. The BAM Manager process view shows that the Notification Services process (nscontrol.exe) is invoked to create the SQL NS databases.

2. In the Monitoring Profile Editor UI, you can find a deployment command that instruments the runtime to monitor and decode data from the runtime systems. In this case willRemarksmoved to the BAM primary import database (when data is retrieved from the messaging system). A tracking profile is injected into the BizTalk Management database, which the BizTalk runtime engine uses to determine what data to publish to the BAM system.

3. The BizTalk Monitoring Deployment command line utility can be used to deploy tracking profiles to the primary BAM import database and the management database, similar to the Tracking Profile Editor. However, the command line utility does not have mapping capabilities.

use of the data

The term data usage refers to the process by which a business user uses the information collected by the BAM infrastructure. It is assumed that the BAM definition has been created (i.e. it defines what data is to be collected and how it is to be displayed) and deployed (i.e. the infrastructure is created dynamically) at this point. It also assumes that the BAM definition has been mapped to the physical implementation (that is, where the data is to be collected has been defined and, in some cases, code is written for placement in the system).

The following diagram illustrates the data usage process.

The following steps describe the process shown in the figure above.

1. The BAM portal is divided into two processes: Internet Explorer and the BAM web portal process (hosted on Internet Information Services).

   1. Internet Explorer uses the security context of the business user connecting to the BAM portal website. The BAM portal website is an ASPX application that collects information using the BAM administration web service.

   2. The BAM portal calls the BAM administration web service to retrieve information such as the BAM activities/views visible to a specific business user. The BAM management web service must be able to impersonate the business user. For this reason, BAM portal and web service are typically hosted on the same machine when deployed.

      Notice

      If your organization supports Kerberos and has Active Directory set up to support delegation, the portal and web service can be hosted on different computers.

   3. The web service calls stored procedures in the primary BAM import database to perform security checks and retrieve metadata and BAM real-time aggregation information.

      Notice

      Currently the BAM portal uses an undocumented oneWeb service named Query Web Service. This is not supported and is expected to be gone in the next release.

   4. Internet Explorer hosts OWC (Office Web Controls) for connecting directly to the BAM Analysis Services cubes.

2. Excel

Duration

Once the BAM definition is in place, the infrastructure is in place, and the developer has instrumented the necessary systems to enable the display, data can flow through the system.

data insertion

One of the main ways data flows into the BAM system is through the BizTalk service (BTSNTSvc.exe). The architecture of the BizTalk service is such that the service can contain logical subsystems.

This process is illustrated using the following diagram.

The following steps describe the process shown in the figure above.

1. One of the subsystems hosts the actual BizTalk Server engine. The engine is responsible for running orchestrations and messaging.

2. Another subsystem is the event bus service. The event bus service is responsible for writing BufferedEventStream data to the primary BAM import database.

Start the runtime engine

When the BizTalk service starts, each subsystem loads metadata that is required for the subsystems to function.

This process is illustrated using the following diagram.

The following steps describe the process shown in the figure above.

1. When the BizTalk engine is running (either orchestrations or messaging), it invokes interceptors that regulate the flow of data from the engine into BAM. When the engine calls the interceptor, it analyzes where the engine is in the execution order for the orchestration or messaging elements. If there is data that needs to be retrieved from the location where the engine is currently running, the interceptor collects the appropriate data. The metadata that determines what data needs to be collected and where is stored in the monitoring profile. When the BizTalk service starts, the engine's interceptors contact the BizTalk Management database and retrieve the appropriate monitoring profile for the items that are running.

2. The main function of the event bus service is to convert data blobs (created by the BizTalk engine) into usable data items and insert the data into the primary BAM import database. The messaging interceptor writes the raw data in a compressed format. This is to avoid creating a large memory working set. In order for the event bus service to convert messaging data into usable data items, it must first load metadata that can interpret the raw data. This metadata is saved asRemarksdesignated. The annotation metadata was placed using the Tracking Profile Editor or the command line utility to deploy the tracking profile. If the deployed solution does not retrieve any messaging data, then there are no annotations. Orchestration interceptors encode enough information in the raw format for the event bus service to do a proper decoding without additional annotations.

storing data

This section explains how data is collected from the BizTalk service.

This process is illustrated using the following diagram.

The following steps describe the process shown in the figure above.

1. In the BizTalk engine (as previously described) there are interceptors that are invoked during the execution of orchestrations and runtime messaging solutions. Based on the monitoring profile information, the interceptors determine what data to collect and then send that data to the BizTalk MessageBox in binary format. There serialized blob data collected at runtime is stored in a "TrackingData" table. During processing, the engine sometimes needs to store data that is in memory during execution. This happens at the so-called persistence storage points. The engine decides when these points occur. At a persistence savepoint, the data collected by the interceptors is flushed and persisted as part of the same transaction. This ensures the consistency of the data in the engine and ultimately the consistency of what is visible to BAM. If an error occurs and the transaction is rolled back, the corresponding BAM data is also rolled back.

2. The event bus service resides in the same executable process as the engine. The data fed into the service comes from the "TrackingData" table stored by the BTS engine (or any "BufferedEventStream" data source). The image shows an assembly - Microsoft.BizTalk.Bam.EventObservation. This assembly does not expose a method for getting the data from the BufferedEventStream. This code resides in the event bus service itself. The data is retrieved and then prepared for storage in the primary import database. The binary data is mostly serialized .NET objects that are reactivated. Then properties are pushed out of the object into formatted SQL statements.

3. When saving the data, the event bus service attempts to create a large batch. The batch contains calls to stored procedures generated by the BAM Manager deployment tool. It is possible that the batch save will fail due to high database activity or other table locking issues. If an error occurs, two more attempts are made to create the batch. If these attempts fail, the batch is split into smaller batches. The attempt is then repeated. If this attempt is also unsuccessful, the batch is moved to the "FailedTracking" table.

Insert custom data

The previous section detailed how BAM retrieves data from the BizTalk Server host using various methods. In an indirect way, this is done using the Monitoring Profile Editor to create a profile. A more direct method is to apply the "EventStream" methods directly in the code. However, not all data relevant to BAM is necessarily available in BizTalk. Therefore, an assembly is provided for custom applications that are written.

This process is illustrated using the following diagram.

The following steps describe the process shown in the figure above.

1. The custom application must have theMicrosoft.BizTalk.Bam.EventObservation-Assemblyload to access the BAM methods for inserting data into a BAM activity. Two primary classes are provided: DirectEventStream and BufferedEventStream. In the image above, the DirectEventStream class is highlighted. This class writes directly to the primary import database. This is the most common way to write data to a BAM activity.

2. Similar to the way BizTalk stores data, the BufferedEventStream class writes binary blobs to an indirect database. In this case, the BizTalk MessageBox is used as a cache. Which class you choose depends on various factors. However, performance considerations should be paramount here. The "BufferedEventStream" class combines data in batches and ensures higher throughput in the long term. The disadvantage is that the data is not written immediately.

3. As with the BizTalk method of inserting data, the event bus service processes the binary data, decodes it, and finally stores it in the BAM primary import database. This is only required for data written through the BufferedEventStream class.

Distributed Navigation

The distributed navigation feature in the BAM portal allows users to view the relationships between activities across remote boundaries.

Further information

Management and Monitoring Architecture