Business Activity Monitoring (BAM) - BizTalk Server (2023)

The following diagram illustrates the architecture for the Business Activity Monitoring (BAM) feature.

You can use the following tools to design, develop, and deploy BizTalk solutions that integrate with BAM.

• Microsoft Excel. The BAM Add-in for Excel provides a user interface that guides business analysts in creating activities and views. Excel serves as both a design tool for business analysts and a data usage tool for business users. For more information about the BAM Add-in for Excel, seeRequirements for using the BAM Add-in for Excel.

• BAM administration program. The BAM Management Utility is a deployment tool that can be used to deploy BAM definitions created in Excel across the enterprise. The BAM management utility creates the necessary SQL Server databases, Analysis Services cubes, SQL Notification Services databases, and DTS or SSIS tasks (depending on the version of SQL Server installed). For more information about BAM Manager, seeBAM administration program.

• Tracking-Profile-Editor. The Tracking Profile Editor enables BizTalk developers to map the data elements defined by business analysts into the BizTalk implementation, including orchestrations and messaging. For more information on the Tracking Profile Editor, seeTracking-Profile-Editor.

• Utility for providing tracking profiles. Tracking Profile Deployment Utility enables IT professionals to deploy new and updated tracking profiles to BAM infrastructure. For more information on the Tracking Profile Provisioning Tool, seeUtility for providing tracking profiles.

presentation

The presentation layer consists of:

• BAM-Portal. The BAM Portal in Microsoft BizTalk Server provides real-time, end-to-end visibility into a business process. It is a web-based feature consisting of a collection of ASP.NET pages. You can customize BAM to improve performance and experience for your users. For more information on the BAM portal, seeBAM-Portal.

• Microsoft Excel. The BAM Add-in for Excel provides a user interface that guides business analysts in creating activities and views. Excel serves as both a design tool for business analysts and a data usage tool for business users. For more information about the BAM Add-in for Excel, seeRequirements for using the BAM Add-in for Excel.

• Custom user interface. ISVs and developers can build custom applications that display BAM data

Processing

The processing layer consists of:

• BAM administration web service. This web service is used by the BAM Portal application to communicate with the BAM Primary Import Tables (PIT). Communication with the database is done using impersonated credentials stored in the registry created during configuration. Methods exposed by this web service can be used by custom clients to retrieve views and their details, associated activities, and pivot table layouts for each user. They can also be used to manage alerts in the database.

• Event-Bus. The BAM Event Bus service processes tracking data (streams) stored in a source database and stores that data in a query table format in the target database.

• SQL Notification Services. SQL Notification Services evaluates the business user-defined instance and aggregated BAM alerts.

  The following diagram illustrates the underlying physical processes for the BAM architecture.

  

design time experience

The following diagram illustrates the design-time experience.

The following steps describe the process shown in the figure above.

1. The Monitoring Profile Editor assumes that BizTalk Server has been configured, that at least one BAM definition has already been deployed (using BM.exe), and that the infrastructure including the primary import database has been created. The Monitoring Profile Editor uses a registry key that is set during BizTalk configuration to determine where the management database is located.

   1. BAM lists the activity within the Primary Import Database as soon as it is detected. BizTalk developers select the deployed activities, which are retrieved from the primary BAM import database.

   2. The BizTalk developer maps this to a physical implementation by browsing the provided assemblies, which are retrieved from the BizTalk Management database.

   3. Once the developer has visually mapped this to the BizTalk artifacts, metadata is passed to the BizTalk Server runtime to collect and store the data. This is done via annotations (for messaging information used to decode the data collected) and tracking profile (used to retrieve the runtime data).

2. Microsoft Excel is used both as a design-time tool and as a data consumption or presentation tool. The Excel design-time experience allows users to create a BAM definition by creating BAM activities and views. It also enables the creation of the pivot controls and charts that will ultimately be displayed in the BAM portal.

Mission

There are two deployment categories

• Building the dynamic infrastructure

• Instrumenting the runtime to collect data.

  The following diagram illustrates BAM deployment.

  

  The following steps describe the process shown in the figure above.

1. The BAM administration utility is used to build the dynamic infrastructure. Using the BAM definition or a design-time Excel workbook plus the BAM configuration XML file, the BAM administration utility creates all the necessary databases and the corresponding DTS or SSIS tasks required for the system to operate.

   1. The primary BAM import database and any supporting stored procedures, triggers, and DTS or SSIS tasks are created.

   2. The BAM archive database is defined but not created until the DTS or SSIS archive task runs.

   3. The BAM Star Schema database if BAM OLAP aggregations have been defined. You can tell if aggregations have been defined if thereal-time aggregationbutton in the Excel spreadsheet is disabled or theCreate OlapCubeis set toTRUEin the BAM definition and configuration XML file has a deployment unit foranalysis database.

   4. The BAM definition and configuration XML must have a non-RTA aggregation defined in order to create a BAM OLAP cube.

   5. The exploded view of the BAM Manager process shows that the Notification Services process (nscontrol.exe) is invoked to create the SQL NS databases.

2. The tracking profile editor UI has a deployment command that instruments the runtime to track and decode data from the runtime systems. In this case it pushesRemarksinto the primary BAM import database (when retrieving data from the messaging system). A tracking profile is injected into the BizTalk Management database, which is used by the BizTalk runtime to determine what data to publish to the BAM system.

3. The BizTalk Tracking Deployment command line tool allows tracking profiles to be published to the primary BAM import database and the management database, similar to the tracking profile editor; However, the command line tool does not allow mapping functions.

data usage

Data usage refers to the process by which a business user consumes the information that the BAM infrastructure has collected. At this point, it is assumed that the BAM definition has been created (defines what data is collected and how it is displayed), that the BAM definition has been deployed (dynamically building the infrastructure), and that the BAM definition has been mapped to the physical Implementation (define where to collect the data and in some cases write code to push into the system).

The following diagram illustrates the process of data consumption.

The following steps describe the process shown in the figure above.

1. The BAM portal is divided into two processes, Internet Explorer and the BAM web portal process hosted on Internet Information Services.

   1. Internet Explorer uses the security context of the business user connecting to the BAM Portal website. The BAM Portal website is an ASPX application that uses the BAM administration web service to collect information.

   2. The BAM portal calls the BAM administration web service to retrieve information such as: B. which BAM activities/views each business user can see. It is important that the BAM administration web service be able to impersonate a business user, so the typical deployment is for the BAM portal and web service to be hosted on the same machine

      note

      If your company supports Kerberos and Active Directory is set up to support delegation, it is possible that the portal and the web service are on different computers.

   3. The web service calls stored procedures in the primary BAM import to perform security checks, retrieve metadata, and retrieve BAM real-time aggregation information.

      note

      Currently, the BAM portal uses an undocumented web service called theQuery web service. This is not supported and will most likely be deprecated in the next release.

   4. Internet Explorer hosts the Office Web Controls (OWC) that connect directly to the BAM Analysis Services cubes.

2. Excel

Duration

After the BAM definition is defined, the infrastructure is built, and the developer has instrumented the necessary systems to enable transparency, data can flow through the system.

insert data

One of the main ways data flows into the BAM system is through the BizTalk service (BTSNTSvc.exe). The BizTalk Service is designed to contain logical subsystems.

The following diagram illustrates this process.

The following steps describe the process shown in the figure above.

1. One subsystem hosts the BizTalk Server engine itself. The engine is responsible for running orchestrations and messaging.

2. Another subsystem is the Event Bus Service. The Event Bus service is responsible for writing buffered event stream data to the BAM primary import database.

boot runtime

When the BizTalk service starts, each subsystem loads metadata required for the subsystems to operate.

The following diagram illustrates this process.

The following steps describe the process shown in the figure above.

1. When the BizTalk engine is running (either orchestrations or messaging), it invokes interceptors that regulate the flow of data from the engine into BAM. When the engine invokes the interceptor, the interceptor analyzes where the engine is in terms of execution order for the orchestration or messaging artifacts. When data needs to be retrieved from where the engine is currently running, the interceptor collects the appropriate data. The metadata that determines where the data needs to be collected and what data needs to be collected is housed in the tracking profile. During BizTalk Service bootup, the engine's interceptors contact the BizTalk Management database to retrieve the appropriate tracking profile for the running artifacts.

2. The main function of the Event Bus service is to convert data blobs (written by the BizTalk engine) into usable data items and insert the data into the primary BAM import database. The messaging interceptor writes the raw data in compressed form. This is done to avoid a large memory set. In order for the Event Bus service to convert message data into usable data items, it must first load metadata that can interpret the raw data. This metadata is known asRemarks. The annotation metadata was placed there via the tracking profile editor or the command line tool for deploying tracking profiles. If the deployed solution does not pull data from messaging, then there are no annotations. Orchestration interceptors encode enough raw information to allow the Event Bus service to properly decode without additional annotations.

save data

This section explains how data is collected by the BizTalk Service.

The following diagram illustrates this process.

The following steps describe the process shown in the figure above.

1. As previously described, the BizTalk Engine has interceptors that are invoked during run-time execution of orchestrations and messaging solutions. Based on the tracking profile information, the interceptors determine what data to capture and then send it in binary form to the BizTalk MessageBox. There is a TrackingData table that contains serialized blob data collected at runtime. During processing, there are times when the engine needs to store data that it keeps in memory during execution. These are known as persistence points, and the engine decides when these points occur. When a persistence point occurs, the data captured by the interceptors is also flushed and persisted within the same transaction. This ensures the consistency of the data in the engine and ultimately what BAM can see. If an error occurs and the transaction is rolled back, the associated BAM data is also rolled back.

2. The Event Bus service resides in the same executable process as the engine. The service feeds itself from data in the TrackingData table stored by the BTS engine (or any BufferedEventStream data source). The image shows an assembly - Microsoft.BizTalk.Bam.EventObservation. This assembly does not provide a method for getting the data from the BufferedEventStream. This code is housed in the Event Bus Service itself. The data is retrieved and then prepared for storage in the primary import database. Most of the binary data is serialized .NET objects that are rehydrated after moving properties out of the object into formatted SQL statements.

3. When the data is saved, the Event Bus service attempts a large batch. The batch contains stored procedure calls generated by the Bam Manager deployment tool. The stack save can fail due to high database activity or other circumstances that lock tables. In case of failure, the stack is tried a few more times. If the retries fail, the batch is split into smaller batches and tried again. If that fails, the batch is moved to the FailedTracking table.

Custom pasting of data

The previous section detailed how BAM retrieves data from the BizTalk Server host using various methods. Indirectly using the Tracking Profile Editor to create a profile and more directly using the Event Stream methods directly in code. However, not all BAM relevant data is necessarily available in BizTalk. Therefore, an assembly is available for custom written applications.

The following diagram illustrates this process.

The following steps describe the process shown in the figure above.

1. The custom application must be able to load theMicrosoft.BizTalk.Bam.EventObservationAssembly to access the BAM methods for inserting data into a BAM activity. There are two primary classes exposed, DirectEventStream and BufferedEventStream. In the image above, the DirectEventStream class is highlighted. This class writes directly to the Primary Import database. This is the most common way to write data to a BAM activity.

2. Similar to the method in which BizTalk stores data, the BufferedEventStream class writes binary blobs to an indirect database. In this case, the BizTalk MessageBox is used as a cache. Which class you choose depends on several things; However, performance is a primary concern. The BufferedEventStream class aggregates data and provides higher throughput. The disadvantage is that the data is not written immediately.

3. As with the BizTalk method of inserting data, the Event Bus service processes the binary data, decodes it, and finally stores it in the BAM primary import database. This is only required for data written through the BufferedEventStream class.

Distributed Navigation

The BAM Portal's distributed navigation capability allows users to view activity relationships across remote boundaries.

See also

Management and tracking architecture