Skip to content

Latest commit

 

History

History
165 lines (137 loc) · 9.9 KB

Analytics and Logging.md

File metadata and controls

165 lines (137 loc) · 9.9 KB

Analytics and Data Logging Formats

Summary

The Bot Framework defines an Actvity Analytics Event. Each event has a globally unique id (the Activity Id) as well as relevant data from the processing of the activty.

The pipeline for the generation and process of these analytics events is:

  1. Pre Processing. When an Activty is received, the the Activity Analytics Event is created, the id is assigned, and the reference is stored in the Context that is passed to all of the middleware processors.
  2. Processing. Each processing step in the middleware pipeline can add objects to the Activity Analytics event. These objects shoud follow json-ld semantics by having @Ids and @types as necessary.
  3. Post Processing. Emitters are run by the middleware pipeline and have a final oppertunity to add relevant contextual data into the analytics events. This may include data such as processing time, final results, or other relevant contextual data.
  4. Analytics Providers. Any registered analytics providers are called, and may serialize the Analytics event in accordance with rules of a particular state store. Providers are envisioned for Trace, Blob Storage, CosmosDB, and Application Insights.

Initial Analytics Record

Upon creation, the activty event is largley empty. The format of the event, represented in JSON-LD, is:

{   
    "@context" : "http://www.microsoft.com/botFramework/schemas/v1",
    "@type" : "botAnalyticsRecord", 
    "@id" : "{activityId}",
    "receivedAtDateTime" : "2017-08-25T10:49:05.234Z",
    "botId" : "{Unique bot identifier}"
}

Properties for botAnalyticsRecord

Property Expected Type Description
receivedAtDateTime DateTime The time the activity was received by the bot.
botId identifier The identifer that uniquely identifes the bot.

Note: Not all records will a Microsoft defined activity Id. Examples of this include bot initiated processing in the context of a timer or other external signals. For these activities, the bot should create a new unique Activity Id and populate the @id field with that result.

Facets

The Analytics object is composed of facets, with each facet being generated by a particular piece of code. A facet is a stand-alone piece of data that is relevent to the broader activity and can be serialized to JSON-LD. For example, an activity that involves calls LUIS would inlcude data about that call such as latency, luis operation id, user utterances, recognized intents, and entities.

In terms of implementation, a facet is just a data bag that supports the "IFacet" interface, which in turn means it can be serialized into the JSON-LD document as part of post processing.

Conversation Facet

For bots that participate in conversations, the conversation id is added into the analytics record. The "@type" attribute for the conversation schema is added in accordance with json-ld convention.

{   
    "@type" : "botAnalyticsRecord, botConversation", 
    "conversationId" : "{converatonId}",
    "conversationTurn" : "{int}"
}

Properties for botConversation

Property Expected Type Description
conversationId identifier The identifer that uniquely defines conversation of which this activity is part. Typically a GUID.
converationTurn integer An increasing number that is incremented with each utterance the user sends to the bot.

Navigation Facet

For bots that navigate users between prompts, a navigation facet is added into the model. This facet treats each prompt as a developer geneated IRI (similar to a web page), and provides origin -> destion nodes. ;

{   
    "@type" : "botAnalyticsRecord, botNavigation",     
    "origin": "{userDefinedIRI}/promptName",
    "destination": "{userDefinedIRI}/promptName"
}

Properties for botNavigation

Property Expected Type Description
origin IRI The developer defined identifer that identifies a prompt within the scope of a given bot. This field represents the prompt the user last visited. In terms of workflow, the user visited this prompt, entered an utterance, and then (as the result of this activity) navigated to the destination prompt. This field is generally expected to be of the form: bot://companyname/botname/dialogname/promptName.
destination IRI The developer defined identifer, in IRI form, that identifies a prompt within the scope of a given bot. This field represents the prompt the user is being redirected to as part of the activity being processed. In terms of workflow, the user visited the origin prompt, entered an utterance, and now is being send to a new prompt. This field is generally expected to be of the form: bot://companyname/botname/dialogname/promptName.

For example:

  1. Bot sends a greeting prompt to the user. "Hello. I'm ComBot. Ask me about COM Interfaces.".
    • The URI for the prompt is: bot://contoso.com/combot/greetingPrompt
  2. User types "What is IUnknown?"
  3. Bot sends a card with details around IUnknown.
    • The URI for the prompt is: bot://contoso.com/combot/IUnknownOverviewCard

LUIS Facet

Bots that call out to LUIS may choose to include the relevant LUIS results in their analytics records. It's important to note that a Bot may call out to LUIS multiple times within the scope of a single activity.

{   
    "@type" : "botAnalyticsRecord, luisOperations",
    "luisOperations" : [
        {
            "@id" : "{LUIS Operation ID}",
            "requestDateTime" : "{datetime}", 
            "duration" : "{timespan}", 
            "responseCode" : "200",
            "languageModelId" : "{Language Model Id}", 
            "q":"turn on the camera",
            "rawResponse":"[{\"intent\":\"OpenCamera\",\"score\":0.976928055},
                        {\"intent\":\"None\",\"score\":0.0230718572}]"
        },
        {
            "@id" : "{LUIS Operation ID}",
            "requestDateTime" : "{datetime}", 
            "duration" : "{timespan}", 
            "responseCode" : "200",
            "languageModelId" : "{Language Model Id}", 
            "q":"turn on the camera",
            "rawResponse":"[{\"intent\":\"takePicture\",\"score\":0.976928055},
                        {\"intent\":\"None\",\"score\":0.0230718572}]"         
        }
    ]
}

Properties for luisOperations Note: LUIS Operations is an array of LuisOperation

Property Expected Type Description
duration Duration The elapsed time by the API call to Luis.
requestDateTime DateTime The time the request to Luis was initiated.
languageModelId identifier The id of the LUIS model used for the operation.
q String The Users Utterance (the "q" parameter passed into the LUIS query)
rawResponse string The raw response from LUIS.
responseCode int The HTTP Response code returned from LUIS

Emitters

Emitters at code that run, look at the context object, and put the relevant objects into the analytics objct. Emitters are specialized, with multiple emitters being run per operation (depending on developer configuration).

Dedicated emitters (such as a Navigation emitter, LUIS emitter, or a Translation Emitter) are registered with the Middleware pipeline and are run using the standard middleware semantics.

Emitters do NOT mutate the underlying Context.

In addition to emitters, because the object model is stored in "context.analytics" any code with access to the context may add data into the analytics stream. This approach is less structured than using emitters, but has the benefits of simplicity for bot developers.

Serialization to JSON-LD

The standard analytics format is JSON-LD with one document per event. The document itself may have any number of entries in it, as defined by the emitters and data stored in the context.analytics object.

Storage Providers

All registered Storage Providers are run by the Middleware as part of the ContextFinalizer step. The Storage Providers walk the context.analytics object and perform relevant steps for the data.

Envisioned "out of the box" storage providers include:

  1. Trace Storage Provider, which serializes the context.analytics object to JSON-LD and then emits is using standard system trace semantics.
  2. Application Insights Provider. This provider creates relevant events in App Insights and uses that as the durable store.
  3. CosmostDB Graph. This stores the JSON-LD documents in a CosmosDB Graph Databases.
  4. CosmosDB. Stores the JSON-Ld document in a the CosmostDB Document Database.
  5. Azure Append Blob. This stores the serialized JSON document in an Azure Append-Only Blob.

Additional Reading

The Google Measurement Protocol (part of Google Analytics) is found here. This protocol focuses on recording events, as seen here:

Measuring Actions

v=1                                   // Version.
&tid=UA-XXXXX-Y                       // Tracking ID / Property ID.
&cid=555                              // Anonymous Client ID.
&t=event                              // Event hit type.
&ec=UX                                // Event Category. Required.
&ea=click                             // Event Action. Required.
&el=Results                           // Event label.

&pa=click                             // Product action (click). Required.
&pal=Search%20Results                 // Product Action List.
&pr1id=P12345                         // Product 1 ID. Either ID or name must be set.
&pr1nm=Android%20Warhol%20T-Shirt     // Product 1 name. Either ID or name must be set.
&pr1ca=Apparel                        // Product 1 category.
&pr1br=Google                         // Product 1 brand.
&pr1va=Black                          // Product 1 variant.
&pr1ps=1                              // Product 1 position.