Dataflows docs and small updates to report

Author: MattFarm

CustomAnalytics/DataFlowVersion/PVA_Dashboard_DataFlow.pbit

@@ -1,57 +1,95 @@
 # Custom analytics solution for Power Virtual Agents - DataFlows version for higher scale bots
 
-If your Bot has significant numbers of monthly sessions, using this version of the report provides improved scalibility. This version does require additional setup, and a Power BI Premium license <https://powerbi.microsoft.com/en-us/pricing/>. The report uses Power BI DataFlows to connect to DataVerse and pre-process some of the compute-intensive transformations of the data.
+If your Bot has significant numbers of monthly sessions, using this version of the report provides improved scalibility. This version does require additional setup, management and a Power BI Premium license <https://powerbi.microsoft.com/en-us/pricing/>. The report uses Power BI DataFlows to connect to DataVerse and pre-process some of the compute-intensive transformations of the data. Egress from Dataverse is done using Azure Synapse Link and Azure Data Lake Storage v2.
+
+<img src="img/PVA_Custom_Analytics_Dataflow_SynapseLink.png" width="797" alt="Custom Analytics with Dataflows and Synapse Link - Diagram">
 
 ## Solution components
 
 - Microsoft Dataverse
 - Microsoft Power BI DataFlows
 - Microsoft Power BI Desktop
-- Chat Transcripts control for Power BI - available at <https://github.com/iMicknl/powerbi-botframework-chat-transcripts>
-- (Optional) Azure Data Lake Storage v2
+- Chat Transcripts control for Power BI - available at <https://github.com/iMicknl/powerbi-botframework-chat-transcripts> - many thanks to Mick Vleeshouwer
+- Azure Data Lake Storage v2
+
+## Solution description
+
+This solution is based on 3 key Dataverse tables associate with Power Virtual Agents:
+
+- bot - a list of PVA bots in your environment (small data size)
+- botcomponent - the components making up each bot (e.g. topics) (small data size)
+- conversationtranscript - the log of conversation activity on all your bots (potentially very large data size)
+
+Data in the conversationtranscript table is streamed using Azure Synapse Link from a Dataverse environment to Azure Data Lake Storage v2, stored in the Common Data Model format. A Power BI Dataflow (Dataflow 1) connects to ADLS to retrieve the conversation transcript records. A second Dataflow (Dataflow 2), pre-processes the data, transforming the values stored in the conversationtranscript table into trace data for consumption in the Power BI report.
 
 ## Solution files
 
-- PVA_Analytics_Export_Transform.json - this is the template for a PowerBI DataFlow
-- PVA_Dashboard.pbit - Power BI template file
+- PVA_Analytics_Export_Transform.json - this is the template for a PowerBI DataFlow - provided with an Azure SQL connection for used to retrieve bot info from Dataverse.
+- PVA_Dashboard_DataFlow.pbit - Power BI template file
 
 ## Installation
 
 ### Installation requirements
 
 - One or more Power Virtual Agents bots
-- A Power BI account
+- A [Power BI Premium Workspace](https://docs.microsoft.com/en-us/power-bi/admin/service-premium-what-is)
 - [Power BI Desktop](https://powerbi.microsoft.com/en-us/downloads/)
 
 ### Installation steps
 
-1. **Edit the DataFlow template**
+#### Connecting Azure Data Lake Storage for data archival
+
+Required - using Azure Synapse Link allows us to export significant telemetry from Dataverse, and has the additional benefit of allowing storage of telemetry longer than it is stored in your Dataverse instance (default 30 days)Steps for configuring this are as follows:
+
+> Important - always test this process on a development environment before applying to your production system.
+
+1. [Set up an Azure Data Lake, and connect to it using Azure Synapse Link](https://docs.microsoft.com/en-us/powerapps/maker/data-platform/azure-synapse-link-data-lake).
+
+2. Configure the conversationtranscript table for export using Azure Synapse Link
+   1. In the Power Apps portal, select Azure Synapse Link - select the link you created in step 1 - Manage tables
+   2. Search for 'conversationtranscript' - Save.
+   3. Transcipts will now be exported to your Azure Data Lake
+
+#### Set up Dataflows
+
+1. Create a new Power BI Dataflow exposing the data stored in Data Lake (we will call this **Dataflow 1**)
+   1. Open your Power BI Workspace - New - Dataflow
+   2. Select 'Attach a Common Data Model' folder
+   3. Browse you Azure Data Lake container, and paste the URL of your 'model.json' file that describes your data.
+
+2. Edit the DataFlow template
    1. [Find your Dataverse environment URL](https://docs.microsoft.com/en-us/powerapps/maker/data-platform/data-platform-powerbi-connector#find-your-dataverse-environment-url), the URL will be in the format: https://yourenvironmentid.crm.dynamics.com/. You will just need the 'yourenvironmentid' part of the URL
    2. Download the file [PVA Analytics_Export_Transform.json](PVA_Analytics_Export_Transform.json?plain=1) and open it in a text editor, e.g. Visual Studio Code.
    3. Perform a find and replace on the file - swapping the placeholder `yourdataverse` with your Dataverse environment URL.
 
-2. **Create the DataFlow from the template**
+3. Create the DataFlow from the template (**Dataflow 2**)
    1. Log in to Power BI at <http://powerbi.com>
    2. Select the Workspace you wish to deploy the DataFlow to or create a new workspace. Please note that you cannot use 'My Workspace' for this purpose. 
    3. Select New > DataFlow
    4. Select Import Model
    5. Select the edited file 'PVA Analytics_Export_Transform.json'. Your DataFlow job should now be ready - test refreshing the data.
    6. You should be prompted for credentials, if not go to Settings -> Data source credentials and select organizational account. You can now login with an Azure Active Directory account that has access to the Dataverse environment.
    7. Whilst in Settings for your DataFlow, be sure to select 'Enhanced compute engine settings' and select 'On'.
-   8. If you want to refresh the content periodically, go to Settings -> Scheduled refresh and select the preferred period.
+   8. You will likely want to refresh the content periodically, go to Settings -> Scheduled refresh and select the preferred period.
+
+4. Add the Dataflow 1 version of conversationtranscript to Dataflow 2 (chaining them together)
+   1. Edit Dataflow 2 - select 'Edit tables'.
+   2. You should now be in the Power Query editor. Select Get data - more... - Power Platform - Power BI Dataflows.
+   3. Sign in, and then select the conversationtranscript table from Dataflow 1.
+   4. You should now return to the Power Query editor. We need to change each reference to conversationtranscript to point to the new table we just imported. Select the original conversationtranscript table - and then look at the Applied Steps window on the right of the screen. Delete any steps after 'Source'. Select the source step, and replace any value with the name of the conversationtranscript table from Dataflow 1. This effectively makes the source of conversationtranscript in Dataflow 2 to be conversationtranscript from Dataflow 1.
 
-3. **Create your Power BI report**
+5. Note that managing the data in your DataLake will require additional attention, outside the scope of these steps.
+
+#### Set up the report
+
+1. Create your Power BI report
    1. Download the file [PVA_Dashboard_DataFlow.pbit](PVA_Dashboard_DataFlow.pbit?plain=1)
    2. Enter the parameters you are prompted for. These are:
       1. The name of the Power BI workspace
-      2. The name of the DataFlow you created
+      2. The name of DataFlow 2
    3. The report should pull in the data and render it
    4. You can now [publish the report from Power BI Desktop](https://docs.microsoft.com/en-us/power-bi/create-reports/desktop-upload-desktop-files) so that other users can access it.
 
-## Connecting Azure Data Lake Storage for data archival
-
-Please follow the instructions in the main [readme] (../readme.md) for details of setting this up. Instead of consuming the DataFlow containing the ConversationTranscript records (stored in Azure Data Lake) in the Power BI report, it should be consumed in the PVA Analytics_Export_Transform DataFlow (change the source of ConversationTranscript from Dataverse to point the Azure Data Lake DataFlow).
-
 ## Troubleshooting
 
 There are a few places where the pipeline may break - troublshooting can normally isolate the problem using these steps:

CustomAnalytics/DataFlowVersion/img/PVA_Custom_Analytics_Dataflow_SynapseLink.png

@@ -32,34 +32,6 @@ This solution allows customers to create a Power BI dashboard for their Power Vi
    3. The report should pull through the data and render it
    4. You can now save the report to share through the Power BI portal.
 
-## Connecting Azure Data Lake Storage for data archival
-
-Optional - if you wish to store and use telemetry longer than it is stored in your Dataverse instance (default 30 days), you can configure export to Azure Data Lake using Azure Synapse Link. Steps for configuring this are as follows:
-
-!Important - always test this process on a development environment before applying to your production system.
-
-1. Set up an Azure Data Lake, and connect to it using Azure Synapse Link. See the steps documented here for details: <https://docs.microsoft.com/en-us/powerapps/maker/data-platform/azure-synapse-link-data-lake>.
-2. Configure the conversationtranscript table for export using Azure Synapse Link
-   1. In the Power Apps portal, select Azure Synapse Link - select the link you created in step 1 - Manage tables
-   2. Search for 'conversationtranscript' - Save.
-   3. Transcipts will now be exported to your Azure Data Lake
-
-3. Create a new Power BI DataFlow exposing the data stored in Data Lake
-   1. Open your Power BI Workspace - New - DataFlow
-   2. Select 'Attach a Common Data Model' folder
-   3. Browse you Azure Data Lake container, and paste the URL of your 'model.json' file that describes your data.
-
-4. Add the DataFlow version of conversationtranscipt to your Power BI report
-   1. In the Power BI report, select 'Transform Data' to open Power Query
-   2. Select Get Data - Power BI DataFlows. Select your new DataFlow
-   3. A copy of the conversationtranscript will now be added.
-   4. Select the original conversation transcript table in Power Query. In 'Applied Steps' select the top option 'Source'
-   5. In the command window, type '= conversationtranscript2' (or the name you have given the new table).
-   6. The orignal table will now point to the table coming from the DataFlow.
-
-5. Refresh the report.
-6. Note that managing the data in your DataLake will require additional attention, outside the scope of these steps.
-
 ## Using the report
 
 The report is based on what is shared through the Power Virtual Agents portal, but with some important differences: