CDS Virtual Entities with Dynamics 365 Finance and Operations
Hello the Community. Hope you are doing well !
Before really start to explain how it works and how this feature can help you, I will start by presenting a little the Virtual Entities in Common Data Service (CDS) ; since I know that a lot of people coming from Finance and Operations are not aware of this global Platform (Power) and was part of Dynamics 365 CE / CRM evolution. Also I suggest to read this other Article talking about Virtual Entities in CDS - by Carina M Claesson :
UPDATE 1st December 2021 : I will really suggest to go through also my new article - much more updated on this topic !
UPDATE 14th March 20213 : I will really suggest to go through also my new article - Again, much more updated on this topic !
What I like is also this schema that presents everything that are part of the Common Data Service :
So, now, what is a “virtual” Entity ?? What is the limitation in comparison of a standard Entity in CDS.
A virtual entity is a custom entity in Common Data Service that has fields containing data from an external data source. Virtual entities appear in your app to users as regular entity records, but contain data that is sourced from an external database, such as an Azure SQL Database / Web API or even here of course Dynamics 365 Finance and Operations. Records based on virtual entities are available in all clients including custom clients developed using the Common Data Service web services.
In the past, to integrate the disparate data sources you would need to create a connector to move data or develop a custom plug-in, either client or server-side. That was time consume, assuming also that you will need to SYNC, handle errors - well like any other integration project. However, with virtual entities you can connect directly with an external data source at runtime so that specific data from the external data source is available in an environment, without the need for data replication.
Virtual entities are made up of three main components, a data provider, a data source record, and a virtual entity. The data provider consists of plug-ins and a data source entity. The data source is an entity record in Common Data Service, which includes metadata that represents the schema of the connection parameters. Each virtual entity references a data source in the entity definition.
Common Data Service includes an OData Data Provider that you can use to connect with an OData v4 web service that accesses the external data.
Alternatively, you can can build your own data providers. Data providers are installed in an environment as a solution in the CDS and of course you can use it as an ALM process to deploy in each environments.
Schema by Microsoft
Virtual entity benefits
Developers can implement plugins to read external data using the Common Data Service web services and Plug-in Registration tool.
System customizers use Power Apps solution explorer to configure the data source record and create virtual entities that are used to access external data without writing any code.
End users work with the records created by the virtual entity to view the data in fields, grids, search results, and Fetch XML-based reports and dashboards.
But, of course there are some limitations. READ it carefully, it’s of course part of my conclusion at the end where I will talk about : Virtual Entities or CDS Dual Write for Finance and Operations. Considering these limitations, this will maybe decide for you which feature you will have to use ; depending on your business cases.
Considerations when you use virtual entities
Virtual entities have these restrictions.
Existing entities cannot be converted to virtual entities.
By default, virtual entities contain only a Name and Id field. No other system managed fields, such as Status or Created On/Modified On are supported.
Virtual entities don't support custom fields with the Currency, Image, or Customer data types.
Virtual entities don't support auditing.
Virtual entity fields can't be used in rollups or calculated fields.
A virtual entity can't be an activity type of entity.
Many features that affect entity table rows cannot be enabled with virtual entities. Examples include queues, knowledge management, SLAs, duplicate detection, change tracking, mobile offline capability, field security, Relevance Search, Portals for Dynamics 365 web portal solutions, and N:N relationships between virtual entities.
Virtual entities are organization owned and don't support the row-level Common Data Service security concepts. You will have to implement your own security model for the external data source !!!
Field metadata properties that validate on update don’t apply to virtual entities. For example, a Whole Number field on a virtual entity field may be set to have a minimum value of zero. However, since the value is coming from an external data source, a query will return values less than zero when retrieved from a virtual entity. The minimum value property is not implied in the query. You would still need to filter the values to be greater than 0 if that’s what is desired.
Virtual entities don't support change tracking and cannot be synchronized by using a Common Data Service feature, such as the Data Export Service.
Having say that…. :) let’s continue or maybe you can read my previous article in LINK where I talked about CDS Dual Write before really going deeper.
Of course this limitation is on CDS Virtual Entity and this will apply for FInance and Operations external data but even for other data sources you want.
Now let’s go to explain more this new Feature CDS Virtual Entities for Dynamics 365 Finance and Operations.
Finance and Operations is a virtual data source in Common Data Service, and enables full create, read, update, delete (CRUD) operations from Common Data Service and Microsoft Power Platform. By definition, the data for virtual entities doesn't reside in Common Data Service. Instead, it continues to reside in the application where it belongs. To enable CRUD operations on Finance and Operations entities from Common Data Service, entities must be made available as virtual entities in Common Data Service. The allows CRUD operations to be performed, from Common Data Service and Microsoft Power Platform, on data that resides in Finance and Operations apps.
Virtual entities for Finance and Operations apps
All Open Data Protocol (OData) entities in Finance and Operations are available as virtual entities in Common Data Service, and therefore also in Power Platform. You can now build experiences in customer engagement apps with data directly from Finance and Operations with full CRUD capability and without copying to Common Data Service. Power Apps Portals can be used to build external-facing websites that enable collaboration scenarios for business processes in Finance and Operations (like Supply Chain Management Template Portal)
Architecture
Virtual entities are a Common Data Service concept that is useful beyond Finance and Operations. The following illustration shows how the Finance and Operations provider for virtual entities is implemented. Six primary methods are implemented by the provider. The first five methods are the standard CRUD operations: Create, Update, Delete, Retrieve, and RetrieveMultiple. The last method, PerformAction, is used to call OData actions.
Calls to the Finance and Operations Virtual Entity Data Provider (shown as "VE Plugin" in the illustration) will cause a Secure Sockets Layer (SSL)/Transport Layer Security (TLS) 1.2 secure web call to the CDSVirtualEntityService web API endpoint of Finance and Operations. This web service then converts the queries into calls to the associated physical entities in Finance and Operations, and invokes CRUD or OData operations on those entities. Because a Finance and Operations entity is directly invoked in all operations, any business logic on the entity or its backing tables is also invoked.
Architecture schema by Microsoft.
During calls, there are two points of translation from Common Data Service to Finance and Operations apps. The first point of translation occurs in the VE Plugin, which translates concepts such as entity physical names into Finance and Operations entity names. It also converts some well-known concepts, such as Company references. The web service call still uses the EntityCollection, Entity, and QueryExpression objects to express the operations that are performed, by using the translated entity names and concepts from the VE Plugin. Finally, the CDSVirtualEntityAdapterService web API in Finance and Operations completes the translation from QueryExpression to QueryBuildDataSource and other internal Finance and Operations language constructs.
Before going to the How-To Setup (stey by step) please read it also the FAQ and Prerequisites
You will need a Finance and Operations instance on version 10.0.12 (coming GA in middle of July 2020)
You will need a Common Data Service instance in version 189 (or later) - v9.1.0.18940
Now let’s go !
UPDATE July 2021, you can install it directly in Power Platform Admin Center (PPAC) - select the right Dataverse (CDS) Environment, and Install the App (Dynamics 365 Apps) and search for “Finance Operations Virtual Entity”
Second, you will need to create an Azure Application in your Active Directory.
For that, go to your Azure Portal. Just pay attention, that you will need 1 app PER instance you want to activate the Virtual Entity feature.
The AAD application must be created on the same tenant as Finance and Operations.
Go to https://portal.azure.com > Azure Active Directory > App registrations.
Select New Registration. Enter the following information:
Name - Enter a unique name.
Account type - Enter Any Azure AD directory (single or multi-tenant).
Redirect URI - Leave blank.
Select Register.
Make note of the Application (client) ID value, you will need it later.
Create a symmetric key for the application.
Select Certificates & secrets in the newly created application.
Select New client secret.
Provide a description and an expiration date.
Select Save. A key will be created and displayed. Copy this value for later use.
The next steps walk through this process in Finance and Operations apps.
In Finance and Operations, go to System Administration > Users > Users.
Select New to add a new user. Enter the following information:
User ID - Enter cdsintegration (or a different value).
User name - Enter cds integration (or a different value).
Provider - Leave at the default value.
Email - Enter cdsintegration (or a different value, does not need to be a valid email account).
Assign the security role CDS virtual entity application to this user.
Remove all other roles including System user.
Go to System Administration > Setup > Azure Active Directory applications to register Common Data Service.
Add a new row.
Client ID - The Application (client) ID created above
Name - Enter CDS Integration (or a different name).
User ID - The user ID created above.
The next step in the process is to provide Common Data Service with the Finance and Operations instance to connect to. The following steps walk through this part of the process.
In Common Data Service, go to Advanced Settings > Administration > Virtual Entity Data Sources.
Select the data source named “Finance and Operations”. It’s part of what you had installed before with the CDS package Deployer tool and the solutions
Fill in the information from the steps above.
Target URL - The URL at which you can access Finance and Operations.
OAuth URL - https://login.windows.net/
Tenant ID - Your tenant, such as “contoso.com”.
AAD Application ID - The Application (client) ID created above.
AAD Application Secret - The secret generated above.
AAD Resource - Enter 00000015-0000-0000-c000-000000000000 (this is the AAD application representing Finance and Operations, and should always be this same value).
Save the changes.
Now the setup is finished yes !!
So, now you can activate & enable virtual entities you want ; enable your custom one (meaning you have done and creating a Custom Data Entity in Finance and Operations or even custom standard one like me)
Enabling virtual entities
Due to the large number of OData enabled entities available in Finance and Operations, by default, the entities are not available as virtual entities in Common Data Service. The following steps allow for enabling entities to be virtual, as needed.
For that you will have to stay on your CDS / CRM instance and open the “Advanced" Find” button. Like this :
Select the entity : “Available Finance and Operation entities”
I recommend before doing a filter on the name of your Finance and Operations Data Entities ; otherwise he will request the whole list and take some times.
You can even create custom saved views in order to see every Virtual Entities that are “Visible” to “True”
Like me I want to activate the PurchPurchaseOrderHeaderV2Entity to be visible as a Virtual Entity.
Click on it.
Like below, click on Visible to “true”.
Refreshing virtual entity metadata
The virtual entity metadata can be force-refreshed when it is expected for the entity metadata in Finance and Operations to have changed. This can be done by setting Refresh to Yes and saving. This will sync the latest entity definition from Finance and Operations to Common Data Service and update the virtual entity.
Maybe Microsoft will add soon a global button somewhere else because it’s a strange process. If we can have like in F&O the button Refresh Data Entity list, it will be better :)
So now here we are, you have your Data Entity of F&O that are visible in your CDS instance, as a Virtual Entity.
Before jumping on use cases, we can start to explain a little bit the process if you want to extend it and also the modeling of fields between CDS and F&O.
Referencing virtual entities
The virtual entities are all generated in the MicrosoftOperationsERPVE solution, which is API Managed. That means the items in the solution change as you make entities visible/hidden, but it is still a managed solution that you can take dependency on. The standard ALM flow would be to just take a standard reference to a virtual entity from this solution with the Add existing option in the ISV solution. It will then show as a missing dependency of the solution and be checked at solution import time. During import if a specified virtual entity does not yet exist, it would automatically be made visible without needing additional work.
To consume virtual entities:
Create a separate solution as usual in Common Data Service in your DEV environment as an Unmanaged solution, which will contain the consuming logic.
Select Entities > Add Existing. Select the virtual entity that you want to reference from the list.
When prompted to select assets to add, select any forms, views, or other elements that you want to customize, then select Finish.
From the development tooling, existing elements such as forms can be modified for the virtual entity. Additionally, new forms, views, and other elements can also be added.
Like me :
When the solution is exported (your custom one) and you want to want to import it - as Managed solution - in your Sandbox or Production CDS instances, it will contain hard dependencies on the virtual entity generated in the MicrosoftOperationsERPVE solution.
Now let’s talk about modeling :
Building an app requires capabilities to perform relational modeling between entities that are being used in the app. In the context of virtual entities, there will be scenarios where virtual entities and native entities in Common Data Service must work together to enable the desired user experience. This topic explains concepts of relational modeling that can be implemented using virtual entities for Finance and Operations.
For that again, you can read carefully the Microsoft DOC ; it’s very well explained ! Really suggest to understand well how it really works behind the scene, since as you now the data type fields (also company structure) are not the same between F&O and the CDS.
Now that we have done the Technical part, let’s talk of how you can use this feature :) - for business cases
The main one for me will be of course to avoid using Business Events in D365 F&O. Since you have your Virtual Entity as an external Data source of Finance and Operations, you can use PowerAutomate directly with the CDS Connector directly. It’s more near real time ; avoid also to build also a custom business event if you don’t have it in standard.
Be aware, it’s just for taking events in Update, Created or Deleted and not for inserting Data. Remember, Virtual Entity is on read-only state.
Like this, since I have activated the Worker Data Entity as a Virtual Entity in my CDS, I can trigger when it’s updated, add condition and of course like always doing a flow with more 300 connectors so… :)
Or course you can build also a PowerApps CanvasApp that are well supported (and huge priority) for Common Data Service as main data source.
So you can use your Virtual Entity, data coming from Finance and Operations, without using any connectors (premium)
Again it’s more for “Display” apps / data. But it’s good to know and more easier & really really faster in this way :
Finally, let’s talk about PowerApps Portal. In my previous article (3months ago) I talked about CDS DualWrite with Dynamics 365 Finance and Operations. First, the sync is not really in real time, sometimes you have to wait few seconds. Also, it will replicated all your DATA of your Entity between FInOps and the CDS. Sometimes, like in an external or internal portal, you just want to display information, that’s all. So in this case, Virtual Entity is really simple to activate and really simple to display in a PowerApps Portal.
You can achieve anonymous access or authentificated users.
I will not go deeper here, since I already do an article also on PowerApps Portal for Finance and Operations :
To conclude, hope you can of course have a better idea on why I had to choose CDS DualWrite or this new feature Virtual Entity for Finance and Operations.
Both are some limitations, both have their pros and cons.
For me, like if you want really to have Dynamics 365 Apps well connected, bidirectionnal (like between an F&O instance and a CRM/Customer Engagement instance or even Field Service) go for CDS DualWrite.
The CDS DualWrite is more complex to configure. Virtual Entity is really simple.
Virtual Entity is very good also if you don’t have a CRM license on your CDS ! Somehow, an Australian partner ask me how to achieve CDS DW but he didn’t have any CRM license and Entity mappings on CDS DW are really dependent on CRM entities.
So here if you want to start building a PowerApps Canvas App on top of CDS, with some DATA that are coming from F&O and you don’t want to replicate DATA,
Finally, you can also do both : CDS DW + Virtual Entities : depending on what you need !
Bye, stay safe and see you next time !
========
For English Community, here is our session with Rachel Profitt from Microsoft FastTrack team, in DynamicsCon 2020 about this feature !
For the Power French Community, here is my How-To Session in French on my Youtube Channel