This is the eighth blog post in this series. You can find the previous posts here:
· Innovations in Integration on the Microsoft Platform
· Lowering Barriers to Innovation: BizTalk IaaS and Walkthrough
· BizTalk Sever 2013 Enhancements
· Windows Azure BizTalk Services EDI Overview Including Portal Configuration
· Windows Azure BizTalk Services - Utilizing EAI Bridge
· Windows Azure BizTalk Services: Integrating with on-premises LOB systems using
the BizTalk Adapter Services
· ReST (Representational State Transfer) in BizTalk 2013
Companies that leverage Dynamics CRM Online want to take full advantage of its rich set of capabilities by integrating with existing lines of business systems that are available on premise. It is common nowadays for organizations to have other lines of business systems that are hosted on premise, behind the firewall.
In this initial blog, I will show one solution that is available for integrating CRM Online with BizTalk 2013 hosted on premise. In the first scenario, I will demonstrate how data can be sent from the BizTalk application to CRM Online.
BizTalk to CRM Online Integration
Fig. 1: Overall Architecture of main components
CRM Online exposes the Organization Service (IOrganizationService). This is a built-in CRM WCF service that is used to interact with CRM data and metadata.
I will be simulating creating a new account via BizTalk and show how to send it to CRM Online. In order to run this scenario, you will need the following:
- CRM Online account
- CRM SDK
- BizTalk Server 2013
Summary of steps (additional details and screenshots below):
1. Navigate to the CRM online organization instance to retrieve the OrganziationService URL.
2. Use BizTalk WCF Service Consuming wizard to generate schema from the Organization Service web service.
3. Create source schema (Account schema).
4. Map source schema to the CRM online schema generated in step 2.
5. Create orchestration (CreateAccount orchestration).
6. Create custom binding and behavior; register them in BizTalk.
7. Configure WCF custom send port.
8. Configure BizTalk.
9. Simulate sending message by dropping the account sample file in the predefined folder.
10. Log into CRM Online and verify that the account has been created.
Using Organization Service to interact with CRM Online Walkthrough:
Step 1 - Log into CRM Online to retrieve Organization Service URL;
Go to Settings - > Customizations -> Developer Resources
Step 2 – Use BizTalk Publish Wizard to generate the schema and the binding file.
In BizTalk solution right-click schema project, select Add and then Add Generated Items; select Consume WCF Service Wizard; navigate through the wizard and enter the URL of your Dynamics Organization Service endpoint ( the url discovered in Step1); after clicking the Get button, click ‘Next’ and select Import.
Step 3 – Replace the generated Organization Service schema with the BizTalk schemas that are provided via CRM SDK (\SDK\Schemas\CRMBizTalkIntegration)
Below is the organizationservice_schemas_microsoft_com_xrm_2011_contracts_services schema. Note that the schema provides CRUD operations in addition to the other CRM-related operations. The nodes that are being used to create a new account are highlighted below. The schema provided by CRM is flexible; it exposes a key-value pair; the key maps to the CRM entity attribute (i.e. name, address1, city, or phone). LogicalName maps to the CRM entity name (i.e. account).
Step 4 – Create account schema.
Step 5 – Map account schema to the expected CRM schema format.
Step 6 – Create orchestration in BizTalk. The orchestration is receiving the message via rcvAccountPort; it then uses Construct Message shape to map the message received to the expected Organization Service schema and sends it to CRM. The response message is captured and written to the file system.
Step 7 – Deploy BizTalk artifacts to BizTalk.
Step 8 – Configure OrganizationService send port.
Import the binding configuration (OrganizationService_Custom.BindingInfo) generated in Step-2. Rename the port from auto generated WcfSendPort_OrganizationService_CustomBinding_IOrganizationService_Custom to OrganizationService.
CRM Online is using Windows Live Id authentication. In order to configure the BizTalk send port to authenticate with CRM Online instance, a custom WCF behavior has to be built and registered in the Global Assembly Cache (GAC) and the configuration file (machine.config or in BizTalk WCF-Custom send handler).
Mikael Håkansson’s blog post provides a detailed explanation with the sample code for creating the custom WCF behavior. I leveraged it for this demo. See below the detailed steps necessary to configure the OrganizationService send port.
OrganizationService send port.
OrganizationService Send Port – General tab view.
Note that no changes have been made to the custom binding generated in Step 2.
OrganizationService Send Port – Binding tab view.
Select customBinding as the Binding Type.
OganizationService Send Port – Behavior tab view
I am using liveIdAuthentication custom behavior which requires the following three properties:
Step 9 – Start BizTalk application.
Drop the sample file that contains account-related information in the predefined incoming folder.
Verify in CRM Online that the account has been added.
Verify the response is received in the expected file system folder.
The response contains the account id which was automatically generated when the new account entity was created in CRM.
The scenario that was presented above used the WCF SOAP Organization Service to integrate with the CRM Online application. Another option typically used for interacting with CRM Online is for BizTalk to leverage a proxy service which in turn uses CRM SDK API to authenticate and communicate with CRM; crmsvcutil tool is part of CRM SDK and can be leveraged to create early bound entity CRM classes that can be used in the proxy service. The proxy service approach is recommended for more complex integration scenarios.
This is the third blog post in this series. You can find the first two posts here:
Innovations in Integration on the Microsoft Platform
Lowering Barriers to Innovation: BizTalk IaaS and Walkthrough
Microsoft BizTalk Server unites enterprise application integration (EAI) and business-to-business (B2B) integration. BizTalk Server is a mature product that has been around for over a decade with a new release every 2-3 years. The picture below shows the main features and enhancements made in previous releases.
According to statistics provided by Microsoft, BizTalk is the most deployed product in its category and is used by 81% of Fortune Global 100 companies.
BizTalk Server 2013 was released in March and features enhancements that were influenced by a combination of industry trends and customer feedback.
At a high level, these enhancements can be grouped into the following categories:
1. Running in the Cloud
2. Connecting to the Cloud
3. Simplifying the Experience
4. Improving Performance
5. Supporting the Latest Platforms and Standards.
Running in the Cloud
BizTalk Server 2013 allows you to run BizTalk Server in an Azure Infrastructure as a Service (IaaS) environment. This can reduce hardware procurement lead times and help reduce the time and cost of setting up and maintaining your BizTalk environments. You can also move applications from on-premises to Azure and back. Refer to Lowering Barriers to Innovation: BizTalk IaaS and Walkthrough for more information on IaaS.
Connecting to the Cloud
BizTalk Server 2013 includes out-of-the box adapters that send and receive messages from Windows Azure Service Bus, simplifying the task of building hybrid applications. BizTalk Server 2013 also provides adapters that invoke REST endpoints and expose BizTalk Server artifacts as RESTful services using cloud adapters like WCF-BasicHttpRelay, WCF-NetTCPRelay and SB-Messaging.
BizTalk Server 2013 also includes an enhanced SharePoint adapter, which makes integrating with SharePoint as simple as integrating with a file share. BizTalk Server 2013 also supports the Azure Access Control Service, which enables customers to move their EDI and EAI based solutions to the cloud.
Simplifying the Experience
BizTalk Server was originally built to use designers and configuration to minimize code writing, and additional investments have been made in this release to make BizTalk Server even easier and more user-friendly.
For instance, the dependencies between artifacts can now be viewed and navigated in the BizTalk Admin console using the Dependency Explorer, which allows you to navigate your solution and find dependency between different artifacts. This helps you easily identify how changes you make might impact other artifacts. The BizTalk Administration Console pictured below shows the view dependencies option and the full dependency information corresponding to the selected artifact.
The ESB capabilities previously introduced in the ESB Toolkit are now fully integrated with BizTalk Server, and the ESB configuration experience is vastly simplified to enable a quick setup.
Integrating with SharePoint using BizTalk Server 2013 is now as simple as integrating with a file share. The dependency on SharePoint forms has been removed, while still providing backward compatibility. You can find more information about the SharePoint Services Adapter here. The picture below shows the configurable properties in the SharePoint Services transport.
BizTalk Server 2013 now comes with out-of-the-box support for SFTP, enabling the sending and receiving of messages from an SFTP server. In the past, users had to either develop their own SFTP solution or use 3rd party adapters. The picture below shows the configurable properties in the SFTP transport.
BizTalk Sever 2013 supports host handler association of dynamic send ports. In past releases, all dynamic send ports executed in the adapter’s default host. Since there was only one default host for an adapter, all messages were routed using the same host, decreasing performance. With BizTalk 2013, it is possible to configure the adapter’s send handler. You can find more information about the dynamic send port handler here. The picture below shows the dynamic send port handler configuration.
Minimal Lower Layer Protocol (MLLP) is the absolute standard for transmitting HL7 messages via TCP/IP. BizTalk Server’s MLLP adapter is widely used by hospitals and medical clinics throughout the world. In the 2013 release, Microsoft has made performance improvements to the MLLP adapter. Tests conducted revealed a performance improvement of up-to 5 times.
The mapping engine in BizTalk Server 2010 and prior releases makes use of XslTransform API for mapping needs. The transformation mapping engine in BizTalk 2013 makes use of the enhanced XslCompiledTransform API, providing performance enhancements. Once the load method completes successfully, the transform method can be called simultaneously from multiple threads. The new XSLT processor compiles the XSLT style sheet to a common intermediate format, and once the style sheet is compiled, it can be cached and reused.
Supporting the Latest Platforms and Standards
BizTalk Server 2013 supports the following platforms and standards:
If you would like to learn more regarding the concepts covered in this post please visit these resources:
· Windows Server 2012, Microsoft Visual Studio 2012, Microsoft SQL Server 2012, Microsoft System Center 2012, the latest version of Microsoft Office
· SAP 7.2 and 7.3, Oracle Database 11.2, Oracle E-Business Suite 12.1, Oracle Siebel 8.1
· Health Level Seven (HL7) 2.5.1 and 2.6
· Society for Worldwide Interbank Financial Telecommunication (SWIFT) 2012 Message Pack
· X12 5030, EDIFACT D05B
If you would like to learn more regarding the concepts covered in this post please visit these resources:
In addition, there are several differences between installing BizTalk Server on the 32-bit and 64-bit editions of Windows. Considerations while installing BizTalk Server 2013 can be found here. Full list of hardware and software requirements can be found here.
If you’d like to learn about our Connected Systems practice at Neudesic, please visit this page: http://www.neudesic.com/What/Expertise/Pages/ConnectedSystems.aspx
We had an Observation when trying to move an AS2 send port across the Applications.
There was an application AS2Test which had 3 AS2 Send ports.
All the 3 send Ports were configured with AS2 Send Pipeline.
AS2Send ::Send Port with AS2 Send Pipeline
AS2MDNSend ::Send Port with AS2 Send Pipeline
AS2TestMDN ::Send Port with AS2 Send Pipeline
Along with this there were 2 applications DemoAS2 and NeudesicBTDF.
DemoAS2 :: This application was having referance to Biztalk EDI Application (the application which has AS2 Pipelines)
NeudesicBTDF:: This application was not having referance to Biztalk EDI Application (the application which has AS2 Pipelines)
we tried to move the send port AS2Send in the application AS2Test,by right clicking on the port in AS2Test Application and selected the option of Move to application
After selecting the Move to application,the destination application name as NeudesicBTDF was selected from the drop down.
We saw that it had all the send ports and receive locations of the AS2 Pipelines listed even though we have selected only one send port i.e AS2Send send port for movement.
In contrary if when we tried to move the AS2Send port to the DemoAS2 application this just listed the AS2Send port selected.
when we have to move any AS2 related send ports to a different application, please make sure the target application has necessary the AS2 pipeline reference’s.
Thanks to all that attended the integration roadshow today! As promised, here is the deck I presented for the last session around Building Hybrid Composite Services Using AppFabric and BizTalk Server 2010. The roadshow started exploring the integration platform and many of the new aspects of Windows Server AppFabric and Windows Azure AppFabric. BizTalk is still a critical part of the integration platform. If you are only using BizTalk though, you are missing out on some great new aspects of the Microsoft platform to build out your solutions. The roadshow is still touring the US so make sure to attend if you still can!
Microsoft Business Integration Roadshow Calendar
Building Composite Enterprise Hybrid Services with BizTalk 2010 and AppFabric Simple.pdf (2.97 mb)
The first sample I put together for creating a message service was fairly simplistic and was missing a few key features necessary to put together a truly valuable message service for the ESB Toolkit. Here are the pieces that were necessary and have been included in this new sample.
The message service built in the original sample had a status property that should not be hardcoded, but configurable based on where the itinerary step is located. This seems like a pretty common need of message services or even orchestration services to have resolver values outside of those provided by the included resolvers. The included resolvers are based on routing or transforming which makes sense, but what if you need other properties? The resolver framework exists to provide information to itinerary services which do the actual work, but it is just a dictionary. What information and how the resolver gets it in there is not limited.
In this case I just wanted something like the STATIC resolver, but based on the values I need for the message service to run. In this case the only property I cared about was a Status of the Order to track to my BAM tables. To do this I put together two potential resolvers. The first is called the ProperyCollectionResolver and the other is the PropertyResolver. I will go into details in the next section on why there are two.
The final piece was to add support in the message service for the resolver. This entails a few steps of invoking the resolver framework to execute and receive the resolver dictionary. This was added to the base BAMMessageService to avoid repeating this in extension classes. Below is the snippet that parses the resolver string and then invokes the resolver framework.
The designer really is the best new feature of the Toolkit from my point of view. I worked on a lot of the core features in the ESB Toolkit around itineraries and thought the changes we made were good, but the designer really adds a significant productivity boost. Key to that productivity is validation, direct support for the ESB engine framework artifacts (resolvers, itinerary services, adapters) and integration with BizTalk. The message service from the original sample achieved some designer support just by registration in the configuration file, but now there are two new resolvers.
The next step was to give the resolvers designer support as well. Using the Designer Extensibility sample as a guide, I included full support for the resolvers. So the question is why two resolvers that basically do the same thing? Both resolvers add whatever key-value pairs are in the resolver string to the resolver dictionary. The reason is pretty simple. The PropertyCollectionResolver just exposes a simple dialogue to add key-value pairs. This could be used with any message service or orchestration created to provide values to the itinerary service. The problem is that there is no design time validation other than correctly formatting the resolver string (which by the way could be a pain in the past when manually creating the itinerary). The advantage is that this can be used right away with any itinerary service. Below is the UI presented for the property collection which takes anything added.
The PropertyResolver does not have a built-in designer support. To correctly utilize it, resolver extenders based on the values the itinerary services are expecting must be added. The advantage is that you can potentially add properties as enums or include other validation with the built-in Enterprise Library Validation block support. The problem is that potentially each itinerary service would require a resolver extender based on the properties it will need. Below is a sample of the properties presented for the resolver extender built specifically for the OrderTrackingMessageService.
Another extension point not well known in the ESB Toolkit is to implement your own itinerary manager responsible for making the right things happen when advancing the itinerary, serializing/de-serializing the itinerary, etc. Implementing your own itinerary manager is not something easily accomplished and probably not a recommended decision. This does enable some interesting possibilities though. The original message service sample was driven by posts on the ESB Toolkit forums about logging to BAM. In the ESBExtensions solution is the ItineraryV2 which extends the base ItineraryV1 from the Toolkit. The purpose of ItineraryV2 is to potentially provide your own BAM logging on itinerary advance. I did not implement this specifically but provide instructions on how to use this itinerary manager and also a snippet on where/how to log BAM data.
I have included installation instructions in the base directory of the sample in a text file called InstallInstructions.txt. Also make sure to unzip the sample to the same directory as your other ESB Toolkit samples. The strong name key file path is the same as other samples which is required to build the project. If you do want to run the tests make sure the resolvers and message service are registered in the esb.config file.
This sample gives a good practical example of the various extensions points in the ESB Toolkit. Please see the zip file with the sample below. I hope to have something on extending adapter providers soon as well!
ESBExtensions.zip (507.66 kb)