get on the bus

The average BizTalk solution has a large number of DLLs that need to reside in the Global Assembly Cache (GAC). As a result, deploying the BizTalk solution can consume more cycles than is necesary. The time factor becomes apparent during troubleshooting (debugging) BizTalk, when the solution must be frequently rebuilt, redeployed, and restarted.

The problem described may be resolved by removing BizTalk DLLs from the GAC and placing them into a common folder. Then the runtime section of the BTSNTSvc.exe.config must be updated to include dependent assembly references to BizTalk DLLs residing in the common folder. Below is the fragment of an updated BTSNTSvc.exe.config runtime section.

<runtime>

 <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">

<probing privatePath="BizTalk Assemblies;Developer Tools;Tracking;Tracking\interop" />

<dependentAssembly>

<assemblyIdentity name="Solution.Project.Module" publicKeyToken="909dd514af6267cd" culture="neutral" />

<bindingRedirect oldVersion="1.0.0.0-9.9.9.9" newVersion="1.0.0.0" />

<codeBase version="1.0.0.0" href="file:///C:\PathToBizTalkDlls\Solution.Project.Module.dll" />

</dependentAssembly>

</assemblyBinding>

</runtime>

This approach can increase development speed and reduce the cost to maintain a BizTalk solution in a development environment. While this is not generally recommended for production environments, we do have clients that have a hard “no GAC” policy; yes, even in production. As such, those customers must have a mechanism for running a BizTalk solution sans GAC.

(This article was first published in BizTalk Hotrod)

The Distinguished Field is often seen as the weaker of the two types of Fields when handling Fields in BizTalk.

After all, the Distinguished Field can't be used as a filter on a message, and it's slower than its big brother the Promoted Field.

Well, today I'm here to dispel the myth of the wimpy Distinguished Field and place in the pantheon of power that equals, and in some ways exceeds the Promoted Field.

MYTH: Getting the value of a Distinguished Field requires loading the entire message into memory.

The first myth that we need to dispel is that the Promoted Field is a quicker field to access than the Distinguished Field.

This is due to the statement in the BizTalk Documentation, and I quote:

The BizTalk Server Message
… Lots of stuff cut out …
"One of the benefits of promoted Fields is that the value of the element that is promoted is available in the context of the message. This means that retrieving that value is inexpensive, as it does not require loading the message into memory to execute an XPath statement on the message."

What is implied here is that for the Promoted Field reading its value doesn't require an XPath read into the message and conversely, that the Distinguished Field is does require loading the message and has a performance cost because it's evaluated when queried.

Nothing could be further from the truth! In fact, the both the Promoted and Distinguished Fields are evaluated at the same time, and both are placed in the message context at the same time. So, let's talk about how fields get into the message context.

About BizTalk Message Context Fields
… Lots of stuff cut out...
"Since Distinguished fields do not require a separate property schema, the evaluation of Distinguished fields by the Orchestration engine consumes less overhead than the evaluation of Property fields by the Orchestration engine. The evaluation of Property fields requires an XPath query, the evaluation of Distinguished fields does not require an XPath query as the pipeline disassembler populates the Distinguished fields in the context and the orchestration engine gets the cached values. However, if the orchestration engine does not find the property in the context, it will then evaluate the XPath query to find the value. Distinguished fields do not have a size limitation.

Now, how does Promoted and Distinguished Fields get into the Message Context? This occurs automatically in the Receive Pipeline by certain pre-built pipeline components?

Out of the box, the BizTalk XML Disassembler, BizTalk Flat File Disassembler and the BizTalk Framework Disassembler Promote Fields to the message context. All other production level Pipelines promote fields, most also support Distinguished Fields.

Distinguished Fields are written to the Message Context if one of these Receive Pipeline Components is used in the Pipeline. Interestingly enough, this explains why the Passthrough pipeline doesn't promote Fields from the message, there are no components in the Passthrough Pipeline, it does nothing to the message content and therefore nothing gets promoted, especially BTS.MessageType.

As far as performance, Distinguished Fields beat out Promoted Fields 9 days each week. This is because both Promoted and Distinguished require the same overhead of writing the message value to the context Field bag in the Message Box, but Promoted Fields have the additional overhead of both being written to the Message Box context database AND the Subscription database. Promoted Fields have an impact every time a message is written to the Message Box because each Promoted Field that exists musts be evaluated in a massive union (very efficiently written union mind you!) that builds the list of matching activation subscriptions. So in short, the more Promoted Fields that you have the costlier the subscription process.

RECOMMENDATION: Use Promoted somewhat sparingly, don't avoid them, but do not use them if you do not need to. Use Promoted Fields as they were designed, to facilitate message routing, but not to make it easy to access a message value. Instead primarily use Distinguished Fields.

MYTH: Its always safe to use a Promoted or Distinguished Field in an Orchestration.

Using Operators in Expressions

exists    test for the existence of a message context property    BTS.RetryCount exists Message_In

Let us talk about how to handle message content that is missing when it is a Promoted Field and a Distinguished Field. What we are talking about specifically is the field that was Promoted or Distinguished did not exist in the inbound message. The XLANG/s xpath statement that was used to query the message for the content during pipeline processing returned a null object.

The first thing to understand is when a Promoted and Distinguished Field comes into existence. They are essentially the same, and this occurs when a Pipeline component parses a message and either Promotes or Writes the value to the context. The simple answer is, when the value does not exist, the Field is not created. A query to the context for the Field returns a null object.

So, if you attempt to access a Promoted or Distinguished Field that didn't exist in the inbound message, you can cause an unhandled exception to be thrown. Specifically in both cases a NullReferenceException.

Promoted Fields, have a special XLang/s test exists (see my previous blog post in this) that you can use to determine if they exist before attempting to use them. In this case, Promoted Fields can always be tested for existence before use and can safely be avoided when they don't exist.

Unfortunately, Distinguished Fields don't have such a special test, and can cause an unpreventable unhandled exception. Specifically if you use a Field that the underlying type is a native non-nullable type. For instance, suppose the value that you have distinguished is a integer. Integers cannot be null (and yes, I am aware of the Nullable<T> generics, but we are talking about what BizTalk XLANG/s has, not what is C# has) and if the underlying value didn't exist and you attempt to use the value, or even test to see if the value exists will cause an unhandled NullReferenceException when BizTalk's XLang engine attempts to convert a null value into an integer by calling the System.Number.Parse(string) method with a null value.

Here comes in the kicker and why a Distinguished field can appear to be fine at design time, but bite you at run-time.

At design-time the expression editor generates a pseudo class-like dotted object for you to use in your expression. At run-time there is simple type-casting that occurs by the run-time engine that inspects the XML datatype of the node in the Schema, retrieves the value as an object… then attempts to call the appropriate ConvertTo method on the Object. When casting a Null to an Int32 or any other intrinsic datatype, a NullReferenceException is thrown and the Orchestration fails.

The primary difference (excluding Routing) between Promoted and Distinguished Fields is the developer design-time experience. Distinguished Fields are easy to use because they emulate .Net Class dotted notation.

RECOMMENDATION: If there is any chance that accessing the Distinguished Field may cause an exception, then place the check in a Scope Shape that has a catch shape to handle the NullReferenceException.

MYTH: Distinguished Fields are only accessible in Orchestrations

WRONG Documentation: Field Schemas
RIGHT Documentation: Distinguished Fields in Disassembler Pipeline Components, Processing the Message, Promote Properties (Node Property of All Schemas)

Another major fallacy about Distinguished Fields is that they are only accessible in the Orchestration. This is also untrue, the BizTalk Server Documentation clearly has an example of how to use Distinguished Fields in any component from the RIGHT Documentation above.

All Distinguished Fields outside of an Orchestration use a fixed schema:
http://schemas.microsoft.com/BizTalk/2003/btsDistinguishedFields

The Field to use is the XPath of the node that is Distinguished such as:
/*[local-name()='PO' and namespace-uri()='http://SendHtmlMessage.PO']/*[local-name()='Price' and namespace-uri()='']

Thus to access this you would use the Read Method:
MyObject = MyMessageContext.Read("/*[local-name()='PO' and namespace-uri()='http://SendHtmlMessage.PO']/*[local-name()='Price' and namespace-uri()='']", " http://schemas.microsoft.com/BizTalk/2003/btsDistinguishedFields");

If the Field exists, then MyObject will contain an object that can be cast to the appropriate type.

RECOMMENDATION: Once a the proper Pipeline Component has processed the message, use the Distinguished Field as you would any Field without the Xpath lookup overhead.

MYTH: Distinguished Fields in Orchestration Expression shapes are actually code.

You have to hand it to the people who did the coding for XLANG/s. It looks like C#, it feels like C# and 99% of the time it pretty much generates standard C#.

In many ways, this is not your father's C#, it is really XLANG/s and it has it's own syntax and special components. Distinguished Fields are a prime example.

Think back on all the times you used a distinguished Field. It feels like it's a C# Object! It uses dotted notation (Node.Node.Node.Attribute). You assign values to it, you use it's value in an expression and it comes out as the correct type. When the node is Boolean, then it behaves like a Boolean. Nothing could be further from the actual behavior as Marty learned recently. Just because it looks like a duck, doesn't mean that it's a duck. It really is a trick, that the Expression Editor parses the XSD on the fly and generates a classlike editor experience, but no actual code ever gets generated.

Further Reading

1. Planning and Architecture > BizTalk Server Architecture > Runtime Architecture > The BizTalk Server Message
2. Planning and Architecture > BizTalk Server Architecture > Runtime Architecture > Processing the Message
3. Developing BizTalk Server Applications > Creating Pipelines Using Pipeline Designer > About Pipelines, Stages, and Components > Distinguished Fields in Disassembler Pipeline Components
4. Creating Orchestrations Using Orchestration Designer > Creating Orchestrations > Using Expressions in Orchestrations > Using Operators in Expressions
5. Creating Schemas Using BizTalk Editor > About Schemas > Different Types of BizTalk Schemas > Property Schemas
6. Creating Schemas Using BizTalk Editor > About Schemas > Ways to Use Message Content to Control Message Processing > About BizTalk Message Context Properties
7. Schema Property Reference > Node Properties - Alphabetical Listings > Node Properties of All Schemas > Promote Properties (Node Property of All Schemas)
8. Creating Schemas Using BizTalk Editor > Creating Schemas > Promoting Properties > How to Copy Data to the Message Context as Distinguished Fields

WCF LOB Adapters are a new technology for integration with back-end systems and applications. WCF LOB adapters will supersede the BizTalk Adapter Framework. If you're running an ESB powered by Neuron, you can use these adapters too. Here's how to configure your bus to talk to a back end system using a WCF LOB adapter.

Step 1: Add a New Binding

Step 1 is to define a new binding to Neuron. A WCF LOB adapter is exposed to consumers as a new transport binding.

• Launch the ESB Explorer, go to the Services > Bindings area, and click New to add a new binding.
• Click the Browse button and navigate to the WFC LOB adapter assembly you want to support. The ESB Explorer will automaically locate and set the class name of the binding.
• Save the new binding under a meaningful name.

Step 2: Add a New Service

Step 2 is to define an endpoint for the WCF LOB system. The WCF LOB adapter makes this look like a service.

• Go to the Services > Endpoints area and click New to add a new service.
• On the General tab, enter the access information for the service including address, binding, and contract. Select the binding you added in Step 1.
• On the Security tab, if the service is secure, enter any credentials required such as username and password.
• On the Ports tab, set up a Send Port to route message traffice to the back end service. Assign a subscriber Id and topic to use for this purpose.

Step 3: Start Messaging

At this point you're really done. Clients can now send messages to the back end service and receive replies. For successful communication check the following:

• Messages need to use the proper schema and expected content
• Messages need to specify support Action values

A technique I used with the samples in the WCF LOB SDK is to turn on WCF diagnostics and message logging when running the test client that comes with the samples. When viewing the logged messages with SvcTraceViewer, I now have examples of valid request and reply messages. Then I can use the .NET test client to send requests and verify that all is in working order.

Ok, so this isn't really too much of a big deal, but Eric Stott found a funny misspelling in the Import MSI Wizard which reminded of something I had noticed.

If you import or export an MSI and during the progress phase click the Progress item on the left list.

In the Export Wizard the "Results:" is blank and the top of the output says "label3"

In the Import MSI Wizard the "Results:" is some generic value and the top says "label1".

What is your easter egg?

List of enhancements to the HL7 Accelerator experience in BizTalk Server.

1. The BizTalk HL7 Accelerator (BTAHL7) is non-compliant with the HL7 2.4 (and 2.3.1) Specification 2.10 in the following manners:
   1. Step 1 Message Construction (serialization):
      1. b.2 if the [field] value is not present, no further characters are required
      2. b.5.iv components that are not present at the end of a field need not be represented by component separators
      3. b.6.iv subcomponents that are not present at the end of a component need not be represented by subcomponent separators
         1. In all of the above cases, the requirement is that the object with no value does not require a delimiter for it, but is certainly gives the implementer the latitude to include it. Incorrectly, BTAHL7 throws a parse error if a sending system sends delimiters at the end of a value that are empty.
         2. The desired behavior is that BTAHL7 quietly accepts any empty field at the end of any object in the system. This behavior is over ridden using a pipeline property or on a party by party basis, but by default the behavior is breaking.
         3. EXAMPLES:

         Field:

         Case 1: Field is not last field in segment:
         NTE|| -> This will cause the parse to fail unless it is the "Allow Trailing Delimiters" flag is set.

         Case 2: Field is last field in segment:

         NTE||| -> This will always fail because BizTalk attempts to parse the extra field in the segment into a non-existent field in the segment.

   2. Step 2 Converting Messages to data values (deserialization):
      1. ignore segments, fields, components, subcomponents, and extra repetitions of a field that are present but were not expected.
         1. BTAHL7 does not ignore any segments, fields, components, subcomponents or extra repetitions if they are not expected. Instead a parse error is thrown and the message is rejected.
         2. The desired behavior is that the HL7 Accelerator retains the extra data for transmission but does not capture the data for processing; additionally this should not cause a parse error.
      2. treat segments that were expected but are not present as consisting entirely of fields that are not present
         1. BTAHL7 does handle this correctly, but the error messages are generally inscrutable if the captured data does not contain required objects.
         2. The desired behavior is that BTAHL7 clearly points to the error.

   The overall impact of the non-compliant behavior is that the HL7 accelerator places a tremendous implementation burden on the implementer. Instead of concerning themselves with values of interest in the message, they spend a tremendous amount of time matching the current system requirements to the HL7 Accelerator defined specification. This increases the implementation time, reduces the flexibility of the solution and in general, makes any BizTalk solution extremely expensive.

2. The BizTalk Server model differentiates between various participants in the entire BizTalk Process.
   1. Business Analyst (BA)
   2. Solution Developer (SD)
   3. Operations Manager (OM)

   The HL7 Accelerator does not match this model. I believe that this lack of separation has made the entire HL7 Accelerator solution difficult to implement and complete solution engaging all of the roles that are involved. The BA has to provide an HL7 Solution definition that does not reflect the desired solution, for instance, they create a spread sheet or a word document that defines the message structure very loosely. There is no was for the BA to tell the SD what message definition and variations are required from the HL7 Standard. The only way to define the message is to directly edit the Schema in the BizTalk Schema editor. The only way to define transformations from Message Schema A to Message Schema B is in the BizTalk Mapper. The BA is completely disassociated from the definition process. They have to communicate via documents (Word or Excel) and then test the results using sample messages. The OM has control over the solution in such a way via the HL7 Configurator that should only be defined at the Development level. This disconnect makes it difficult for the appropriate roles to be applied and frustrates the users at each level.

3. The BizTalk Accelerator solution is difficult to debug and troubleshoot. The Developer has to deploy the schemas and pipelines and then manually submit the messages.
   1. Cannot test Schemas and Maps in the IDE you must deploy the entire solution to validate them.
   2. The HL7 Tables/Segments/Datatypes always have a global impact and are difficult to separate out. The relationship between them is not obvious and the XML editor is extremely difficult to master. This is resolved by using namespaces to separate the schemas, but even that solution is extremely complex.
   3. Editing of table values is difficult and not obvious. There is no way to easily import BA defined values into the table definitions.
   4. The MLLP Test tools are difficult to use and inflexible and seem to be as if they were written at the last minute. When errors are encountered, the tools simply throw an undecipherable exception message instead of reporting a helpful error message.
   5. The MLLP Test tools cannot help the solution test high volume message loading. The tools allow you to either send a single message repeatedly very quickly, or you must simply rerun the tool in some kind of loop in the console which processes relatively slowly.
4. The HL7 Accelerator does not expose any helpful .Net Classes to help decode/encode and handle message content.
   1. The HL7 DateTime field does not map exactly to a .Net class. For instance, the HL7 DT field has 4 digits to the right of the decimal point (ss.mmmm) and the .Net DateTime Type only had 3 (ss.mmm) and so either you must accept a loss of precision (not acceptable for certain lab testing results where the value is a critical measurement) or construct a custom .Net HL7 DateTime datatype to hold this value, deserialize/serialize it and handle it.
   2. The HL7 Serialization/Deserialization does not allow you to choose to accept invalid messages. In other words, the user may want to accept a message that will parse, and then run a second pass on it to validate the data. To do this requires two schemas to be deployed, a loose schema for initial validation and a second tighter schema that must be called to revalidate the message for deeper content validation. It would help that you could turn off validation at the field level using configuration and then determine at which pass the validation will occur.
   3. Promoting message content properties is difficult. For instance, there are many message properties that clients almost invariably want to monitor. Patient Information, Lab Test information, MSH Tracking fields like MSH7 and MSH10 to correlate the message back to the sending system and Provider information. It is certainly possible to build an XPath based property promoter and construct a Property Schema to do this, but pipeline components are always problematic and considering the likelihood that client will want to do this, it seems that a simple pre-built component should already exist and be inside the HL7 Deserializer/Serializer to handle this.
5. The HL7 Accelerator and BizTalk do not work well together to allow multiple message types that have very similar structure to be routed to a single orchestration and then sent to end points for processing.
   1. The ADT and ORM Messages have many trigger types, but need to be handled similarly. To process the messages the implementer must create many send ports for each message type, but they all go to the same physical send port. Every change to the Orchestration send port, typically requires the implementer to make as many as numerous port changes in a single orchestration. Also, the Send Orchestration ended up being extremely complex to handle all the various message triggers for the same message.
   2. The HL7 Send Ports cannot handle System.Xml documents. The message context contains the schema to use, but the Send port cannot handle the generic message type and cast it to the correct message type. This means that every send port in an orchestration must be strongly typed.
   3. The MLLP Send Port does not allow dynamic send ports making it difficult to use a single send port to send to multiple end points programmatically.
   4. Because of the limitation on the System.Xml documents, it is not possible to use some of the more advanced features in BizTalk like dynamic mapping. Thus all messages must be strongly typed and constructed.
6. The System does not interact with some of the HL7.org tools and schemas well.
   1. HL7.org provides a powerful tool that allows a Business Analyst to modify, design, diff and export to RIM HL7 Message definitions. Just looking at the feature set of Message Work Bench (http://www.hl7.org/), at a minimum this tool is a tremendous guide to what kind of features a Business Analyst would need.
      1. Diff two schemas
      2. Capture a message and generate a best guess at a schema
      3. Capture a message and verify conformance to a user defined schema
      4. Extensible Table editing for HL7 conformance values
      5. export of a BA Defined schema in xsd format that is not compatible with the BizTalk Schemas.
      6. HTML (XML with XSLT) generation of a readable message specification
      7. Innumerable reports about a message definition
      8. Capture a message and insert sample values from the message into a specification.
7. Users define custom schemas that have both subtle and widely variant message definitions in comparison to the specification
   1. Make it easier to capture a running message stream and have a wizard that automatically picks a "best matching" schema and adjusts the schema automatically to the actual message data. So that during the design phase, a BizTalk solution could be running in learn mode and then as it runs reports what messages that have failed and make it easy for the implementer to update the schema to allow those messages.
   2. Define reporting tools and mechanisms that update the schema on the fly, suspending messages and then letting the Operations team report and then adjust the schema and resubmit the message automatically.

I've been working on understanding the BizTalk Security model for quite a while and I keep working my Visio diagram over and over.

For BizTalk Server 2004 simply do not use the Level 0 Security Membership column.

Please feel free to send me any feed back if this is not clear.

NOTE:

The variable PropExists as bool has been already created

The Property of interest is BTS.RetryCount

The Message is Message_In

The list from Using Operators in Expressions (http://msdn2.microsoft.com/en-us/library/aa561847.aspx) has the typical list of stuff that you expect in C#, multiplication, bit operations (shift left and right) and Boolean operators, but a couple of extremely useful constructs are available that are unique to BizTalk.

The most important of these (in my humble opinion) is the exists operator.

As you are all aware, to even check whether a property exists in an expression throws an exception… as in the following case:

PropExists = (Message_In(BTS.RetryCount) != null && Message_In(BTS.RetryCount) != "");

If BTS.RetryCount does not exist in the message context, then the MissingPropertyException (Windows Event Log Event ID 10019) is thrown.

Without having to resort to a scope shape and exception handler, the exists operator allows you to check if a property exists in a message and is used in the following format:

PropExists = BTS.RetryCount exists Message_In;

OR

if (BTS.RetryCount exists Message_In)

{

     …;

}

Conclusion:

Using the XLANG/s exists operator in your orchestration allows you to test for the existence of a property in a message without having to resort to a scope shape and exception handler.

Below are a few of more XLANG/s functions that can provide some value in your Orchestrations:

Applies to: BizTalk Server 2006 with the HL7 1.3 Accelerator

1. Outline of the problem
   1. Trailing Delimiters are empty values at the end of an object in a HL7 ER7 formatted message.
   2. Examples:
      1. Empty Field
         1. NTE|P|
         2. NTE|P||
      2. Empty component
         1. ORC|1|725^
      3. Empty Subcomponent
         1. ORC|1|||||27&
      4. Empty repeat
         1. OBR|1||||||||027~
   3. Trailing delimiters indicate the following object exists and is empty, which is quite different from null, null is an explicit value indicated by a pair of double quotes -> "".
   4. The BizTalk HL7 Accelerator by default does not allow trailing delimiters.
2. There are three methods to allow trailing delimiters.

   NOTE: All Schemas always allow trailing delimiters in the MSH Segment

   1. Using party identifiers
      1. MSH3.1 – Receive/inbound processing, using this value as a party allows you to configure the system to allow inbound trailing delimiters.
      2. MSH5.1 – Send/outbound processing, using this value as a party allows you to configure the system to allow outbound trailing delimiters.
      3. Generally, if you allow inbound trailing delimiters, unless you are willing to programmatically remove all trailing delimiters, then you need to configure the send to allow trailing delimiters.
      4. Add the appropriate parties to the BizTalk Parties list from these two fields in your message stream.
      5. Open the BizTalk HL7 Configuration tool and for each party check the "Allow trailing delimiters (separators)" check box on the Validation tab.

      Disadvantage – Each MSH3.1 and MSH5.1 value must be represented in the parties list and configured.

      Advantage – granular control over system behavior for each inbound/outbound system.

   2. Using instance properties of a pipeline used in a send port or receive location.
      1. Open the BizTalk Server Administration console
      2. locate the send port or receive location that contains the BTAHL72XReceivePipeline or BTAHL72XSendPipeline pipeline.
      3. Open the properties
      4. To the right of the pipeline selected locate the […] ellipses button
      5. In the property list, locate the "TrailingDelimiterAllowed" property and set it to True.

      Advantage – All messages through a particular Send Port or Receive Location will allow trailing delimiters.

      Disadvantage – Must configure each Send Port or Receive Location. No granular control over which remote parties will send or receive messages with trailing delimiters.

   3. Using a custom pipeline that uses a pre-configured BTA HL7 Pipeline component.
      1. Use Visual Studio to construct a custom receive and send pipeline using the appropriate assembler or dissasembler.
      2. Set the component property to "TrailingDelimitersAllowed" to True
      3. Compile and deploy the custom pipeline
      4. Use the custom pipeline instead of the standard pipeline for all HL7 message processing

      Advantage – All messages using the custom pipeline will automatically allow trailing delimiters.

      Disadvantage – Requires custom coding and development to create and deploy the custom pipeline. No granular control over which remote parties will send or receive messages with trailing delimiters.

   4. What does a Trailing Delimiter do to the XML Schema?

      Allowing trailing delimiters does not have the impact often expected in the actual XML Schema.
      The Schema reproduces the message with no data loss.
      Thus, the message when represented in XML must contain the extra fields, in order to reproduce the outbound message.
      Thus, a trialing delimiter results in an empty XML field.
      Trailing Delmiters are not stripped from the inbound message.

      Example:
      <PID_21>44172</PID_21>
      <PID_21>9257</PID_21> -> the original maximum number of repeats
      <PID_21></PID_21> -> The empty repeated field

   5. Allowing trailing delimiters not remove the trailing delimiters from the message, it simply suppresses the check that will cause the message to fail parse with trailing delimiters.
3. When can you not fix the problem by enabling trailing delimiters
   1. Each object in a message must have a location in the target BTAHL7 schema for its content to reside.
      If you have more objects in the message than are contained at that location, then enabling trailing delimiters will not resolve the problem. The schema must be extended to accommodate the empty message content.
      Examples:
      1. Extra Field

         NTE|P||||
         Only 4 fields in NTE Segment, the 4^th field exists, but is empty.

      2. Extra component

         PID|1|1523|47^^^^^^^
         Only 5 components in a CX data type, the 5^th component exists, but is empty

      3. Extra subcomponent

         ORC|1|||||27&&
         Only 2 subcomponents in a CQ data type, the 3^rd subcomponent is empty, but exists.

      4. Extra Repeat

         PID|1||||||||||||||||||||4419~5217~
         Only 2 repeats allowed for the field "Mother's identifier", the repeat is empty, but exists.

   2. In each of these cases, you must locate the failing object and extend the type to allow an additional object of that type.

• Field
  Add a field of ST to the end of the segment with a suitable name in the segments_nnn.xsd
• Component
• Create a new Custom CX data type (i.e. CX_XtraComp) in the datatypes_nnn.xsd and add a new component to the custom CX data type. Update the field in the segments_nnn.xsd file to use the custom data type instead of the standard datatype.
• Subcomponent
• Create a new Custom CQ data type that accepts an additional TS value at the end of the data type. Create a custom TQ data type that uses the new custom CQ data type as the first subcomponent. Modify the ORC segment to use the new CQ data type at ORC.7 instead of the standard CQ data type.
• Repeat
  Modify the Field definition for PID.21 in the segments_nnn.xsd to allow more repeats in the field.

I'm sure that over time you've run into the dreaded "File transport does not have read/write privileges for receive location "C:\ Flatfile\SAPTestIn\".".

Usually you simply go to the folder and either give the BizTalk account full permission (bad) or Everyone full permission (really bad).

So for a production environment, what is the absolute minimum permissions required?

For the Receive File Adapter the explicit permission are:

How does this translate into what to do in the System?

Right clicking on the folder and in the security tab,
setting "Modify" is not enough, though you would think so:

Strangely enough the Delete Subfolders and Files attribute is not set when the Modify property is set, you need to add the
FILE_DELETE_CHILD "Delete Subfolders and Files" Attribute:

Once you have added the Delete Subfolders and Files check box you will have the minimum permissions for the file receive adapter:

For the Send Adapter

The permission for the File Send adapter depend on what properties you have set in the Adapter Advance properties:

If you have the "Use temporary file while writing" flag un-checked then all you need are:

If you have the "Use temporary file while writing" flag checked then the flags you need are:

I've been reviewing and studying the CrossReferencing APIs because it looks like no one is using them, and I like to find new things to play with.

I got a hint from Marty Wasznicky that they were written for a customer years ago "to provide a way to easily take an inbound value and cross it over to an outbound value."

As I thought about it, I realized that I do this all the time. For example, at a health care provider, I had inbound LabCorp Medical Testing values that had a one-to-one relationship with the electronic medical records (EMR) value. So, I could possibly use this to map values from one system to value to another system.

UC1 – Map inbound values to outbound values in a Map (pre-built functoids!)

Then I was thinking about the fact that in my SAP System, the target system has values in the BAPI that are different between environments. For example, the OBJ_SYS value in the message changed. Instead of two maps with hard-coded values in them, I could use the xRef lookup to return the value that is appropriate for each system.

UC2 – Environmental Constants that change from system to system, but are shared across all servers in the same group used in Maps, Pipelines, or Orchestrations.

Then I got interested in using the APIs directly in the Orchestration. I realized that I could set a flag that turned on and off Debugging output that was specific for each component in the system Orchestration, Expression Shape, Exception, and so on. System.Diagnostics.Debug.WriteLineIf(Boolean, Debugstring, Category)

UC3 – Optional configuration values that could be used in Orchestrations to change behavior

Then I finally realized that I could use these APIs to set variables in the Orchestration for use. For example, I wanted the Category in the Debug statement to be configurable. Once I figured that out, I realized that almost any variable in an Orchestration could be initialized and changed on the fly, constrained by some limits, that I've listed below.

UC4 – Variables that can be changed in the Orchestration or Pipelines (YES PIPELINES or Custom CODE!), as desired.

The advantages of this approach are:

1. Variables are read at run time and require no restart of the Host Instance to apply the change.
2. The APIs and functoids are brutally easy to use.
3. The values take effect immediately.
4. This is the ONLY way I know of to change values in a map at run time because they provide built-in functoids for the APIs.
5. The security is controlled by the standard BizTalk Security model. No additional security management is required.
6. Unlike the BTSNTSvc.exe.config file, erroneous changes to the data will not cause the service to fail to start.
7. The config is shared simultaneously across all the servers automatically.

Disadvantages are:

1. The fields are limited to 50 characters.
2. The Name (AppID) and Values (CommonID) must be each unique. You cannot have duplicate values in either column.
3. The method to feed values into the system leaves much to be desired. It's a complex set of XML config files, and the development of them is an arcane art -> I may be induced to build a creator tool that generates them.
4. Once a value is into the system, there is only one class of values that can be deleted. A lot of the remainder can't be deleted or even updated. I have concentrated my efforts on the ones that are updatable due to this reason.
5. The data is stored in the BizTalkMgmtDb, which is not normally optimized heavily because that it is a low usage database. So you have to be careful that there is sufficient room on the where that the database resides if you plan on putting many records into the system.

Here are the four APIs that I am concentrating on:

Basically two ways to Get, by Name (AppID) and Value (CommonID)

One way to delete by Name (AppID) which deletes the Pair

One way to Create by Name (AppID) which can generate a Base64 encoded GUID for the Value so that you can simply use it as a flag to indicate that you saw the value.

                PENDING-> How to use the CrossReferencing APIs to guarantee unique messages!

Microsoft.BizTalk.CrossReferencing Namespace (some of the interesting APIs, and YES I expect someone to read about them)

public static string GetAppID (

       string idXRef,

       string appInstance,

       string commonID

)

public static string GetCommonID (

       string idXRef,

       string appInstance,

       string appID

)

public static bool RemoveAppID (

       string idXRef,

       string appInstance,

       string appID

)

public static string SetCommonID (

       string idXRef,

       string appInstance,

       string appID,

       string commonID

)

How to use these APIs?

Below is a representation of the table structure:

Ok, so you have to have an AppType, it is required for the AppInstance and xRef datatable.

• ListOfAppType.xml

    <appType>

        <name>ApplicationOne</name>

    </appType>

Then you create the App Instance that has one and only one AppInstance per AppType and is never used anywhere else.

• ListOfAppInstance.xml

    <appInstance>

        <instance>ApplicationOne_01</instance>

        <type>ApplicationOne</type>

    </appInstance>

Now we start on the right side of the tree that contains the "Group" values, idXRef

• ListOfValueXRef.xml

    <valueXRef>

        <name>Credit Term</name>

    </valueXRef>

Then we build the common table that relates idXRef (Group) to AppInstance (Application)

• ListOfValueXRef_Data.xml

    <valueXRef name="Credit Term"> <- This is the Group

        <appType name="ApplicationOne"> <- This is the Application

            <appValue commonValue="Credit Term 1">001</appValue>

            <appValue commonValue="Credit Term 2">002</appValue>

            <appValue commonValue="Credit Term 3">003</appValue>

            <appValue commonValue="Credit Term 4">004</appValue>

        </appType>

    </valueXRef>

The AppValue is the Name field and the CommonValue is the V alue field.

In the application I wrote:

Group: Credit Term

Application: ApplicationOne_01

Name: 001

Value: Credit Term 2

To create or update a value in the table:

if (CrossReferencing.GetCommonID("Credit Term", "ApplicationOne_01", "001").Length > 0)

{

CrossReferencing.RemoveAppID("Credit Term", "ApplicationOne_01", "001");

}

TheReturnString = CrossReferencing.SetCommonID("Credit Term", "ApplicationOne_01", "001", "Credit Term 2");

TheReturnString will equal "Credit Term 2" if successful.

How do I use this in an Orchestration?

One way to use this is to drive your Debugging code at run time.

For instance, I created the following entries:

<Setupfile.xml>

<?xml version="1.0" encoding="UTF-8"?>

<Setup-Files>

    <!--btsxrefimport –file=setupfile.xml-->

    <App_Type_file>ListOfAppType.xml</App_Type_file>

    <App_Instance_file>ListOfAppInstance.xml</App_Instance_file>

    <IDXRef_file>ListOfIDXRef.xml</IDXRef_file>

    <IDXRef_Data_file>ListOfIDXRefData.xml</IDXRef_Data_file>

</Setup-Files>

<ListOfAppType.xml>

<?xml version="1.0" encoding="utf-8"?>

<listOfAppType>

    <appType>

        <name>Neudesic.Integration.BizTalk.GE.InvXRef</name>

    </appType>

</listOfAppType>

<ListOfAppInstance.xml>

<?xml version="1.0" encoding="UTF-8"?>

<listOfAppInstance>

    <appInstance>

        <instance>Orchestration.Variables</instance>

        <type>Neudesic.BizTalk.GE.Variables</type>

    </appInstance>

</listOfAppInstance>

<ListOfIDXRef.xml>

<?xml version="1.0" encoding="utf-8" ?>

<listOfIDXRef>

    <idXRef>

        <name>Acetta.Janus.Integration.BizTalk.GE.Invoicing</name>

    </idXRef>

</listOfIDXRef>

<ListOfIDXRefData.xml>

<?xml version="1.0" encoding="utf-8" ?>

<listOfIDXRefData>

    <idXRef name="Neudesic.Integration.BizTalk.GE.Invoicing">

        <appInstance name="Orchestration.Variables">

            <appID commonID="true">Debug</appID>

            <appID commonID="ReceiveInvoiceFromSalesforceAndTransform">DebugCategory</appID>

        </appInstance>

    </idXRef>

</listOfIDXRefData>

Now in the expression shape access the Debug and Debug category Variables:

Finally, let's use it to determine that Debugging is enabled and we should print out to the Debug Console:

How about Maps?

The CrossReference APIs were designed with Map usage in mind.

Here I set values that I need for BAPI calls to SAP:

<?xml version="1.0" encoding="utf-8" ?>

<listOfIDXRefData>

    <idXRef name="Neudesic.Integration.BizTalk.GE.Invoicing">

        <appInstance name="X_InvoiceAPApprovedToBAPI_ACC_INVOICE_RECEIPT_POST">

            <appID commonID="2300000000">SAPDocFormatString</appID>

            <appID commonID="SFDOCNO000000000">REF_DOC_NO_FormatString</appID>

            <appID commonID="BKPFD">OBJ_TYPE</appID>

            <appID commonID="SAPDEP004">OBJ_SYS</appID>

            <appID commonID="42">COMP_CODE</appID>

            <appID commonID="T1">DOC_TYPE</appID>

        </appInstance>

    </idXRef>

</listOfIDXRefData>

This returns the OBJ_SYS parameter that varies from Production and Development, this means I don't have to recompile the Map between production and development:

At the end of the day the CrossReferencing API's and using them effectively will reduce custom code and recompilation in your Maps, Orchestrations and pipelines.

Removing DTA (Tracking) Data permanently

First, I would like to thank my co-BizTalker at Neudesic, Mo Tan who helped prepare this blog… way to go, Mo!

While reading the BizTalk Documentation the other day (my favorite read next to Dilbert), I ran across an interesting little feature—Permanently deleting tracking without archiving it.

Background

Historically (BizTalk 2004 SP1 and earlier), there was no easy way to remove aged data from the Tracking Database (BizTalkDTADb), and so managing the amount of data that was tracked was critical to the long-term health of the environment.

You had to avoid long-running Orchestrations that handled many messages (Convoy Orchestrations that loop on a single message type) needed to be avoided because the number of message they tracked would grow and fill the Message Box database (BizTalkMsgBoxDb) with an unbounded number of message instances, potentially slowing the system performance significantly.

 So, we had two issues that limited the design of our solution design:

• Long-running Convoy orchestrations had to be shut down regularly to allow the live message history to be moved to the DTA Db.
• Tracking data had to be limited to ensure that the Tracking database would not grow too large.

 In BizTalk 2004 SP2 and BizTalk 2006, there is a SQL Job call "DTA Purge and Archive (BizTalkDTADb)."

The new DTA Purge and Archive job performs two new critical tasks to solve these limits:

1. Archive and remove aged data from the Tracking (DTA) database OR simply remove aged data from the Tracking Database. This second option must be manually configured in the Job to enable it.
2. Forcibly remove aged data from the Message Box database – and this is critical – even if there is a running Service that is associated with the data by doing a "Hard Delete" (one of the parameters to the job). This date must be after the tracking date to ensure the data is in the DTA database before being purged.

This second new functionality to the job is an underappreciated feature. Removing data from the Message box means that essentially, you can run a Convoy orchestration for much longer time periods without shutdown.

By default, when this job is installed, it is not enabled.

Steps to enable the Job (this is a review of the BizTalk Documentation):

1.    Within the SQL Server Management Studio, find the "DTA Purge and Archive (BizTalkDTADb)" job.

2.    Open the first (and only) step named "Archive and Purge."

3.    Update the command field, which executes a stored procedure "dtasp_BackupAndPurgeTrackingDatabase", in order to configure the frequency of archiving from the MessageBox to the tracking database (BizTalkDTADb) and purging any track messages thereafter.

4.    The parameters as defined below:

5.    Here's an example:

•    exec dtasp_BackupAndPurgeTrackingDatabase 12, 1, 3, '\\MyBizTalkServer\backup', null, 0

6.    Save and enable the job.

Steps to change the job to permanently delete the tracking data

1.    Within the SQL Server Management Studio, find the "DTA Purge and Archive (BizTalkDTADb)" job

2.    Open the first (and only) step named "Archive and Purge"

3.    Update the command field, from "dtasp_BackupAndPurgeTrackingDatabase" to "dtasp_PurgeTrackingDatabase" which does not archive to the tracking database, BizTalkDTADb, instead purges directly from the MessageBox.

4.    The parameters are defined below:

5.    Here's an example:

•    exec dtasp_ PurgeTrackingDatabase 12, 1, 3, GetUTCDate()

6.    Save and enable the job.

At a recent client, the Salesforce developer (Doug Auerbach of Avanade) pointed out that the newest release (What's New in Apex Web Services API Winter '07) has the capability for Salesforce to call external WebServices with Notification messages. You can wire up a large range of events using various rules and use them to generate custom WSDL that, when exposed by your application, Salesforce will call with the data object specified. Tim Dutcher, the Technical Director at Acetta, agreed to pursue the possibility of using this new functionality.

In BizTalk Server 2006, we approached the solution from the viewpoint of exposing a BizTalk Orchestration as a WebService that consumed the WSDL generated by the Salesforce wizard. The WebService is a two-way send-receive port. The initiating event is a notificationsRequest ,and then we respond with a notificationsResponse message. The notificationsRequest is a sample api (not a real world sample) that contains two objects:

Id – the ID of the sent object
sObject – a repeating xml:Any node that contains a node of the types specified in the Salesforce wizard as a generic sObject and a RecordTypeId that specifies the type of the incoming object.

After the WSDL was in place, I created a simple Orchestration that received the Notification message and returned the response message consisting of a simple XML boolean flag (Ack) indicating success or failure.

We pointed the Salesforce engine at the exposed WebService and immediately got an error response in the Salesforce outbound message queue:

Server did not recognize the value of HTTP Header SOAPAction (URL).

Inspection of the WebService that was provided by Salesforce indicated that the SOAPAction property was set to "", an empty string.

After some searching for the error message Tim Dutcher found a web site (there are many) that told us to add the following attribute to the WebService Class:

[SoapDocumentServiceAttribute(RoutingStyle=SoapServiceRoutingStyle.RequestElement)]

After appending this attribute to the existing attribute list, we were able to complete a round trip communication between BizTalk and Salesforce.

Microsoft has already started incorporating the SysInternals product line into the MS line.

http://www.microsoft.com/technet/sysinternals/default.mspx

They have released an update of Procmon

Announcing Process Monitor v1.0

We're excited to announce the release of Process Monitor, a system monitoring tool that not only replaces Regmon and Filemon by including file system and registry monitoring, but adds process, thread, and DLL monitoring as well as advanced filtering, event information, and basic data mining capabilities.

I highly recommend it!

Required Ports for BizTalk Server

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/bts06coredocs/html/a11cec7a-de97-47f9-9153-0be82d694fab.asp

Also listing the:

Ports for the Receive and Send Servers

http://msdn.microsoft.com/library/en-us/bts06coredocs/html/19cafe74-6676-4779-8ced-de0841dba19f.asp

Ports for the Administration Server

http://msdn.microsoft.com/library/en-us/bts06coredocs/html/dc0094b2-5ba2-4b8f-84aa-b6232be358ee.asp

Ports for the Tracking Server

http://msdn.microsoft.com/library/en-us/bts06coredocs/html/4b4913a4-7d8d-4fd0-a116-ba868c4ad7da.asp

Ports for the Processing Servers

http://msdn.microsoft.com/library/en-us/bts06coredocs/html/0207b68f-8769-4674-aadc-b3a67b6f210b.asp

Ports for the Enterprise Single Sign-On Servers

http://msdn.microsoft.com/library/en-us/bts06coredocs/html/30eb99f9-02cb-43c9-b038-d7bd898e3a7a.asp

Ports for the Human Workflow Services Server

http://msdn.microsoft.com/library/en-us/bts06coredocs/html/15c46422-adc0-4d0c-a66e-6cbc162b4b9c.asp

Ports for the Business Activity Services and Administration Computer in the Corporate Domain

http://msdn.microsoft.com/library/en-us/bts06coredocs/html/757b7b02-6ace-4db1-a403-2170f1205d2b.asp

Ports for the BAM Portal Server

http://msdn.microsoft.com/library/en-us/bts06coredocs/html/df67c31c-a7a3-4bff-9374-0d8a0cf0ad21.asp

You probably already know of:

Service overview and network port requirements for the Windows Server system

http://support.microsoft.com/default.aspx?scid=kb;en-us;832017

I bet you are all drooling over the new VirtualStream support in BizTalk Server 2006. With this, you can access a forward-read only stream as if it was a fully implemented stream with backward and forward access to the data.

The VIrtualStream is implemented in the undocumented API's but is also exposed in the SDK as VirtualStream.cs, so I feel pretty safe in using the deployed version.

But, the use of the VirtualStream is not what this about, this is about the security implications of the VirtualStream. The VS supports a disk caching scheme to store large streams…

As such, this means that the stream size in the cache can be quite large and you _generally_ don't have to worry about it, but in fact you do because this can fail in subtle an unexpected ways.

Two things can go wrong in this scenario.

Understand that the stream is cached to the BizTalk Server Host Instance account's %TEMP% folder, generally under:

C:\Documents and Settings\<BTSHostInstanceName>\Local Settings\Temp

The first problem I see here is that the C: Drive is NOT a good place to put large files that grow unexpectedly. The IO performance is poor and in addition, you could exhaust the drive space unexpectedly. So I recommend setting the BizTalk Host Instance's TEMP folder manually to a separate drive, preferably a non-backed up high speed disk.

Ok, so now you have the TEMP folder set to a separate drive, the second problem that could occur is that the BizTalk Host Instance account may not have Read-Write access to that folder. When the account is logged on the first time, the OS builds the account's local folder structure and gives it the correct access, but you need to manually set this when you manually set the TEMP folder location.

So:

1)      Try to move the TEMP folder of the BizTalk Host Instance account to a large and non-OS used drive

2)      Make sure that BizTalk Host Instance account has full control of the folder

That's it, then you won't get surprising failures in your pipeline when processing large files.