get on the bus

Well, I am certainly excited this week.

I am also (Jan Eliasen's post) really excited about the book I am co-authoring on.

The list of absolute rock stars that are working on it is amazing:

I’ve been running Windows 7 for only a short time now.

But in that, as a developer, I find that I avoid doing my development on my actual desktop.

Specifically, I have several customers today that run different versions of BizTalk Server from 2004 to 2009.

In addition, many of these customers require you to install VPN software that can wreck havoc with your most critical hardware asset you bring to the table, your personal laptop.

Let us examine a typical work flow that I have encountered several times.

I develop a clean bootable Windows Server 2008 Virtual machine on my Windows 7 Desktop

I ship the VHD to corporate, and they deploy it on our Hyper-V environment

Me and my team working remotely, share and update the system

I pull back the VHD and give it to the customer who is running Virtual PC 2007 SP1

For each of these stages, the migration process is the same.

1. Uninstall the Integration Components (IC)
2. At the command prompt, run the following command:

bcdedit /set {current} detecthal yes

What happened here?

The {current} value tells BCD to modify the currently running boot configuration.

The detecthal yes value tells to force a redetect of the of the hardware during boot

What will happen, is that at the next boot, the system will determine the best HAL to boot from and update the configuration to use it.

Thus, at boot, if the system has been moved from Hyper-V to Virtual PC, the HAL needed will change and the system will locate and update itself to use the new HAL.

This procedure should work for most optimizing Hyper Visors as they all require somewhat different HALs.

My wife this weekend noted as she was reading CNET that WordPress had announced a vulnerability to a worm.

Well, actually, they announced that if you had skipped the last two releases, failed to do your updates, that you would be vulnerable to the worm.

The question often happens, not so much in the actual software development shops, but rather from the common user

How does this happen? They should find all the bugs before they ship!

I’ve thought about this question a lot. It really does seem somewhat straightforward.

Simply put, test the product before it ships, and don’t allow it to have bugs!

hmmm… let me thing about that for a second.

A novel idea, but how do we actually do this? I’ve done professional software development for almost 20 years, and before that, self taught software development for 10 years. I should be the perfect candidate for building bug-free software, I can test it myself, I understand all the tricks.

I can’t do it, and it simply comes down to numbers.

Let me tell you a story.

About 12 years or so, as a fairly senior software tester, I was given the task of verifying a bug had been fixed in the software.

The test seemed pretty simple.

I had a field on a form, and the field was 16 characters long.

The bug was that if the first character in the form was not one of your US based letters, but rather something from a non-english language like French, that the letter would be ignored.

So, if the user typed in:

[ÅÅÅÅuter hansten]

The actual value would be:

[uter hansten    ]

This was pretty easy to test, I fired up my trusty GUI test engine, and wrote a quick little program that looped through all the valid characters and checked to see if they worked.

Being Friday afternoon, I thought giving the program the entire weekend to run would be ample time. I turned away from my test PC and back to my real machine, I suddenly paused.

I wondered, how many tests would this little program actually do?

Lets see… this math looks pretty simple.

Q: How many letters are in the alphabet that need to be tested?

A: This looks like I can safely say we need all the special letters, that was part of the problem, so 256 – 32 = 224 (this is a-Z, numbers, special characters, and the extended characters)

Q: How many characters in the field?

A: 16

Q: So, how many loops will I have to do?

A: 16e+224 = 5.2e+269

SCREECHING HALT… WHAT? WHAT WAS THAT NUMBER?

I happened to recall that a GOOGOL = 10e+100 was a number larger  than the number of atoms in the observable universe.

This number, the loops I had to do would NEVER complete by the end of the weekend, as a matter of fact, it would be quicker to count all those atoms in the observable universe than to run this test.

I turned back to my test PC and canceled the test. (oh and for you code heads, I simply inspected the software and was able to reduce my test count to TWO test cases, rather than 5.2e+259)

First off, I want the casual user to understand some very important things.

No software can bug free. Just look at the current software you are using to view this blog. It has many places that you can type in more than 16 characters. Every one of those places needs to be tested, but there is only so much time before the end of the universe as we know it (which is somewhere in the neighborhood of 37 years).

Secondly, even testing the software, and fixing every bug does not actually improve the quality of the software. Most bug fixing actually adds more bugs than it fixes.

Wait, what do you mean, when you fix a bug you have one less bug!

Yes, that bug may be fixed (and this is maybe, some attempts to fix a bug actually do not fix anything) but the new code to fix the bug may have bugs in itself.

I call this the (very similar to the Neutron Lifecycle in a nuclear reactor) the defect life-cycle.

Each cycle of the software starts with a fixed number of bugs.

As the software build for the next release approaches:

Current bugs exist in the software, let use call this value Bc (Bugs current)

Attempts are made to fix bugs that are known (the Bug List) and some of those are actually removed, let us call this value Pr (planned and removed)

Review of the software during the development cycle discovers unknown bugs, and these are attempted to be fixed, let us call this value Rr (reviewed and removed)

New features are added to the software, and these new features have bugs in them, let us call this value Na (New and added)

Attempts to fix bugs that are know cause additional bugs in the software, let use call this value Pa (planned and added)

Review of the software that discovers bugs, and the attempts to fix the bugs cause new bugs, let use call this Ra (reviewed and added)

The Bugs that remain in the product after a build is the result of this formula, and let us call this value Ba (Bugs after cycle)

So, for each internal build, the rate at which bugs are fixed is:

Ba = Bc – Pr – Rr + Na + Pa + Ra

So, if the Sum of the added bugs is less than Sum of the Bugs removed, then the quality of the software has improved.

Unfortunately this condition is actually rare! Most cycles of the build actually increase the number of the bugs, because the constant pressure of adding new features to the product, and attempts by the junior developers to fix bugs introduced by senior developers (who can’t be bothered to fix that old code). Often overwhelm the any bugs that are actually removed.

Please note, nowhere in this do the actual bugs found have a positive impact on the quality of the software, rather you could strongly argue that by finding bugs the team is practically guaranteed to be adding bugs and making the software worse.

How do we get out of this?

First off, make sure that you understand that software testing will never improve the quality of the software.

Actually, it is a measuring tool, a sounding rod.

If you are testing 10% of the product and you find 100 critical bugs, then it seems that your total critical bug count would be in the neighborhood of 1,000 bugs.

Make a judgement call at this point, can you, as the software company handle that support load?

Are you putting peoples lives at risk?

Many other relevant questions…

BUT at this point do not turn to your software test team and ask them… the sounding rod measures, it does not judge.

I’ve talked about some features and issues with where I want to take the HL7 Accelerator.

Today’s post is more about why we should all care. After all, HL7 is a little protocol that some computers use to communicate, who cares?

In part, I want to talk about where HL7 stands in the world of it’s personal impact to you, your family and friends.

Have you seen a doctor recently? Maybe a family member had a check for swine flu, cholesterol or rheumatoid arthritis?

You have been touched by the HL7 Protocol. After all, where do you think your doctors gets the information from? That’s right, there is a diligent worker out there, that takes that sample of blood, skin or sputum and does some magic with it. The result is that they type in information into their Electronic Lab Notebook, or the device feeds it to the system.

From there, the path is somewhat convoluted. The LAB company (Quest, LABCORP and so on) then attempts to transmit that information electronically to your health care company… yes via HL7.

It then arrives, and they need to find you! What was your medical record number (oops, someone typed it in wrong) and update your electronic healthcare record (EHR).

There are many things that the EHR does, including billing your insurance company and so on, but as far as I’m concerned the NUT of the problem is, am I healthy!

The HL7 protocol was the first system to allow automatic updates on the patient health status.

When the CDC needs to be updated about a regional outbreak of some “notifiable” condition they get those results via an HL7 message.

So, the HL7 messaging system also tells us, is the country healthy?

HL7 Messages are the only electronic communication on the planet that could literally kill someone!

What if the HL7 message is delay, misrouted or misapplied.

Aunt Sally could be diagnosed with a terrible disease that requires aggressive treatment, when actually all she had was gout!

The HL7 protocol is not an idle, irrelevant hard to understand messaging protocol, but instead is a critical element in the rapid, and accurate diagnosis and management of the entire patient experience.

Science fiction has predicted killer computer programs, I have the honor of actually working with that on a daily basis.

HL7, take care, we love it, but we know it’s value.

I’ve been invited to speak this year at TechReady9 about the HL7 Message stack and how to sell it to customers.

If you are one of my Microsoft friends, I hope you can make it.

For others, I will gladly post my opinion about where and when to sell HL7.

The average BizTalk solution has a large number of DLLs that need to reside in the Global Assembly Cache (GAC). As a result, deploying the BizTalk solution can consume more cycles than is necesary. The time factor becomes apparent during troubleshooting (debugging) BizTalk, when the solution must be frequently rebuilt, redeployed, and restarted.

The problem described may be resolved by removing BizTalk DLLs from the GAC and placing them into a common folder. Then the runtime section of the BTSNTSvc.exe.config must be updated to include dependent assembly references to BizTalk DLLs residing in the common folder. Below is the fragment of an updated BTSNTSvc.exe.config runtime section.

<runtime>

 <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">

<probing privatePath="BizTalk Assemblies;Developer Tools;Tracking;Tracking\interop" />

<dependentAssembly>

<assemblyIdentity name="Solution.Project.Module" publicKeyToken="909dd514af6267cd" culture="neutral" />

<bindingRedirect oldVersion="1.0.0.0-9.9.9.9" newVersion="1.0.0.0" />

<codeBase version="1.0.0.0" href="file:///C:\PathToBizTalkDlls\Solution.Project.Module.dll" />

</dependentAssembly>

</assemblyBinding>

</runtime>

This approach can increase development speed and reduce the cost to maintain a BizTalk solution in a development environment. While this is not generally recommended for production environments, we do have clients that have a hard “no GAC” policy; yes, even in production. As such, those customers must have a mechanism for running a BizTalk solution sans GAC.

(This article was first published in BizTalk Hotrod)

The Distinguished Field is often seen as the weaker of the two types of Fields when handling Fields in BizTalk.

After all, the Distinguished Field can't be used as a filter on a message, and it's slower than its big brother the Promoted Field.

Well, today I'm here to dispel the myth of the wimpy Distinguished Field and place in the pantheon of power that equals, and in some ways exceeds the Promoted Field.

MYTH: Getting the value of a Distinguished Field requires loading the entire message into memory.

The first myth that we need to dispel is that the Promoted Field is a quicker field to access than the Distinguished Field.

This is due to the statement in the BizTalk Documentation, and I quote:

The BizTalk Server Message
… Lots of stuff cut out …
"One of the benefits of promoted Fields is that the value of the element that is promoted is available in the context of the message. This means that retrieving that value is inexpensive, as it does not require loading the message into memory to execute an XPath statement on the message."

What is implied here is that for the Promoted Field reading its value doesn't require an XPath read into the message and conversely, that the Distinguished Field is does require loading the message and has a performance cost because it's evaluated when queried.

Nothing could be further from the truth! In fact, the both the Promoted and Distinguished Fields are evaluated at the same time, and both are placed in the message context at the same time. So, let's talk about how fields get into the message context.

About BizTalk Message Context Fields
… Lots of stuff cut out...
"Since Distinguished fields do not require a separate property schema, the evaluation of Distinguished fields by the Orchestration engine consumes less overhead than the evaluation of Property fields by the Orchestration engine. The evaluation of Property fields requires an XPath query, the evaluation of Distinguished fields does not require an XPath query as the pipeline disassembler populates the Distinguished fields in the context and the orchestration engine gets the cached values. However, if the orchestration engine does not find the property in the context, it will then evaluate the XPath query to find the value. Distinguished fields do not have a size limitation.

Now, how does Promoted and Distinguished Fields get into the Message Context? This occurs automatically in the Receive Pipeline by certain pre-built pipeline components?

Out of the box, the BizTalk XML Disassembler, BizTalk Flat File Disassembler and the BizTalk Framework Disassembler Promote Fields to the message context. All other production level Pipelines promote fields, most also support Distinguished Fields.

Distinguished Fields are written to the Message Context if one of these Receive Pipeline Components is used in the Pipeline. Interestingly enough, this explains why the Passthrough pipeline doesn't promote Fields from the message, there are no components in the Passthrough Pipeline, it does nothing to the message content and therefore nothing gets promoted, especially BTS.MessageType.

As far as performance, Distinguished Fields beat out Promoted Fields 9 days each week. This is because both Promoted and Distinguished require the same overhead of writing the message value to the context Field bag in the Message Box, but Promoted Fields have the additional overhead of both being written to the Message Box context database AND the Subscription database. Promoted Fields have an impact every time a message is written to the Message Box because each Promoted Field that exists musts be evaluated in a massive union (very efficiently written union mind you!) that builds the list of matching activation subscriptions. So in short, the more Promoted Fields that you have the costlier the subscription process.

RECOMMENDATION: Use Promoted somewhat sparingly, don't avoid them, but do not use them if you do not need to. Use Promoted Fields as they were designed, to facilitate message routing, but not to make it easy to access a message value. Instead primarily use Distinguished Fields.

MYTH: Its always safe to use a Promoted or Distinguished Field in an Orchestration.

Using Operators in Expressions

exists    test for the existence of a message context property    BTS.RetryCount exists Message_In

Let us talk about how to handle message content that is missing when it is a Promoted Field and a Distinguished Field. What we are talking about specifically is the field that was Promoted or Distinguished did not exist in the inbound message. The XLANG/s xpath statement that was used to query the message for the content during pipeline processing returned a null object.

The first thing to understand is when a Promoted and Distinguished Field comes into existence. They are essentially the same, and this occurs when a Pipeline component parses a message and either Promotes or Writes the value to the context. The simple answer is, when the value does not exist, the Field is not created. A query to the context for the Field returns a null object.

So, if you attempt to access a Promoted or Distinguished Field that didn't exist in the inbound message, you can cause an unhandled exception to be thrown. Specifically in both cases a NullReferenceException.

Promoted Fields, have a special XLang/s test exists (see my previous blog post in this) that you can use to determine if they exist before attempting to use them. In this case, Promoted Fields can always be tested for existence before use and can safely be avoided when they don't exist.

Unfortunately, Distinguished Fields don't have such a special test, and can cause an unpreventable unhandled exception. Specifically if you use a Field that the underlying type is a native non-nullable type. For instance, suppose the value that you have distinguished is a integer. Integers cannot be null (and yes, I am aware of the Nullable<T> generics, but we are talking about what BizTalk XLANG/s has, not what is C# has) and if the underlying value didn't exist and you attempt to use the value, or even test to see if the value exists will cause an unhandled NullReferenceException when BizTalk's XLang engine attempts to convert a null value into an integer by calling the System.Number.Parse(string) method with a null value.

Here comes in the kicker and why a Distinguished field can appear to be fine at design time, but bite you at run-time.

At design-time the expression editor generates a pseudo class-like dotted object for you to use in your expression. At run-time there is simple type-casting that occurs by the run-time engine that inspects the XML datatype of the node in the Schema, retrieves the value as an object… then attempts to call the appropriate ConvertTo method on the Object. When casting a Null to an Int32 or any other intrinsic datatype, a NullReferenceException is thrown and the Orchestration fails.

The primary difference (excluding Routing) between Promoted and Distinguished Fields is the developer design-time experience. Distinguished Fields are easy to use because they emulate .Net Class dotted notation.

RECOMMENDATION: If there is any chance that accessing the Distinguished Field may cause an exception, then place the check in a Scope Shape that has a catch shape to handle the NullReferenceException.

MYTH: Distinguished Fields are only accessible in Orchestrations

WRONG Documentation: Field Schemas
RIGHT Documentation: Distinguished Fields in Disassembler Pipeline Components, Processing the Message, Promote Properties (Node Property of All Schemas)

Another major fallacy about Distinguished Fields is that they are only accessible in the Orchestration. This is also untrue, the BizTalk Server Documentation clearly has an example of how to use Distinguished Fields in any component from the RIGHT Documentation above.

All Distinguished Fields outside of an Orchestration use a fixed schema:
http://schemas.microsoft.com/BizTalk/2003/btsDistinguishedFields

The Field to use is the XPath of the node that is Distinguished such as:
/*[local-name()='PO' and namespace-uri()='http://SendHtmlMessage.PO']/*[local-name()='Price' and namespace-uri()='']

Thus to access this you would use the Read Method:
MyObject = MyMessageContext.Read("/*[local-name()='PO' and namespace-uri()='http://SendHtmlMessage.PO']/*[local-name()='Price' and namespace-uri()='']", " http://schemas.microsoft.com/BizTalk/2003/btsDistinguishedFields");

If the Field exists, then MyObject will contain an object that can be cast to the appropriate type.

RECOMMENDATION: Once a the proper Pipeline Component has processed the message, use the Distinguished Field as you would any Field without the Xpath lookup overhead.

MYTH: Distinguished Fields in Orchestration Expression shapes are actually code.

You have to hand it to the people who did the coding for XLANG/s. It looks like C#, it feels like C# and 99% of the time it pretty much generates standard C#.

In many ways, this is not your father's C#, it is really XLANG/s and it has it's own syntax and special components. Distinguished Fields are a prime example.

Think back on all the times you used a distinguished Field. It feels like it's a C# Object! It uses dotted notation (Node.Node.Node.Attribute). You assign values to it, you use it's value in an expression and it comes out as the correct type. When the node is Boolean, then it behaves like a Boolean. Nothing could be further from the actual behavior as Marty learned recently. Just because it looks like a duck, doesn't mean that it's a duck. It really is a trick, that the Expression Editor parses the XSD on the fly and generates a classlike editor experience, but no actual code ever gets generated.

Further Reading

1. Planning and Architecture > BizTalk Server Architecture > Runtime Architecture > The BizTalk Server Message
2. Planning and Architecture > BizTalk Server Architecture > Runtime Architecture > Processing the Message
3. Developing BizTalk Server Applications > Creating Pipelines Using Pipeline Designer > About Pipelines, Stages, and Components > Distinguished Fields in Disassembler Pipeline Components
4. Creating Orchestrations Using Orchestration Designer > Creating Orchestrations > Using Expressions in Orchestrations > Using Operators in Expressions
5. Creating Schemas Using BizTalk Editor > About Schemas > Different Types of BizTalk Schemas > Property Schemas
6. Creating Schemas Using BizTalk Editor > About Schemas > Ways to Use Message Content to Control Message Processing > About BizTalk Message Context Properties
7. Schema Property Reference > Node Properties - Alphabetical Listings > Node Properties of All Schemas > Promote Properties (Node Property of All Schemas)
8. Creating Schemas Using BizTalk Editor > Creating Schemas > Promoting Properties > How to Copy Data to the Message Context as Distinguished Fields

WCF LOB Adapters are a new technology for integration with back-end systems and applications. WCF LOB adapters will supersede the BizTalk Adapter Framework. If you're running an ESB powered by Neuron, you can use these adapters too. Here's how to configure your bus to talk to a back end system using a WCF LOB adapter.

Step 1: Add a New Binding

Step 1 is to define a new binding to Neuron. A WCF LOB adapter is exposed to consumers as a new transport binding.

• Launch the ESB Explorer, go to the Services > Bindings area, and click New to add a new binding.
• Click the Browse button and navigate to the WFC LOB adapter assembly you want to support. The ESB Explorer will automaically locate and set the class name of the binding.
• Save the new binding under a meaningful name.

Step 2: Add a New Service

Step 2 is to define an endpoint for the WCF LOB system. The WCF LOB adapter makes this look like a service.

• Go to the Services > Endpoints area and click New to add a new service.
• On the General tab, enter the access information for the service including address, binding, and contract. Select the binding you added in Step 1.
• On the Security tab, if the service is secure, enter any credentials required such as username and password.
• On the Ports tab, set up a Send Port to route message traffice to the back end service. Assign a subscriber Id and topic to use for this purpose.

Step 3: Start Messaging

At this point you're really done. Clients can now send messages to the back end service and receive replies. For successful communication check the following:

• Messages need to use the proper schema and expected content
• Messages need to specify support Action values

A technique I used with the samples in the WCF LOB SDK is to turn on WCF diagnostics and message logging when running the test client that comes with the samples. When viewing the logged messages with SvcTraceViewer, I now have examples of valid request and reply messages. Then I can use the .NET test client to send requests and verify that all is in working order.

Ok, so this isn't really too much of a big deal, but Eric Stott found a funny misspelling in the Import MSI Wizard which reminded of something I had noticed.

If you import or export an MSI and during the progress phase click the Progress item on the left list.

In the Export Wizard the "Results:" is blank and the top of the output says "label3"

In the Import MSI Wizard the "Results:" is some generic value and the top says "label1".

What is your easter egg?

List of enhancements to the HL7 Accelerator experience in BizTalk Server.

1. The BizTalk HL7 Accelerator (BTAHL7) is non-compliant with the HL7 2.4 (and 2.3.1) Specification 2.10 in the following manners:
   1. Step 1 Message Construction (serialization):
      1. b.2 if the [field] value is not present, no further characters are required
      2. b.5.iv components that are not present at the end of a field need not be represented by component separators
      3. b.6.iv subcomponents that are not present at the end of a component need not be represented by subcomponent separators
         1. In all of the above cases, the requirement is that the object with no value does not require a delimiter for it, but is certainly gives the implementer the latitude to include it. Incorrectly, BTAHL7 throws a parse error if a sending system sends delimiters at the end of a value that are empty.
         2. The desired behavior is that BTAHL7 quietly accepts any empty field at the end of any object in the system. This behavior is over ridden using a pipeline property or on a party by party basis, but by default the behavior is breaking.
         3. EXAMPLES:

         Field:

         Case 1: Field is not last field in segment:
         NTE|| -> This will cause the parse to fail unless it is the "Allow Trailing Delimiters" flag is set.

         Case 2: Field is last field in segment:

         NTE||| -> This will always fail because BizTalk attempts to parse the extra field in the segment into a non-existent field in the segment.

   2. Step 2 Converting Messages to data values (deserialization):
      1. ignore segments, fields, components, subcomponents, and extra repetitions of a field that are present but were not expected.
         1. BTAHL7 does not ignore any segments, fields, components, subcomponents or extra repetitions if they are not expected. Instead a parse error is thrown and the message is rejected.
         2. The desired behavior is that the HL7 Accelerator retains the extra data for transmission but does not capture the data for processing; additionally this should not cause a parse error.
      2. treat segments that were expected but are not present as consisting entirely of fields that are not present
         1. BTAHL7 does handle this correctly, but the error messages are generally inscrutable if the captured data does not contain required objects.
         2. The desired behavior is that BTAHL7 clearly points to the error.

   The overall impact of the non-compliant behavior is that the HL7 accelerator places a tremendous implementation burden on the implementer. Instead of concerning themselves with values of interest in the message, they spend a tremendous amount of time matching the current system requirements to the HL7 Accelerator defined specification. This increases the implementation time, reduces the flexibility of the solution and in general, makes any BizTalk solution extremely expensive.

2. The BizTalk Server model differentiates between various participants in the entire BizTalk Process.
   1. Business Analyst (BA)
   2. Solution Developer (SD)
   3. Operations Manager (OM)

   The HL7 Accelerator does not match this model. I believe that this lack of separation has made the entire HL7 Accelerator solution difficult to implement and complete solution engaging all of the roles that are involved. The BA has to provide an HL7 Solution definition that does not reflect the desired solution, for instance, they create a spread sheet or a word document that defines the message structure very loosely. There is no was for the BA to tell the SD what message definition and variations are required from the HL7 Standard. The only way to define the message is to directly edit the Schema in the BizTalk Schema editor. The only way to define transformations from Message Schema A to Message Schema B is in the BizTalk Mapper. The BA is completely disassociated from the definition process. They have to communicate via documents (Word or Excel) and then test the results using sample messages. The OM has control over the solution in such a way via the HL7 Configurator that should only be defined at the Development level. This disconnect makes it difficult for the appropriate roles to be applied and frustrates the users at each level.

3. The BizTalk Accelerator solution is difficult to debug and troubleshoot. The Developer has to deploy the schemas and pipelines and then manually submit the messages.
   1. Cannot test Schemas and Maps in the IDE you must deploy the entire solution to validate them.
   2. The HL7 Tables/Segments/Datatypes always have a global impact and are difficult to separate out. The relationship between them is not obvious and the XML editor is extremely difficult to master. This is resolved by using namespaces to separate the schemas, but even that solution is extremely complex.
   3. Editing of table values is difficult and not obvious. There is no way to easily import BA defined values into the table definitions.
   4. The MLLP Test tools are difficult to use and inflexible and seem to be as if they were written at the last minute. When errors are encountered, the tools simply throw an undecipherable exception message instead of reporting a helpful error message.
   5. The MLLP Test tools cannot help the solution test high volume message loading. The tools allow you to either send a single message repeatedly very quickly, or you must simply rerun the tool in some kind of loop in the console which processes relatively slowly.
4. The HL7 Accelerator does not expose any helpful .Net Classes to help decode/encode and handle message content.
   1. The HL7 DateTime field does not map exactly to a .Net class. For instance, the HL7 DT field has 4 digits to the right of the decimal point (ss.mmmm) and the .Net DateTime Type only had 3 (ss.mmm) and so either you must accept a loss of precision (not acceptable for certain lab testing results where the value is a critical measurement) or construct a custom .Net HL7 DateTime datatype to hold this value, deserialize/serialize it and handle it.
   2. The HL7 Serialization/Deserialization does not allow you to choose to accept invalid messages. In other words, the user may want to accept a message that will parse, and then run a second pass on it to validate the data. To do this requires two schemas to be deployed, a loose schema for initial validation and a second tighter schema that must be called to revalidate the message for deeper content validation. It would help that you could turn off validation at the field level using configuration and then determine at which pass the validation will occur.
   3. Promoting message content properties is difficult. For instance, there are many message properties that clients almost invariably want to monitor. Patient Information, Lab Test information, MSH Tracking fields like MSH7 and MSH10 to correlate the message back to the sending system and Provider information. It is certainly possible to build an XPath based property promoter and construct a Property Schema to do this, but pipeline components are always problematic and considering the likelihood that client will want to do this, it seems that a simple pre-built component should already exist and be inside the HL7 Deserializer/Serializer to handle this.
5. The HL7 Accelerator and BizTalk do not work well together to allow multiple message types that have very similar structure to be routed to a single orchestration and then sent to end points for processing.
   1. The ADT and ORM Messages have many trigger types, but need to be handled similarly. To process the messages the implementer must create many send ports for each message type, but they all go to the same physical send port. Every change to the Orchestration send port, typically requires the implementer to make as many as numerous port changes in a single orchestration. Also, the Send Orchestration ended up being extremely complex to handle all the various message triggers for the same message.
   2. The HL7 Send Ports cannot handle System.Xml documents. The message context contains the schema to use, but the Send port cannot handle the generic message type and cast it to the correct message type. This means that every send port in an orchestration must be strongly typed.
   3. The MLLP Send Port does not allow dynamic send ports making it difficult to use a single send port to send to multiple end points programmatically.
   4. Because of the limitation on the System.Xml documents, it is not possible to use some of the more advanced features in BizTalk like dynamic mapping. Thus all messages must be strongly typed and constructed.
6. The System does not interact with some of the HL7.org tools and schemas well.
   1. HL7.org provides a powerful tool that allows a Business Analyst to modify, design, diff and export to RIM HL7 Message definitions. Just looking at the feature set of Message Work Bench (http://www.hl7.org/), at a minimum this tool is a tremendous guide to what kind of features a Business Analyst would need.
      1. Diff two schemas
      2. Capture a message and generate a best guess at a schema
      3. Capture a message and verify conformance to a user defined schema
      4. Extensible Table editing for HL7 conformance values
      5. export of a BA Defined schema in xsd format that is not compatible with the BizTalk Schemas.
      6. HTML (XML with XSLT) generation of a readable message specification
      7. Innumerable reports about a message definition
      8. Capture a message and insert sample values from the message into a specification.
7. Users define custom schemas that have both subtle and widely variant message definitions in comparison to the specification
   1. Make it easier to capture a running message stream and have a wizard that automatically picks a "best matching" schema and adjusts the schema automatically to the actual message data. So that during the design phase, a BizTalk solution could be running in learn mode and then as it runs reports what messages that have failed and make it easy for the implementer to update the schema to allow those messages.
   2. Define reporting tools and mechanisms that update the schema on the fly, suspending messages and then letting the Operations team report and then adjust the schema and resubmit the message automatically.

I've been working on understanding the BizTalk Security model for quite a while and I keep working my Visio diagram over and over.

For BizTalk Server 2004 simply do not use the Level 0 Security Membership column.

Please feel free to send me any feed back if this is not clear.

NOTE:

The variable PropExists as bool has been already created

The Property of interest is BTS.RetryCount

The Message is Message_In

The list from Using Operators in Expressions (http://msdn2.microsoft.com/en-us/library/aa561847.aspx) has the typical list of stuff that you expect in C#, multiplication, bit operations (shift left and right) and Boolean operators, but a couple of extremely useful constructs are available that are unique to BizTalk.

The most important of these (in my humble opinion) is the exists operator.

As you are all aware, to even check whether a property exists in an expression throws an exception… as in the following case:

PropExists = (Message_In(BTS.RetryCount) != null && Message_In(BTS.RetryCount) != "");

If BTS.RetryCount does not exist in the message context, then the MissingPropertyException (Windows Event Log Event ID 10019) is thrown.

Without having to resort to a scope shape and exception handler, the exists operator allows you to check if a property exists in a message and is used in the following format:

PropExists = BTS.RetryCount exists Message_In;

OR

if (BTS.RetryCount exists Message_In)

{

     …;

}

Conclusion:

Using the XLANG/s exists operator in your orchestration allows you to test for the existence of a property in a message without having to resort to a scope shape and exception handler.

Below are a few of more XLANG/s functions that can provide some value in your Orchestrations:

Applies to: BizTalk Server 2006 with the HL7 1.3 Accelerator

1. Outline of the problem
   1. Trailing Delimiters are empty values at the end of an object in a HL7 ER7 formatted message.
   2. Examples:
      1. Empty Field
         1. NTE|P|
         2. NTE|P||
      2. Empty component
         1. ORC|1|725^
      3. Empty Subcomponent
         1. ORC|1|||||27&
      4. Empty repeat
         1. OBR|1||||||||027~
   3. Trailing delimiters indicate the following object exists and is empty, which is quite different from null, null is an explicit value indicated by a pair of double quotes -> "".
   4. The BizTalk HL7 Accelerator by default does not allow trailing delimiters.
2. There are three methods to allow trailing delimiters.

   NOTE: All Schemas always allow trailing delimiters in the MSH Segment

   1. Using party identifiers
      1. MSH3.1 – Receive/inbound processing, using this value as a party allows you to configure the system to allow inbound trailing delimiters.
      2. MSH5.1 – Send/outbound processing, using this value as a party allows you to configure the system to allow outbound trailing delimiters.
      3. Generally, if you allow inbound trailing delimiters, unless you are willing to programmatically remove all trailing delimiters, then you need to configure the send to allow trailing delimiters.
      4. Add the appropriate parties to the BizTalk Parties list from these two fields in your message stream.
      5. Open the BizTalk HL7 Configuration tool and for each party check the "Allow trailing delimiters (separators)" check box on the Validation tab.

      Disadvantage – Each MSH3.1 and MSH5.1 value must be represented in the parties list and configured.

      Advantage – granular control over system behavior for each inbound/outbound system.

   2. Using instance properties of a pipeline used in a send port or receive location.
      1. Open the BizTalk Server Administration console
      2. locate the send port or receive location that contains the BTAHL72XReceivePipeline or BTAHL72XSendPipeline pipeline.
      3. Open the properties
      4. To the right of the pipeline selected locate the […] ellipses button
      5. In the property list, locate the "TrailingDelimiterAllowed" property and set it to True.

      Advantage – All messages through a particular Send Port or Receive Location will allow trailing delimiters.

      Disadvantage – Must configure each Send Port or Receive Location. No granular control over which remote parties will send or receive messages with trailing delimiters.

   3. Using a custom pipeline that uses a pre-configured BTA HL7 Pipeline component.
      1. Use Visual Studio to construct a custom receive and send pipeline using the appropriate assembler or dissasembler.
      2. Set the component property to "TrailingDelimitersAllowed" to True
      3. Compile and deploy the custom pipeline
      4. Use the custom pipeline instead of the standard pipeline for all HL7 message processing

      Advantage – All messages using the custom pipeline will automatically allow trailing delimiters.

      Disadvantage – Requires custom coding and development to create and deploy the custom pipeline. No granular control over which remote parties will send or receive messages with trailing delimiters.

   4. What does a Trailing Delimiter do to the XML Schema?

      Allowing trailing delimiters does not have the impact often expected in the actual XML Schema.
      The Schema reproduces the message with no data loss.
      Thus, the message when represented in XML must contain the extra fields, in order to reproduce the outbound message.
      Thus, a trialing delimiter results in an empty XML field.
      Trailing Delmiters are not stripped from the inbound message.

      Example:
      <PID_21>44172</PID_21>
      <PID_21>9257</PID_21> -> the original maximum number of repeats
      <PID_21></PID_21> -> The empty repeated field

   5. Allowing trailing delimiters not remove the trailing delimiters from the message, it simply suppresses the check that will cause the message to fail parse with trailing delimiters.
3. When can you not fix the problem by enabling trailing delimiters
   1. Each object in a message must have a location in the target BTAHL7 schema for its content to reside.
      If you have more objects in the message than are contained at that location, then enabling trailing delimiters will not resolve the problem. The schema must be extended to accommodate the empty message content.
      Examples:
      1. Extra Field

         NTE|P||||
         Only 4 fields in NTE Segment, the 4^th field exists, but is empty.

      2. Extra component

         PID|1|1523|47^^^^^^^
         Only 5 components in a CX data type, the 5^th component exists, but is empty

      3. Extra subcomponent

         ORC|1|||||27&&
         Only 2 subcomponents in a CQ data type, the 3^rd subcomponent is empty, but exists.

      4. Extra Repeat

         PID|1||||||||||||||||||||4419~5217~
         Only 2 repeats allowed for the field "Mother's identifier", the repeat is empty, but exists.

   2. In each of these cases, you must locate the failing object and extend the type to allow an additional object of that type.

• Field
  Add a field of ST to the end of the segment with a suitable name in the segments_nnn.xsd
• Component
• Create a new Custom CX data type (i.e. CX_XtraComp) in the datatypes_nnn.xsd and add a new component to the custom CX data type. Update the field in the segments_nnn.xsd file to use the custom data type instead of the standard datatype.
• Subcomponent
• Create a new Custom CQ data type that accepts an additional TS value at the end of the data type. Create a custom TQ data type that uses the new custom CQ data type as the first subcomponent. Modify the ORC segment to use the new CQ data type at ORC.7 instead of the standard CQ data type.
• Repeat
  Modify the Field definition for PID.21 in the segments_nnn.xsd to allow more repeats in the field.

I'm sure that over time you've run into the dreaded "File transport does not have read/write privileges for receive location "C:\ Flatfile\SAPTestIn\".".

Usually you simply go to the folder and either give the BizTalk account full permission (bad) or Everyone full permission (really bad).

So for a production environment, what is the absolute minimum permissions required?

For the Receive File Adapter the explicit permission are:

How does this translate into what to do in the System?

Right clicking on the folder and in the security tab,
setting "Modify" is not enough, though you would think so:

Strangely enough the Delete Subfolders and Files attribute is not set when the Modify property is set, you need to add the
FILE_DELETE_CHILD "Delete Subfolders and Files" Attribute:

Once you have added the Delete Subfolders and Files check box you will have the minimum permissions for the file receive adapter:

For the Send Adapter

The permission for the File Send adapter depend on what properties you have set in the Adapter Advance properties:

If you have the "Use temporary file while writing" flag un-checked then all you need are:

If you have the "Use temporary file while writing" flag checked then the flags you need are:

I've been reviewing and studying the CrossReferencing APIs because it looks like no one is using them, and I like to find new things to play with.

I got a hint from Marty Wasznicky that they were written for a customer years ago "to provide a way to easily take an inbound value and cross it over to an outbound value."

As I thought about it, I realized that I do this all the time. For example, at a health care provider, I had inbound LabCorp Medical Testing values that had a one-to-one relationship with the electronic medical records (EMR) value. So, I could possibly use this to map values from one system to value to another system.

UC1 – Map inbound values to outbound values in a Map (pre-built functoids!)

Then I was thinking about the fact that in my SAP System, the target system has values in the BAPI that are different between environments. For example, the OBJ_SYS value in the message changed. Instead of two maps with hard-coded values in them, I could use the xRef lookup to return the value that is appropriate for each system.

UC2 – Environmental Constants that change from system to system, but are shared across all servers in the same group used in Maps, Pipelines, or Orchestrations.

Then I got interested in using the APIs directly in the Orchestration. I realized that I could set a flag that turned on and off Debugging output that was specific for each component in the system Orchestration, Expression Shape, Exception, and so on. System.Diagnostics.Debug.WriteLineIf(Boolean, Debugstring, Category)

UC3 – Optional configuration values that could be used in Orchestrations to change behavior

Then I finally realized that I could use these APIs to set variables in the Orchestration for use. For example, I wanted the Category in the Debug statement to be configurable. Once I figured that out, I realized that almost any variable in an Orchestration could be initialized and changed on the fly, constrained by some limits, that I've listed below.

UC4 – Variables that can be changed in the Orchestration or Pipelines (YES PIPELINES or Custom CODE!), as desired.

The advantages of this approach are:

1. Variables are read at run time and require no restart of the Host Instance to apply the change.
2. The APIs and functoids are brutally easy to use.
3. The values take effect immediately.
4. This is the ONLY way I know of to change values in a map at run time because they provide built-in functoids for the APIs.
5. The security is controlled by the standard BizTalk Security model. No additional security management is required.
6. Unlike the BTSNTSvc.exe.config file, erroneous changes to the data will not cause the service to fail to start.
7. The config is shared simultaneously across all the servers automatically.

Disadvantages are:

1. The fields are limited to 50 characters.
2. The Name (AppID) and Values (CommonID) must be each unique. You cannot have duplicate values in either column.
3. The method to feed values into the system leaves much to be desired. It's a complex set of XML config files, and the development of them is an arcane art -> I may be induced to build a creator tool that generates them.
4. Once a value is into the system, there is only one class of values that can be deleted. A lot of the remainder can't be deleted or even updated. I have concentrated my efforts on the ones that are updatable due to this reason.
5. The data is stored in the BizTalkMgmtDb, which is not normally optimized heavily because that it is a low usage database. So you have to be careful that there is sufficient room on the where that the database resides if you plan on putting many records into the system.

Here are the four APIs that I am concentrating on:

Basically two ways to Get, by Name (AppID) and Value (CommonID)

One way to delete by Name (AppID) which deletes the Pair

One way to Create by Name (AppID) which can generate a Base64 encoded GUID for the Value so that you can simply use it as a flag to indicate that you saw the value.

                PENDING-> How to use the CrossReferencing APIs to guarantee unique messages!

Microsoft.BizTalk.CrossReferencing Namespace (some of the interesting APIs, and YES I expect someone to read about them)

public static string GetAppID (

       string idXRef,

       string appInstance,

       string commonID

)

public static string GetCommonID (

       string idXRef,

       string appInstance,

       string appID

)

public static bool RemoveAppID (

       string idXRef,

       string appInstance,

       string appID

)

public static string SetCommonID (

       string idXRef,

       string appInstance,

       string appID,

       string commonID

)

How to use these APIs?

Below is a representation of the table structure:

Ok, so you have to have an AppType, it is required for the AppInstance and xRef datatable.

• ListOfAppType.xml

    <appType>

        <name>ApplicationOne</name>

    </appType>

Then you create the App Instance that has one and only one AppInstance per AppType and is never used anywhere else.

• ListOfAppInstance.xml

    <appInstance>

        <instance>ApplicationOne_01</instance>

        <type>ApplicationOne</type>

    </appInstance>

Now we start on the right side of the tree that contains the "Group" values, idXRef

• ListOfValueXRef.xml

    <valueXRef>

        <name>Credit Term</name>

    </valueXRef>

Then we build the common table that relates idXRef (Group) to AppInstance (Application)

• ListOfValueXRef_Data.xml

    <valueXRef name="Credit Term"> <- This is the Group

        <appType name="ApplicationOne"> <- This is the Application

            <appValue commonValue="Credit Term 1">001</appValue>

            <appValue commonValue="Credit Term 2">002</appValue>

            <appValue commonValue="Credit Term 3">003</appValue>

            <appValue commonValue="Credit Term 4">004</appValue>

        </appType>

    </valueXRef>

The AppValue is the Name field and the CommonValue is the V alue field.

In the application I wrote:

Group: Credit Term

Application: ApplicationOne_01

Name: 001

Value: Credit Term 2

To create or update a value in the table:

if (CrossReferencing.GetCommonID("Credit Term", "ApplicationOne_01", "001").Length > 0)

{

CrossReferencing.RemoveAppID("Credit Term", "ApplicationOne_01", "001");

}

TheReturnString = CrossReferencing.SetCommonID("Credit Term", "ApplicationOne_01", "001", "Credit Term 2");

TheReturnString will equal "Credit Term 2" if successful.

How do I use this in an Orchestration?

One way to use this is to drive your Debugging code at run time.

For instance, I created the following entries:

<Setupfile.xml>

<?xml version="1.0" encoding="UTF-8"?>

<Setup-Files>

    <!--btsxrefimport –file=setupfile.xml-->

    <App_Type_file>ListOfAppType.xml</App_Type_file>

    <App_Instance_file>ListOfAppInstance.xml</App_Instance_file>

    <IDXRef_file>ListOfIDXRef.xml</IDXRef_file>

    <IDXRef_Data_file>ListOfIDXRefData.xml</IDXRef_Data_file>

</Setup-Files>

<ListOfAppType.xml>

<?xml version="1.0" encoding="utf-8"?>

<listOfAppType>

    <appType>

        <name>Neudesic.Integration.BizTalk.GE.InvXRef</name>

    </appType>

</listOfAppType>

<ListOfAppInstance.xml>

<?xml version="1.0" encoding="UTF-8"?>

<listOfAppInstance>

    <appInstance>

        <instance>Orchestration.Variables</instance>

        <type>Neudesic.BizTalk.GE.Variables</type>

    </appInstance>

</listOfAppInstance>

<ListOfIDXRef.xml>

<?xml version="1.0" encoding="utf-8" ?>

<listOfIDXRef>

    <idXRef>

        <name>Acetta.Janus.Integration.BizTalk.GE.Invoicing</name>

    </idXRef>

</listOfIDXRef>

<ListOfIDXRefData.xml>

<?xml version="1.0" encoding="utf-8" ?>

<listOfIDXRefData>

    <idXRef name="Neudesic.Integration.BizTalk.GE.Invoicing">

        <appInstance name="Orchestration.Variables">

            <appID commonID="true">Debug</appID>

            <appID commonID="ReceiveInvoiceFromSalesforceAndTransform">DebugCategory</appID>

        </appInstance>

    </idXRef>

</listOfIDXRefData>

Now in the expression shape access the Debug and Debug category Variables:

Finally, let's use it to determine that Debugging is enabled and we should print out to the Debug Console:

How about Maps?

The CrossReference APIs were designed with Map usage in mind.

Here I set values that I need for BAPI calls to SAP:

<?xml version="1.0" encoding="utf-8" ?>

<listOfIDXRefData>

    <idXRef name="Neudesic.Integration.BizTalk.GE.Invoicing">

        <appInstance name="X_InvoiceAPApprovedToBAPI_ACC_INVOICE_RECEIPT_POST">

            <appID commonID="2300000000">SAPDocFormatString</appID>

            <appID commonID="SFDOCNO000000000">REF_DOC_NO_FormatString</appID>

            <appID commonID="BKPFD">OBJ_TYPE</appID>

            <appID commonID="SAPDEP004">OBJ_SYS</appID>

            <appID commonID="42">COMP_CODE</appID>

            <appID commonID="T1">DOC_TYPE</appID>

        </appInstance>

    </idXRef>

</listOfIDXRefData>

This returns the OBJ_SYS parameter that varies from Production and Development, this means I don't have to recompile the Map between production and development:

At the end of the day the CrossReferencing API's and using them effectively will reduce custom code and recompilation in your Maps, Orchestrations and pipelines.