This is really a note to self, as I have a habit of forgetting the numerous issues I have overcome and feel my blog can be a great repository for such information, as it may also assist someone else out there. I haven't been as rigorous with this as I should have recently. Well, for a long time actually!

Anyway, quite a simple one really, but one I have managed to forget. If you are setting up a new WCF web service hosted within IIS you may receive errors such as that below:

Event Type: Error
Event Source: ASP.NET 2.0.50727.0
Event Category: None
Event ID: 1088
Description:
Failed to execute request because the App-Domain could not be created. Error: 0x80070005 Access is denied.

Event Type: Warning
Event Source: ASP.NET 2.0.50727.0
Event Category: None
Event ID: 1073
Description:
Failed to initialize the AppDomain:/LM/W3SVC/1/Root/<your app domain>

Exception: System.IO.FileLoadException
Message: Could not load file or assembly 'System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. Access is denied.
StackTrace:    at System.Reflection.Assembly.nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, Assembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection)
   at System.Reflection.Assembly.InternalLoad(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Reflection.Assembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Activator.CreateInstance(String assemblyName, String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, Evidence securityInfo, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(String assemblyName, String typeName)
   at System.AppDomain.CreateInstance(String assemblyName, String typeName)
   at System.AppDomain.CreateInstance(String assemblyName, String typeName)
   at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters)
   at System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors(String appId, IApplicationHost appHost, HostingEnvironmentParameters hostingParameters

I am sure there are other reasons for receiving these errors, but the times I have come across it have been due to the account defined withint the associated application pool not having read permission to the application's web.config file.

A large percentage of issues in the numerous BizTalk 2004 environments I have encountered with at my current client have been related to configuration. The current build of the environments are conducted in a semi automated fashion; the initial build is performed by Altiris and then there are a number of steps carried out either manually or through automation scripts thereafter.

This is obviously far from ideal, as any manual configuration is prone to errors. The favoured solution would be to script the build of the entire environment and then provide a simple button for some operational employee to click that would go off and build the whole thing for you.

There are products out there that can achieve this for you, an example being Microsoft’s APF (Automated Purposing Framework) that they apparently use to do just that internally. However, your client may not be in a position to implement an external product to conduct any automation of builds for various reasons, so you would be stuck in a similar predicament whereby the build is a mishmash of manual and automated steps.

To complicate matters further, the environment at my current client is built to be a shared service. A standard integration platform based on BizTalk 2004 is deployed in the numerous environments (dev, system test, pre-production and production) and then applications are installed on top of this platform. The complexity here is that each of those applications may also have some environment configuration requirements and in one particular case we have a large, business critical application that makes numerous changes to this generic platform.

That is enough of the background information. The point to this narrative is that managing configuration of integration platform environments can be quite complex. Tools to consider when attempting mitigating the risks involved include automated build tools as mentioned above, but also configuration management tools such as the product that I have been pushed to evaluate for this purpose at my current client – Microsoft DCM (Desired Configuration Monitoring).

Now I will hold my hands up here. Until recently Configuration Management had been a concept for others to contend with. In my professional career I have been primarily involved with development until getting involved with BizTalk since the 2004 version. Therefore my exposure to technologies such as SMS (Systems Management Server), MOM (Microsoft Operations Manager) and the like had also been limited. During the last few years I have found MOM to be a very useful solution and due to necessity I am now finding SMS and DCM to also be equally useful when attempting to solve the numerous challenges involved when managing an Integration Services platform.

My initial focus has been on evaluating DCM and was the main reason for this post. This evaluation has simply entailed investigating the potential use by focusing on the manifests used by the DCM tool to report upon a server’s configuration. In case you have not come across DCM before, manifests are simply XML files that define a number of rules according to the DCM schema that are then used by the tool to report on any differences found on a specified server.

So for example, you may wish to create a rule that ascertains whether the HTTP Batch size on your BizTalk 2004 HTTP Send Host server is set to 1. Perhaps this setting is essential for the applications hosted on that environment to function as expected. In this case you would use the freely supplied Authoring Tool to enter the configuration rule, save the manifest and then execute DCM, passing in this manifest to use. As a quick note, for our investigation we are conducting this manually through the executable directly rather than utilising SMS where the functionality can be automated.

Anyway, I have been using this tool for a number of days and have to say in some frustration. I understand that this whole solution is freely available from Microsoft so you should not expect too much, but the whole authoring process seems unnecessarily complex and convoluted. The XML schema in particular seems to be illogical and the authoring tool hampers progress as much as aids it. This may lead the cynic in some people to think this was done on purpose so that Microsoft could sell their consulting services to produce ready built manifests and/or assist in the creation of those that are not already available.

However, once you have defined your manifest(s) the tool is certainly a useful addition to your armoury when dealing with the issues described earlier. Due to the amount of configuration applied to our environments I was never entirely convinced that they had been configured as per the instructions. Even the automated steps could potentially have incorrect parameters passed to them due to process. So being able to use DCM to check the most obvious configurations were applied correctly certainly assisted in providing a higher level of confidence.

So for me, this freely available utility has given me an immediate benefit even when a reasonable amount of investment time up front when authoring the manifests has to be factored in. This has been gained simply by executing the tool manually, the next step will be to look at the automation of this through SMS and the generation of reports based upon the results.

However, we may not be taking this particular route just yet. We may decide to stay with the simple manual execution for the meantime and look at establishing Microsoft Systems Center at the client, which introduces Microsoft System Center Configuration Manager 2007 – the successor to DCM. I have had a relatively brief presentation of this product by Microsoft and have to say I was quite impressed. It seems to encapsulate MOM, SMS and DCM within a more integrated solution while providing much needed enhancements to those individual products.

If you have any vested interest in making sure your environments are managed affectively and have any investments in those products, if you have not already spent some time investigating Microsoft System Center I would strongly suggest you do.

Firstly, I knew it had been a long time since I posted on my blog, as I have been snowed under with work for my current client and none of this work warranted any posts on the subject. However, I must admit I didn’t realise how long ago it was since I actually posted. Doesn’t time fly huh? I can’t say I know where this year has gone, but we all say that every year don’t we?

Anyway, I solved a relatively simple (but annoying) problem today that I hadn’t encountered before that didn’t have much information available on the internet; well, not for my particular scenario. So I thought it would be worth popping on my blog, firstly as a simple wave to say I am still alive and secondly as a reference for me in the future. You know how it is, you spend time solving problems then encounter them a few months later only to forget how you solved it. Well, I feel the blog is a good repository for this kind of information.

So, what was the issue? Well, one of our testing environments had taken a bit of a pounding recently with various non-functional tests. This is never healthy if you wish to come out at the end with the exact same environment as you started without some form of rebuild be it automated or not. While trying to deploy a custom adapter to this environment numerous errors were occurring, which led to the conclusion that there was in issue with SSO. A highly likely conclusion considering one of the non-functional tests included the restoration and movement of SSO.

During the investigation it was observed that whenever you tried to register a new adapter you would receive the following error:

SSO AUDIT
 Function: CreateApplication
 Tracking ID: e902339c-b431-40e5-a880-337bb873725b
 Client Computer: <Computer Name> (wmiprvse.exe:1232)
 Client User: <User Name>
 Application Name: {11C54F41-4C5A-4929-B828-7CD6E550BEB8}
 Error Code: 0xC0002A18, The format of the account name is not valid. Domain accounts must include the domain name. Local accounts must not include a domain or computer name.

This naturally led to an examination of all things related to SSO, in particular the SSO services. You would have thought from the error message that the service account used for SSO was incorrect. However, after too much time it transpired that the default BizTalk Host was registered using an incorrect domain group; it actually wasn’t in the domain as per the above error.

And the conclusion to this little story? Spend a little more time studying the GUIDs in the error. Only in hindsight did I think about searching out the details to find they would have guided me to the answer more quickly.

I have had to park my work on BizTalk 2006 while I assist another team at my current client on some BizTalk 2004 work. I was helping to install and configure some BizTalk groups and remembered just how much of a pain this can be and how much this has evolved in BizTalk 2006.

Anyway, I had an issue during the pre-installation of Visual Studio 2003 on a Windows 2003 SP1 server that took me some time to fathom out. Checking the error log highlighted the following errors in particular:

[03/13/06,10:57:24] Microsoft FrontPage 2000 Web Extensions Client: ***ERRORLOG EVENT*** : Error code 1603 for this component means "Fatal error during installation.

[03/13/06,10:57:24] Microsoft FrontPage 2000 Web Extensions Client: ***ERRORLOG EVENT*** : Setup Failed on component Microsoft FrontPage 2000 Web Extensions Client

[03/13/06,10:57:27] Setup Runtime Files: ***ERRORLOG EVENT*** : Error code 1603 for this component means "Fatal error during installation.

[03/13/06,10:57:27] Setup Runtime Files: ***ERRORLOG EVENT*** : Setup Failed on component Setup Runtime Files

[03/13/06,10:57:30] Microsoft Visual J# .NET Redistributable Package 1.1: ***ERRORLOG EVENT*** : Error code 1603 for this component means "Fatal error during installation.

[03/13/06,10:57:30] Microsoft Visual J# .NET Redistributable Package 1.1: ***ERRORLOG EVENT*** : Setup Failed on component Microsoft Visual J# .NET Redistributable Package 1.1

[03/13/06,10:57:32] setup.exe: ***ERRORLOG EVENT*** : Unable to open g_szRegkeyMSIntegration in CSetupManager::RemoveSetupFiles()

[03/13/06,10:57:32] setup.exe: ***ERRORLOG EVENT*** : CSetupManager::RemoveSetupFiles() failed in CSetupManager::RunInstall()

[03/13/06,10:57:32] VS70BaseUI: ***ERRORLOG EVENT*** : DepCheck indicates Microsoft FrontPage 2000 Web Extensions Client is not installed.

[03/13/06,10:57:32] VS70BaseUI: ***ERRORLOG EVENT*** : DepCheck indicates Setup Runtime Files is not installed.

[03/13/06,10:57:32] VS70BaseUI: ***ERRORLOG EVENT*** : DepCheck indicates Microsoft Visual J# .NET Redistributable Package 1.1 is not installed.

After some considerable amount of time researching what could be the root cause, I stumbled across the following KB article that led me to the answer http://support.microsoft.com/default.aspx?scid=kb;en-us;327763

In my scenario the servers that I received had an incorrect mapping for the “My Documents” folder, which pointed to a network folder that did not exist. I changed this to point to a temporary folder located on the local machine and the installation continued successfully.

I hope this post may help save someone else time.

At my current client they have been relatively slow on the uptake of Windows Server 2003 SP1, so those of us working within the Integration Services team have not had to deal with any complications that this service pack has introduced to BizTalk. Recently however we have been installing SP1 and have noticed a number of errors that have been introduced because of this.

We observed that BizTalk Host Instances were being stopped and restarted automatically. Following this up in the Windows Event Log we noticed a number of error messages relating to DBNETLIB and other entries indicating that there were issues relating to network connectivity. Some captures of these are shown below.

After some research we came across the following Knowledge Base article at Microsoft: http://support.microsoft.com/default.aspx?scid=kb;en-us;899599

Now the symptoms described in this article are exactly what we were experiencing. Apparently Windows SP1 introduces some extra security feature that reduces the size of the queue for concurrent TCP/IP connections to the server, which helps to prevent denial of service attacks.

All well and good, however the article suggests the actual issue only materialises when TCP/IP is under heavy load, sometimes incorrectly identifying valid TCP/IP connections as a denial of service attack. However, we have been receiving the errors when the BizTalk Group was under no load whatsoever.

After implementing the registry modifications in that article the problems did disappear, but I am interested why the BizTalk Group was experiencing the condition at all when there was no load. I spent some time investigating what may be the root cause and the only conclusion I could come to was that it may be related to the number of host instances that we had within the BizTalk Group; there were 40 in this case.

I have not had the time to investigate this any further, although a simple enough test would be to see whether the errors materialise with less host instances. One other thing that I did notice after installing Windows 2003 SP1 though was the number of security log entries. Is it just me or do we now see a huge increase in the number of entries? Something else I would like to investigate time permitting.

Just a final note, I have seen this occur with both BizTalk 2004 and BizTalk 2006.

We recently encountered an issue on one of our BizTalk 2004 Groups that took quite a long time to figure out, so I thought I would post about this for self future reference and also in case anyone else ever stumbles across the same problem.

The group configuration consisted of two BizTalk servers, two SQL Servers clustered in an active/passive mode, one of these containing the Message Box the other hosting the other BizTalk databases. We also had a separate Analysis Services database.

Once everything had been installed and BizTalk was configured we noticed we were getting intermittent errors simply when browsing the adapters within the BizTalk Administration Console. These were related to transactions, as shown in the following capture:

Numerous checks were made to ascertain that MSDTC was functioning correctly and had the correct security settings. Generally, if you experience any errors that mention transaction in the description you first point of call should be to make sure that the MSDTC security settings are not restricting you. However, in our case this was all working fine and running tests using DTCPing, DTCTester and the like did not reveal anything either.

The other interesting (or should I say frustrating) aspect of this error was due to its intermittent regularity. This made it particularly difficult to debug. However, once we installed an orchestration on the server the error become more prevalent. We could now replicate a similar error on demand by simply attempting to stop an orchestration within BizTalk Administrator, with an error “A connection with the transaction manager was lost”.

After various investigations and discussions with PSS a short-term fix was found where taking the Management Database cluster resource offline, bringing it back online and restarting the SQL Server would solve the problem.

However the actual issue has been addressed with Windows 2003 SP1 (which was not installed on these servers), so if you have that applied you are in luck. If you cannot apply Windows 2003 SP1 though you must install the following HotFix that addresses the issue:

http://support.microsoft.com/default.aspx?scid=kb;en-us;895250

(Shameless plug)

Looking to work in a high quality position for a high quality consultancy? Look to the left where the latest positions at Conchango are available.

Oh, and if you do happen to get a job here don't forget to say where you found out about the position :-)

I don't think I need to say any more...

Further to my previous post on the issues I was experiencing with the configuration of BizTalk 2006 with cross domain local groups, we now have a workaround that is acceptable to our client.

A temporary account should be created in the group domain that will be used for the configuration so we can get past this stage. Once the configuration is completed, the correct accounts can be set-up and the temporary account can be removed. This takes form of the following steps:

SSO Configuration

• Configure SSO using the temporary account
• Change the account that the SSO Service executes under to the correct, permanent account
• Restore the master secret and restart the server (using ssoconfig or the MMC)

BizTalk Runtime Configuration