RPC errors

One of the areas I find a little bit hard to understand, is the RPC exceptions logged in the event log (application). This is of course specific to AX 4 and AX 2009, but in my opinion Microsoft has done a pore job explaining the prerequisites and relationship between the RPC errors logged as Information and Warnings in the event log.

Sample (AX 2009 on Windows Server 2008)

I have studied this somewhat, but I have never found a good solution to generalize into best practice when installing AX. I understand that something impacts the way a client calls a Remote Procedure being executed on the AOS, but I don’t have a real understanding of the possible reasons. It could be a number of factors ranging from authentication, x++ code/logic (run at property), network packets, time outs, network drivers etc. but I really miss some basic guidelines on how to account for this. Could it also be some registry settings/tweaks needed? If yes, Microsoft should in my opinion indeed help the partners to better understand the prerequisites since this is all about making AX operating without any disturbance and “hidden” issues.

My challenge to eventual readers is to share your experience and help defining a best practice for this area. I would happily share the result of a joint community effort on this blog. If anyone remember the old “Damgaard Technet”, I really miss a common channel for addressing technical issues that I think a lot of partners around the world are experiencing, I don’t understand why every partner must invest in a lot of R&D activities to solve these kind of issues especially since it’s often about not documented prerequisites!

So please donate your experiences as comments to this blog and I promise to compile a best practice that everyone can use to better avoid RPC issues. Since a lot of unpleasant spam occurs in blogs, I have to moderate your comments before publishing but anonymous comments are allowed so keep them coming.

Experiences with the AX BizTalk Adapter

We have now encountered our first issue with the AX 2009 BizTalk Adapter. I will try not to go into every detail, but instead focus on the principal guidelines.

#1 If you experience a warning in the eventlog (application), Event ID 110 on the BizTalk Server saying "An X++ exception has occurred. Unable to lock resource channel '<channel name>'" (see sample below), I would suggest to look at how you have defined channels, action policies, document services etc. with regards to what is defined in the different company accounts since this can in fact lead to conflicts. We first related this to installing AX 2009 SP1 since it was the only change recorded in the same time frame and we didn't see this while we worked with the RTM release. The solution was to clean up in the AIF configuration using the AX client, but the cause is at the time beeing not known (it could be SP1, but that's only a theory at the time beeing).
























Update 2008/12/17:
After looking into the details, I also noticed a change of the warning logged by BizTalk for the same issue. Sample of the warning logged initially (figure 1) and a new version of the same message (figure 2) is shown below. I guess that something changed in SP1 and that the problem can be related to a required reboot of the BizTalk Server after applying AX 2009 SP1 (not logged or notifyed by the installer).


#2 Disparsed documentation. The documentstion for AX 2009 is quite good for AIF configuration, but is's very sparse when you are looking for details about the configuration of the transport properties in BizTalk (Visual Studio). If you want to understand the difference between the various authentication types, you have to look up the corresponding documentation for AX 4. At least this was where I found the answers to my general questions.

Convergence 2008 in Copenhagen

After attending the annual Convergence conference for the first time, I feel even more energic about the future for AX and Microsoft is clearly ramping up the fight against SAP and Oracle. The product is ready to grow both horizontally and vertically (broader and deeper functionality) and Microsoft has done a lot with the product since they bought Navision back in 2002 (USD 1.55 billion in cash and share).

One good example is the Enterprise Portal that has been part of the product since version 2.x. In my opinion it's more a revolution than en evolution despite the fact that SharePoint was introduced as the framework in AX 4. The really big change is the introduction of the Role Centers and the rather bold ambition to build a true portal solution from the ERP perspective. And it's really about role tailored "Business Intelligence to everyone". Some deep changes in the runtime and a much better developer experience in the crossing between MorphX and Visual Studio, opens up for a lot of interesting possibilities. I really hope to get started with some serious implementations pretty soon, because I'm convinced that the Enterprise Portal framework will bring much value to the customers.

The second top news for me was the announcement of the AX Data Management framework (ADM). If you attended the conference and missed this session, you really missed one exiting and excellent session given by Mr. Sri Srinivasan! I attended 6 AX technical sessions, but this was by far the best one in every aspect. And what's it all about? Finally customers and partners will have a framework to clean up the database schema, to categorize the data and to move data out of the primary production database. This maybe doesn't seem like something important to most people, but if you think about customers running the product for 8-10 years doing every main upgrade, you will probably recognize the fact that not all changes in the database schema has been high quality. Combined with different developers not always (seldom?) paying attention to the impact of their adjustments on the database back end, it shouldn't come as a surprise to anyone that the database schema grows into a jungle of different objects with broad (many columns) and deep (many million rows) tables with a lot of indexes hurting the performance big time. The Data Management framework also addresses this dilemma. Add the fact that 5-10% percent of the tables contributes to 80-85% of the total database size and you probably see the need for some serious clean up work. The roadmap for the ADM consists of 3 phases starting with a stand alone toolset and ending as an integrated part of AX. Microsoft also said that they would utilize the management functionality in SQL Server even more in the future. The final area of functionality also covers index maintenance with alerts being raised on given fragmentation thresholds. So all in all rules for archiving fiscal years still making the figures available through SSRS and special forms in AX for handling online and archive data, should really sound like good news to both existing and new customers. And this is in my opinion another sign of Microsoft ramping up the battle against SAP and Oracle by filling the administration and management gap to lower the TCO. Studies and analysis of existing customers, was reported to give effects in the 40+% range and I find it hard to questioning this numbers after attending the session - Sri is clearly an expert within this topic.

The last of my top 3 take a ways, was actually not related to AX and technology, but it was taken from the Business Leadership track. Session BL01 "A Macro view of Business Trends" was lead by Mr. Thomas Mendel from Forrester and supported by a representative from both HP - EDS and Microsoft. It was arranged like a prepared discussion session with Forrester giving their thoughts about what they considers to be the three megatrends that will change the IT business over the next years and ending up with the view as seen from HP - EDS and Microsoft. It was really both exiting and refreshing to listen to them and to get some confirmations about my own opinions. And the tree megatrends are Globalization 2.0, Invisible IT and how the consumer marked influences the Enterprise IT. A lot of dimensions and elements was brought into the discussion ranging from the CHINDIA effects via Cloud computing (did you know that Microsoft buys 10 000 servers every month to power their Azure platform from data centers all over the world?) to innovation driven from the consumer market. They all agreed on that the traditional office space was history and that the y generation will bring a totally new set of requirements to the Enterprise IT departments.

So all in all I returned with a great feeling abut the future for DAX and it will be very interesting to see how the battle on the ERP arena develops in the rather turbulent world economic situation we are facing right now. I think a lot of new businesses will discover that the Dynamics branding really is more than yet another market initiative. Finally the announcement of DAX SP1 didn't come as a suprise at all...

AX 2009 SP1

I recently got information telling that Service Pack 1 for AX 2009 is pretty close to RTM. I don't have any information about what it contains, but let's hope SSAS and SSRS for SQL Server 2008 is fully supported.

Maybe some more information will be published during the upcoming Convergence conference in Copenhagen (November 19 - 20).

In addition Microsoft has released at least one kernel hot fix for AX 2009 with reference to the "non existing" KB article 958328. It' still hard to find any information about this hot fix, but I know Microsoft is working on the KB article. As usual, hot fixes are provided directly from Premier Support (or other official Microsoft support channels), while the KB articles still are published on Partner and/or Customer Source. Kernel hot fixes are distributed as a set of MSP files which also is a good thing as I see it.

With this said, our experience so far has been pretty good and the quality of the RTM release seems to be better that 80/20 (we haven't yet experienced any big issues except for the technical issues described in an earlier post). So at this stage and after working with the release for almost 5 months, it looks as a big step in the right direction. We are still working with AIF and the BizTalk adapter, and except for some minor issues, we have been able to proceed as planned on our current project deliveries.

Dynamics AX and virtualization

Early in September, Microsoft changed their support policy for several systems including Dynamics AX from only supporting their own virtualization software (Virtual Server and Hyper-V) to other virtualization vendors including the market leader VMWare. According to the News Release dated Sept. 3 2008 on http://www.vmware.com/, their hypervisor VMware ESX 3.5 update 2 (ESX 3.5u2) passed the Microsoft Server Virtualization Validation Program (SVVP).

This is old news on the Internett highway, but the effect of this is that customers now can get full support both from Microsoft and VMWare for systems running on the validated release of the VMWare ESX 3.5 hypervisor. This can seem like a minor detail, but a lot of businesses have in fact been running most Microsoft software on an unsupported virtualization platform for a long time without any security or guaranteed support. The change to include VMWare as a supported platform, is important information and should indeed be emphasized.

For everyone involved in using and delivering products in the Dynamics product line, I think it’s good news that Microsoft now officially supports several Dynamics products (AX, CRM and NAV) running on this release of VMWare ESX 3.5 and it actually shows that Microsoft has opened up for other well established players in the virtualization market.

Links
http://support.microsoft.com/kb/957006/
http://www.vmware.com/company/news/releases/svvp.html
http://windowsservercatalog.com/svvp.aspx?svvppage=svvp.htm

AX 2009 BizTalk Adapter

I recently did my first experiences with installing the AX 2009 BizTalk Adapter. "As usual" some discoveries were done, but this time it was more related to my less than minor knowledge of BizTalk...

After the BizTalk Server 2006 R2 Standard Edition (BT) was installed and verified, I went away installing the adapter "by the book" (that's the Installation Guide and the Server and Database Administration Guide). The documentation doesn't really contain a lot of information for BT newbie’s like me and after Googling about the error message I received, it seems like several people has stumbled across this one both for AX 4 and AX 2009. Surprisingly no solution was given and I thought this was a good opportunity to share my experiences.

As mentioned I went away installing the BizTalk adapter (custom installation, check the BizTalk adapter which not surprisingly also checks the .NET Business Connector). I then entered the necessary information the installation wizard asked for and hit the button labeled "Install". After some minutes watching the progress bar for installation of AX Components, I noticed the progress bar doing a "negative" progress followed by a summary saying "Setup was not completed" with a red square in front of both .NET Business Connector and BizTalk adapter. Ok, as normal open the log file and look for errors... Scrolling through the log file I noticed that the .NET Business Connector installed just fine. After some more scrolling down, I found the following error message: "An error occurred during the BizTalk adapter install custom action step within the Microsoft Dynamics AX components installer. For details, see the previous messages in the log. >>Install: BizTalk Adapter registration failed: Access denied". Being under some pressure getting the task done, I though "Oh no, not another day of hunting down a resolution". As mentioned I goggled about the message in bold above and I found several hits to similar experiences, but without a solution.

So my other really good friend in my professional life - the Event log - was my next stop. After filtering the event log - application, I found some warnings with source ENTSSO replicating the Access denied message. This is in my mind always a good indication and I then read a little bit about this on Microsoft TechNet. ENTSSO is short for Enterprise Single Sign On and this service is a vital part of the security mechanism in BizTalk Server with regards to authentication. After reading some more, I decided to open the Microsoft Single Sign On Admin Console. At the end of the summary saying Errors, I once again saw the "Access denied" message under the category RPC. Then I was "bold enough" to add my installation user to the security group called SSO Administrators. Reopened the SSO Admin Console and the RPC error was gone!

I also added the installation user to the BizTalk Administrators security group (required to add new adapters). After doing a new installation of the BizTalk adapter, everything went as expected and the log file looked just fine. Checking inside the BizTalk Admin Console, I found the adapter listed as a resource under "Application.1" (the default application in BizTalk).

To summarize this Blog entry, you have to understand some of the basics of BizTalk before installing the BizTalk adapter for AX 2009 (and also AX 4 as far as I can understand). If this is something Microsoft should mention in the documentation or not, is really open for discussion. As far as I know, they are working on some kind of detailed information right now and maybe this will be included. But all in all, this is another example of the increased demand for broad technical knowledge to install every component available in AX 2009. In my opinion it's a new ballgame compared to previous versions of this packaged application from Microsoft.

We will carry on the next days trying to get AX to communicate with BizTalk utilizing Application Intergration Framework (AIF) and I expect to be able to share some information from this journy quite soon.

So long!

First Experiences

First of all; Dynamics AX 2009 is really exiting with regards to the tight integration with the Microsoft Technology Stack. The product is really starting to take shape in the technology area and it's starting to look as a Microsoft product (remember the history... Damgaard, Navision, Microsoft?). At the same time the complexity has increased and this of course both affects the time and skills needed to implement AX.

My journy so far is based on experiences from one customer installation (Core solution by now) running Windows Server 2003 R2 Standard x64 and from a single computer installation running on Windows Server 2008 Standard x32.

NetBIOS over TCP/IP: At a customer site I experienced problems connecting clients to the AOS instance ("A connection to the Axapta Object Server could not be established") - the initial startup of the AOS service for the instance, went fine. After trying some tweaks, I had to ask the hosting partner for a network trace. It turned out that the problem was that NetBIOS over TCP/IP was deactivated. After activating this (network card, Properties, Advanced, WINS, Enable NetBIOS over TCP/IP) on the AOS and servers with AX client installed, it all went smooth. After looking into the details, I think the reason is that AX from version 4 uses Remote Procedure Call (RPC) for communication between clients and AOS instances, and that RPC requires NetBIOS over TCP/IP. Perhaps Microsoft should put this into the system requirements for AX?

Installing the application component on 64-bits Windows Servers: I had a hard time trying to change the location for the application files from the installer. I almost always installs the application component to a different location than \program files\..\, but this turned out to be very hard on a 64-bit Windows plattform... I did'nt have the time to try out a silent install utilizing a parameter file and the solution could be to specify the location here. No matter what I changed, it defaulted to \program files\..\ and I could'nt change it because it was disabled for editing. I ended up doing the installation followed by copying the files to the correct location as separate application instances. By doing it this way, I have all the original setup information left as a master. This does'nt happen when installing on a 32-bit Windows plattform and here you can change the location freely.

SQL Server 2005/2008: If you only are going to implement AX CORE components (application, AOS instance and database) SQL Server 2008 is supported. If you on the other hand are going to use Reporting Extentions (the AX integration against SQL Server Reporting Services - SSRS) and/or Analysis Extentions (AX integration against SQL Server Analysis Services - SSAS), you have to use these components from SQL Server 2005. Also remember that both SSRS and SSAS are required if you want to use Enterprise Portal (the Web application module in AX) since the Role centres are composed of Web parts that shows SSRS reports (Report Viewer) and that the KPIs also are produced through SSRS reports utilizing SSAS. A former collegue has already an entry about this in his blog, but unfortenately I didn't read his post until after doing the exact same experience. But I totally agree with him that Microsoft should fix this compatibility issue as soon as possible, especially since SQL Server 2008 now are in RTM and supposed to fit very well with AX 2009. Another thing worth mentioning if you are going to install SQL Server 2005 on Windows Server 2008, is that you have to install the IIS6 compatibility components before installing SQL Server 2005.

.NET Framework 3.5: This is a general requirement, but I went away installing .NET Framework 3.5 SP1 as this is the latest release... After installing Enterprise Portal and doing the initial configuration, I ended up with an error message saying "Unhandled Exception" when browsing the site and an repeating error logged in the eventlog (application) saying "An unexpected error has occurred - A ProgressTemplate must be specified on UpdateProgress control with ID 'AxProgressControl'". As usual when encountering these kind of errors, I used Google to search for references. I found one link saying that the solution was to uninstall .NET 3.5 SP1 and reinstall .NET 3.5. All I can say is that it worked... This is also explained in the System Requirements page for Dynamics AX 2009 (says .net 3.5 and not .net 3.5 sp1), but at the same time Microsoft tells you to run Microsoft Update to get the latest security updates for your installation. I think many will be trapped by this one either when installing or later. Anyway I strongly advise you to inform your customers not to install .NET 3.5 SP1 (or any components like Visual Studio 2008 SP1 that updates .net) on the server hosting Enterprise Portal until this component is supported on .NET 3.5 SP1. If you arrive at the office one morning facing a non functional Enterprise Portal site, I would start by checking if .NET Framework 3.5 SP1 is installed...

Installation order: As explained by Microsoft in the AXInstallationGuide, you should always install and configure the Core solution first. Configuration is the same as running through the Installation Check List. After this, it all depends on the features or options required for the implementation. For a full implementation, it worked well for me by running the installation of each component in the following sequence: 1) Workflow 2) Reporting Extentions 3) Analysis Extentions and 4) Enterprise Portal. I installed each component individually and made the necessary configuration from the AX client both before and after the installation. One important part of the installation is to verify the setup logs to make shure everything was properly installed. Also read the documentation carefully to make shure all the pre requesites are met. I used Windows Sharepoint Services 3.0 SP1 (WSS) and installed this separately prior to installing Enterprise Portal (AX setup can do this for you). I ran the configuration, but did'nt create any applications or sites at this stage - just created the main service and the main database. My message is that you can end up with a lot of tricky errors if you install every option in one operation after installing and configuring the core solution. Also remember to fill in the information for the System Accounts before installing any options or additional components.

SSRS 2005 on Windows Server 2008: Remember the manual configuration required before installing Reporting Extentions. This is described in detail in the AXInstallationGuide. What you also should remember is to map the account for the AX SSRS application pool as a user in the SSRS databases.

Role centres: Some of the default Role centres still has errors with missing initial values for report parameters (I have'nt spent time trying to fix this yet). What I find a little bit more suprising is that a couple of the Role centres gives a "stack" of errors and messages saying that the dataset returned is larger that the default value. I don't have the details available right now, but I'll post on this later. The solution is rather simple - you have to increase the buffer size in the AOS instance configuration. I think it defaults to 24 Kb and by changing it to 25 Kb, I got rid of the errors and warnings. This is another area Microsoft should pay attention to and fix, since this is standard out-of-the box functionality that partners should'nt need to spend time on fixing.

This wraps up my first entry in this blog. Hopefully someone will find this information useful and please, don't leave a lot of "pingback" comments - it's impossible to have complete control over all the content out there. I promise to do my best to link to the source when this is applicable...