A few observations 3 weeks after GO LIVE

Ok, now I've been part of a Go Live for AX 2009 Sp1 and I have gained 3 weeks of operational experience. It's to early to draw any definitive conclusions, but this is a short summary of my key observations in random order:

• The decision to go for Windows Server 2008 x64 Standard was right
• The decision to go for SQL Server 2005 Sp2 x64 Standard in a active-passive failover cluster was probably right, except for some breaks in the database communication (exact reason still unknown)
• AIF performs better than expected even without utilizing paralell processing on several AOS instances (2 000 messages can easily be processed in a couple of minutes utilizing one uni directional channel)
• The BizTalk Adapter haven't caused us any spesific issues so far
• Pay extra attention when submitting batch jobs to the batch queue (or when constructing new batch jobs) and don't expect all logic to execute automatically under the new Batch Framework (look out for logic tied to the client tier even in the standard application and plan for adding some extra logic to keep the code compatible in both interactive and batch mode)
• Look out for hot fixes from Microsoft (check Partnersource or Customersource freqently) and plan for some delays getting a response (new installations should evaluate the roll up package released late in February)

Bottom line; the Batch Framework is worth paying special attention to during analysis, design, build and testing.

LIVE, update

Well we had a breakthrough this morning! At least the batch execution now seems to run as expected. We will conclude after running the batch through the night and tomorrow. Anyway all news are good news right now.

After investigating the AOS server used to run the AIF batch(es), we discovered some deviations in Performance Manager (high number of page faults and hence PF Delta). This led us to change the Batch Group and switch to another AOS server. After this we have been able to automatically process inbound and outbounde messages. If the batch survives the night and tomorrow, we will take actions on the faulting AOS by unistalling/cleaning up and installing a new AOS instance. The only load on the suspicious server right now, is generated by the BizTalk Adapter (.NET Business Connector).

After spending long days in the office since last Friday, I'm heading home more optimistic than ever.

LIVE

Ok, we have been LIVE since last Friday and I thought it was time for a short update.

Basically things have gone quite well, except for a lot of problems with AIF and message processing. We have identified some issues with customizations, without coming close to solving the biggeste issue: Batch execution of the AIF message processing services. The wierd thing is that they work for some time and then suddenly they stop working, and not leaving a trace of evidence of what went wrong (just sets the status to 'Error' without any error message in QueueManager or in the Exception log). We have done a lot of debugging but this is very time consuming and the underlying logic is hard to follow. We have set up the Batch Groups as per MS definition routing the batch execution to the correct AOS server (2 non clustered) and we have tried a lot of different setups for the Batch jobs with and without dependencies, without any success. The event logs are looking good and the overall utilization of the server resources, are under control.

Part of the complexity is tied to the impersonation logic (RunAs), but this seems to be under control after some issues the first days (permissions). We have ended up implementing a manual processing routine that bypasses the impersonation logic to allow debugging. The manual routine works well, but needs a dedicated resource to act as Queue Manager (not a good solution, but we are able to keep the queue in a controlled state).

I can't go into further details right now, but the new Batch Framework is causing us some pain and this is not what we where hoping for. When the batch jobs run successfully, the performance is acceptable and we are able to process a decent number of messages in any direction. To add some more mystic to the issue, we sometimes have the inbound processing running without problems, while we have problems with outbound processing. Suddenly this changes without any clue or trace of what happened. And we are monitoring the AIF lock table closely and we are also looking for database locks, without seeing any issues in these areas.

So the haunt continues and you can expect some more updates about this later when the issue hopefully is tracked down and solved.

If anyone have some more on this, you are welcome to leave a comment (I'm still optimistic).

So long

The haunt for knowledge

This is a follow up of the post regarding RPC errors.

First step to obtain a basic understanding of how MSRPC works can be found here. Microsoft has implemented their own version of RPC that they call MSRPC. Wikipedia puts the MSRPC implementation into a historical perspective and relates it to the source spesification called Distributed Computing Environment (DCE) from the Open Software Foundation.

I haven't yet found any specific information about how the Client- and Server Stubs are implemented in the Dynamics AX 3-tier architecture, but I'll try to dig a little bit more into this.

In addition to this, Microsoft recently also have published some KB articles on Partnersource that throws some more light on the issue I initially was blogging about. I'll try to find the URL's and update this post with the direct links when I get the time.

Update 2009-02-18:

Valuable links

Technet Magazine (article written by Zubair Alexander)
MSDN (Error Codes 1700 - 3999)
Florian's Weblog

Approaching GO LIVE

For those of you that have read my prior posts regarding AX 2009 and AIF with the AX BizTalk adapter, we have been working with our first implementation since June 2008. Without going into the details, we are approaching GO LIVE and the latest configuration is done this week such as defining the final endpoints. We basically have a solution with a set of front line services (FTP) in the perimeter network and BizTalk Server 2006 R2 togheter with AX 2009 in the local network tied together with some middleware. Not revolutionary or innovative, but a simple, cost effective and robust solution based on proven technology.

The test results so far are good and we are ready to GO LIVE tying a lot of trading partners to the client. We will have sales ordres, purchase orders, packing slips and picking lists flowing, in addition to invoices in different shapes and flavors. First phase is roll out in one country and two additional roll out phases are planned for the next months. By roll out in this context, we talk about markets with a set of different trading partners for each market.

Stay tuned for general updates!

Microsoft Dynamics AX 2009 AIF BizTalk Adapter Configuration White Paper

Finally Microsoft has released a White paper explaining how to configure AIF together with the AX 2009 BizTalk adapter! You can find it here.

This documentation is clear and right to the point compared to the corresponding White paper for AX 4 that is all we have had available until now. It also touches the necessary batch jobs for supporting the out- and inbound message flow (async/sync).

Well done Microsoft; let's hope for more useful and formal documentation for AX 2009! This will avoid a lot of R&D activities and increase the quality of the product. We also need best practise documentation outside the scope of the compiler :-)

RPC errors

One of the areas I find a little bit hard to understand, is the RPC exceptions logged in the event log (application). This is of course specific to AX 4 and AX 2009, but in my opinion Microsoft has done a pore job explaining the prerequisites and relationship between the RPC errors logged as Information and Warnings in the event log.

Sample (AX 2009 on Windows Server 2008)

I have studied this somewhat, but I have never found a good solution to generalize into best practice when installing AX. I understand that something impacts the way a client calls a Remote Procedure being executed on the AOS, but I don’t have a real understanding of the possible reasons. It could be a number of factors ranging from authentication, x++ code/logic (run at property), network packets, time outs, network drivers etc. but I really miss some basic guidelines on how to account for this. Could it also be some registry settings/tweaks needed? If yes, Microsoft should in my opinion indeed help the partners to better understand the prerequisites since this is all about making AX operating without any disturbance and “hidden” issues.

My challenge to eventual readers is to share your experience and help defining a best practice for this area. I would happily share the result of a joint community effort on this blog. If anyone remember the old “Damgaard Technet”, I really miss a common channel for addressing technical issues that I think a lot of partners around the world are experiencing, I don’t understand why every partner must invest in a lot of R&D activities to solve these kind of issues especially since it’s often about not documented prerequisites!

So please donate your experiences as comments to this blog and I promise to compile a best practice that everyone can use to better avoid RPC issues. Since a lot of unpleasant spam occurs in blogs, I have to moderate your comments before publishing but anonymous comments are allowed so keep them coming.