Watch out for the combination of reports as PDF with graphics and batch execution

We have seen this issue in our design and finally Microsoft has now verified (newsgroup microsoft.public.axapta.programming) that AX 2009 has an issue when you use the buildt in PDF functionality for reports containing graphics (logos etc.) and the logic executes as part of a batch job. The reason is that the PDF AX logic uses the Image class to handle the graphic and the Image class is in fact (documented) bound to the client tier. So this is an impossible combination and in my oppinion, a good example of issues in the shadowlands of the changes in the Batch Framework and the default application logic provided by Microsoft.

The solution is to use a PDF printer driver to produce the PDF file, but this should'nt really be necessary since AX has built in support for PDF generation. If you have read some of my earlier blog posts, you will probably notice that we have experienced some major issues when running the application logic in batch under the new Batch Framework and the lesson learned is that you should pay extra attention and put in extra effort to make the batch jobs run as expected.

My best tip is to plan for this as an extra test activity during build (or as an additional payload during testing) and reflect this in the estimates given to the customers. I conclude that the comlexity is rapidly increasing as a consequence of a very good architectural change in AX (server bound batch execution and impersonation), but the application lacks some conditional checks and maybe also some best practise checks in the compiler ("Dear developer, you are trying to call a class on the server that is bound to the client. Please consider your design and look for alternative solutions").

Staying on top of things - AX 2009 hot fixes

To be honest, I find it very hard to keep track of the available hot fixes (and cumulative rollup packages). By very hard, I'm mostly thinking about the time spent on 1) searching for possible hot fixes and 2) if one possible related hot fix is found, understanding what the hot fix actually solves.

Partnersource (or Customersource) is the primary source for this kind of information by searching the Knowledge base. Maybe it's only me, but I would like a much more effecient tool and much more detailed information about the individual hot fixes to keep our customers happy and of course spend as little time as possible on this activity. The real trouble arises when you find KB number without a published KB article. What do to in these situations? The answer is to ask for unpublished information through your support channel and hereby allocating even more time and resources.

So people; take the opportunity to vote by answering the recently published poll on the right hand side in this blog.

I promise to share the results with Microsoft.

Bottom line is that it's all about quality of the software by easily providing current information about available hot fixes to the partners (and customers).

Debugging Batch Jobs

We have a hard time getting the AX 2009 debugger to work when breakpoints are set in code executing in Batch Jobs. Microsoft has described this in several documents without any special requirements, but we are unable to get the debugger to work as expected. All settings are correct in the server configuration and in the client when the breakpoint is defined. We have also tried to use the keywork breakpoint in the code without any difference. It seems like the debugger has a hard time attaching to the process. We have tried this in several environments and solutions, and in different combinations, and basically we are stucked.

The fundamental changes in the Batch Framework requires more debugging than ever due to the increased complexity inherited from the changes around impersonation (RunAs) and the prmiary tool in this situation is in fact the debugger.

We have also checked with several other sources and they have done the same experience.

All experience around this issue is highly welcome.

Update 2009/04/03:

Microsoft has confirmed that they have received a problem report (4652) for this issue, but no solution or fix is available yet. The problem was reported in January 2009.

A few observations 3 weeks after GO LIVE

Ok, now I've been part of a Go Live for AX 2009 Sp1 and I have gained 3 weeks of operational experience. It's to early to draw any definitive conclusions, but this is a short summary of my key observations in random order:

• The decision to go for Windows Server 2008 x64 Standard was right
• The decision to go for SQL Server 2005 Sp2 x64 Standard in a active-passive failover cluster was probably right, except for some breaks in the database communication (exact reason still unknown)
• AIF performs better than expected even without utilizing paralell processing on several AOS instances (2 000 messages can easily be processed in a couple of minutes utilizing one uni directional channel)
• The BizTalk Adapter haven't caused us any spesific issues so far
• Pay extra attention when submitting batch jobs to the batch queue (or when constructing new batch jobs) and don't expect all logic to execute automatically under the new Batch Framework (look out for logic tied to the client tier even in the standard application and plan for adding some extra logic to keep the code compatible in both interactive and batch mode)
• Look out for hot fixes from Microsoft (check Partnersource or Customersource freqently) and plan for some delays getting a response (new installations should evaluate the roll up package released late in February)

Bottom line; the Batch Framework is worth paying special attention to during analysis, design, build and testing.

LIVE, update

Well we had a breakthrough this morning! At least the batch execution now seems to run as expected. We will conclude after running the batch through the night and tomorrow. Anyway all news are good news right now.

After investigating the AOS server used to run the AIF batch(es), we discovered some deviations in Performance Manager (high number of page faults and hence PF Delta). This led us to change the Batch Group and switch to another AOS server. After this we have been able to automatically process inbound and outbounde messages. If the batch survives the night and tomorrow, we will take actions on the faulting AOS by unistalling/cleaning up and installing a new AOS instance. The only load on the suspicious server right now, is generated by the BizTalk Adapter (.NET Business Connector).

After spending long days in the office since last Friday, I'm heading home more optimistic than ever.

LIVE

Ok, we have been LIVE since last Friday and I thought it was time for a short update.

Basically things have gone quite well, except for a lot of problems with AIF and message processing. We have identified some issues with customizations, without coming close to solving the biggeste issue: Batch execution of the AIF message processing services. The wierd thing is that they work for some time and then suddenly they stop working, and not leaving a trace of evidence of what went wrong (just sets the status to 'Error' without any error message in QueueManager or in the Exception log). We have done a lot of debugging but this is very time consuming and the underlying logic is hard to follow. We have set up the Batch Groups as per MS definition routing the batch execution to the correct AOS server (2 non clustered) and we have tried a lot of different setups for the Batch jobs with and without dependencies, without any success. The event logs are looking good and the overall utilization of the server resources, are under control.

Part of the complexity is tied to the impersonation logic (RunAs), but this seems to be under control after some issues the first days (permissions). We have ended up implementing a manual processing routine that bypasses the impersonation logic to allow debugging. The manual routine works well, but needs a dedicated resource to act as Queue Manager (not a good solution, but we are able to keep the queue in a controlled state).

I can't go into further details right now, but the new Batch Framework is causing us some pain and this is not what we where hoping for. When the batch jobs run successfully, the performance is acceptable and we are able to process a decent number of messages in any direction. To add some more mystic to the issue, we sometimes have the inbound processing running without problems, while we have problems with outbound processing. Suddenly this changes without any clue or trace of what happened. And we are monitoring the AIF lock table closely and we are also looking for database locks, without seeing any issues in these areas.

So the haunt continues and you can expect some more updates about this later when the issue hopefully is tracked down and solved.

If anyone have some more on this, you are welcome to leave a comment (I'm still optimistic).

So long

The haunt for knowledge

This is a follow up of the post regarding RPC errors.

First step to obtain a basic understanding of how MSRPC works can be found here. Microsoft has implemented their own version of RPC that they call MSRPC. Wikipedia puts the MSRPC implementation into a historical perspective and relates it to the source spesification called Distributed Computing Environment (DCE) from the Open Software Foundation.

I haven't yet found any specific information about how the Client- and Server Stubs are implemented in the Dynamics AX 3-tier architecture, but I'll try to dig a little bit more into this.

In addition to this, Microsoft recently also have published some KB articles on Partnersource that throws some more light on the issue I initially was blogging about. I'll try to find the URL's and update this post with the direct links when I get the time.

Update 2009-02-18:

Valuable links

Technet Magazine (article written by Zubair Alexander)
MSDN (Error Codes 1700 - 3999)
Florian's Weblog