Current challenges and issues

In my new role I have been busy working with a couple of hosted solutions experiencing various stability and performance issues. It’s easy to conclude that hosting AX solutions externally is demanding and that it requires a lot of attention from the customer to establish a good working relation between the hosting partner and the AX partner.

Even more important is the ability to regulate the responsibilities between the different parties. Simple things like for instance database maintenance must be defined in a way that separates regular maintenance not requiring AX knowledge and the opposite. Typical maintenance tasks like reorganizing and re indexing falls into the first category since it can be performed by any DBA (no changes of the definitions or design). When it comes to maintenance requiring AX knowledge (requiring changes in tables node in the AOT), it’s as important to makes sure the AX partner has the necessary access to SQL Server to be able to utilize all the valuable information provided in the form of Dynamics Management Views (I’m not mentioning Oracle here, since it seems like SQL Server is the dominating RDBMS for AX at least here in Norway).

SQL Server 2005 brought a lot of good news in this area and SQL Server 2008 took this even further with the introduction of a brand new Activity Monitor and the Performance Data Collection. Developers should in my opinion pay more attention to the load that their customizations put on the database server and examination of query plans, should be obligatory before releasing changes in the production solution.

In addition to the simple examples mentioned above, my experience says that even if each solution has different characteristics, some common areas can be defined to guide the approach. Without going into the details, the following summarizes what I normally define as the key sources for each category:

Stability
• Network related issues (AOS -> database, AOS -> application share)
• Operational knowledge (description of the relationships between the different server roles in the AX solution and routines describing how to perform a controlled stop – start sequence)
• Pro active maintenance of the AX application and kernels (implementing hot fixes and roll ups for current SP level, new SP)

Performance
• Physical disk I/O at the database level (separation and isolation, Raid levels, sector alignment, block size) regardless of DAS, SAN, NAS heads etc. (general rules apply)
• General load on database instance and utilization of system resources (both OS and SQL Server internals)
• Database configuration, usage and maintenance (best practice configuration, indexes, transactions, space allocation, index maintenance)
• Customizations (caching, run on, query width and selectivity)
• Identify and implement hot fixes related to performance issues (both application and kernels)

While talking about database configuration, I would very much recommend the Check lists that the AX Performance Team has compiled and published on their blog. This is vital information for everyone involved in installing and configuring AX, but also as a general source for hosting partners and customers. The best thing is that this is based on experience and best practice, and in my opinion the value of their recommendations goes beyond AX (general for OLTP applications).

In addition to working with troubled solutions (time consuming, but very motivating), I have implemented “my first” AX solution in a pure virtualized environment including SQL Server 2008 Enterprise. I’m excited about this solution mainly because it’s always interesting to make new experiences. Many people have probably walked this path already, but it’s a first time for everything.
This finalizes my first blog entry in several months, but I will be paying more attention to my blog from now on.

Happy weekend!

Opportunities

This time I'll post a short professional status - I'm currently unemployed. My company decided to end the Dynamics business after 2 years. Not the nicest entry to the summer holliday, but at least a opportunity to get close to the Norwegian Dynamics business talking to customers, partners and Microsoft in a new setting. This business seems to be going very well at the moment despite the economic downturn. One of my reflections is why my company decided to end this business when the market is very positive around both AX and CRM... Without going into details, ERP in general seems to be a challenge for my company and paired with the history of the company, it's quite obvious that ERP was never a part of the culture. This is always difficult to uncover during the interviews and you need some time within a company to really see this.

I hope to keep this blog going and to have a closure of my next professional move as soon as possible followed by a long summer holliday together with my family.

Happy summer to the WWW!

Update 09/06/26:

I'm closing the hunt and I'm signing a new work agreement early next week. And this blog will still be alive after a long summer holliday. Stay tuned for updates in August/September.

Gartner Magic Quadrant

In a recently published research report (June 4), Gartner places Microsoft Dynamics AX as the leader in the Midmarket and Tier 2-Oriented ERP for Product-Centric Companies.

"Gartner concludes that only one offering qualifies as a leader in the
market at this time: Microsoft Dynamics AX."

This is good news for everyone involved in both using and delivering services around AX, especially if you are working with AX 2009. I have been blogging about some of the features and changes in AX 2009 for some time and I find my own oppinion to be aligned with what Garter expresses around the technological aspects.

After reading the report, I thought about my experiences with Axapta/AX and I found it worth summarizing my history with Axapta and AX to put the product envolvement in a subjective perspective (from Damgaard via Navision to Microsoft as software vendor):

I first looked at Axapta 2.1 in 2001 and back then, it was considered to be a product with quite rough edges (a very young product born around 1998). The company I worked for at this time, decided to wait until version 2.5 before doing the first implementations. We got a lot of experience from these implementations and discovered that the product still had some rough edges (for instance the returning issues around the famous axdat.udb file espesially in solutions with clustered Application Object Servers). Then we got Axapta 3.0 adding more functionality. Axapta was already branded as a true international solution, but the requirements for doing a central implementation supporting users in different time zones, was driving the requirement for the number of AOS licensed heavily and such solutions did'nt support access to the data stored in the AX database across time zones (it was at best Unicode enabled if you remembered to enable this before synchronizing the database for the first time). This was also in the same period where Microsoft bought Navision. After using more time on finding work arounds to technical issues compared to bringing value to the customers, I decided to do something else for the next 3 years (I was a little bit fed up to be honest). When the opportunity to return to what then was called AX (4) back in 2007, I first considered the architectural changes (eliminated the axdat.udb file and put the license/session handling into the database, replaced the AOCP protocol with RPC, buildt a new Web application based on SharePoint, a pure 3-tier architecture and a greater range for the very important RecId value) and found it very promising. Based on this, I decided to give AX a second try. My experience with AX 4, proved that AX 4x was a big step in the right direction with regards to architecture. Now we only lacked support for handling users accross different time zones on one (or several) AOS instances with scaling and redundancy beeing the drivers for the number of AOS licenses required. This was finally introduced in AX 2009 and for the first time, I considered AX to live up to the promise of beeing a true, international solution. Add even tighter integration to other Microsoft products and technologies, and a Web Application "bringing BI to everyone" through a role based Enterprise Portal, and AX 2009 was positioned to compete with the other 2 main rivals also in the Enterprise Market.

I don't regret returning to AX and I'm rest assured that AX 2009 and later releases, will move further up and to the right in the Magic Quadrant "cementing" it's position as the most agile and TCO effective ERP solution on the market. SAP Business One is of course a serious player and it will be interesting to see how the competition evolves the next years.

So what's my point here? Given the history of AX and the fact that Microsoft now has done the necessary and required changes with regards to the architecture (not a small task), the product enters a new era. Companies looking for a new ERP solution, should indeed evaluate AX 2009 in line with both SAP and Oracle. And existing AX customers running on a version prior to AX 4, should work through their exisiting solution either aiming at eliminating as many customizations as possible or in fact re implement AX 2009 with a clear strategy around utilizing standard functionality to lower TCO over time and keep in pace with new/added functionality. It's in my oppinion all about positioning for a very exiting product cycle where I expect a lot of new functionality to be introduced (both horizontal and vertical) and less architectural changes!

Some information about the next release of AX (6) is already available (AOD files moved from file system to the database, increased range for ObjectId etc.). Some will probably argue that this is architectural changes, but as I see it yet not ground ground breaking compared to the changes already implemented in AX 4 and AX 2009. If you have access to the Product Roadmap, you can read what Microsoft is planning for the future and my final word, is the fact that Gartner concludes that Microsoft is delivering on their vision and that this is one of the key reasons for Gartners conclusion!

Happy reading.

Mounting the AX 2009 Demo VPC under Hyper-V

I recently was asked to make sure that we could demonstrate AX 2009 outside MS Virtual PC 2007. I admit that Virtual PC is not the ideal virtualization environment for demos especially if you don't have a secondary hard drive with sufficent performance (a fast disk) and sufficent memory resources. Since most people carry a standard business configured laptop computer, it's hard to run a virtual machine with satisfactory performance.

The other issue is whether to utilize the standard demo VPC's (Microsoft has 2 demo VPC for AX 2009) or to create your own. Since setting up and configuring a complete AX 2009 solution has become an hour intensive task due to the number of components and supporting software required, I decided to use the standard VPC provided from Microsoft (17 files for download for demo VPC 1 aka AX-SRV-01).

An additional requriement was that users should be able to connect to the virtual machine remotely.


Here are the tasks I perfomed to provide the standard demo VPC under Hyper-V:

1. Downloaded 17 files for demo VPC 1 on my laptop


2. Exctracted the VHD to my laptop


3. Mounted the VHD under Virtual PC 2007 and started the VM with public networking


4. Updated the VM with the latest security updates and verified that the firewall was running


5. (Now I could have uninstalled the Virtual Machine Additions, but I decided to to this after mounting the VHD under Hyper-V)


6. After stopping the VM, I copied the VHD to the Hyper-V host (approxemaetly 30 Gb)


7. Mounted the VHD under Hyper-V and started it through the Hyper-V console with public networking (no networking will be enabled until Integration Services is installed)


8. Uninstalled Virtual Machine Additions and restarted the VM (some creativity was needed at this stage to be able to access Windows Explorer)


9. Installed Hyper-V Integration Services and restarted the VM


10. At this point the VM was fully operational including Remote Desktop Connections , but none of the configured Web Sites was working...


11. After investigating the Web Site configuration, I noticed that they where configured to use host headers which led me to further investigate the DNS configuration. Without beeing a DNS expert, I concluded that the relationship between the defined DNS records and the original IP configuration (192.168.0.1) on the one side and the forward lookup zone on the other, was important. I did NOT want to tamper or alter the DNS configuration to avoid a lot of reconfiguration (remember that the server runs several roles including Domain Controller).


12. After stopping the VM, I created a new private virtual network (Hyper-V networking) and I added a second network card that I bound to the newly defined network


13. After starting the VM once again, I defined a static IP (192.168.0.1) on the new network interface and voila - all Web Sites where operational and accessible again without any reconfiguration.

I allocated 3 Gb of memory and one virtual CPU to the VM under Hyper-V.

The overall performance of the VM was supprisingly good even when executed on a Hyper-V host with local disks (no SAN) and sharing resources with a large number of other virtual machines.

The major limitation with this setup, is the number of concurrent RDP sessions (WTS running in admin mode).

Watch out for the combination of reports as PDF with graphics and batch execution

We have seen this issue in our design and finally Microsoft has now verified (newsgroup microsoft.public.axapta.programming) that AX 2009 has an issue when you use the buildt in PDF functionality for reports containing graphics (logos etc.) and the logic executes as part of a batch job. The reason is that the PDF AX logic uses the Image class to handle the graphic and the Image class is in fact (documented) bound to the client tier. So this is an impossible combination and in my oppinion, a good example of issues in the shadowlands of the changes in the Batch Framework and the default application logic provided by Microsoft.

The solution is to use a PDF printer driver to produce the PDF file, but this should'nt really be necessary since AX has built in support for PDF generation. If you have read some of my earlier blog posts, you will probably notice that we have experienced some major issues when running the application logic in batch under the new Batch Framework and the lesson learned is that you should pay extra attention and put in extra effort to make the batch jobs run as expected.

My best tip is to plan for this as an extra test activity during build (or as an additional payload during testing) and reflect this in the estimates given to the customers. I conclude that the comlexity is rapidly increasing as a consequence of a very good architectural change in AX (server bound batch execution and impersonation), but the application lacks some conditional checks and maybe also some best practise checks in the compiler ("Dear developer, you are trying to call a class on the server that is bound to the client. Please consider your design and look for alternative solutions").

Staying on top of things - AX 2009 hot fixes

To be honest, I find it very hard to keep track of the available hot fixes (and cumulative rollup packages). By very hard, I'm mostly thinking about the time spent on 1) searching for possible hot fixes and 2) if one possible related hot fix is found, understanding what the hot fix actually solves.

Partnersource (or Customersource) is the primary source for this kind of information by searching the Knowledge base. Maybe it's only me, but I would like a much more effecient tool and much more detailed information about the individual hot fixes to keep our customers happy and of course spend as little time as possible on this activity. The real trouble arises when you find KB number without a published KB article. What do to in these situations? The answer is to ask for unpublished information through your support channel and hereby allocating even more time and resources.

So people; take the opportunity to vote by answering the recently published poll on the right hand side in this blog.

I promise to share the results with Microsoft.

Bottom line is that it's all about quality of the software by easily providing current information about available hot fixes to the partners (and customers).

Debugging Batch Jobs

We have a hard time getting the AX 2009 debugger to work when breakpoints are set in code executing in Batch Jobs. Microsoft has described this in several documents without any special requirements, but we are unable to get the debugger to work as expected. All settings are correct in the server configuration and in the client when the breakpoint is defined. We have also tried to use the keywork breakpoint in the code without any difference. It seems like the debugger has a hard time attaching to the process. We have tried this in several environments and solutions, and in different combinations, and basically we are stucked.

The fundamental changes in the Batch Framework requires more debugging than ever due to the increased complexity inherited from the changes around impersonation (RunAs) and the prmiary tool in this situation is in fact the debugger.

We have also checked with several other sources and they have done the same experience.

All experience around this issue is highly welcome.

Update 2009/04/03:

Microsoft has confirmed that they have received a problem report (4652) for this issue, but no solution or fix is available yet. The problem was reported in January 2009.