AX 2009 Technical Journal

AX 2012 Technical Journal

2011-11-03T10:08:00.006+01:00

Just a short message - future entries regarding AX 2012 will be published in a separate blog titled AX 2012 Technical Journal.

Stay tuned for new updates.

AX2009 SP1 secret statistics

2011-02-17T15:16:00.000+01:00

I while ago (September 2010) I asked a question based on a simple graph. I thought it was time for an update leaving the question open.

Looking forward (read: AX 2012)

2011-02-09T10:56:00.000+01:00

Back in November 2008 I blogged about AX 2009 after attending the Convergence conference in Copenhagen and I was pretty excited about the possibilities. After working with AX 2009 since it was released in 2008, I'm still excited but also more realistic about several aspects of implementing and operating AX. After gaining hands on experience, you discover all the rough edges and you get a feel for the quality of the software with regards to things not woring as expected for instance measured as the number of hotfixes released. All in all it seems like AX 2009 has succeeded in the market.

So whats next? AX 2012 was officially published as the name of the next version of AX (aka AX 6) at the Microsoft Dynamics Technical Conference taking place in Redmund in January. I wasn't able to participate, but based on the information published after the conference, it certainly seems like a huge (giant?) shift both with regards to added and extended functionalty, and technical with some architectural changes as well (like the possibility to replace the classic AOD files in the file system with storage in the database).

The post by Jake at AxaptaSource gives a good summary of what was presented at the conference and it should be pretty clear to everyone that we are talking about the biggest shift in the history of AX and a shift that will have an impact on everyone involved in the value chain. This is of cource in the category of un official information, but still quite trustworthy as far as I can understand.

Modelling is one example that will influence developers and people involved in deploying changes. Much tighter integration to Visual Studio is another. The security system is re designed with roles becoming a real part of the security system (a role in AX 2009 is the same as the Role Center assigned to a user at least within AX) and the requirement of defining a user in Active Directory, seems not to be a requirement anymore (replaced by Claims based security in SharePoint for non AD users). Setup is made easier and system management is prioritzed with a new add in for System Center Operation Manager. The BizTalk adapter is gone and replaced by the WCF adapter (Web Services). The (data) caching machanisms are re designed (much needed in my mind). Include columns are finally supported for SQL Server 2008 togheter with support for full text indexes. Huge performance improvements mainly in the area of CIL execution (Common Intermediate Language). Drag and drop transformation of forms to web pages in Enterprise Portal (indicates big changes in the area of forms in general). Workflow executing as part of the AOS service and shared workflow editor (designer) withVisual Studio 2010. Management through PowerShell. AIF service contracts have changed (possibly influencing the upgrade effort) and less complexity (hey, that sounds like a step in the right direction). Very thigh integration to Excel supporting data entry and synchronization from Excel (big time mental change for a lot of people I think). Big improvements in the area of debugging and batch execution in addition to AOS beeing able to host WCF processes (a real enabler).

Add a module for Public Sector (be aware Agresso and Visma :-)) and 1200 new features in addition to some major rewrites (rumors), and you probably start to see the big picture. I think the AX Community must start preparing for AX 2012 on the mental level since this release will require a shift in most of the ways we have been used to treat new releases of AX.

Virtualization of AOS - VMWare versus Hyper-V

2011-02-01T14:58:00.001+01:00

A very interesting thread on the Dynamics AX Community forum regarding virtualization of AOS.

As stated, the only important information missing right now, is a description of the differences in the underlying infrastructure between the VMWare and Hyper-V implementations.

Update:
Te original poster has published information regarding the infrastructure and this information only increases the value of this subject - he has managed to test virtualization of the AOS role under VMWare and Hyper-V on the same (identical) infrastructure! This makes his findings extremely interesting (at least to me) partly because it leaves room for possible reasons/explanations/theories and maybe the biggest paradox - if his findings are true in general, a lot of customers running AOS under VMWare are possibly influenced by decreased performance caused by differences in how the AOS behaves on VMWare. Since most customers most likely are using VMWare, the scope of this subject with regards to the number of customers possibly affected, makes it one of the most interesting subjects in a long time.

It could of course be tightly coupled to the OS and how the OS performs under the two hypervisors (as mentioned as a possible reason in the thread) and I will be light on this until some more research has been done.

Comments are welcome as usual.

AX 2009 and edition of SQL Server Analysis Services

2011-01-28T18:20:00.000+01:00

I recently came across an issue when trying to build a SSAS project containing modifications to the standard AX Cubes.

When building or rebuilding the project, I got 1 047 (!) errors with the description "Dimension [x] : Only dimension attributes and attribute relationships can have translations in Standard Edition."

The only solution to this is to (try to) modify the SSAS database in SQL Server Management Studio by some heavy scripting. Or of course upgrade SSAS to Enterprise Edition which I really don't know is legal without licensing Enterprise Edition (I'm trying to clarify this). Anyway it's a valid upgrade path regarding to the information given by Microsoft for SQL Server 2008 R2 (Version and Edition Upgrades) on MSDN.

The system requirements for AX 2009 doesn't contain any information saying SSAS Enterprise Edition is needed and neither does the general SQL Server reqiurements.Until further noticed and official information is given from Microsoft, it's worth having this in mind when deciding which edition of SQL Server to use together with AX 2009.

The only information from Microsoft I have found on this issue, is in a draft document titled "How to add financial dimension to Dynamics AX default cubes" (PDF) from the EMEA Dynamics Support blog on MSDN. On the bottom of page 3 and the beginning of page 4, the following information is given:

This is the only information from Microsoft I have found and despite this beeing a draft document, it clearly states that the system requirements should be updated alternatively that another soultion is provided. From this I understand that you have to backup the OLAP database and restore it to an instance of SQL Server 200x Developer Edition (equals Enterprise Edition in functionality), create a project in BIDS, build and deploy the changes to Developer Edition, and finish doing a final backup - restore sequence back to the production instance of SSAS.

In general, I think most customers will end up designing their own cubes and modifying the needed Role Centers accordingly, to avoid all the struggle with the default cubes.

Experience doing an AX 2009 Complete Install

2011-01-18T09:48:00.000+01:00

A short update on my experience from doing a complete install of AX 2009 with dedicated servers for each server role.

First of all, the process went quite well and the only struggle was to get all the Service Principal Names (SPNs) correct including named instances and the SQL Server Browser service on the SSAS server.

Main take aways:

Spend time on planning and account for some issues related to Kerberos authentication (hard to estimate)
Follow the latest version of the AX Installation Guide
Make sure you deploy the latest Cumulative Rollup package for AX (and other services)
Bookmark the AX EMEA Support Team Blog (very valuable source of information)
Make sure each component work as expected before you proceed the install and configuration sequence

After installing SharePoint 2010 for the first time, I also have a suggestion for Microsoft regarding AX setup - please have a look at the SharePoint 2010 Products Preparation Tool (shown below) and incorporate this idea into the next release of AX.

AX 2009 Rollup 6 and KB 2278963 (SharePoint 2010)

2010-12-22T15:52:00.001+01:00

Finally I'm doing a complete install of Dynamics AX 2009 on a new plattform for a customer! While performing the installation of the core solution, my verified plan (verified by Microsoft) was to install in this sequence:

AX 2009 RTM patched with binaries from the KB
AX 2009 SP1
AX 2009 SP1 RU6
Initialize
Apply application fix for the KB

Everything went according to plan until step 5. This step failed with the following message:

This update is not applicable to your installation of Microsoft Dynamics AX because wrong version of application files is installed. This update is applicable to version 5.0.1000.52-5.0.1500.3560 of application files.

Setup could not complete the installation.

After checking my sources and discussing with Microsoft without any progress, I decided to install a second application in this sequence:

AX 2009 RTM
AX 2009 SP1
Initialize
KB 2278963
AX 2009 SP1 RU6

This time everything went according to plan and based on my experience, one of the following are true for this issue:

KB2278963 application hotfix is forgotten with regards to RU6

KB2278963 is included but not documented as part of RU6 (or RU5)

Wrong version is used when the axupdate process verifies the application version (max is 5.0.1500.3760 - RU6 gives 3761)

The worst thing is that I had to spend extra time and effort doing a new install and 2 compilations in addition to installing a separate AOS and a Client to work on the same versionlevel.

AX 2009 64-bit AOS instance memory usage

2010-10-29T12:01:00.001+02:00

I have seen some people asking if the 64-bit "edition" of the AOS kernel is capable of utilizing memory above 2 Gb (search microsoft.public.axapta which by the way is or is going to be retired). The snapshot below is taken from a customer site.

Working Set equal to 2 Gb and Peak Working Set equal to 4,1 Gb - say no more.

Question

2010-09-21T14:53:00.000+02:00

A little challenge to my readers:

Can You tell me what I'm trying to illustrate in the graph below?

A couple of tips along the way

it's closely related to Microsoft and Dynamics AX 2009 SP1...
the average is almost 69...

Please leave your answer as a comment.

Follow Up on Event ID 110 and 117 (annoying database error)

2010-09-15T11:24:00.000+02:00

I wrote about this a while ago and since this still is what I call an annoying database error (noice) and something very common to AX 4.0/AX 2009 solutions, I would like to call on my readers to vote for a change in the AOS kernels (x32/x64) to filter out this information to prevent bombing the event log with errors not beeing errors at all. The voting can be done at Microsoft Connect.

Finally I would like to quote a paragraph in Microsoft KB article 143339:

Applications need to code their error handling routines for SQL Server connections to call SQLError() until it returns SQL_NO_DATA_FOUND, and act on any messages other than the ones that return a pfNative code of 5701 or 5703.

Maybe it's impossible to change this behaviour, but as long as Microsoft don't state this in their feedback on Connect, it's still an issue that I find hard to accept...

AX 2009 - deadlocks

2010-09-07T14:39:00.000+02:00

I recently conducted a system review at a customer site and discovered a message totally new to me beeing logged in on all AOS: "Object Server xx: Dialog issued for client-less session 1: Cannot edit a record in LastValue (SysLastValue). User ID: ,AdminUserSetup. Deadlock where one or more users have simultaneously locked the hole table or part of it."

The database at hand was not configured to use Read Committed Snapshot Isolation (RCSI) as recommended by the AX Performance Team. Enabling RCSI on the database solved the issue.

Missing information

2010-08-27T10:58:00.000+02:00

I recently was asked about the difference between the 4 alternative values for "Cache Settings" found on the tab labeled "Performance" in the AX Configuration Utility (client) - se example below.

The different options are explained directly in the GUI, but I would like to know how this mechanism is implemented (technically). I have searched all possible sources (formal and informal) without finding any information at all. It's not unusual to hit "blind spots" like this regarding formal documentation around Dynamics AX, but it should not be necessary to open a support case every time this happens.

Anyone having a reference or are willing to share some experience regarding the effect of this setting?

Comments

2010-06-24T11:39:00.001+02:00

I have changed the possibility to post comments from anonymous (all) to Registered Users. Reason: loads of spam comments in my mailbox.

Event ID 110 and 117 (annoying database error)

2010-03-08T13:09:00.006+01:00

I observe the issue described below in both AX 4.0.x and AX 2009 SP1 x solutions running SQL Server 2005 and 2008 (the version of SQL Server Native Client is the same). This is also reported by others in several Communities like microsoft.public.axapta (last one beeing a post by Mike titled "AOS errors" from February 25 2010 where one suggestion is to syncronize the SYSLASTVALUE table).

Eventlog, application

#1 Event ID 110

Object Server 01: Dialog issued for client-less session 1: Cannot edit a record in LastValue (SysLastValue). User ID: , AdminUserSetup.
The SQL database has issued an error.

#2 Event ID 117

Object Server 01:
The database reported (session 1 (-AOS-)):
[Microsoft][SQL Native Client][SQL Server]Changed language setting to us_english.
The SQL statement was:
"UPDATE SYSLASTVALUE SET VALUE=?,RECVERSION=? WHERE (((((((USERID=?) AND (RECORDTYPE=?)) AND (ELEMENTNAME=?)) AND (DESIGNNAME=?)) AND (ISKERNEL=?)) AND (COMPANY=?)) AND (RECVERSION=?))"

I don't know if this has any impact on the solutions at all and I have also found some information on Partnersource describing this as "noice". And I also find it a little bit hard to understand why a syncronization of the system table SYSLASTVALUE should be needed since the real error message in fact is stating "Changed language setting to us_english.".

The reason for posting this is that I find it a little bit hard to understand why something is logget as an error situation if this is "just noice". AX is in general bad at categorizing events in event log application, but I would like to share some experience with others regarding this. Since it's reported from the ODBC layer it could be related to settings like default language for the DBO user (aka AOS Service Account) or inproper settings for Regional Settings for the same Service Account. This is a couple of my own teories left to verify, but bottom line I find it hard to be "just noice".

Hopefully we can start a good discussion around this and bring the issue forward to the system vendor instead of reporting it individually (perhaps a bigger impact).

Update March 10 2010:
For those of you not reading the comments, I would like to point out that Chris pointed to an important KB-article from Microsoft. After looking a little bit more at the details (Googled for NativeError 5703), I found another reference on MSDN. Quote:

The SQL Server ODBC driver returns SQL_SUCCESS_WITH_INFO on a successful SQLConnect, SQLDriverConnect, or SQLBrowseConnect. When an ODBC application calls SQLError after getting SQL_SUCCESS_WITH_INFO, it can receive the following messages:

5701—indicates SQL Server initially putting the user's context into the default database defined at the server for the login ID used in the connection
5703—indicates the language being used on the server

If either the ODBC data source has a default database specified or the application specified the DATABASE keyword on SQLDriverConnect or SQLBrowseConnect, there will be a second 5701 message that indicates the user's context has been switched to the database requested.

My conclusion is that the AOS kernel treat every error raised from the ODBC layer as an error. If Microsoft could implement a change to filter out NativeError 5701 and 5703, a lot of confusion could be saved and not at least a lot of time investigating this issue could also be saved.

Until further notice, this is my conclusion.

Again, thank you Chris for leaving this valuable information.

Update #2 March 11 2010:

I have today posted this as a suggestion on the MS Dynamics AX Suggestions Home page at Microsoft Connect. Please sign in at Connect and vote this up if you find this issue to be a valuable improvement.

Update #3 May 11 2010:

The EMEA Dynamics AX Support team recently released a post giving another explanation. It can be found here.

AX Load Balancing

2010-03-05T13:01:00.001+01:00

A tree tiered architecture has been around since AX 4.0. In AX 2009 you have 3 possible AOS configurations:

Single AOS instance(s)
Load Balanced AOS instances without a dedicated load balancer
Load Balanced AOS instances with a dedicted load balancer

After experimenting a little bit with all of the possible configurations, I find it a little bit difficult to separate configuration 2 and 3 above. The only difference is the presence of one (or several) AOS instance(s) defined as dedicated load balancers only handling the load balancing functionality (no end user sessions) for AX clients that whish to log on to AX (handshake). So the big question is when to use a configuration with a dedicated load balancer. I can't see any reason or benefit other than limiting the need to adjust the client configurations when AOS instances move out and in of the cluster. The only benefit deploying a dedicated load balancer is that this will be the only AOS instance referenced by the clients and that this makes the client configuration more static compared to configuration 2 when every AOS instance in fact both is acting as a load balancer and handling client sessions.

And information from Microsoft states that a dedicated load balancer only is needed when you have more than 4 AOS instances in the same cluster.

All combinations in the client configurations will work (no difference wheter you only reference the dedicated load balancing instance and/or all AOS instances).

Any comments around this matter will be appreciated.

Short note on AX and database mirroring in SQL Server 2008

2010-02-03T23:55:00.001+01:00

The last days, I have been working with a customer running AX 2009 and SQL Server 2008 SP1 Standard (x64) on Windows Server 2008 SP2. The solution suffered from general performance issues that possibly could have a lot of sources. As usual my attention was around SQL Server and this time I started looking at Wait Stats which showed a waste amount of waits related to mirroring. After some Googling, I was a little bit confused about this beeing normal or not. The nature of database mirroring could in fact result in high Wait Stat values since the processes involved, is mostly sleeping (suspended). But I choosed to consider this as a possible source since the nature of database mirroring in my oppinion, does'nt fit the nature of AX as a classic OLTP application. And since disabling database mirroring is an isolated and low risk operation, we choosed to stop mirroring for all database (around 10). After this, the overall performance increased and the customer reported this almost immediately. It's a little bit early to conclude, but the initial responce seems pretty promising.

I will not go into the details around how mirroring was set up and the underlying infrastructure, but since AX is a OLTP application with a lot of inserts, deletes and updates within small, repeating and identical transactions during a normal business day, the processes responsible for doing the mirroring seems to have a big payload on the source server. Remember that database mirroring in fact is a repliction technology and that changes in the primary database, must be replicated to the secondary database (add requirements for low network latency and quality of service). It could of course be other applications that contributes more to the payload, but since time is important in situations like this, we only focused on the SQL Server instance and less on the individual databases.

Another lesson is to be very restrictive when deploying several databases under the same instance as your primary AX database. The payload of each database should be evaluated and categorized to avoid resource conflicts between databases (for instance queries in the wild, long running transactions and cursor intensive applications). Despite all the positive effects of consolidating databases, the complexity increases a lot when performance issues arise for AX since the source could be another application. My general advise is to prioritize a dedicated databaseserver for your AX production database since this makes it a lot easier to optimize AX. Yes, it will have an impact on the cost (hardware and licenses), but it will normally pay off in the long for customers relying on AX as a mission critical solution. This is an advice I generally bring to my customers.

If interesting, I can throw in some more details. And yes - I'm aware of database mirroring beeing one of the supported HA configurations for AX. Please add a comment to indicate further details.

So long.

Minor observation installing AX 2009

2010-01-06T11:03:00.000+01:00

Some time back, I did a QA run on one installation suffering some performance issues. The consultant originally installing AX choosed to use the installer to create the database (shown in the picture below). Personally I prefer to create the database manually, both to get control over every aspect and it usually don't take more than 30 minutes to get everything in place manually. The solution at hand was running SQL Server 2008.

While looking into the database configuration (sp_helpdb), I noticed that the compatibility level was set to 90 which corresponds to SQL Server 2005. At this stage, I asked the consultant doing the installation why he had choosen to set the compatibility level to 90 and he then answered that he always used the AX installer to create the database.

Without testing this option when installing AX, it seems like Microsoft has provided a template database beeing attached to the specified SQL Server instance by the installer in the security context of the setup user. I guess the installer asks for some information like collation, file locations etc. but compatibility level keeps untouched.

Compatibility level is a database level configuration option and it is used by the database instance to decide which functionality the database instance should "activate" for each database. It's mostly used to provide backwards compatibility during migration of databases, but it could also impact the ability to utilize new features.

A good reference can be found at MSDN and compatibility level could possibly be a source for some time consuming investigations. It seems like compatibility level = 90 is OK even for an AX database running on SQL Server 2008, but I don't see any reason for not setting it to 100 to make sure all new features in SQL Server 2008 is enabled for the AX database.

Or always create the database manually using the system database model as the template (will in most situations match the version of SQL Server).

A new year approaching

2009-12-23T23:51:00.000+01:00

With another week to go and the Cristmas hollidays coming, it's time to reflect on the past and coming year. I think 2009 was a good year for AX 2009 despite the product beeing released in 2008. The main reason for 2009 beeing a good year for AX 2009 is mainly based on the Gartner report positioning AX 2009 the number one (#1) leader in the magic quadrant for what we call SMB here in Norway. This could of course be marketing hoopla, but if you read the report, Gartner pays special attention to "Microsoft delivering on their vision". Personally I find this interesting and I suggest that every partner study the current Roadmap and Statement of Direction (SOD) to see what Microsoft is planning (visionary) for AX in the future. In my mind, AX 2009 did'nt bring a lot of news on the techology side (the new batch framework and the support for UTC is perhaps the most important improvements), but overall MS managed to position AX as a real challenger and maybe also a winner in the combat against SAP and Oracle/PeopleSoft.

Based on the number of hotfixes released for AX 2009 and SP1, MS still has some challenges regarding quality. Maybe a slowdown in the release cycle would be a good advise and maybe also broaden the early adoption program to gain some more experience from the field. Again this is really not special to AX, since we have to remember that all software is manufactured by humans and no human is free of errors. All in all I think AX 2009 was a great step forward at least with regards to technology and architecture, since the product is well positioned to compete with the biggest rivals even in the upper, right quadrant. Finally "MS got rid of most of (every?) not so industry standard implementation as expected" and lifted the product up to the MS level of integration (still some ground to cover, but yet greatly improved).

So it's exiting times generally and with regards to AX. I'm positive about the year to come and also the next decade. From the Roadmap and SOD, I read a steady growth both vertically and horizontaly with regards to functionality. We are already seeing some evidence of this with MS buying verticals from partners (like the POS vertical from a Danish world wide partner financing? a rather special twist in buing a highly specialized ERP solution called Guideix A/S). The future story of this evolution of one of my previous employers, will be very interesting to follow. Driven by market, customers or product/technology? Who knows...

Anyway I'm optimistic about the future of AX and I seriously meen that AX 2009 brought AX closer (really close) to what the product has been marketed as since the beginning (Damgaard back in 1998 - more than 10 years ago). At the same time I have to admire the Damgaard brothers for the introduction of a true 3-tired solution (was it 2.5?) that MS now has adopted and brought forward to a pure 3-tiered solution with no other choices. Add the rather annoying AOCP beeing replaced with MS RPC in AX 4.0 and you get the picture.

One final message to all x++ developers out there, is to take the time to study and understand the difference between the different caching schemes available in AX (look at the DEV III course documentation or Inside AX 4.0/2009 book). This is key knowledge to utilize the 3-tier architecture in AX 4.0 and 2009 (also valid for earlier versions when running in 3-tire thin configurations) and at the same time, rather complicated. From my experience, this is an area that get far to less attention and that has a big potential for optimizations (and performance gains).

Without digging deeper into the philosopical area, I would take the opportunity to wish my few (but valuable) readers, a happyy Christmas and an interesting new AX year (and decade)!

See you in 2010! So long...

DAS or SAN for SQL Server part II

2009-12-16T09:04:00.001+01:00

I thought a follow up was adequate for this one. After looking for some more views, oppinions and recommendations, I found the following posts (3 part) on the blog for the Microsoft SQL Server Development Customer Advisory Team:

Deploying SQL Server with SAN #1
Deploying SQL Server with SAN #2
Deploying SQL Server with SAN #3

There are many different oppinions out there, but I usually pay special attention to what Microsoft says since they are as close to the products as possible. At least I found this information valuable as kind of a formal reference. Look at separation of I/O based on the characteristics for each category of file types that SQL Server ulimately read and write from/to as a good example (often argued by the SAN guys - "we want to maintain a single array of disks instead of hooking a set of disks to a LUN"). And as argued earlier, SAN's complicates and often blurres the picture when dealing with performance issues when you are sharing the SAN with other applications/usage patterns (dedicated SAN to SQL Server is a good thing).

In my oppinion it's important to get every piece in a solution configured as optimal as possible from the start to secure a best practise deployment. A best practise deployment will again make sure that you have control over the deployment with regards to configuration and setup, when for instance performance issues arise. Under these circumstances, it's a lot easier to narrow down the scope for the activities necessary to diagnose the issue (similar to what the doctor does when you feel ill).

And the different SAN vendors add some more to the table where each vendor has it's own set of recommendations and of course some invaluable piece of magical software solving every challenge.

As you probably have noticed, I'm a simple man trying to live by the prinicple "Keep It Simple Stupid", but this is not an easy rule to apply in the world where IT technology plays a key role in every business.

It would be interesting to gather some comments around this matter, so come on and post your comments preferably in the form of real world experiences.

So long.

AX Application DBA

2009-12-15T13:19:00.001+01:00

Yesterday I recieved a newsletter from SSWUG titled

Are You an application DBA?

My first reflection was that I have been and that I partly still is an application DBA for AX (and also some Axapta solutions). My second thought was that this is an interesting question, since I very often see a total lack of DBA responsibilities and naturally even less DBA's that focus on AX. When I meet someone with a DBA role, it's often in relationship to hosted solutions (outsourcing) and these guys are general DBA's focusing on "the big picture" (at best).

One example was a customer with a consolidated SQL Server solution counting almost 60 user databases supporting the same number of applications. When facing these kinds of situations, the potential for optimizing the AX database is rather small. Take tempdb as an example. AX 4.0 and later benefits from Read Committed Snapshot Isolation as a consequence of introducing versioning and optimistic locking. This again increases the load on tempdb, since the version store are held in tempdb. Since tempdb is a system database shared between every database on the same instance, the total load on tempdb again impacts AX. And I can say that different applications uses tempdb differently and often not following best practice. Mixing OLTP and OLAP load on the same instance or database server, is another classic.

This is some examples of when you need to know how the application uses SQL Server and maybe not something a general DBA would pay attention to without looking into how each application actually utilizes tempdb. Another example is locking and lock escalations in other applications impacting for instance AX. Add databases consuming a considerable amount of CPU in combination with high I/O load.

So in my oppinion it's a great need for application DBA's and AX is a good example since AX is a ERP solution (very often mission critical). I would bet that all customers running AX would see a good return on investment (ROI) hiring an application DBA for AX responsible for pro active maintenance and follow up on queries not performing at the full potential, long running queries etc. But the economic downturn probably impact the willingness of customers spending money on this and this again justifies for calling in people like my self for short term activities (fire fighting). This is a bit of a paradox to me...

AX and virtualization (to do or not)

2009-12-04T15:00:00.004+01:00

I have only done one completely virtualized implementation of AX 2009 (including SQL Server 2008 EE x64). This solution is not yet in production, but in the meanwhile I have looked at several other virtualized implementations done by other partners and often operated by another vendor (hosted).

One of them (AX 4.0) consists of 2 AOS servers in a normal AX load-balancing cluster. The interesting part in this setup is that one server is a dedicted, physical server with DAS, while the other is virtualized (ESX 3.5). When talking to the users, they complain about overall performance when having sessions against the AOS instance on the virtualized server. I suspect that the most clever users always looks at the caption in the main AX window to see which AOS server they hit (and maybe they also start one additional session immediately to hit the physical one followed by closing the first session against the virtual). Everything seems ok from the server console (PerMon, TaskMan, Event log etc.), but I also feel (subjective) that the loading of the AX client also takes some more time when hitting the virtualized one.

Again I have to talk about complexity. Ideeally AX AOS should be the perfect candidate to virtualize since it's a CPU, memory and network intensive process. The AOS server will never pay a high physical I/O load and disk I/O is in general low (of cource some batches could impact this in AX 2009). And AX 2009 is now fully supported on several virtualized platforms. So exactly why do I bring in complexity again? As with Storage Area Networks, a virtualized environment is simple to utilize (when it's working as expected), but the technology and the levels of infrastructure behind adds a lot of potential sources when things are not running as expected (most virtualized environments also utilizes a SAN). I guess the odds for getting down to all the needed details is a lot better when implemented at a customer site, but customers nowdays tend to outsource this since "it's not part of their core business". In this scenario, the complexity is very visible since the AX partner (or the consultant doing the performance audit) don't even get all the details or access to the parts of the system necessary for defining the big picture. Seen from the hosting partner, it's all about utilizing the underlying platform as much as possible trying to maximize the potential of the platform. This often means mixing customers on the same platform (logically isolated on every level), but ulitmately sharing the exact same resources at a certain level. This again typically leads to traditional bottlenecks, but they are well hidden for both the customer and the man in the midle.

So what's the lesson here? Always differentiate between hosted solutions and locally implemented ones! The main principles are the same, but it's cruical to bring in the hole army of partners in the planning when the solution is operated by a hosting partner. Without this, you are basically left on your own and you'll probably never be able to see the big picture consisting of all the details, to judge where the reall issues causing the problems are hiding. I guess this is true for every application or solution.

DAS or SAN for SQL Server?

2009-11-18T09:06:00.002+01:00

Performance issues is always complicated, but a common approach is possible (in fact the only way to work through this is to define the problem area and structure the different factors into groups). In my approach the operation platform is one key group while the database is another. Based on the work I have done since September, I have found it very hard to get real and trustworthy information about the utilization of SAN storage when this is used for SQL Server. And as a advisor I don't have deep knowledge about the solutions from the indivdual SAN vendors (expertice and experience required). In my mind the old rules about separation of physical I/O from the I/O characteristics still applies despite the fact that SAN's normally have high end controllers capable of handling huge I/O loads. DAS (Direct Attached Storage) as the traditional way of providing storage, seems to be much easier to work with when analysing performance issues at the database level mainly because good old Performance Monitor will tell you all you have to know to conclude and defined possible counter measures. This is NOT the same for SAN's! Since a SAN mainly is used for storage provision and consolidation, you have to work your way through the hole SAN setup and identify every piece of software that is generatig I/O against the SAN. Add some interconnect (normally fiber switches, HBA's etc.) in between and you have multiplied the complexity by at least PI.

Add another overhead for hosted solutions where several customers normally share the same infrastructure and the picture starts to be rather though to control. During my search I found another blog discussing the same issues and his summary resembles much of my own experience.

So this is yet another example of how developments in technology actually complicates the everyday work for both customers and consultants, and this is clearly something to account for when doing the TCO matrix.

Based on this, customers should evaluate SQL Server storage and consider to implemented database storage as DAS. Do you agree?

Current challenges and issues

2009-11-06T14:10:00.003+01:00

In my new role I have been busy working with a couple of hosted solutions experiencing various stability and performance issues. It’s easy to conclude that hosting AX solutions externally is demanding and that it requires a lot of attention from the customer to establish a good working relation between the hosting partner and the AX partner.

Even more important is the ability to regulate the responsibilities between the different parties. Simple things like for instance database maintenance must be defined in a way that separates regular maintenance not requiring AX knowledge and the opposite. Typical maintenance tasks like reorganizing and re indexing falls into the first category since it can be performed by any DBA (no changes of the definitions or design). When it comes to maintenance requiring AX knowledge (requiring changes in tables node in the AOT), it’s as important to makes sure the AX partner has the necessary access to SQL Server to be able to utilize all the valuable information provided in the form of Dynamics Management Views (I’m not mentioning Oracle here, since it seems like SQL Server is the dominating RDBMS for AX at least here in Norway).

SQL Server 2005 brought a lot of good news in this area and SQL Server 2008 took this even further with the introduction of a brand new Activity Monitor and the Performance Data Collection. Developers should in my opinion pay more attention to the load that their customizations put on the database server and examination of query plans, should be obligatory before releasing changes in the production solution.

In addition to the simple examples mentioned above, my experience says that even if each solution has different characteristics, some common areas can be defined to guide the approach. Without going into the details, the following summarizes what I normally define as the key sources for each category:

Stability
• Network related issues (AOS -> database, AOS -> application share)
• Operational knowledge (description of the relationships between the different server roles in the AX solution and routines describing how to perform a controlled stop – start sequence)
• Pro active maintenance of the AX application and kernels (implementing hot fixes and roll ups for current SP level, new SP)

Performance
• Physical disk I/O at the database level (separation and isolation, Raid levels, sector alignment, block size) regardless of DAS, SAN, NAS heads etc. (general rules apply)
• General load on database instance and utilization of system resources (both OS and SQL Server internals)
• Database configuration, usage and maintenance (best practice configuration, indexes, transactions, space allocation, index maintenance)
• Customizations (caching, run on, query width and selectivity)
• Identify and implement hot fixes related to performance issues (both application and kernels)

While talking about database configuration, I would very much recommend the Check lists that the AX Performance Team has compiled and published on their blog. This is vital information for everyone involved in installing and configuring AX, but also as a general source for hosting partners and customers. The best thing is that this is based on experience and best practice, and in my opinion the value of their recommendations goes beyond AX (general for OLTP applications).

In addition to working with troubled solutions (time consuming, but very motivating), I have implemented “my first” AX solution in a pure virtualized environment including SQL Server 2008 Enterprise. I’m excited about this solution mainly because it’s always interesting to make new experiences. Many people have probably walked this path already, but it’s a first time for everything.
This finalizes my first blog entry in several months, but I will be paying more attention to my blog from now on.

Happy weekend!

Opportunities

2009-06-25T07:54:00.004+02:00

This time I'll post a short professional status - I'm currently unemployed. My company decided to end the Dynamics business after 2 years. Not the nicest entry to the summer holliday, but at least a opportunity to get close to the Norwegian Dynamics business talking to customers, partners and Microsoft in a new setting. This business seems to be going very well at the moment despite the economic downturn. One of my reflections is why my company decided to end this business when the market is very positive around both AX and CRM... Without going into details, ERP in general seems to be a challenge for my company and paired with the history of the company, it's quite obvious that ERP was never a part of the culture. This is always difficult to uncover during the interviews and you need some time within a company to really see this.

I hope to keep this blog going and to have a closure of my next professional move as soon as possible followed by a long summer holliday together with my family.

Happy summer to the WWW!

Update 09/06/26:

I'm closing the hunt and I'm signing a new work agreement early next week. And this blog will still be alive after a long summer holliday. Stay tuned for updates in August/September.

Gartner Magic Quadrant

2009-06-09T16:29:00.005+02:00

In a recently published research report (June 4), Gartner places Microsoft Dynamics AX as the leader in the Midmarket and Tier 2-Oriented ERP for Product-Centric Companies.

"Gartner concludes that only one offering qualifies as a leader in the
market at this time: Microsoft Dynamics AX."

This is good news for everyone involved in both using and delivering services around AX, especially if you are working with AX 2009. I have been blogging about some of the features and changes in AX 2009 for some time and I find my own oppinion to be aligned with what Garter expresses around the technological aspects.

After reading the report, I thought about my experiences with Axapta/AX and I found it worth summarizing my history with Axapta and AX to put the product envolvement in a subjective perspective (from Damgaard via Navision to Microsoft as software vendor):

I first looked at Axapta 2.1 in 2001 and back then, it was considered to be a product with quite rough edges (a very young product born around 1998). The company I worked for at this time, decided to wait until version 2.5 before doing the first implementations. We got a lot of experience from these implementations and discovered that the product still had some rough edges (for instance the returning issues around the famous axdat.udb file espesially in solutions with clustered Application Object Servers). Then we got Axapta 3.0 adding more functionality. Axapta was already branded as a true international solution, but the requirements for doing a central implementation supporting users in different time zones, was driving the requirement for the number of AOS licensed heavily and such solutions did'nt support access to the data stored in the AX database across time zones (it was at best Unicode enabled if you remembered to enable this before synchronizing the database for the first time). This was also in the same period where Microsoft bought Navision. After using more time on finding work arounds to technical issues compared to bringing value to the customers, I decided to do something else for the next 3 years (I was a little bit fed up to be honest). When the opportunity to return to what then was called AX (4) back in 2007, I first considered the architectural changes (eliminated the axdat.udb file and put the license/session handling into the database, replaced the AOCP protocol with RPC, buildt a new Web application based on SharePoint, a pure 3-tier architecture and a greater range for the very important RecId value) and found it very promising. Based on this, I decided to give AX a second try. My experience with AX 4, proved that AX 4x was a big step in the right direction with regards to architecture. Now we only lacked support for handling users accross different time zones on one (or several) AOS instances with scaling and redundancy beeing the drivers for the number of AOS licenses required. This was finally introduced in AX 2009 and for the first time, I considered AX to live up to the promise of beeing a true, international solution. Add even tighter integration to other Microsoft products and technologies, and a Web Application "bringing BI to everyone" through a role based Enterprise Portal, and AX 2009 was positioned to compete with the other 2 main rivals also in the Enterprise Market.

I don't regret returning to AX and I'm rest assured that AX 2009 and later releases, will move further up and to the right in the Magic Quadrant "cementing" it's position as the most agile and TCO effective ERP solution on the market. SAP Business One is of course a serious player and it will be interesting to see how the competition evolves the next years.

So what's my point here? Given the history of AX and the fact that Microsoft now has done the necessary and required changes with regards to the architecture (not a small task), the product enters a new era. Companies looking for a new ERP solution, should indeed evaluate AX 2009 in line with both SAP and Oracle. And existing AX customers running on a version prior to AX 4, should work through their exisiting solution either aiming at eliminating as many customizations as possible or in fact re implement AX 2009 with a clear strategy around utilizing standard functionality to lower TCO over time and keep in pace with new/added functionality. It's in my oppinion all about positioning for a very exiting product cycle where I expect a lot of new functionality to be introduced (both horizontal and vertical) and less architectural changes!

Some information about the next release of AX (6) is already available (AOD files moved from file system to the database, increased range for ObjectId etc.). Some will probably argue that this is architectural changes, but as I see it yet not ground ground breaking compared to the changes already implemented in AX 4 and AX 2009. If you have access to the Product Roadmap, you can read what Microsoft is planning for the future and my final word, is the fact that Gartner concludes that Microsoft is delivering on their vision and that this is one of the key reasons for Gartners conclusion!

Happy reading.

Mounting the AX 2009 Demo VPC under Hyper-V

2009-06-09T07:59:00.004+02:00

I recently was asked to make sure that we could demonstrate AX 2009 outside MS Virtual PC 2007. I admit that Virtual PC is not the ideal virtualization environment for demos especially if you don't have a secondary hard drive with sufficent performance (a fast disk) and sufficent memory resources. Since most people carry a standard business configured laptop computer, it's hard to run a virtual machine with satisfactory performance.

The other issue is whether to utilize the standard demo VPC's (Microsoft has 2 demo VPC for AX 2009) or to create your own. Since setting up and configuring a complete AX 2009 solution has become an hour intensive task due to the number of components and supporting software required, I decided to use the standard VPC provided from Microsoft (17 files for download for demo VPC 1 aka AX-SRV-01).

An additional requriement was that users should be able to connect to the virtual machine remotely.

Here are the tasks I perfomed to provide the standard demo VPC under Hyper-V:

Downloaded 17 files for demo VPC 1 on my laptop

Exctracted the VHD to my laptop

Mounted the VHD under Virtual PC 2007 and started the VM with public networking

Updated the VM with the latest security updates and verified that the firewall was running

(Now I could have uninstalled the Virtual Machine Additions, but I decided to to this after mounting the VHD under Hyper-V)

After stopping the VM, I copied the VHD to the Hyper-V host (approxemaetly 30 Gb)

Mounted the VHD under Hyper-V and started it through the Hyper-V console with public networking (no networking will be enabled until Integration Services is installed)

Uninstalled Virtual Machine Additions and restarted the VM (some creativity was needed at this stage to be able to access Windows Explorer)

Installed Hyper-V Integration Services and restarted the VM

At this point the VM was fully operational including Remote Desktop Connections , but none of the configured Web Sites was working...

After investigating the Web Site configuration, I noticed that they where configured to use host headers which led me to further investigate the DNS configuration. Without beeing a DNS expert, I concluded that the relationship between the defined DNS records and the original IP configuration (192.168.0.1) on the one side and the forward lookup zone on the other, was important. I did NOT want to tamper or alter the DNS configuration to avoid a lot of reconfiguration (remember that the server runs several roles including Domain Controller).

After stopping the VM, I created a new private virtual network (Hyper-V networking) and I added a second network card that I bound to the newly defined network

After starting the VM once again, I defined a static IP (192.168.0.1) on the new network interface and voila - all Web Sites where operational and accessible again without any reconfiguration.

I allocated 3 Gb of memory and one virtual CPU to the VM under Hyper-V.

The overall performance of the VM was supprisingly good even when executed on a Hyper-V host with local disks (no SAN) and sharing resources with a large number of other virtual machines.

The major limitation with this setup, is the number of concurrent RDP sessions (WTS running in admin mode).

Watch out for the combination of reports as PDF with graphics and batch execution

2009-05-08T10:38:00.002+02:00

We have seen this issue in our design and finally Microsoft has now verified (newsgroup microsoft.public.axapta.programming) that AX 2009 has an issue when you use the buildt in PDF functionality for reports containing graphics (logos etc.) and the logic executes as part of a batch job. The reason is that the PDF AX logic uses the Image class to handle the graphic and the Image class is in fact (documented) bound to the client tier. So this is an impossible combination and in my oppinion, a good example of issues in the shadowlands of the changes in the Batch Framework and the default application logic provided by Microsoft.

The solution is to use a PDF printer driver to produce the PDF file, but this should'nt really be necessary since AX has built in support for PDF generation. If you have read some of my earlier blog posts, you will probably notice that we have experienced some major issues when running the application logic in batch under the new Batch Framework and the lesson learned is that you should pay extra attention and put in extra effort to make the batch jobs run as expected.

My best tip is to plan for this as an extra test activity during build (or as an additional payload during testing) and reflect this in the estimates given to the customers. I conclude that the comlexity is rapidly increasing as a consequence of a very good architectural change in AX (server bound batch execution and impersonation), but the application lacks some conditional checks and maybe also some best practise checks in the compiler ("Dear developer, you are trying to call a class on the server that is bound to the client. Please consider your design and look for alternative solutions").

Staying on top of things - AX 2009 hot fixes

2009-04-30T17:03:00.002+02:00

To be honest, I find it very hard to keep track of the available hot fixes (and cumulative rollup packages). By very hard, I'm mostly thinking about the time spent on 1) searching for possible hot fixes and 2) if one possible related hot fix is found, understanding what the hot fix actually solves.

Partnersource (or Customersource) is the primary source for this kind of information by searching the Knowledge base. Maybe it's only me, but I would like a much more effecient tool and much more detailed information about the individual hot fixes to keep our customers happy and of course spend as little time as possible on this activity. The real trouble arises when you find KB number without a published KB article. What do to in these situations? The answer is to ask for unpublished information through your support channel and hereby allocating even more time and resources.

So people; take the opportunity to vote by answering the recently published poll on the right hand side in this blog.

I promise to share the results with Microsoft.

Bottom line is that it's all about quality of the software by easily providing current information about available hot fixes to the partners (and customers).

Debugging Batch Jobs

2009-04-02T16:21:00.003+02:00

We have a hard time getting the AX 2009 debugger to work when breakpoints are set in code executing in Batch Jobs. Microsoft has described this in several documents without any special requirements, but we are unable to get the debugger to work as expected. All settings are correct in the server configuration and in the client when the breakpoint is defined. We have also tried to use the keywork breakpoint in the code without any difference. It seems like the debugger has a hard time attaching to the process. We have tried this in several environments and solutions, and in different combinations, and basically we are stucked.

The fundamental changes in the Batch Framework requires more debugging than ever due to the increased complexity inherited from the changes around impersonation (RunAs) and the prmiary tool in this situation is in fact the debugger.

We have also checked with several other sources and they have done the same experience.

All experience around this issue is highly welcome.

Update 2009/04/03:

Microsoft has confirmed that they have received a problem report (4652) for this issue, but no solution or fix is available yet. The problem was reported in January 2009.

A few observations 3 weeks after GO LIVE

2009-03-30T19:15:00.004+02:00

Ok, now I've been part of a Go Live for AX 2009 Sp1 and I have gained 3 weeks of operational experience. It's to early to draw any definitive conclusions, but this is a short summary of my key observations in random order:

The decision to go for Windows Server 2008 x64 Standard was right
The decision to go for SQL Server 2005 Sp2 x64 Standard in a active-passive failover cluster was probably right, except for some breaks in the database communication (exact reason still unknown)
AIF performs better than expected even without utilizing paralell processing on several AOS instances (2 000 messages can easily be processed in a couple of minutes utilizing one uni directional channel)
The BizTalk Adapter haven't caused us any spesific issues so far
Pay extra attention when submitting batch jobs to the batch queue (or when constructing new batch jobs) and don't expect all logic to execute automatically under the new Batch Framework (look out for logic tied to the client tier even in the standard application and plan for adding some extra logic to keep the code compatible in both interactive and batch mode)
Look out for hot fixes from Microsoft (check Partnersource or Customersource freqently) and plan for some delays getting a response (new installations should evaluate the roll up package released late in February)

Bottom line; the Batch Framework is worth paying special attention to during analysis, design, build and testing.

LIVE, update

2009-03-12T17:29:00.003+01:00

Well we had a breakthrough this morning! At least the batch execution now seems to run as expected. We will conclude after running the batch through the night and tomorrow. Anyway all news are good news right now.

After investigating the AOS server used to run the AIF batch(es), we discovered some deviations in Performance Manager (high number of page faults and hence PF Delta). This led us to change the Batch Group and switch to another AOS server. After this we have been able to automatically process inbound and outbounde messages. If the batch survives the night and tomorrow, we will take actions on the faulting AOS by unistalling/cleaning up and installing a new AOS instance. The only load on the suspicious server right now, is generated by the BizTalk Adapter (.NET Business Connector).

After spending long days in the office since last Friday, I'm heading home more optimistic than ever.

LIVE

2009-03-11T22:19:00.003+01:00

Ok, we have been LIVE since last Friday and I thought it was time for a short update.

Basically things have gone quite well, except for a lot of problems with AIF and message processing. We have identified some issues with customizations, without coming close to solving the biggeste issue: Batch execution of the AIF message processing services. The wierd thing is that they work for some time and then suddenly they stop working, and not leaving a trace of evidence of what went wrong (just sets the status to 'Error' without any error message in QueueManager or in the Exception log). We have done a lot of debugging but this is very time consuming and the underlying logic is hard to follow. We have set up the Batch Groups as per MS definition routing the batch execution to the correct AOS server (2 non clustered) and we have tried a lot of different setups for the Batch jobs with and without dependencies, without any success. The event logs are looking good and the overall utilization of the server resources, are under control.

Part of the complexity is tied to the impersonation logic (RunAs), but this seems to be under control after some issues the first days (permissions). We have ended up implementing a manual processing routine that bypasses the impersonation logic to allow debugging. The manual routine works well, but needs a dedicated resource to act as Queue Manager (not a good solution, but we are able to keep the queue in a controlled state).

I can't go into further details right now, but the new Batch Framework is causing us some pain and this is not what we where hoping for. When the batch jobs run successfully, the performance is acceptable and we are able to process a decent number of messages in any direction. To add some more mystic to the issue, we sometimes have the inbound processing running without problems, while we have problems with outbound processing. Suddenly this changes without any clue or trace of what happened. And we are monitoring the AIF lock table closely and we are also looking for database locks, without seeing any issues in these areas.

So the haunt continues and you can expect some more updates about this later when the issue hopefully is tracked down and solved.

If anyone have some more on this, you are welcome to leave a comment (I'm still optimistic).

So long

The haunt for knowledge

2009-02-04T14:22:00.006+01:00

This is a follow up of the post regarding RPC errors.

First step to obtain a basic understanding of how MSRPC works can be found here. Microsoft has implemented their own version of RPC that they call MSRPC. Wikipedia puts the MSRPC implementation into a historical perspective and relates it to the source spesification called Distributed Computing Environment (DCE) from the Open Software Foundation.

I haven't yet found any specific information about how the Client- and Server Stubs are implemented in the Dynamics AX 3-tier architecture, but I'll try to dig a little bit more into this.

In addition to this, Microsoft recently also have published some KB articles on Partnersource that throws some more light on the issue I initially was blogging about. I'll try to find the URL's and update this post with the direct links when I get the time.

Update 2009-02-18:

Valuable links

Technet Magazine (article written by Zubair Alexander)
MSDN (Error Codes 1700 - 3999)
Florian's Weblog

Approaching GO LIVE

2009-01-28T18:13:00.005+01:00

For those of you that have read my prior posts regarding AX 2009 and AIF with the AX BizTalk adapter, we have been working with our first implementation since June 2008. Without going into the details, we are approaching GO LIVE and the latest configuration is done this week such as defining the final endpoints. We basically have a solution with a set of front line services (FTP) in the perimeter network and BizTalk Server 2006 R2 togheter with AX 2009 in the local network tied together with some middleware. Not revolutionary or innovative, but a simple, cost effective and robust solution based on proven technology.

The test results so far are good and we are ready to GO LIVE tying a lot of trading partners to the client. We will have sales ordres, purchase orders, packing slips and picking lists flowing, in addition to invoices in different shapes and flavors. First phase is roll out in one country and two additional roll out phases are planned for the next months. By roll out in this context, we talk about markets with a set of different trading partners for each market.

Stay tuned for general updates!

Microsoft Dynamics AX 2009 AIF BizTalk Adapter Configuration White Paper

2009-01-26T16:53:00.007+01:00

Finally Microsoft has released a White paper explaining how to configure AIF together with the AX 2009 BizTalk adapter! You can find it here.

This documentation is clear and right to the point compared to the corresponding White paper for AX 4 that is all we have had available until now. It also touches the necessary batch jobs for supporting the out- and inbound message flow (async/sync).

Well done Microsoft; let's hope for more useful and formal documentation for AX 2009! This will avoid a lot of R&D activities and increase the quality of the product. We also need best practise documentation outside the scope of the compiler :-)

RPC errors

2008-12-23T15:19:00.003+01:00

One of the areas I find a little bit hard to understand, is the RPC exceptions logged in the event log (application). This is of course specific to AX 4 and AX 2009, but in my opinion Microsoft has done a pore job explaining the prerequisites and relationship between the RPC errors logged as Information and Warnings in the event log.

Sample (AX 2009 on Windows Server 2008)

I have studied this somewhat, but I have never found a good solution to generalize into best practice when installing AX. I understand that something impacts the way a client calls a Remote Procedure being executed on the AOS, but I don’t have a real understanding of the possible reasons. It could be a number of factors ranging from authentication, x++ code/logic (run at property), network packets, time outs, network drivers etc. but I really miss some basic guidelines on how to account for this. Could it also be some registry settings/tweaks needed? If yes, Microsoft should in my opinion indeed help the partners to better understand the prerequisites since this is all about making AX operating without any disturbance and “hidden” issues.

My challenge to eventual readers is to share your experience and help defining a best practice for this area. I would happily share the result of a joint community effort on this blog. If anyone remember the old “Damgaard Technet”, I really miss a common channel for addressing technical issues that I think a lot of partners around the world are experiencing, I don’t understand why every partner must invest in a lot of R&D activities to solve these kind of issues especially since it’s often about not documented prerequisites!

So please donate your experiences as comments to this blog and I promise to compile a best practice that everyone can use to better avoid RPC issues. Since a lot of unpleasant spam occurs in blogs, I have to moderate your comments before publishing but anonymous comments are allowed so keep them coming.

Experiences with the AX BizTalk Adapter

2008-12-08T17:19:00.006+01:00

We have now encountered our first issue with the AX 2009 BizTalk Adapter. I will try not to go into every detail, but instead focus on the principal guidelines.

#1 If you experience a warning in the eventlog (application), Event ID 110 on the BizTalk Server saying "An X++ exception has occurred. Unable to lock resource channel '<channel name>'" (see sample below), I would suggest to look at how you have defined channels, action policies, document services etc. with regards to what is defined in the different company accounts since this can in fact lead to conflicts. We first related this to installing AX 2009 SP1 since it was the only change recorded in the same time frame and we didn't see this while we worked with the RTM release. The solution was to clean up in the AIF configuration using the AX client, but the cause is at the time beeing not known (it could be SP1, but that's only a theory at the time beeing).

Update 2008/12/17:
After looking into the details, I also noticed a change of the warning logged by BizTalk for the same issue. Sample of the warning logged initially (figure 1) and a new version of the same message (figure 2) is shown below. I guess that something changed in SP1 and that the problem can be related to a required reboot of the BizTalk Server after applying AX 2009 SP1 (not logged or notifyed by the installer).

#2 Disparsed documentation. The documentstion for AX 2009 is quite good for AIF configuration, but is's very sparse when you are looking for details about the configuration of the transport properties in BizTalk (Visual Studio). If you want to understand the difference between the various authentication types, you have to look up the corresponding documentation for AX 4. At least this was where I found the answers to my general questions.

Convergence 2008 in Copenhagen

2008-11-21T21:14:00.002+01:00

After attending the annual Convergence conference for the first time, I feel even more energic about the future for AX and Microsoft is clearly ramping up the fight against SAP and Oracle. The product is ready to grow both horizontally and vertically (broader and deeper functionality) and Microsoft has done a lot with the product since they bought Navision back in 2002 (USD 1.55 billion in cash and share).

One good example is the Enterprise Portal that has been part of the product since version 2.x. In my opinion it's more a revolution than en evolution despite the fact that SharePoint was introduced as the framework in AX 4. The really big change is the introduction of the Role Centers and the rather bold ambition to build a true portal solution from the ERP perspective. And it's really about role tailored "Business Intelligence to everyone". Some deep changes in the runtime and a much better developer experience in the crossing between MorphX and Visual Studio, opens up for a lot of interesting possibilities. I really hope to get started with some serious implementations pretty soon, because I'm convinced that the Enterprise Portal framework will bring much value to the customers.

The second top news for me was the announcement of the AX Data Management framework (ADM). If you attended the conference and missed this session, you really missed one exiting and excellent session given by Mr. Sri Srinivasan! I attended 6 AX technical sessions, but this was by far the best one in every aspect. And what's it all about? Finally customers and partners will have a framework to clean up the database schema, to categorize the data and to move data out of the primary production database. This maybe doesn't seem like something important to most people, but if you think about customers running the product for 8-10 years doing every main upgrade, you will probably recognize the fact that not all changes in the database schema has been high quality. Combined with different developers not always (seldom?) paying attention to the impact of their adjustments on the database back end, it shouldn't come as a surprise to anyone that the database schema grows into a jungle of different objects with broad (many columns) and deep (many million rows) tables with a lot of indexes hurting the performance big time. The Data Management framework also addresses this dilemma. Add the fact that 5-10% percent of the tables contributes to 80-85% of the total database size and you probably see the need for some serious clean up work. The roadmap for the ADM consists of 3 phases starting with a stand alone toolset and ending as an integrated part of AX. Microsoft also said that they would utilize the management functionality in SQL Server even more in the future. The final area of functionality also covers index maintenance with alerts being raised on given fragmentation thresholds. So all in all rules for archiving fiscal years still making the figures available through SSRS and special forms in AX for handling online and archive data, should really sound like good news to both existing and new customers. And this is in my opinion another sign of Microsoft ramping up the battle against SAP and Oracle by filling the administration and management gap to lower the TCO. Studies and analysis of existing customers, was reported to give effects in the 40+% range and I find it hard to questioning this numbers after attending the session - Sri is clearly an expert within this topic.

The last of my top 3 take a ways, was actually not related to AX and technology, but it was taken from the Business Leadership track. Session BL01 "A Macro view of Business Trends" was lead by Mr. Thomas Mendel from Forrester and supported by a representative from both HP - EDS and Microsoft. It was arranged like a prepared discussion session with Forrester giving their thoughts about what they considers to be the three megatrends that will change the IT business over the next years and ending up with the view as seen from HP - EDS and Microsoft. It was really both exiting and refreshing to listen to them and to get some confirmations about my own opinions. And the tree megatrends are Globalization 2.0, Invisible IT and how the consumer marked influences the Enterprise IT. A lot of dimensions and elements was brought into the discussion ranging from the CHINDIA effects via Cloud computing (did you know that Microsoft buys 10 000 servers every month to power their Azure platform from data centers all over the world?) to innovation driven from the consumer market. They all agreed on that the traditional office space was history and that the y generation will bring a totally new set of requirements to the Enterprise IT departments.

So all in all I returned with a great feeling abut the future for DAX and it will be very interesting to see how the battle on the ERP arena develops in the rather turbulent world economic situation we are facing right now. I think a lot of new businesses will discover that the Dynamics branding really is more than yet another market initiative. Finally the announcement of DAX SP1 didn't come as a suprise at all...

AX 2009 SP1

2008-11-12T17:13:00.002+01:00

I recently got information telling that Service Pack 1 for AX 2009 is pretty close to RTM. I don't have any information about what it contains, but let's hope SSAS and SSRS for SQL Server 2008 is fully supported.

Maybe some more information will be published during the upcoming Convergence conference in Copenhagen (November 19 - 20).

In addition Microsoft has released at least one kernel hot fix for AX 2009 with reference to the "non existing" KB article 958328. It' still hard to find any information about this hot fix, but I know Microsoft is working on the KB article. As usual, hot fixes are provided directly from Premier Support (or other official Microsoft support channels), while the KB articles still are published on Partner and/or Customer Source. Kernel hot fixes are distributed as a set of MSP files which also is a good thing as I see it.

With this said, our experience so far has been pretty good and the quality of the RTM release seems to be better that 80/20 (we haven't yet experienced any big issues except for the technical issues described in an earlier post). So at this stage and after working with the release for almost 5 months, it looks as a big step in the right direction. We are still working with AIF and the BizTalk adapter, and except for some minor issues, we have been able to proceed as planned on our current project deliveries.

Dynamics AX and virtualization

2008-10-16T22:14:00.003+02:00

Early in September, Microsoft changed their support policy for several systems including Dynamics AX from only supporting their own virtualization software (Virtual Server and Hyper-V) to other virtualization vendors including the market leader VMWare. According to the News Release dated Sept. 3 2008 on http://www.vmware.com/, their hypervisor VMware ESX 3.5 update 2 (ESX 3.5u2) passed the Microsoft Server Virtualization Validation Program (SVVP).

This is old news on the Internett highway, but the effect of this is that customers now can get full support both from Microsoft and VMWare for systems running on the validated release of the VMWare ESX 3.5 hypervisor. This can seem like a minor detail, but a lot of businesses have in fact been running most Microsoft software on an unsupported virtualization platform for a long time without any security or guaranteed support. The change to include VMWare as a supported platform, is important information and should indeed be emphasized.

For everyone involved in using and delivering products in the Dynamics product line, I think it’s good news that Microsoft now officially supports several Dynamics products (AX, CRM and NAV) running on this release of VMWare ESX 3.5 and it actually shows that Microsoft has opened up for other well established players in the virtualization market.

Links
http://support.microsoft.com/kb/957006/
http://www.vmware.com/company/news/releases/svvp.html
http://windowsservercatalog.com/svvp.aspx?svvppage=svvp.htm

AX 2009 BizTalk Adapter

2008-10-15T22:02:00.004+02:00

I recently did my first experiences with installing the AX 2009 BizTalk Adapter. "As usual" some discoveries were done, but this time it was more related to my less than minor knowledge of BizTalk...

After the BizTalk Server 2006 R2 Standard Edition (BT) was installed and verified, I went away installing the adapter "by the book" (that's the Installation Guide and the Server and Database Administration Guide). The documentation doesn't really contain a lot of information for BT newbie’s like me and after Googling about the error message I received, it seems like several people has stumbled across this one both for AX 4 and AX 2009. Surprisingly no solution was given and I thought this was a good opportunity to share my experiences.

As mentioned I went away installing the BizTalk adapter (custom installation, check the BizTalk adapter which not surprisingly also checks the .NET Business Connector). I then entered the necessary information the installation wizard asked for and hit the button labeled "Install". After some minutes watching the progress bar for installation of AX Components, I noticed the progress bar doing a "negative" progress followed by a summary saying "Setup was not completed" with a red square in front of both .NET Business Connector and BizTalk adapter. Ok, as normal open the log file and look for errors... Scrolling through the log file I noticed that the .NET Business Connector installed just fine. After some more scrolling down, I found the following error message: "An error occurred during the BizTalk adapter install custom action step within the Microsoft Dynamics AX components installer. For details, see the previous messages in the log. >>Install: BizTalk Adapter registration failed: Access denied". Being under some pressure getting the task done, I though "Oh no, not another day of hunting down a resolution". As mentioned I goggled about the message in bold above and I found several hits to similar experiences, but without a solution.

So my other really good friend in my professional life - the Event log - was my next stop. After filtering the event log - application, I found some warnings with source ENTSSO replicating the Access denied message. This is in my mind always a good indication and I then read a little bit about this on Microsoft TechNet. ENTSSO is short for Enterprise Single Sign On and this service is a vital part of the security mechanism in BizTalk Server with regards to authentication. After reading some more, I decided to open the Microsoft Single Sign On Admin Console. At the end of the summary saying Errors, I once again saw the "Access denied" message under the category RPC. Then I was "bold enough" to add my installation user to the security group called SSO Administrators. Reopened the SSO Admin Console and the RPC error was gone!

I also added the installation user to the BizTalk Administrators security group (required to add new adapters). After doing a new installation of the BizTalk adapter, everything went as expected and the log file looked just fine. Checking inside the BizTalk Admin Console, I found the adapter listed as a resource under "Application.1" (the default application in BizTalk).

To summarize this Blog entry, you have to understand some of the basics of BizTalk before installing the BizTalk adapter for AX 2009 (and also AX 4 as far as I can understand). If this is something Microsoft should mention in the documentation or not, is really open for discussion. As far as I know, they are working on some kind of detailed information right now and maybe this will be included. But all in all, this is another example of the increased demand for broad technical knowledge to install every component available in AX 2009. In my opinion it's a new ballgame compared to previous versions of this packaged application from Microsoft.

We will carry on the next days trying to get AX to communicate with BizTalk utilizing Application Intergration Framework (AIF) and I expect to be able to share some information from this journy quite soon.

So long!

First Experiences

2008-08-22T13:20:00.002+02:00

First of all; Dynamics AX 2009 is really exiting with regards to the tight integration with the Microsoft Technology Stack. The product is really starting to take shape in the technology area and it's starting to look as a Microsoft product (remember the history... Damgaard, Navision, Microsoft?). At the same time the complexity has increased and this of course both affects the time and skills needed to implement AX.

My journy so far is based on experiences from one customer installation (Core solution by now) running Windows Server 2003 R2 Standard x64 and from a single computer installation running on Windows Server 2008 Standard x32.

NetBIOS over TCP/IP: At a customer site I experienced problems connecting clients to the AOS instance ("A connection to the Axapta Object Server could not be established") - the initial startup of the AOS service for the instance, went fine. After trying some tweaks, I had to ask the hosting partner for a network trace. It turned out that the problem was that NetBIOS over TCP/IP was deactivated. After activating this (network card, Properties, Advanced, WINS, Enable NetBIOS over TCP/IP) on the AOS and servers with AX client installed, it all went smooth. After looking into the details, I think the reason is that AX from version 4 uses Remote Procedure Call (RPC) for communication between clients and AOS instances, and that RPC requires NetBIOS over TCP/IP. Perhaps Microsoft should put this into the system requirements for AX?

Installing the application component on 64-bits Windows Servers: I had a hard time trying to change the location for the application files from the installer. I almost always installs the application component to a different location than \program files\..\, but this turned out to be very hard on a 64-bit Windows plattform... I did'nt have the time to try out a silent install utilizing a parameter file and the solution could be to specify the location here. No matter what I changed, it defaulted to \program files\..\ and I could'nt change it because it was disabled for editing. I ended up doing the installation followed by copying the files to the correct location as separate application instances. By doing it this way, I have all the original setup information left as a master. This does'nt happen when installing on a 32-bit Windows plattform and here you can change the location freely.

SQL Server 2005/2008: If you only are going to implement AX CORE components (application, AOS instance and database) SQL Server 2008 is supported. If you on the other hand are going to use Reporting Extentions (the AX integration against SQL Server Reporting Services - SSRS) and/or Analysis Extentions (AX integration against SQL Server Analysis Services - SSAS), you have to use these components from SQL Server 2005. Also remember that both SSRS and SSAS are required if you want to use Enterprise Portal (the Web application module in AX) since the Role centres are composed of Web parts that shows SSRS reports (Report Viewer) and that the KPIs also are produced through SSRS reports utilizing SSAS. A former collegue has already an entry about this in his blog, but unfortenately I didn't read his post until after doing the exact same experience. But I totally agree with him that Microsoft should fix this compatibility issue as soon as possible, especially since SQL Server 2008 now are in RTM and supposed to fit very well with AX 2009. Another thing worth mentioning if you are going to install SQL Server 2005 on Windows Server 2008, is that you have to install the IIS6 compatibility components before installing SQL Server 2005.

.NET Framework 3.5: This is a general requirement, but I went away installing .NET Framework 3.5 SP1 as this is the latest release... After installing Enterprise Portal and doing the initial configuration, I ended up with an error message saying "Unhandled Exception" when browsing the site and an repeating error logged in the eventlog (application) saying "An unexpected error has occurred - A ProgressTemplate must be specified on UpdateProgress control with ID 'AxProgressControl'". As usual when encountering these kind of errors, I used Google to search for references. I found one link saying that the solution was to uninstall .NET 3.5 SP1 and reinstall .NET 3.5. All I can say is that it worked... This is also explained in the System Requirements page for Dynamics AX 2009 (says .net 3.5 and not .net 3.5 sp1), but at the same time Microsoft tells you to run Microsoft Update to get the latest security updates for your installation. I think many will be trapped by this one either when installing or later. Anyway I strongly advise you to inform your customers not to install .NET 3.5 SP1 (or any components like Visual Studio 2008 SP1 that updates .net) on the server hosting Enterprise Portal until this component is supported on .NET 3.5 SP1. If you arrive at the office one morning facing a non functional Enterprise Portal site, I would start by checking if .NET Framework 3.5 SP1 is installed...

Installation order: As explained by Microsoft in the AXInstallationGuide, you should always install and configure the Core solution first. Configuration is the same as running through the Installation Check List. After this, it all depends on the features or options required for the implementation. For a full implementation, it worked well for me by running the installation of each component in the following sequence: 1) Workflow 2) Reporting Extentions 3) Analysis Extentions and 4) Enterprise Portal. I installed each component individually and made the necessary configuration from the AX client both before and after the installation. One important part of the installation is to verify the setup logs to make shure everything was properly installed. Also read the documentation carefully to make shure all the pre requesites are met. I used Windows Sharepoint Services 3.0 SP1 (WSS) and installed this separately prior to installing Enterprise Portal (AX setup can do this for you). I ran the configuration, but did'nt create any applications or sites at this stage - just created the main service and the main database. My message is that you can end up with a lot of tricky errors if you install every option in one operation after installing and configuring the core solution. Also remember to fill in the information for the System Accounts before installing any options or additional components.

SSRS 2005 on Windows Server 2008: Remember the manual configuration required before installing Reporting Extentions. This is described in detail in the AXInstallationGuide. What you also should remember is to map the account for the AX SSRS application pool as a user in the SSRS databases.

Role centres: Some of the default Role centres still has errors with missing initial values for report parameters (I have'nt spent time trying to fix this yet). What I find a little bit more suprising is that a couple of the Role centres gives a "stack" of errors and messages saying that the dataset returned is larger that the default value. I don't have the details available right now, but I'll post on this later. The solution is rather simple - you have to increase the buffer size in the AOS instance configuration. I think it defaults to 24 Kb and by changing it to 25 Kb, I got rid of the errors and warnings. This is another area Microsoft should pay attention to and fix, since this is standard out-of-the box functionality that partners should'nt need to spend time on fixing.

This wraps up my first entry in this blog. Hopefully someone will find this information useful and please, don't leave a lot of "pingback" comments - it's impossible to have complete control over all the content out there. I promise to do my best to link to the source when this is applicable...