Website multivariate testing is increasingly common as companies look to optimise website performance.  However, multivariate testing for websites with Content Management Systems (CMS) is far from straight-forward.

What is multivariate testing?

For those who are not familiar with the concept of multivariate testing, the basic principle is to have two (or more) separate variants of the same page on a website and track click-through/conversion on each variant of the page separately.  This allows a website manager to evaluate the effectiveness of a particular change to a page vs. the previous version of the same page. For example; it will provide definitive answers to the questions "has the change improved conversion rates" and "has the change improved click-through ratio".  

This effectively allows some form of "trial and error" optimisation whereby changes can be piloted and then either made permanent or revoked depending on their impact.  Google Analytics provides great facilities for split testing pages linked to pay-per-click (PPC) marketing campaigns.

What are the challenges when multivariate testing in a CMS?

Most website managers will now be considered at somewhat of a handicap if they are unable to take advantage of multivariate (or split) testing tweaks to pages.  The principles behind a CMS mean that the structure of a page is normally defined in a template and the content (editable by CMS users) is rendered within this set structure.  

So is it possible to retain a relatively transparent experience for CMS content authors whilst facilitating split testing?  I think the answer to this question depends on the flexibility of the CMS you are using.

How can you overcome these challenges?

In my case I was looking at the Umbraco open source ASP.NET content-management system and I was able to provide a solution, but only by adding some custom code and replacing the standard Umbraco default page.  From version 4, Umbraco has used ASP.NET masterpages as the underlying technology behind it's page templates.  Each page entity must be tied to a single template and therefore a specific masterpage file.  

For example; in the custom version of the Umbraco default page that I created the OnPreInit event of the ASP.NET page lifecycle was handled; and within the handler the MasterPageFile property of the page was set dependant on a querystring argument.  Therefore regardless of the template that was assigned to the page within the CMS I was able to apply whatever masterpage file I wished.  After overcoming this hurdle it was relatively easy to create a macro within the CMS to allow users to create separate content for the A, B or C variants of each page as they desired.

Article written by Mark Reed, Technical Consultant, MSM Software.