Official Harlan Tytus Beverly Blog: 3 ways to do Google A/B Testing or Multivariate testing in a Website or App

AB Testing with Google Analytics

Summary

This page details the methods by which we can use Google Analytics to perform A/B Testing such that it can help us determine if one asset or algorithm or text is better than another.

Multivariate testing can be done by running multiple A/B tests at the same time.

This document explains three methods of A/B Testing...

  1. Redirect Method (not ideal, but very easy)
  2. Javascript Method (ideal way!!!)
  3. Dynamic HTML Method (PHP/Python templates)... (less ideal as you will see).

Step 1: Google Analytics Configuration

In order to do A/B testing Google Analytics must be setup and installed on all pages of a site. In addition, you must establish 2 extra things.

a.) A Goal.

b.) An Experiment.

Setting up a Goal

Google Analytics (at http://google.com/analytics) lets you easily setup goals such as time on site, certain URL reached, or even certain Event reached.

To set up a goal, sign in to analytics and clic the ADMIN button (currently top-right of page). From here, select the site, property, and then under profile, select "Goals".

Click "Create a Goal" and you will be guided to setup a Destination Goal (you need a destination URL, a Duration Goal, a #Pages Goal, or an Event Goal!

THE EASIEST GOALS ARE DESTINATION URLS! TRY TO DESIGN THE PAGES TO LOAD A "THANK YOU PAGE" as A SEPERATE URL>>> THIS MAKES GOAL SETTING EASY!

To do an event goal, you must setup an event on your site. Setting up an event is pretty easy, you just have to do a "push" with javascript when the goal is achieved. See:https://developers.google.com/analytics/devguides/collection/gajs/eventTrackerGuide

here's some real sample code for setting up an event that's been tested athttp://www.keyingredient.com/blog/events:

  //When person submits their URL, load this html
  <script>
  _gaq.push(['_trackEvent', 'Conversions', 'Converted', 'SigneupCCCEmail', 1]);
  </script>

Setting up an Experiment

To setup an experiment you have to go to a different area than goals... Click on the website from the GA homepage, then click on CONTENT in the menu on the left. Next, click on Experiments. (this is also where you can see the RESULTS of your experiment). Next, click on "Create Experiment".

Now you have 2 choices... either you are going to use "REDIRECT METHOD" or you are NOT... (I prefer NOT, but redirect is easier).

If you will do REDIRECT METHOD, then SKIP TO NEXT SECTION.

If you will NOT do redirects (which is better on users IMHO), then to setup the experiment...

  1. enter a fake url that does not exist.. (such as asdasdjjj.com), click Start Setup
  2. give it a descriptive name (like fake_test_hoempage_banner_experiment)
  3. now select a metric.  (THIS IS THE GOAL YOU SETUP IN THE LAST STEP!!!!)  
  4. hit next step.
  5. now enter a fake url for each variation you will use... feel free to make it descriptive if you wish.
  6. hit Next step.
  7. Select "Manually insert the code" .. you can ignore the code.. (it's for redirect method).  INSTEAD, get the EXPERIMENT_ID  (this is important for later steps).
  8.  Hit next step and then Start Experiment... (Any warnings can be ignored, the urls you gave were fake.. and that's okay. it's what you want for a 'non-redirect' method.).  NOTE: if you were doing redirect method, you wouldn't want any warnings here..

Step 2: Implementing the Experiment in Code

Before you start implementing new experiment, need to declare it's ID in keysocial/settings/shared.py.

This operation is important. It later will allow unit-tests (they connecting to Google API) check experiments statuses. And if they are ended, tests will notify about this while failing.

# Google experiments list
GOOGLE_EXPERIMENTS = {
    'holiday_banner': 'IK__c5dVRNqF2fDB9sZdOA',
    'experiment_description': 'EXPERIMENT_ID', <----
}

To use this experiment ID, you need to pass it while rendering jinja template.

import settings
context = {
    'recipe': recipe,
    'experiments': settings.GOOGLE_EXPERIMENTS,  <----
}
self.render_jinja(template, context)

Then use it target place i.e.

<div class="experiment" data-bind="googleExperiment: {id: '{{ experiments['experiment_description'] }}'>

Method 1: Redirect Method

If you are choosing the redirect method, then you have to do things a little differently in Google Analytics and on your site.

  1. FIRST, you MUST have 2 unique URL pages (a passed in parameter IS sufficient for uniqueness).  EXAMPLE:  http://mysite.com/home   http://mysite.com/althome  would work,  or http://mysite.com and http://mysite.com?var=1  would work....  BUT THEY MUST BE DIFFERENT< AND YOUR EXPERIMENT IS SIMPLY GOING TO RANDOMLY REDIRECT PEOPLE TO ONE PAGE VS THE OTHER (that is the redirect experiment).
  2. Next, during the Experiment Setup, you must input these 'real' urls into the URL fields for each variation..

Now, all you do is when you hit "Manually insert the code", put the redirect code they provide there onto the page that gets loaded first (it does not go on both pages...just on the one that gets loaded first).. such ashttp://mysite.com/home (but NOT on http://mysite.com/althome).

BOTH PAGES (and this is always true of all pages), must have the normal Google Analytics Tracking codes installed...

That's it.... your experiment will now run, pick a winner and you are done!

Method 2: Javascript Method

This is the ideal way, because it is client side, requires no redirect, and can use Google's Multi-armed-bandit technique to decide which will automatically figure out a winner better AND remember to whom what version was showed (for consistency)

Full instructions here:https://developers.google.com/analytics/devguides/collection/gajs/experiments

First, Setup the GA Goal and Experiment as described in Step 1.

Now, simply add the following Javascript...

Got Feedback? Email us at <a href="mailto:help@keyingredient.com">help@keyingredient.com</a> and let us know how we are doing!</a>

   <script src="//www.google-analytics.com/cx/api.js?experiment=WfCFFL4gS_YOUR_EXPERIMENT_ID_HERE_z2ie7J_Vqg3nw"></script>
   <script>

     // 2. Choose the Variation for the Visitor
     var variation = cxApi.chooseVariation();
   
         // Show the HTML or Not!
        var content = document.getElementById("testhtml");
   
    if (variation == 1) {
        content.style.display = "none";
          
            } else {
        content.style.display = "block";
   
        }
   
     cxApi.setChosenVariation(
       variation,             // The index of the variation shown to the visitor
       'WfCFFL4gS_YOUR_EXPERIMENT_ID_HERE_z2ie7J_Vqg3nw'                // The id of the experiment the user has been exposed to
     );
     _gaq.push(['_trackEvent', 'WfCFFL4gS_YOUR_EXPERIMENT_ID_HERE_z2ie7J_Vqg3nw', 'WfCFFL4gS_YOUR_EXPERIMENT_ID_HERE_z2ie7J_Vqg3nw',         'WfCFFL4gS_YOUR_EXPERIMENT_ID_HERE_z2ie7J_Vqg3nw', variation, true]);
   
   </script>

or this way... (if you want to move things around instead:

   <script src="//www.google-analytics.com/cx/api.js?experiment=8eynxD_YOUR_EXPERIMENT_ID_HERE_JCSC2X3lOOS8IyhQ"></script>
   <script>
   var $j = jQuery.noConflict();
   $j(window).bind("load", function() {
     // 2. Choose the Variation for the Visitor
     var variation = cxApi.chooseVariation();
   
         // Show the HTML or Not!
        var content = document.getElementById("theccclogo");
   
    if (variation == 1) {
        content.style.width = "400px";
        content.style.height = "295px";
        content.style.top="-270px";
        content.style.left="270px";
        content.style.marginBottom="-250px";
          
            } else {
   
        }
   
     cxApi.setChosenVariation(
       variation,             // The index of the variation shown to the visitor
       '8eynxD_YOUR_EXPERIMENT_ID_HERE_JCSC2X3lOOS8IyhQ'                // The id of the experiment the user has been exposed to
     );
     _gaq.push(['_trackEvent', '8eynxD_YOUR_EXPERIMENT_ID_HERE_JCSC2X3lOOS8IyhQ', '8eynxD_YOUR_EXPERIMENT_ID_HERE_JCSC2X3lOOS8IyhQ',     '8eynxD_YOUR_EXPERIMENT_ID_HERE_JCSC2X3lOOS8IyhQ', variation, true]);
   });
   </script>

Method 3: Dynamic HTML Method (PHP/Python templates)

This method is less ideal because it is more complex, and it requires that the PHP or PYTHON determine which experiment to show.. and keep consistency (via cookie or session or something).

Full Details Here:https://developers.google.com/analytics/devguides/collection/gajs/experimentsand here:https://developers.google.com/analytics/devguides/platform/features/experimentsmore info here:https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtExperimentsGuideand here:https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/experiments

NOTE: Sometimes these guy say you need to use the EXPERIMENT API, to set the setServingFramework to API or External.. this is NOT TRUE, and it's a hassle to do. just use the Google analytics interface, it's much easier... and you don't need to set the framework to anything.. I've tested it to work in all methods.

So here's my short explanation of how to do it. [PHP focused method, but PYTHON is basically the same deal, just use your preferred PYTHON templating language]

First, load the needed Experiment Javascript:

  <script src="//www.google-analytics.com/cx/api.js"></script>

Next, setup your experimentID somehow.. I did this in PHP using:

  <?php
  // The Id of the experiment running on the page
  $experimentId = 'YByMKfprRCStcMvK8zh1yw';
  
  // The default variation chosen for the visitor
  // I DON’T SET This here, because I set it later on in the code … $chosenVariation= 0;
  ?>

Then, select a way to determine which version to show.. (random? based on cookie? etc.).. I just used IP Address, because it was easy and fairly consistent...

  <?php
  $ipaddr = $_SERVER['REMOTE_ADDR'];
  $lastval = substr($ipaddr, -1);
  $lastvalmod2 = $lastval%2;
  if($lastvalmod2 === 1) {
  $chosenVariation= 0;
  } else { $chosenVariation= 1; }
  ?>

A better way in Python would be to check user_ids or randomly assign a number then store that in the user’s session… (or the not logged in user’s session)

Next, you have to send this chosenVariation to GA... here's the code to do that in PHP:

  <script>
    cxApi.setChosenVariation(
      <?php echo $chosenVariation; ?>,             // The index of the variation shown to the visitor
      '<?php echo $experimentId ?>'                 // The id of the experiment the user has been exposed    to
    );
    _gaq.push(['_trackEvent', '<?php echo $experimentId ?>', '<?php echo $experimentId ?>', '<?php echo    $chosenVariation; ?>', <?php echo $chosenVariation; ?>, true]);
  </script>

That's it!

Step 3: Look at Results and Pick a Winner

Now, you can see the experiments and results (explained more fully here:https://support.google.com/analytics/answer/2365329>

NOTE ABOUT SETTING PERCENTAGE OF TRAFFIC TO EXPERIMENT

When you set the % of traffic to experiment, you are only telling google what % will see the experiment... (e.g. in the experiment they might see A or B)...

Such that if you had 2 options A/B, setting this value to 100%, means that on average A will be chosen 50% and B 50%....

If you want to set it higher, you will need to have more variations (like A,B,C,D).. even if B,C,D are actually the same variant... thus at 100% in experiment, A=25% likely, B,C,D combined to be 75% likely.

To set it lower on an A/B test.. setting the %inexperiment to 10%, would yield A at 5% of traffic and B at 5% of traffic... and 'origional (usually A) at 90%. thus A combined would be 95%. e.g. %inexperiment * #testvariations/(origional+variations)

SIDENOTE: For any of these numbers to work like you expect, you must select ON the option to: "Distribute traffic evenly across all variations". Or else, google will adjust the weightings based on goal success.

APPENDIX: MORE INFORMATION ON STUFF

This section is just misc. information from my own notes while I was testing this stuff.

YOU CAN SEE EXPERIMENTS RUNNING ON OUR BLOG

HOMEPAGE EXAMPLE USING PHP: http://blog.keyingredient.comEXAMPLE USING JAVASCRIPT: http://blog.keyingredient.com/events

Here's my full notes.. where I tried to use the Experiments API system.. (but learned that the GA Site is easier than this). This document is intended to help with understanding of the Google A/B Testing System. By: Harlan T. Beverly 8/17/13

NOTE: This section uses PHP as sample source code… however Python API and examples are also available via Google… You should READ this section anyways, because it will help understand the Python Implementation as well.

1. Understanding the Experiments API system for Google Analytics. This link explains the system in overview…https://developers.google.com/analytics/devguides/platform/features/experiments

And this is a link to explain the Client-Side redirect method….https://developers.google.com/analytics/devguides/collection/gajs/experiments

However, half-way down the above page… it talks generally about a SERVER side method, that doesn’t require a redirect (for dynamic content pages)…

This document is all about the DYNAMIC CONTENT METHOD of GOOGLE EXPERIEMENTS A/B Testing.

2. To get started (python or php), you will need a Google Account with access to the Google Analytics for the website… request this access from lucina@keyingredient.com

A goal must be setup (or provided for you). Marketing could provide this goal for you and have it set up, or else you must set up at least 1 goal for the site using analytics system… ideally this is the goal of the A/B test (such as made it to the next page, registered, etc.).

You can do this here: https://www.google.com/analytics and click on ADMIN

While you are there you will also need the Account_Id (click on account settings), the Tracking ID (like UA-322432-22) (click on View Property Settings), and the profile_id (click on view profile settings) .

3. Now decide on the experiment (like show a different “join” button, or something). https://support.google.com/analytics/answer/1745154?hl=en

4. Next, you’ll need to enable Google Analytics API for an app:https://code.google.com/apis/console/

Full setup instructions are here:https://developers.google.com/analytics/solutions/articles/hello-analytics-apiBUT BE WARNED: Much of the code google posts has errors and won’t run!! (Old version numbers I think). Anyways, do the step called 1. Register a project in the Google APIs Console

If you get stuck, this might help: https://code.google.com/p/google-api-php-client/

4. You will also need the latest API files for the server to host!!! See…https://code.google.com/p/google-api-php-client/ Or for Pythonhttps://code.google.com/p/google-api-python-client/

- - - WARNING make sure you set your paths right to where you install these libraries.

5. Now, you will need to setup a “ConfigureExperiments” page of some kind (possibly build a backend interface for this)… My example for this is built using PHP (see below: SetupAnalytics.php)… however, Python uses the same basic idea. (Oauth2 is required for this).

See my SetupAnalytics.php file below for showing how to do Oauth2…

For Python, you will have to look at their sample code:https://developers.google.com/analytics/solutions/articles/hello-analytics-api

GOTCHA: The oauth2 callback url must be set to your file (e.g. SetupAnalytics.php) or whatever.

NOTE: Their sample code is just a “Hello” to the api… which I’ve built on in my example to actually setup an actual experiment.

This ConfigureExperiments stuff is pretty complex at first, but basically:

         You are doing a POST to google using an oauthed user to create the experiment.

It returns an object that has the id in it (or you can print it)

  In my sample code…

   I made 2 variations of a page… and I gave them names… and most importantly,

   I did the setServingFramework(‘API’)  to the experiment…. This is needed!
      Here’s the full definitions of the Management.experiments view: https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtReference/management/experiments

This page tries to explain stuff.. but my code is simply easier to understand:https://developers.google.com/analytics/devguides/config/mgmt/v3/mgmtExperimentsGuide

SOME GOTCHAS!

a. you must have a goal setup in the google analytics b. you have to set you ga_id, ga_name, and ga_profile_id right for the goal! c. goals have funky names… it must be ga:goal1Copmletions or ga:goal2Completions etc… nothing else.

6. Okay, your experiments are setup… now it’s time to hook up the dynamic content to the experiment….

Basically you have two options… option 1: simply report to GA the version you chose….https://developers.google.com/analytics/devguides/collection/gajs/experiments

<?php // The Id of the experiment running on the page $experimentId = 'YByMKfprRCStcMvK8zh1yw';

// The default variation chosen for the visitor // I DON’T SET This here, because I set it later on in the code … $chosenVariation= 0; ?>

• THEN run the google analytics script.

 cxApi.setChosenVariation(
   $chosenVariation,             // The index of the variation shown to the visitor
   $experimentId                 // The id of the experiment the user has been exposed to
 );

HTML; ?> _gaq.push(['_trackEvent', '<?php echo $experimentId ?>', '<?php echo $experimentId ?>', '<?php echo $chosenVariation; ?>', <?php echo $chosenVariation; ?>, true]); </script>

Or <script>

 cxApi.setChosenVariation(
   <?php echo $chosenVariation; ?>,             // The index of the variation shown to the visitor
   '<?php echo $experimentId ?>'                 // The id of the experiment the user has been exposed to
 );
 _gaq.push(['_trackEvent', '<?php echo $experimentId ?>', '<?php echo $experimentId ?>', '<?php echo $chosenVariation; ?>', <?php echo $chosenVariation; ?>, true]);

</script>

THIS OPTION IS NICE because you can store your choice in the user’s session/cookie too and then they don’t see different ones…. BUT it doesn’t self-optimize.

NOTE YOU MUST CALL ga again after you set Chosen Variation or else it doesn’t count page views right! _gaq.push(['_trackEvent', '<?php echo $experimentId ?>', '<?php echo $experimentId ?>', '<?php echo $chosenVariation; ?>', <?php echo $chosenVariation; ?>, true]); Option 2.https://developers.google.com/analytics/devguides/collection/gajs/methods/gaJSApiEventTracking?csw=1#_gat.GA_EventTracker_._trackEvent&utm_campaign=eventTracking?utm_medium=blog

Let Google pick which one to show… To do this requires the user to tell you which variation to load…. (or else requires a refresh, which we are trying to avoid).

Heres some PHP to pick a variation based on IP address..

$ipaddr = $_SERVER['REMOTE_ADDR']; $lastval = substr($ipaddr, -1); $lastvalmod2 = $lastval%2; if($lastvalmod2 === 1) { $chosenVariation= 0; } else { $chosenVariation= 1; }

A better way in Python would be to check user_ids or randomly assign a number then store that in the user’s session… (or the not logged in user’s session)

To do this without a refresh, you’ll need to have the user load the javascript that would pick a variation and then pass that variation into the server.. (this may not be possible!). so probably best not to try this..

So either: use the redirect method OR use the method where we pick which experiment to show.

IF we do the redirect method, you could just add a hook to certain pages like ?var=1 which the PHP or Python looks at to load different content. This method may be best of all worlds, but does require a redirect. And remember to CHECK THE BOX to combine urls if you use this method.

7. Now you can see the experiments:https://support.google.com/analytics/answer/2365329

8. There is an easier way to setup Experiments….http://www.lunametrics.com/blog/2012/06/04/google-website-optimizer-dead-long-live-google-analytics-content-experiments/#utm_source=google&utm_medium=organic&utm_campaign=(organic)&utm_content=-slb&utm_term=(not%20provided)&ts=1376954225

https://developers.google.com/analytics/solutions/experiments-server-side

• IT IS Be possible to build an analytics entry without all the fancy code, just loggin in and getting an experiment id (set the urls to 1.com 2.com , etc.) • All you really want is the Experiment ID… and the above all works correctly (no need to set the Serving Framework or do all that fancy backend authentication.

Now that you have the experiment ID and the variation IDs… you can simply proceed with setting up the experiment.

9. There is another way to do all this… using Javascript for the experiment! This way is good because it doesn’t require a refresh (javascript is doing the work) But it does require you to code up the experiment in javascript each time)https://developers.google.com/analytics/devguides/collection/gajs/experiments

Here’s how I did the Javascript method.

a. setup the experiment in analytics web interface with fake urls (oneki.com twoki.com etc.) * NOTE: I set the goal correctly though.. setting goals is really important this way, since you are relying on google to “chooseVariation”. b. add the following javascript and html (notice this is an example where I show or hide the “got feedback’ stuff). NOTE: I’ve added the experiment ID manually here since I am not using PHP

Got Feedback? Email us at <a href="mailto:help@keyingredient.com">help@keyingredient.com</a> and let us know how we are doing!</a>

 // 2. Choose the Variation for the Visitor
 var variation = cxApi.chooseVariation();

     // Show the HTML or Not!
    var content = document.getElementById("testhtml");

if (variation == 1) {

    content.style.display = "none";
      
        } else {
    content.style.display = "block";

 cxApi.setChosenVariation(
   variation,             // The index of the variation shown to the visitor
   'WfCFFL4gSz2ie7J_Vqg3nw'                // The id of the experiment the user has been exposed to
 );
 _gaq.push(['_trackEvent', 'WfCFFL4gSz2ie7J_Vqg3nw', 'WfCFFL4gSz2ie7J_Vqg3nw', 'WfCFFL4gSz2ie7J_Vqg3nw', variation, true]);

</script>

---

Testing experiments

For test particular variation in experiments add GET parameters to url. Pattern for url looks like this:

http://keyingredient.com/?test_[experiment_id]=[variation_value]&test_[experiment2_id]=[variation_value]

for example if experiment_id=jFaGW6l6Rny0tjL_lsOPeQ and we want test first variation:

http://keyingredient.com/?test_jFaGW6l6Rny0tjL_lsOPeQ=1

After enter on above link, the variation for this experiment will be save to a cookie.

For reset the variation value enter to link looks like this:

http://keyingredient.com/?test_[experiment_id]

In above case:

http://keyingredient.com/?test_jFaGW6l6Rny0tjL_lsOPeQ

Implementation for this feature is in:

assets/js/src/models/googleexperiment.js

Google Experiments documentation

Official Harlan Tytus Beverly Blog

Wednesday, June 18, 2014

3 ways to do Google A/B Testing or Multivariate testing in a Website or App

Contents

Summary

Step 1: Google Analytics Configuration

Setting up a Goal

Setting up an Experiment

Step 2: Implementing the Experiment in Code

Method 1: Redirect Method

Method 2: Javascript Method

Method 3: Dynamic HTML Method (PHP/Python templates)

Step 3: Look at Results and Pick a Winner

NOTE ABOUT SETTING PERCENTAGE OF TRAFFIC TO EXPERIMENT

APPENDIX: MORE INFORMATION ON STUFF

Testing experiments

1 comment:

Blog Archive

Followers

Wednesday, June 18, 2014

3 ways to do Google A/B Testing or Multivariate testing in a Website or App

Contents

Summary

Step 1: Google Analytics Configuration

Setting up a Goal

Setting up an Experiment

Step 2: Implementing the Experiment in Code

Method 1: Redirect Method

Method 2: Javascript Method

Method 3: Dynamic HTML Method (PHP/Python templates)

Step 3: Look at Results and Pick a Winner

NOTE ABOUT SETTING PERCENTAGE OF TRAFFIC TO EXPERIMENT

APPENDIX: MORE INFORMATION ON STUFF

Testing experiments

1 comment:

Blog Archive

Subscribe To

Followers