June Dershewitz on Web Analytics

How to Get Bounce Rate for Google Website Optimizer A/B Test Variations

Suppose you’re planning to run a simple A/B test on a Web page and you’d like to use bounce rate as your measure of success. Can you do it with Google Website Optimizer (GWO)?

I’ve had this issue crop up a number of times recently, so I decided to research it. Here’s what I found out.

Can I set bounce rate as a conversion goal within GWO?

No. GWO itself does not report bounce rate, nor does it allow bounce rate to be set as a conversion goal. It’s possible to approximate bounce rate as a goal by coding every single link click on every single experiment page as a conversion event, but that gives you exit rate, not bounce rate - plus the setup can take a lot of effort.

Can I get bounce rate for GWO A/B test variations from Google Analytics or Omniture?

Yes. Although bounce rate isn’t reported within GWO, you can get it from Web analytics tools such as Google Analytics and Omniture. Integrating the data turns out to be very easy, and in fact requires zero coding as long as all experiment pages contain your standard Web analytics page tags.  

Here’s why it’s easy: As GWO serves variations in an A/B test, visitors are redirected to unique URLs (indexA.html vs. indexB.html vs. indexC.html, for example). This behavior was contrary to my initial assumption that all visitors got the same URL, as is the case for GWO multivariate tests. Since URLs are unique in an A/B test, you can simply view the pages report within your Web analytics tool and filter on (for example) index*.html to see one row per variation.

Do I need to pass any custom variables from GWO to my Web analytics tool?

No. You don’t need to pass any custom variables to get reporting for an A/B test.

If you're running a multivariate test, on the other hand, you must pass the GWO variation ID to your Web analytics tool as a custom variable, since the URL remains the same for all variations. If you’re interested in this technique, there are good discussion threads on this topic in the GWO support forum.

If I optimize for bounce rate outside of GWO, do I still need to set up a goal within GWO?

Yes, but it can be a simple placeholder. When you create a GWO A/B test you’re required to provide a conversion goal before you can launch. However, if you're planning to optimize for bounce rate or some other metric in your standard Web analytics tool, you can spoof a GWO conversion goal by specifying a dummy page somewhere on your site. In effect, you're simply using GWO to manage the serving of page variations.

To prove that it can be done, here are some screen shots from a test I set up:

Now it's your turn. Got any tips, tricks, admonishments?

The Evolution of SFGate (and Thoughts on Measuring Site Redesign)

Like many San Franciscans I read SFGate every single day.  A cup of coffee, my local news fix, and then I'm ready to face the world.  But when I reached for the paper earlier this week my routine was derailed - the home page had undergone a major redesign.

After a reluctant attempt to embrace the new look I thought, "Gee, seems like SFGate has been the same forever.  Has it really changed that much?"  So I visited the Wayback Machine, dug out old copies of SFGate, and made this timeline of the site's evolution:

Here's a close-up of last year versus current:

Conclusion: Until this latest redesign the SFGate home page had remained virtually unchanged since 2002.

To the design team's credit they are doing a fantastic job of accepting feedback and addressing questions in the wake of the redesign.  As of this writing there are 164 public comments about the redesign; some are amusing but snarky critiques ("Annoying, busy, fluffy and broken..."), others make useful observations ("The new page crashes Internet Explorer on my smartphone...").  The comments are worth browsing both as a consumer and as a web analytics professional.

My relationship with SFGate is simply as a consumer; I know nothing of their corporate approach to web measurement.  And yet I can't help but wonder how they are measuring redesign effectiveness.  Or, for that matter, how any of us are measuring redesign effectiveness whenever we do redesigns.

I am full of questions:

• Did the design team do any multivariate or A/B testing? 
• If not, was it a conscious decision or just the path of least resistance?
  
• Do they have a plan in place for measuring the impact of redesign on visitor behavior (and ultimately, on the bottom line)? 
• Was measurement baked into the redesign project plan or was it more of an afterthought? 
• Were web analysts involved early on, and if so, in what capacity?

I want to know!

I believe, as an industry, we are now more inclined to consider measurement as a valid and necessary part of redesign than we were a few years ago.  But I still think we've got some evolving to do. 

Just the other day, by way of Future Now's Bryan Eisenberg, I read that 76.7% of internet retailers do not do multivariate or A/B testing.  And that's just retail, arguably the most evolved of web verticals!  What about the rest of us?  Are we making the most of site redesign measurement, or is there still room for improvement?

As always I'd love your feedback.

[I have written about the site redesign measurement process before: here are 3 ways you can prepare for site redesign measurement.]

How to Prep for Site Redesign Measurement

I've been blogging for a few months now and it's been really fun, especially when it comes to choosing topics to write about.  So far I've posted about career development, events I attend and host, data integration, user-generated content, ... heck, I've even posted a picture of my family's pet chicken.

Now, in an attempt to say more about what I actually do at Semphonic, I thought I'd tell you about one project I'm currently working on.  Without giving out any identifying details about my client I'd like to share some tidbits that will be generally useful if you find yourself in a similar position.

So I'm doing a site redesign analysis project.  My client has recently made some major structural, interface and content changes to their site, and I'm comparing visitor behavior (and outcomes and sentiment) before and after the  launch of the new site. 

Redesign analysis is something I've done numerous times throughout my career in web analytics.  Simple fact:  sites get redesigned over and over.  Analyzing the impact of redesign is mostly about picking the right things to measure and then coming up with an interesting story to tell project stakeholders.  Let it be clear that I'm talking about sweeping changes to a whole site that happen at the flip of a switch, not incremental changes made by way of testing. 

Sometimes a redesign launches and I - as the measurement person - have looked back and said, "D'oh!  I should have thought of that in advance!"  In the remainder of this post I'll list out 3 ways I prepare for a site redesign measurement project.  Setting up these things ahead of time will allow you to avoid some known pitfalls.

Redesign Prep #1: Take Screenshots

What to do:
Take "before" screenshots of all the major pages on your site and put them aside for safekeeping.  Yes, you may be able to dig archived pages out of a content management system, but screenshots are a more reliable bet.

Let me just say I absolutely love the screen capture utility called SnagIt, especially the scrolling window functionality.  I recently discovered that SnagIt can also preserve links on a web page, which is quite useful. 

Why to do it:
Because you'll need to know what the site looked like prior to redesign, and  you may want to include these screenshots in your analysis presentation.

Redesign Prep #2: Decide on Timing

What to do:
Take out your calendar, pick "pre" and "post" windows of time to analyze, then pick dates a little farther out from that when you'll be able to share your analysis findings. 

It's a bit of a balancing act.  Invariably site owners want to know immediately whether or not their effort was a success, but from the analysis standpoint it's advisable to wait until a sufficient period of time has passed before you draw any conclusions about the impact of redesign. 

In my current project I'm doing a 2-week quick check-in followed by a 1-month post final analysis - that way I'm able to give some immediate feedback while I wait for the data to roll in and then use the longer time period in my final presentation.

Why to do it:
Because being proactive rather than reactive about data analysis is simply good form, plus it gives redesign project stakeholders some dates to look forward to.

Redesign Prep #3: Grab Data

What to do:
Figure out  if there's any historical data you won't be able to get after the site launches.  This varies depending on what tool you're using, but browser overlay and "Next Page" reports are the most commonly affected.   

What I mean is, you may only be able to view the browser overlay for the period of time when your page looks exactly like it does today.  For dates in the past when the page looked different, your browser overlay report may be unintelligible.

Know what's fleeting, then go in and grab data for your "pre" time period(s) while the data is still available.  If, like me, you choose to analyze two different windows of time - a short period and a long period - be sure to collect both of these snapshots.  Be thorough.

Why to do it:
Because you don't want to be faced with gaps in data as you pull together your analysis. 

So, how do you prepare for measuring redesign?  Anything you'd like to add to what I've mentioned here?  I welcome your comments.