June Dershewitz on Web Analytics

The Evolution of SFGate (and Thoughts on Measuring Site Redesign)

Like many San Franciscans I read SFGate every single day.  A cup of coffee, my local news fix, and then I'm ready to face the world.  But when I reached for the paper earlier this week my routine was derailed - the home page had undergone a major redesign.

After a reluctant attempt to embrace the new look I thought, "Gee, seems like SFGate has been the same forever.  Has it really changed that much?"  So I visited the Wayback Machine, dug out old copies of SFGate, and made this timeline of the site's evolution:

Here's a close-up of last year versus current:

Conclusion: Until this latest redesign the SFGate home page had remained virtually unchanged since 2002.

To the design team's credit they are doing a fantastic job of accepting feedback and addressing questions in the wake of the redesign.  As of this writing there are 164 public comments about the redesign; some are amusing but snarky critiques ("Annoying, busy, fluffy and broken..."), others make useful observations ("The new page crashes Internet Explorer on my smartphone...").  The comments are worth browsing both as a consumer and as a web analytics professional.

My relationship with SFGate is simply as a consumer; I know nothing of their corporate approach to web measurement.  And yet I can't help but wonder how they are measuring redesign effectiveness.  Or, for that matter, how any of us are measuring redesign effectiveness whenever we do redesigns.

I am full of questions:

• Did the design team do any multivariate or A/B testing? 
• If not, was it a conscious decision or just the path of least resistance?
  
• Do they have a plan in place for measuring the impact of redesign on visitor behavior (and ultimately, on the bottom line)? 
• Was measurement baked into the redesign project plan or was it more of an afterthought? 
• Were web analysts involved early on, and if so, in what capacity?

I want to know!

I believe, as an industry, we are now more inclined to consider measurement as a valid and necessary part of redesign than we were a few years ago.  But I still think we've got some evolving to do. 

Just the other day, by way of Future Now's Bryan Eisenberg, I read that 76.7% of internet retailers do not do multivariate or A/B testing.  And that's just retail, arguably the most evolved of web verticals!  What about the rest of us?  Are we making the most of site redesign measurement, or is there still room for improvement?

As always I'd love your feedback.

[I have written about the site redesign measurement process before: here are 3 ways you can prepare for site redesign measurement.]

How to Prep for Site Redesign Measurement

I've been blogging for a few months now and it's been really fun, especially when it comes to choosing topics to write about.  So far I've posted about career development, events I attend and host, data integration, user-generated content, ... heck, I've even posted a picture of my family's pet chicken.

Now, in an attempt to say more about what I actually do at Semphonic, I thought I'd tell you about one project I'm currently working on.  Without giving out any identifying details about my client I'd like to share some tidbits that will be generally useful if you find yourself in a similar position.

So I'm doing a site redesign analysis project.  My client has recently made some major structural, interface and content changes to their site, and I'm comparing visitor behavior (and outcomes and sentiment) before and after the  launch of the new site. 

Redesign analysis is something I've done numerous times throughout my career in web analytics.  Simple fact:  sites get redesigned over and over.  Analyzing the impact of redesign is mostly about picking the right things to measure and then coming up with an interesting story to tell project stakeholders.  Let it be clear that I'm talking about sweeping changes to a whole site that happen at the flip of a switch, not incremental changes made by way of testing. 

Sometimes a redesign launches and I - as the measurement person - have looked back and said, "D'oh!  I should have thought of that in advance!"  In the remainder of this post I'll list out 3 ways I prepare for a site redesign measurement project.  Setting up these things ahead of time will allow you to avoid some known pitfalls.

Redesign Prep #1: Take Screenshots

What to do:
Take "before" screenshots of all the major pages on your site and put them aside for safekeeping.  Yes, you may be able to dig archived pages out of a content management system, but screenshots are a more reliable bet.

Let me just say I absolutely love the screen capture utility called SnagIt, especially the scrolling window functionality.  I recently discovered that SnagIt can also preserve links on a web page, which is quite useful. 

Why to do it:
Because you'll need to know what the site looked like prior to redesign, and  you may want to include these screenshots in your analysis presentation.

Redesign Prep #2: Decide on Timing

What to do:
Take out your calendar, pick "pre" and "post" windows of time to analyze, then pick dates a little farther out from that when you'll be able to share your analysis findings. 

It's a bit of a balancing act.  Invariably site owners want to know immediately whether or not their effort was a success, but from the analysis standpoint it's advisable to wait until a sufficient period of time has passed before you draw any conclusions about the impact of redesign. 

In my current project I'm doing a 2-week quick check-in followed by a 1-month post final analysis - that way I'm able to give some immediate feedback while I wait for the data to roll in and then use the longer time period in my final presentation.

Why to do it:
Because being proactive rather than reactive about data analysis is simply good form, plus it gives redesign project stakeholders some dates to look forward to.

Redesign Prep #3: Grab Data

What to do:
Figure out  if there's any historical data you won't be able to get after the site launches.  This varies depending on what tool you're using, but browser overlay and "Next Page" reports are the most commonly affected.   

What I mean is, you may only be able to view the browser overlay for the period of time when your page looks exactly like it does today.  For dates in the past when the page looked different, your browser overlay report may be unintelligible.

Know what's fleeting, then go in and grab data for your "pre" time period(s) while the data is still available.  If, like me, you choose to analyze two different windows of time - a short period and a long period - be sure to collect both of these snapshots.  Be thorough.

Why to do it:
Because you don't want to be faced with gaps in data as you pull together your analysis. 

So, how do you prepare for measuring redesign?  Anything you'd like to add to what I've mentioned here?  I welcome your comments.