Where We're at with Data Integration

I recently had the opportunity to ask 20 or so fellow web analytics practitioners what they're doing with respect to data integration.  Although each answer was as unique as the person who spoke, there was a common theme:  we want to do more in the future than we are doing today.

The few who are now farthest along in data integration efforts are all doing it in-house, with real data warehouses and technical teams who manage the underlying hardware and software and load processes.  Others are struggling to get systems like this up and running, all the while wondering if there is a better way.  The vast majority of us are sitting on the sidelines knowing data integration is something we want to do eventually but not knowing quite where to start.

Web analytics vendors have begun to address this common desire: see, for instance, WebTrends Visitor Intelligence and Omniture Genesis.  As hosted services, such products could make data integration possible for those of us who are not willing or able to maintain our own technical infrastructure.  If it's not the final answer, at least it's a step in the right direction. 

Assuming we get our tool issues sorted out, we'll still have people issues to contend with.  Data integration almost certainly involves the coordinated efforts of several groups within an organization.  Sometimes our counterparts don't want to share - for political reasons, or privacy reasons, or who knows what.  Getting everyone to work together may end up being the biggest challenge we'll face.

Yes, folks, that's what we're up against.  And yet, it's something we all want and I believe it's something we'll manage to do in the months and years to come.

Comments

[Eric T. Peterson]

June,

I've been thinking a lot about data integration lately, mostly in the context of bringing together qualitative and quantitative data to form a more complete view of visitor behavior. I've been hearing more and more great stories about this integration being accomplished --- sometimes with out-of-box implementations managed by companies who play well together, other times via more complex integrations built around some pretty sophisticated homegrown tools.

Clearly this is the wave of the future, but I have a question: Did you have a sense when you talked to your huddle about whether the primary interest is multi-channel data integration, or was it more integration of data from within the online channel?

I'm very exited to see you blogging and will add you to the blogroll at Web Analytics Demystified ASAP.

Welcome to it!

Eric T. Peterson
http://www.webanalyticsdemystified.com

[June]

Thank you for commenting, Eric! Your online vs. multi-channel question is a good one. Incorporating qualitative online data is a decent start, and there was definitely some talk of that in my huddle (both as a desire and as an accomplishment). In the long run, though, I believe we want to take *anything* we happen to know about our customer and cross it with web behavior. If the business extends significantly beyond the web channel, the level of integration will, too.

[Keith Holloway]

Hi June,

The only integration I see is that being done in-house, too. We find we need to do it ourselves, and I am happy to do so because it represents opportunity. In fact, I see tremendous opportunities here, and I wish I had unlimited people and resources to make the integrations I need happen, but unfortunately I don't!

Getting the help of 3rd parties can be very difficult, too. I've called WebTrends to discuss integration options with our email marketing software, but they didn't return my call. Even as a WebTrends customer, I can't even get a competent person on the phone to discuss technical issues. I agree that getting everyone to work together could be the biggest challenge.

I have seen a few email marketing companies "integrate" with Google Analytics recently, although they are simply generating links that GA understands and therefore are reported on in the GA interface - it doesn't and didn't require any actual "integration" with GA.

RE: Eric's question about what is being integrated... we are integrating mostly online data - conversions, search ranks, email clicks and campaigns, and most recently voice calls.

I can see more of the latter (voice) and other offline data appearing more and more on the integration list.

[June]

Keith, thanks for your thoughtful comments. I believe we see eye-to-eye on this issue - it's not that data integration is impossible, it's just far more complex than we'd like!

[Charlie Ballard]

Hi, June. Found your blog from a recommendation by Eric, great stuff.

As an M&A manager here in Boston at an interactive agency, my biggest frustration over the past two years has been trying to find technology that can help me answer one simple question: "what lift does running Display advertising concurrently with Paid Search and SEO have on the others?" Banner ads don't get the ROI results that clients have grown to love from PPC, but if they can have an effect on other channels then it's far easier to justify budget. And we've seen directionally that they DO seem to have some effect, but dammit, we just can't prove it.

I asked the chair of the IAB this question last year and was rewarded with the kinda snarky reply: "well, you run one alone for a month or two, and then you run them together for a month or two and then measure the difference". No good. Our clients experience too much seasonality.

Products like Omniture Genesis seem like they might approach this "2 Plus 2 Might Equal 5" equation, but fall just short, as the goal of Genesis seems to be more to provide centralized reporting than it is on measuring true cross-channel lift. And as Eric says above, this doesn't even begin to describe qualitative results.

If you have any good suggestions or have heard of anyone successfully pulling this off, I would *love* to hear about it.

Thanks!

[June Dershewitz]

Hi Charles! I understand your frustration. Here's another approach to measuring cross-channel lift:

First I want to rephrase your question slightly. Let's ask, "Is someone who sees (but does not necessarily click) a banner ad more likely to reach my site by way of search?" In order to answer this question I will assume that you have 1) access to visitor-level web activity, and 2) integrated banner impression data.

For the purposes of this example, let's consider the pool of visitors who came to your site yesterday. Split these visitors into 2 groups: those who've seen 1 or more banners in the past 7 days (for example), and those who've seen no banners at all.

Now calculate the % of visitors in each group who clicked through to your site from search results 1 or more times yesterday. You'll wind up with 2 numbers: % search visitors with banner, and % search visitors without banner. There you have it.

As I mentioned, this is just an example. You'll want to tweak it to suit your needs. Hopefully it'll give you what you need to justify (or not) your display ad spend.

[Fernando Labastida]

Hi June,

The fact that web analytics professionals are considering or embarking on data integration projects shows how far this segment has come in the last 10-12 years. The interesting thing is, data integration is such an old problem, but is now being applied to the relatively new discipline of web analytics.

The problem is that most traditional data integration solutions are "big iron" enterprise solutions, and are the domain of the application and IT experts. It's a good thing that WebTrends and Omniture are getting into the game. There are a crop of agile, lightweight data integration solutions that are cost-effective and can be used by business analysts, so the web analysts should easily be able to "get off the fence" and start the ETL processes going!

Fernando Labastida
Fernando's Integration Blog
http://labastida.com