Integrating Marketing Mix Modeling with Data-driven Attribution for Holistic Insights

Today’s marketers have more opportunities than ever to drive business success. They also face increasing pressure to prove, manage, and optimize marketing performance. 

A relentless push towards accountability has driven the adoption of ever-more-sophisticated measurement tools. Many marketers use marketing mix modeling (MMM), some use data-driven attribution, while others consult a separate solution for each. 

Tools continue to evolve. Now, solutions that merge and substantively improve both of these measurement best practices promise faster, more efficient, more holistic insights. To find out more, we commissioned Forrester Consulting to survey 150 companies in order to explore how marketers are evaluating, adopting, and using these emerging tools. Key learnings will be presented in our Dec 8th webinar hosted by Google and featuring Tina Moffett, Senior Analyst at Forrester along with Dave Barney, Product Manager, Adometry at Google. Sign up here.

Why consider a merger?

While separate MMM and data-driven attribution tools offer cross-channel measurement, each has limitations:

• Speed and Granularity. Traditional MMM offers high-level analysis on a quarterly or yearly basis, which can limit more granular, or on-the-fly optimization
  
• Data Limitations. Data-driven attribution requires a wealth of granular, user-level  data, which can limit offline channel visibility

When the two measurement practices are combined, however, they improve the outputs from each. Data-driven attribution informs MMM models. MMM data feeds attribution analysis. Resulting insights allow marketers to see the impact of each marketing element in near real-time.

Pending or trending?

Today, many marketers get the optimization benefit from separate MMM and data-driven attribution tools. Will merged tools become a new marketing performance measurement standard?

While it may be too early to tell, there is a growing desire for tools that help marketers move beyond channel-based optimization to larger strategic cross-channel planning. Forrester reports that many respondents have already moved, or plan to move, on the merged measurement trend and the most common approach has been to purchase a solution from a vendor, and to make use of the vendor’s implementation support. 

“There will be a paradigm shift in understanding for the marketing channels. I think it gives them an opportunity to think holistically rather than in a silo, like, ‘this is my world, this is my budget, as long as I get this much traffic in my channel, I am ok.’ It’s no longer the case. Getting that understanding is going to be key. It gives us better understanding of how our customers navigate through different touch-points.”

— Director of Marketing And Automation Systems at a major global retailer

Benefits and challenges

Integrated MMM and data-driven attribution tools are enabling marketers to make strategic planning decisions and precisely measure individual-level interactions in near real-time. Satisfaction with integrated tools is high among those who have implemented them.

Faster access to insights has more companies looping in more stakeholders from marketing execs and analysts to customer insights or analytics, brand managers, and eCommerce professionals. 

At the same time, early adopters report challenges. Integrating tools and data sources is a big ask, learning when to make changes based on new insights takes time, and setting expectations about timelines and results is paramount. 

Ensuring that the entire organization is on board with using a merged measurement platform is critical, as is supporting stakeholders in changing business practices as a result.

Proceed with insight

As merged tools come on strong, the experiences of early adopters may be instructive to those moving to embrace a merged solution. Recommendations on best practices, processes, and supports, are examined in the full whitepaper. 

Making the right move

While companies cite common barriers to adoption, respondents suggest that a number of challenges that are stopping them today would be resolved in the near future including, skills, understanding of benefits and technology blockers.  

As merged tools mature and become more commonplace, technology concerns will abate. More marketers will know about these solutions, and about how to use them to drive marketing optimization and strategy. Staying informed is the key to making the right call on whether, when, and how to adopt merged measurement tools for your business.

To learn more, sign up for our upcoming webinar with Forrester Research on December 8th.

Posted by Natasha Moonka, Product Marketing Manager, Google Analytics team

TensorFlow - Google’s latest machine learning system, open sourced for everyone

Posted by Jeff Dean, Senior Google Fellow, and Rajat Monga, Technical Lead

Deep Learning has had a huge impact on computer science, making it possible to explore new frontiers of research and to develop amazingly useful products that millions of people use every day. Our internal deep learning infrastructure DistBelief, developed in 2011, has allowed Googlers to build ever larger neural networks and scale training to thousands of cores in our datacenters. We’ve used it to demonstrate that concepts like “cat” can be learned from unlabeled YouTube images, to improve speech recognition in the Google app by 25%, and to build image search in Google Photos. DistBelief also trained the Inception model that won Imagenet’s Large Scale Visual Recognition Challenge in 2014, and drove our experiments in automated image captioning as well as DeepDream.

While DistBelief was very successful, it had some limitations. It was narrowly targeted to neural networks, it was difficult to configure, and it was tightly coupled to Google’s internal infrastructure - making it nearly impossible to share research code externally.

Today we’re proud to announce the open source release of TensorFlow -- our second-generation machine learning system, specifically designed to correct these shortcomings. TensorFlow is general, flexible, portable, easy-to-use, and completely open source. We added all this while improving upon DistBelief’s speed, scalability, and production readiness -- in fact, on some benchmarks, TensorFlow is twice as fast as DistBelief (see the whitepaper for details of TensorFlow’s programming model and implementation).

TensorFlow has extensive built-in support for deep learning, but is far more general than that -- any computation that you can express as a computational flow graph, you can compute with TensorFlow (see some examples). Any gradient-based machine learning algorithm will benefit from TensorFlow’s auto-differentiation and suite of first-rate optimizers. And it’s easy to express your new ideas in TensorFlow via the flexible Python interface.

Inspecting a model with TensorBoard, the visualization tool

TensorFlow is great for research, but it’s ready for use in real products too. TensorFlow was built from the ground up to be fast, portable, and ready for production service. You can move your idea seamlessly from training on your desktop GPU to running on your mobile phone. And you can get started quickly with powerful machine learning tech by using our state-of-the-art example model architectures. For example, we plan to release our complete, top shelf ImageNet computer vision model on TensorFlow soon.

But the most important thing about TensorFlow is that it’s yours. We’ve open-sourced TensorFlow as a standalone library and associated tools, tutorials, and examples with the Apache 2.0 license so you’re free to use TensorFlow at your institution (no matter where you work).

Our deep learning researchers all use TensorFlow in their experiments. Our engineers use it to infuse Google Search with signals derived from deep neural networks, and to power the magic features of tomorrow. We’ll continue to use TensorFlow to serve machine learning in products, and our research team is committed to sharing TensorFlow implementations of our published ideas. We hope you’ll join us at www.tensorflow.org.

Wordsmith for Marketing: Using the Reporting API to automate agency client reports

This is a guest post by Cole Faloon, a developer for Wordsmith for Marketing at Automated Insights.

Digital marketing professionals live and breathe Google Analytics, AdWords and social media, constantly measuring just how well their strategies are performing. But communicating successes in client reports takes an inordinate amount of time. Enter Wordsmith for Marketing, the client reporting solution from Automated Insights that automatically transforms Google Analytics, AdWords and social data into plain-English reports.

The vastness of data in Google Analytics made it an obvious foundation for Wordsmith for Marketing. Our app is built around the Google Analytics Core Reporting API. The app pulls down metrics like visits, page views, and conversions for different periods, comparing the data across spans of time.

The API is flexible enough for us to receive dates at the ranges we need. We can slice up the data by pre-defined dimensions by week, month, and quarter.

Another feature we love? Google's implementation of the OAuth 2.0 Authorization Framework. It allows users of our solution to sign in with their Google account, getting us access to their Analytics data right away and creating a fluid user experience. They just log in and they’re ready to go.

Empowered by Google Analytics, we give marketers a clear explanation of how their clients’ digital marketing efforts are performing and advice on how to improve; they have the option of editing the reports to add finishing touches or comments before sending them on to their clients. Wordsmith for Marketing automatically produces insightful client-ready analysis, saving marketing agencies hundreds of hours and thousands of dollars while allowing them to better serve their clients. 

For more information, visit the Wordsmith for Marketing app listing in the Google Analytics Partner Gallery or the Wordsmith for Marketing website.

- The Google Analytics Developer Relations team, on behalf of Wordsmith for Marketing

Computer, respond to this email.

Posted by Greg Corrado*, Senior Research Scientist

Machine Intelligence for You

What I love about working at Google is the opportunity to harness cutting-edge machine intelligence for users’ benefit. Two recent Research Blog posts talked about how we’ve used machine learning in the form of deep neural networks to improve voice search and YouTube thumbnails. Today we can share something even wilder -- Smart Reply, a deep neural network that writes email.

I get a lot of email, and I often peek at it on the go with my phone. But replying to email on mobile is a real pain, even for short replies. What if there were a system that could automatically determine if an email was answerable with a short reply, and compose a few suitable responses that I could edit or send with just a tap?

Some months ago, Bálint Miklós from the Gmail team asked me if such a thing might be possible. I said it sounded too much like passing the Turing Test to get our hopes up... but having collaborated before on machine learning improvements to spam detection and email categorization, we thought we’d give it a try.

There’s a long history of research on both understanding and generating natural language for applications like machine translation. Last year, Google researchers Oriol Vinyals, Ilya Sutskever, and Quoc Le proposed fusing these two tasks in what they called sequence-to-sequence learning. This end-to-end approach has many possible applications, but one of the most unexpected that we’ve experimented with is conversational synthesis. Early results showed that we could use sequence-to-sequence learning to power a chatbot that was remarkably fun to play with, despite having included no explicit knowledge of language in the program.

Obviously, there’s a huge gap between a cute research chatbot and a system that I want helping me draft email. It was still an open question if we could build something that was actually useful to our users. But one engineer on our team, Anjuli Kannan, was willing to take on the challenge. Working closely with both Machine Intelligence researchers and Gmail engineers, she elaborated and experimented with the sequence-to-sequence research ideas. The result is the industrial strength neural network that runs at the core of the Smart Reply feature we’re launching this week.

How it works

A naive attempt to build a response generation system might depend on hand-crafted rules for common reply scenarios. But in practice, any engineer’s ability to invent “rules” would be quickly outstripped by the tremendous diversity with which real people communicate. A machine-learned system, by contrast, implicitly captures diverse situations, writing styles, and tones. These systems generalize better, and handle completely new inputs more gracefully than brittle, rule-based systems ever could.

Diagram by Chris Olah

Like other sequence-to-sequence models, the Smart Reply System is built on a pair of recurrent neural networks, one used to encode the incoming email and one to predict possible responses. The encoding network consumes the words of the incoming email one at a time, and produces a vector (a list of numbers). This vector, which Geoff Hinton calls a “thought vector,” captures the gist of what is being said without getting hung up on diction -- for example, the vector for "Are you free tomorrow?" should be similar to the vector for "Does tomorrow work for you?" The second network starts from this thought vector and synthesizes a grammatically correct reply one word at a time, like it’s typing it out. Amazingly, the detailed operation of each network is entirely learned, just by training the model to predict likely responses.

One challenge of working with emails is that the inputs and outputs of the model can be hundreds of words long. This is where the particular choice of recurrent neural network type really matters. We used a variant of a "long short-term-memory" network (or LSTM for short), which is particularly good at preserving long-term dependencies, and can home in on the part of the incoming email that is most useful in predicting a response, without being distracted by less relevant sentences before and after.

Of course, there's another very important factor in working with email, which is privacy. In developing Smart Reply we adhered to the same rigorous user privacy standards we’ve always held -- in other words, no humans reading your email. This means researchers have to get machine learning to work on a data set that they themselves cannot read, which is a little like trying to solve a puzzle while blindfolded -- but a challenge makes it more interesting!

Getting it right

Our first prototype of the system had a few unexpected quirks. We wanted to generate a few candidate replies, but when we asked our neural network for the three most likely responses, it’d cough up triplets like “How about tomorrow?” “Wanna get together tomorrow?” “I suggest we meet tomorrow.” That’s not really much of a choice for users. The solution was provided by Sujith Ravi, whose team developed a great machine learning system for mapping natural language responses to semantic intents. This was instrumental in several phases of the project, and was critical to solving the "response diversity problem": by knowing how semantically similar two responses are, we can suggest responses that are different not only in wording, but in their underlying meaning.

Another bizarre feature of our early prototype was its propensity to respond with “I love you” to seemingly anything. As adorable as this sounds, it wasn’t really what we were hoping for. Some analysis revealed that the system was doing exactly what we’d trained it to do, generate likely responses -- and it turns out that responses like “Thanks", "Sounds good", and “I love you” are super common -- so the system would lean on them as a safe bet if it was unsure. Normalizing the likelihood of a candidate reply by some measure of that response's prior probability forced the model to predict responses that were not just highly likely, but also had high affinity to the original message. This made for a less lovey, but far more useful, email assistant.

Give it a try

We’re actually pretty amazed at how well this works. We’ll be rolling this feature out on Inbox for Android and iOS later this week, and we hope you’ll try it for yourself! Tap on a Smart Reply suggestion to start editing it. If it’s perfect as is, just tap send. Two-tap email on the go -- just like Bálint envisioned.

---

* This blog post may or may not have actually been written by a neural network.^↩

How to measure translation quality in your user interfaces

Posted by Javier Bargas-Avila, User Experience Research at Google

Worldwide, there are about 200 languages that are spoken by at least 3 million people. In this global context, software developers are required to translate their user interfaces into many languages. While graphical user interfaces have evolved substantially when compared to text-based user interfaces, they still rely heavily on textual information. The perceived language quality of translated user interfaces (UIs) can have a significant impact on the overall quality and usability of a product. But how can software developers and product managers learn more about the quality of a translation when they don’t speak the language themselves?

Key information in interaction elements and content are mostly conveyed through text. This aspect can be illustrated by removing text elements from a UI, as shown in the the figure below.

Three versions of the YouTube UI: (a) the original, (b) YouTube without text elements, and (c) YouTube without graphic elements. It gets apparent how the textless version is stripped of the most useful information: it is almost impossible to choose a video to watch and navigating the site is impossible.

In "Measuring user rated language quality: Development and validation of the user interface Language Quality Survey (LQS)", recently published in the International Journal of Human-Computer Studies, we describe the development and validation of a survey that enables users to provide feedback about the language quality of the user interface.

UIs are generally developed in one source language and translated afterwards string by string. The process of translation is prone to errors and might introduce problems that are not present in the source. These problems are most often due to difficulties in the translation process. For example, the word “auto” can be translated to French as automatique (automatic) or automobile (car), which obviously has a different meaning. Translators might chose the wrong term if context is missing during the process. Another problem arises from words that behave as a verb when placed in a button or as a noun if part of a label. For example, “access” can stand for “you have access” (as a label) or “you can request access” (as a button).

Further pitfalls are gender, prepositions without context or other characteristics of the source text that might influence translation. These problems sometimes even get aggravated by the fact that translations are made by different linguists at different points in time. Such mistranslations might not only negatively affect trustworthiness and brand perception, but also the acceptance of the product and its perceived usefulness.

This work was motivated by the fact that in 2012, the YouTube internationalization team had anecdotal evidence which suggested that some language versions of YouTube might benefit from improvement efforts. While expert evaluations led to significant improvements of text quality, these evaluations were expensive and time-consuming. Therefore, it was decided to develop a survey that enables users to provide feedback about the language quality of the user interface to allow a scalable way of gathering quantitative data about language quality.

The Language Quality Survey (LQS) contains 10 questions about language quality. The first five questions form the factor “Readability”, which describes how natural and smooth to read the used text is. For instance, one question targets ease of understanding (“How easy or difficult to understand is the text used in the [product name] interface?”). Questions 6 to 9 summarize the frequency of (in)consistencies in the text, called “Linguistic Correctness”. The full survey can be found in the publication.

Case study: applying the LQS in the field

As the LQS was developed to discover problematic translations of the YouTube interface and allow focused quality improvement efforts, it was made available in over 60 languages and data were gathered for all these versions of the YouTube interface. To understand the quality of each UI version, we compared the results for the translated versions to the source language (here: US-English). We inspected first the global item, in combination with Linguistic Correctness and Readability. Second, we inspected each item separately, to understand which notion of Linguistic Correctness or Readability showed worse (or better) values. Here are some results:

• The data revealed that about one third of the languages showed subpar language quality levels, when compared to the source language.
• To understand the source of these problems and fix them, we analyzed the qualitative feedback users had provided (every time someone selected the lower two end scale points, pointing at a problem in the language, a text box was surfaced, asking them to provide examples or links to illustrate the issues).
• The analysis of these comments provided linguists with valuable feedback of various kinds. For instance, users pointed to confusing terminology, untranslated words that were missed during translation, typographical or grammatical problems, words that were translated but are commonly used in English, or screenshots in help pages that were in English but needed to be localized. Some users also pointed to readability aspects such as sections with old fashioned or too formal tone as well as too informal translations, complex technical or legal wordings, unnatural translations or rather lengthy sections of text. In some languages users also pointed to text that was too small or criticized the readability of the font that was used.
• In parallel, in-depth expert reviews (so-called “language find-its”) were organized. In these sessions, a group of experts for each language met and screened all of YouTube to discover aspects of the language that could be improved and decided on concrete actions to fix them. By using the LQS data to select target languages, it was possible to reduce the number of language find-its to about one third of the original estimation (if all languages had been screened).

LQS has since been successfully adapted and used for various Google products such as Docs, Analytics, or AdWords. We have found the LQS to be a reliable, valid and useful tool to approach language quality evaluation and improvement. The LQS can be regarded as a small piece in the puzzle of understanding and improving localization quality. Google is making this survey broadly available, so that everyone can start improving their products for everyone around the world.

Introducing the Definitive Guide to Data-Driven Attribution

Originally Posted on the Adometry M2R Blog

For as many dollars organizations invest in marketing, it never ceases to amaze me how many of those organizations are willing to make guesses about how effectively those dollars are being used. Even when those guesses are educated, they can be way off. We live in a world where data-driven attribution can take the guesswork out of your marketing program to gain a clear and comprehensive view into the customer journey.

It can be intimidating to get started with data-driven attribution. Many marketers are already inundated with data from marketing mix modeling, real-time bidding, website analytics, CRM and more. But the genius of data-driven attribution is that it makes all that other data better, more relevant and actionable to improve the bottom line.

With our Definitive Guide to Data-Driven Attribution, we’ve laid out just how your organization can approach marketing attribution. We’ve made it easy to understand what data-driven attribution does, how it fits in with what you’re already doing and how to get started.

What Is Attribution and What Are the Benefits?

Let’s start with the basics. There are a number of basic models such as first touch, last touch, even and custom attribution. Those models offer general answers across a basic marketing mix, but they fail to provide the true value of each marketing asset as the marketing campaigns get more complex. Today’s cross-channel marketers need a more scientific approach.

Data-driven attribution models use sophisticated algorithms to determine which touch points are the most influential. That means marketers can see the benefits of each touch point and adjust future spending to maximize results.

How Does Data-Driven Attribution Fit into my Analytics Toolset?

Odds are you’re already collecting a ton of marketing and advertising data. That’s great! Data-driven attribution doesn't replace that information. It greatly enhances it.

As an example, let’s look at marketing mix modeling. At the end of a campaign, you look back and assess performance. With data-driven attribution, you can accurately see how each tactic performed so you can plan better for the next campaign. Extending that to the next step, accurate attribution gives you insight that your real-time bidding partners can use to buy top performing ad placements.

Another example is your CRM. As you gain customers, your CRM captures transaction, contact and segment data, but CRMs tend to focus more on customer service and support, not marketing. And although CRMs track multiple channels, they look at lower-funnel activities and offer limited visibility into acquisition and cross-channel marketing in non-direct channels. CRM data is an input that can feed your data-driven attribution solution to yield a more complete picture of customer behavior.

As the graphic above shows (and details more within the guide), data-driven attribution ties all of your other marketing analytics together and improves what you’ve been getting from each one.

Getting Started

Data-driven solutions vary. To get the benefits, you’ll need to ask the right questions about your organization, solidify the right budgets and motivate the right people. In the guide we outline five key steps to getting started.

1. Define Goals: Consider your current pain points and business goals. Determine the value that all of your marketing activities must deliver for the business and take a holistic view of the data-driven changes you’ll make to meet those goals. That will help determine marketing’s impact on revenue so you can formulate budgets that will yield the highest returns.


2. Justify Budget: The right solution will pay for itself by creating cross-department efficiencies and increasing the return on each marketing investment, but change can be difficult. Check out the full Definitive Guide for a real-world budgeting exercise to help you promote the benefits of data-driven attribution to key stakeholders.


3. Be Selective: There are a number of attribution providers. Evaluate them by asking the right questions about their ease of implementation, breadth of services, methodology, capabilities and technology roadmap. Can they handle your data? How will they work with your existing partners, including your ad agency? Do they provide a consultative partnership? Is their model data-driven or rules-based? Are they media agnostic? How is their model validated? Can they measure online and offline activities? How do they account for multi-screen customer journeys? How often do they upgrade their solution?


4. Get Prepared: Picking a provider is a good start, but you also must get ready for integration. Prepare both human and data resources to hit the ground running. Evaluating data readiness and preparing stakeholders ahead of time will help you determine how much support you’ll need during implementation.


5. Evaluate Success: Your stakeholders will be more invested in driving success with data-driven attribution if they can envision what success looks like, and concretely evaluate whether goals are being achieved. Show them the way. Leverage your goals to evaluate your provider’s performance on marketing performance, enterprise ability, ease and flexibility, quality of output, total cost of ownership and an innovative roadmap.

There’s no doubt that today’s marketers need better performance measures to know whether they are producing the best results for the organization. Data-driven attribution requires investment on the front end, but it pays big rewards that will have you asking why you didn’t take the plunge sooner.

We encourage you to dive deeper to help your organization understand the true benefits and implications of data driven attribution through our definitive guide.

AlfaStrakhovanie Doubles Transaction Rates With Google Analytics Enhanced Ecommerce

AlfaStrakhovanie LLC is one of Russia's largest insurance companies, and they need a way to measure all the complexities of an insurance business in an online platform. For example, insurance companies can only count profit after a policy expires, not when it is purchased, as accident claims and customer payouts need to be taken into account.

The company partnered with Agima, a Google Analytics Certified Partner, to find the best solution for a comprehensive online acquisition and retention measurement. Using Custom Dimensions and advanced Enhanced Ecommerce, AlfaStrakhovanie was able to have a better understanding of the purchase funnel per customer segment.

"AGIMA Interactive Agency is our long-term partner, helping us grow our profit on the internet and avoid the online pitfalls of the insurance business. It combines quantitative with qualitative research methods, so all our decisions now are data-driven and fully conscious. We have a strong confirmation of success and profit growth, and this allows us to continue our collaborative work." Tatyana Puchkova, VP of Marketing, AlfaStrakhovanie

As a result of the new implementation and ongoing data analysis, the company’s transaction rate has more than doubled, and revenue trends are on the rise while average order size remains untouched. The company now understands which customers and car owners it reaches with its marketing and (with the help of competitive price analysis) how the price affects transaction rates. The company is aware of the loss of potential profit because it knows the demand and can adjust prices accordingly.

To learn more read the full case study

Posted by Daniel Waisberg, Analytics Advocate