How to measure translation quality in your user interfaces

Posted by Javier Bargas-Avila, User Experience Research at Google

Worldwide, there are about 200 languages that are spoken by at least 3 million people. In this global context, software developers are required to translate their user interfaces into many languages. While graphical user interfaces have evolved substantially when compared to text-based user interfaces, they still rely heavily on textual information. The perceived language quality of translated user interfaces (UIs) can have a significant impact on the overall quality and usability of a product. But how can software developers and product managers learn more about the quality of a translation when they don’t speak the language themselves?

Key information in interaction elements and content are mostly conveyed through text. This aspect can be illustrated by removing text elements from a UI, as shown in the the figure below.

Three versions of the YouTube UI: (a) the original, (b) YouTube without text elements, and (c) YouTube without graphic elements. It gets apparent how the textless version is stripped of the most useful information: it is almost impossible to choose a video to watch and navigating the site is impossible.

In "Measuring user rated language quality: Development and validation of the user interface Language Quality Survey (LQS)", recently published in the International Journal of Human-Computer Studies, we describe the development and validation of a survey that enables users to provide feedback about the language quality of the user interface.

UIs are generally developed in one source language and translated afterwards string by string. The process of translation is prone to errors and might introduce problems that are not present in the source. These problems are most often due to difficulties in the translation process. For example, the word “auto” can be translated to French as automatique (automatic) or automobile (car), which obviously has a different meaning. Translators might chose the wrong term if context is missing during the process. Another problem arises from words that behave as a verb when placed in a button or as a noun if part of a label. For example, “access” can stand for “you have access” (as a label) or “you can request access” (as a button).

Further pitfalls are gender, prepositions without context or other characteristics of the source text that might influence translation. These problems sometimes even get aggravated by the fact that translations are made by different linguists at different points in time. Such mistranslations might not only negatively affect trustworthiness and brand perception, but also the acceptance of the product and its perceived usefulness.

This work was motivated by the fact that in 2012, the YouTube internationalization team had anecdotal evidence which suggested that some language versions of YouTube might benefit from improvement efforts. While expert evaluations led to significant improvements of text quality, these evaluations were expensive and time-consuming. Therefore, it was decided to develop a survey that enables users to provide feedback about the language quality of the user interface to allow a scalable way of gathering quantitative data about language quality.

The Language Quality Survey (LQS) contains 10 questions about language quality. The first five questions form the factor “Readability”, which describes how natural and smooth to read the used text is. For instance, one question targets ease of understanding (“How easy or difficult to understand is the text used in the [product name] interface?”). Questions 6 to 9 summarize the frequency of (in)consistencies in the text, called “Linguistic Correctness”. The full survey can be found in the publication.

Case study: applying the LQS in the field

As the LQS was developed to discover problematic translations of the YouTube interface and allow focused quality improvement efforts, it was made available in over 60 languages and data were gathered for all these versions of the YouTube interface. To understand the quality of each UI version, we compared the results for the translated versions to the source language (here: US-English). We inspected first the global item, in combination with Linguistic Correctness and Readability. Second, we inspected each item separately, to understand which notion of Linguistic Correctness or Readability showed worse (or better) values. Here are some results:

• The data revealed that about one third of the languages showed subpar language quality levels, when compared to the source language.
• To understand the source of these problems and fix them, we analyzed the qualitative feedback users had provided (every time someone selected the lower two end scale points, pointing at a problem in the language, a text box was surfaced, asking them to provide examples or links to illustrate the issues).
• The analysis of these comments provided linguists with valuable feedback of various kinds. For instance, users pointed to confusing terminology, untranslated words that were missed during translation, typographical or grammatical problems, words that were translated but are commonly used in English, or screenshots in help pages that were in English but needed to be localized. Some users also pointed to readability aspects such as sections with old fashioned or too formal tone as well as too informal translations, complex technical or legal wordings, unnatural translations or rather lengthy sections of text. In some languages users also pointed to text that was too small or criticized the readability of the font that was used.
• In parallel, in-depth expert reviews (so-called “language find-its”) were organized. In these sessions, a group of experts for each language met and screened all of YouTube to discover aspects of the language that could be improved and decided on concrete actions to fix them. By using the LQS data to select target languages, it was possible to reduce the number of language find-its to about one third of the original estimation (if all languages had been screened).

LQS has since been successfully adapted and used for various Google products such as Docs, Analytics, or AdWords. We have found the LQS to be a reliable, valid and useful tool to approach language quality evaluation and improvement. The LQS can be regarded as a small piece in the puzzle of understanding and improving localization quality. Google is making this survey broadly available, so that everyone can start improving their products for everyone around the world.

Introducing the Definitive Guide to Data-Driven Attribution

Originally Posted on the Adometry M2R Blog

For as many dollars organizations invest in marketing, it never ceases to amaze me how many of those organizations are willing to make guesses about how effectively those dollars are being used. Even when those guesses are educated, they can be way off. We live in a world where data-driven attribution can take the guesswork out of your marketing program to gain a clear and comprehensive view into the customer journey.

It can be intimidating to get started with data-driven attribution. Many marketers are already inundated with data from marketing mix modeling, real-time bidding, website analytics, CRM and more. But the genius of data-driven attribution is that it makes all that other data better, more relevant and actionable to improve the bottom line.

With our Definitive Guide to Data-Driven Attribution, we’ve laid out just how your organization can approach marketing attribution. We’ve made it easy to understand what data-driven attribution does, how it fits in with what you’re already doing and how to get started.

What Is Attribution and What Are the Benefits?

Let’s start with the basics. There are a number of basic models such as first touch, last touch, even and custom attribution. Those models offer general answers across a basic marketing mix, but they fail to provide the true value of each marketing asset as the marketing campaigns get more complex. Today’s cross-channel marketers need a more scientific approach.

Data-driven attribution models use sophisticated algorithms to determine which touch points are the most influential. That means marketers can see the benefits of each touch point and adjust future spending to maximize results.

How Does Data-Driven Attribution Fit into my Analytics Toolset?

Odds are you’re already collecting a ton of marketing and advertising data. That’s great! Data-driven attribution doesn't replace that information. It greatly enhances it.

As an example, let’s look at marketing mix modeling. At the end of a campaign, you look back and assess performance. With data-driven attribution, you can accurately see how each tactic performed so you can plan better for the next campaign. Extending that to the next step, accurate attribution gives you insight that your real-time bidding partners can use to buy top performing ad placements.

Another example is your CRM. As you gain customers, your CRM captures transaction, contact and segment data, but CRMs tend to focus more on customer service and support, not marketing. And although CRMs track multiple channels, they look at lower-funnel activities and offer limited visibility into acquisition and cross-channel marketing in non-direct channels. CRM data is an input that can feed your data-driven attribution solution to yield a more complete picture of customer behavior.

As the graphic above shows (and details more within the guide), data-driven attribution ties all of your other marketing analytics together and improves what you’ve been getting from each one.

Getting Started

Data-driven solutions vary. To get the benefits, you’ll need to ask the right questions about your organization, solidify the right budgets and motivate the right people. In the guide we outline five key steps to getting started.

1. Define Goals: Consider your current pain points and business goals. Determine the value that all of your marketing activities must deliver for the business and take a holistic view of the data-driven changes you’ll make to meet those goals. That will help determine marketing’s impact on revenue so you can formulate budgets that will yield the highest returns.


2. Justify Budget: The right solution will pay for itself by creating cross-department efficiencies and increasing the return on each marketing investment, but change can be difficult. Check out the full Definitive Guide for a real-world budgeting exercise to help you promote the benefits of data-driven attribution to key stakeholders.


3. Be Selective: There are a number of attribution providers. Evaluate them by asking the right questions about their ease of implementation, breadth of services, methodology, capabilities and technology roadmap. Can they handle your data? How will they work with your existing partners, including your ad agency? Do they provide a consultative partnership? Is their model data-driven or rules-based? Are they media agnostic? How is their model validated? Can they measure online and offline activities? How do they account for multi-screen customer journeys? How often do they upgrade their solution?


4. Get Prepared: Picking a provider is a good start, but you also must get ready for integration. Prepare both human and data resources to hit the ground running. Evaluating data readiness and preparing stakeholders ahead of time will help you determine how much support you’ll need during implementation.


5. Evaluate Success: Your stakeholders will be more invested in driving success with data-driven attribution if they can envision what success looks like, and concretely evaluate whether goals are being achieved. Show them the way. Leverage your goals to evaluate your provider’s performance on marketing performance, enterprise ability, ease and flexibility, quality of output, total cost of ownership and an innovative roadmap.

There’s no doubt that today’s marketers need better performance measures to know whether they are producing the best results for the organization. Data-driven attribution requires investment on the front end, but it pays big rewards that will have you asking why you didn’t take the plunge sooner.

We encourage you to dive deeper to help your organization understand the true benefits and implications of data driven attribution through our definitive guide.

AlfaStrakhovanie Doubles Transaction Rates With Google Analytics Enhanced Ecommerce

AlfaStrakhovanie LLC is one of Russia's largest insurance companies, and they need a way to measure all the complexities of an insurance business in an online platform. For example, insurance companies can only count profit after a policy expires, not when it is purchased, as accident claims and customer payouts need to be taken into account.

The company partnered with Agima, a Google Analytics Certified Partner, to find the best solution for a comprehensive online acquisition and retention measurement. Using Custom Dimensions and advanced Enhanced Ecommerce, AlfaStrakhovanie was able to have a better understanding of the purchase funnel per customer segment.

"AGIMA Interactive Agency is our long-term partner, helping us grow our profit on the internet and avoid the online pitfalls of the insurance business. It combines quantitative with qualitative research methods, so all our decisions now are data-driven and fully conscious. We have a strong confirmation of success and profit growth, and this allows us to continue our collaborative work." Tatyana Puchkova, VP of Marketing, AlfaStrakhovanie

As a result of the new implementation and ongoing data analysis, the company’s transaction rate has more than doubled, and revenue trends are on the rise while average order size remains untouched. The company now understands which customers and car owners it reaches with its marketing and (with the help of competitive price analysis) how the price affects transaction rates. The company is aware of the loss of potential profit because it knows the demand and can adjust prices accordingly.

To learn more read the full case study

Posted by Daniel Waisberg, Analytics Advocate 

New Book: Learning Google AdWords and Google Analytics

This is a guest post by Benjamin Mangold. Benjamin is the Co-Founder of Loves Data, a Google Analytics Certified Partner. When he’s not seeking out insights, you will find Benjamin blogging, presenting or playing with the Measurement Protocol. 

A new book showing you how to make the best use of both Google Analytics and Google AdWords is now available. The book is called Learning Google AdWords and Google Analytics by Benjamin Mangold of Loves Data (a Google Analytics Certified Partner). Here’s what Benjamin says about the book:

“Google Analytics is an incredibly powerful business tool and the focus of this new book is to show you how to unlock the hidden value of your reports. There are lots of techniques and tips covered, making the book a very practical resource to get more out of your website data and your online advertising campaigns.” 

Google Analytics Advocate, Justin Cutroni says, “Benjamin brings all of the information you need to get started and to grow and take advantage of these powerful tools. What I really like is how he brings all of the information about both systems – AdWords and Analytics – together in one central place. He makes it easy to understand how to measure your AdWords campaigns in Google Analytics data. This makes the information very actionable, which is exactly what you want.”

The book includes a foreword by Avinash Kaushik who says, “It balances the critical strategic elements that need to be present in any digital discussion (jump to Chapter 5), and the tactical elements that you’ll find useful every day (for example, Chapter 13 or, my favorite, Chapter 21).”

Google Analytics topics covered in the book include:

• Using the Multi-Channel Funnels reports
  
• Interpreting reports to improve your website and marketing
  
• Comprehensive overview of reports and interface features
  
• Introduction to Google Tag Manager

The way you can use Google AdWords and Google Analytics together is also covered, showing you how you can take your search and display campaigns to the next level. 

Google AdWords topics include:

• How to run successful Google AdWords campaigns
  
• Advanced campaign configuration opportunities
  
• Reporting on campaign performance and optimization
  
• Setting up and running display campaigns

You can grab a copy now on Amazon in paperback or for your Kindle.

Posted by Benjamin Mangold, Google Analytics Certified Partner

Why You Should Care About Attribution

Originally Posted on the Adometry M2R Blog

This is a guest post by Brian Sim, Product Marketing Manager, Marin Software

Attribution is sometimes perceived as being too complex, too technical, and too cold. However, when you look beyond the advanced-level math that goes into attribution algorithms and consider consumer behavior and buying tendencies, the importance of attributing revenue across the customer purchase path becomes very apparent.

Consider your own purchase behavior. How many steps did it take to get you from awareness to purchase the last time you bought something? Chances are it took you at least a few steps to get from “I think I could use a new lawn mower” to deciding “This 200cc self-driving, side-discharge robot lawnmower is the exact one that I need, and I’m going to purchase it this weekend because there’s going to be a seasonal sale.”

With that in mind, the next logical question is, based on your customer journey, do you think it makes sense for the last advertisement you saw prior to buying to get 100% of the credit for your purchase? If your answer is “no, that doesn’t make sense,” then you’ve uncovered the problem that attribution is trying to address.

On the analysis end, attribution modeling platforms like Adometry are tackling one of the most grounded-in-reality problems marketers face: “Is my multi-channel marketing budget being spent on the right channels?” On the execution end, revenue management platforms like Marin Software enable marketers to optimize their campaigns based on their advanced attribution data and answer the question, “How can I take that attribution data and improve my future ROI?”

Three Reasons Why You Should Care About  Attribution

Reason 1: It helps you understand your customer’s path from discovery to purchase.

As a recent Google study showed, consumer purchase paths are rarely straightforward; 60% of purchases take multiple steps, and depending on the industry you’re in, up to 84% of total revenue can come from purchases that required multiple steps across several days. Advanced attribution models can quantify each step of the customer purchase pathway. Armed with this knowledge, marketers can begin to associate ROI with specific marketing channels, understand the time lag for customer decision-making, and optimize spending across different marketing channels.

Reason 2: It allows you to understand and quantify performance across channels.

The multi-step customer purchase path may not be an issue if every step occurred within a single channel, but alas that’s not the case. The path from discovery to purchase typically involves multiple disparate marketing channels, each playing a slightly different role.

In order to optimize your marketing spend, you first need to understand the interplay amongst the various channels. Data-driven attribution allows marketers to assign proper credit to each touch point along the buyer journey. This allows the marketer to understand the proper valuation of channel and budget, and bid and tailor creative more effectively.

For example, within the travel vertical, Social acts as an assistive interaction and is many steps displaced from the actual purchase decision. In contrast, Display is almost as close to the customer’s purchase decision as the paid Search channel. In this case, direct marketers may optimize their campaigns to weigh the Display and Search channels more significantly. In many other cases, Display plays much more of an assistive role, and direct marketers may optimize their campaigns towards Search or another channel that is closer to the customer’s purchase decision.

Reason 3: It gives you the insight you need to make smarter decisions.

As the saying goes, “knowing is half the battle.” But knowing is only half the battle. The value of attribution is only realized once marketers can act upon their data. Adometry’s Programmatic Connector enables marketers to seamlessly incorporate attribution data into day-to-day decision-making workflows. Additionally, this is where an open stack execution partner like Marin Software helps complete the circle. Marin’s Revenue Connect is an open, flexible platform that enables advertisers to integrate data from any of their sources, including advanced attribution data, to improve campaign performance.

Activating your attribution data can help achieve real results. MoneySuperMarket, the UK’s leading price comparison site, partnered with Marin Software and Adometry to activate their attribution data in their marketing campaigns. By marrying their search intent and first-party audience data and then applying an algorithmic multi-click attribution model, MoneySuperMarket increased CTR by 12% and reduced CPC 7% across their motor insurance campaign, and increased profit margins 14% across all insurance campaigns.

Yes, attribution can be complex. But the value in unlocking that data can provide sustainable, competitive advantages across all of your marketing decisions.

Improving YouTube video thumbnails with deep neural nets

Posted by Weilong Yang and Min-hsuan Tsai, Video Content Analysis team and the YouTube Creator team

Video thumbnails are often the first things viewers see when they look for something interesting to watch. A strong, vibrant, and relevant thumbnail draws attention, giving viewers a quick preview of the content of the video, and helps them to find content more easily. Better thumbnails lead to more clicks and views for video creators.

Inspired by the recent remarkable advances of deep neural networks (DNNs) in computer vision, such as image and video classification, our team has recently launched an improved automatic YouTube "thumbnailer" in order to help creators showcase their video content. Here is how it works.

The Thumbnailer Pipeline

While a video is being uploaded to YouTube, we first sample frames from the video at one frame per second. Each sampled frame is evaluated by a quality model and assigned a single quality score. The frames with the highest scores are selected, enhanced and rendered as thumbnails with different sizes and aspect ratios. Among all the components, the quality model is the most critical and turned out to be the most challenging to develop. In the latest version of the thumbnailer algorithm, we used a DNN for the quality model. So, what is the quality model measuring, and how is the score calculated?

The main processing pipeline of the thumbnailer.

(Training) The Quality Model

Unlike the task of identifying if a video contains your favorite animal, judging the visual quality of a video frame can be very subjective - people often have very different opinions and preferences when selecting frames as video thumbnails. One of the main challenges we faced was how to collect a large set of well-annotated training examples to feed into our neural network. Fortunately, on YouTube, in addition to having algorithmically generated thumbnails, many YouTube videos also come with carefully designed custom thumbnails uploaded by creators. Those thumbnails are typically well framed, in-focus, and center on a specific subject (e.g. the main character in the video). We consider these custom thumbnails from popular videos as positive (high-quality) examples, and randomly selected video frames as negative (low-quality) examples. Some examples of the training images are shown below.

Example training images.

The visual quality model essentially solves a problem we call "binary classification": given a frame, is it of high quality or not? We trained a DNN on this set using a similar architecture to the Inception network in GoogLeNet that achieved the top performance in the ImageNet 2014 competition.

Results

Compared to the previous automatically generated thumbnails, the DNN-powered model is able to select frames with much better quality. In a human evaluation, the thumbnails produced by our new models are preferred to those from the previous thumbnailer in more than 65% of side-by-side ratings. Here are some examples of how the new quality model performs on YouTube videos:

Example frames with low and high quality score from the DNN quality model, from video “Grand Canyon Rock Squirrel”.

Thumbnails generated by old vs. new thumbnailer algorithm.

We recently launched this new thumbnailer across YouTube, which means creators can start to choose from higher quality thumbnails generated by our new thumbnailer. Next time you see an awesome YouTube thumbnail, don’t hesitate to give it a thumbs up. ;)

1stdibs Luxury Marketplace Hits New Heights With Google Analytics Premium

1stdibs.com is a global marketplace connecting art, design, jewelry dealers to potential buyers online. As such, the company has a complex digital ecosystem that requires an advanced analytical capability. In order to create a data-driven business, the company worked with Cardinal Path, a Google Analytics Certified Partner and Premium Authorized Reseller, in order to collect, process, store, and visualize all their data.

The companies started by discussing the most important key performance indicators (KPIs), which would be used to measure both failures and successes in their efforts. This understanding led to more customized and effective data collection, including features such as Custom Dimensions, User ID and Enhanced Ecommerce.

“You only need to look at the growth of our data and analytics team—which has quadrupled in the past year—to see what a critical role data now plays in our business. We just continue to unlock more and more value from our digital data assets.” - Adam Karp, CMO at 1stdibs

Using Google Analytics Premium to power their decisions, 1stdibs saw a 47% lift in transactions on paid media campaigns and a 10% gain in overall return on ad spend (ROAS). Plus, newly optimized email strategies led to a 34% increase in email click-through rates. 

To learn more, read the full case study.

Posted by Daniel Waisberg, Analytics Advocate