The burning questions on every buyer's mind (Full report) Machine translation
Today, machines translate more in a day than all human translators on the planet combined translate in a year. As machine translation (MT) becomes more widespread, its uses expand too. You could argue MT has replaced human translators in some scenarios. On the other hand, it also created new ones. The old questions of cost, speed of translation or quality have perhaps taken a backseat in favor of a new one – how to best leverage available MT technology.
For a lot of companies, the question how to use MT is, however, not a straightforward one, especially as there is a multitude of solutions available. Navigating the options becomes a challenge unto itself – and you haven’t even started testing the different solutions in your work stream!
Nimdzi has put together a guide which answers some of the burning questions on everyone’s minds – whether they are localization buyers eager to get a head start or MT specialists looking for the latest developments in the world of MT.
Information contained in this report
- MT use cases
- Selecting MT
- Best practices
- The development of multi-engine middleware platforms
- Development of predictive quality metrics
- Improved support for terminology in neural machine translation
- MT brands developing their own software connectors
- MT post-editing: primarily used to drive cost down and achieve a faster turnaround for large quantities of content
- Using raw MT output: for translation of e-commerce product information, reviews, forums, support articles, etc.
- MT for the use of machines: MT output is being leveraged by other automated tools and platforms in various fields such as multilingual search, machine interpreting, eDiscovery in litigation, etc.
MT has been all the rage for a while now, and the pace at which it is developing in 2019 is not slowing down. As more information is passed through engines, algorithms are continuously being updated, resulting in improved MT output.
Today, the entire spectrum of the language industry is developing or using MT technology. Tech giants such as Google and Microsoft remain the largest drivers of MT development and have the most popular tools. Small companies specialize in dedicated MT solutions. Language service providers (LSPs) offer a framework to support their clients with MT, however, only a handful of them create their own in-house MT solutions (i.e. UK-based firm Capita Translation and Interpreting or Chinese company GTCom). Some buyers create and train their own MT engines from toolkits or MT servers purchased from their suppliers.
MT solutions also vary due to the vast combination of potential language pairs – some MT technology may perform better in a selected language pair and for a certain content type as opposed to another. That is why a number of firms combine multiple MT engines in one workflow, leading to the need for middleware platforms capable of supporting this setup.
Predictive quality metrics is another area a number of firms are investigating. For example, Memsource has a solution for comparing the MT output against the database of human translation to predict whether it is a good or bad match. This, however, requires a large set of data, and not all firms have access to it. Some are trying to develop predictive quality scoring based on confidence scores in neural toolkits. We expect this technology to become more widespread in a few months.
Developing support for terminology in neural machine translation (NMT) is another current trend. This support already exists in Microsoft Custom Translator, IBM, Google AutoML, and Amazon and is coming up in a handful of other NMT solutions too.
The final trend we identified is MT companies are adding connectors, which integrate directly into the user software. This pits MT brands against companies developing Translation Memory Systems (TMS), which have been embedding human translation into buyer systems. The underlying idea is for the buyers to access the MT from their applications and not rely on translation memory (TM). MT would then be capable of identifying the context and speaker in order to choose the best output for that specific situation.
The underlying idea is for the buyers to access the MT from their applications and not rely on translation memory (TM).
MT would then be capable of identifying the context and speaker in order to choose the best output for that specific situation.
The actual use of MT is changing too. MT was originally designed to help people understand the content in a foreign language (or at least to offer the gist). However, thanks to improvements in quality, MT is now primarily seen as a tool to improve productivity. Nonetheless, efficiency is not the only use case for MT.
MT use cases
Potential buyers seeking to integrate MT into their technology stack may not know all the potential uses for MT. This is why at Nimdzi we have classified MT use cases, aiming to provide insight into their real-life applications. These can naturally vary from company to company and from product to product. We expect new use cases will develop as new product types and services continue to emerge. So far, we have identified the following three groups of use cases.
When buyers turn to MT, they are usually looking to cut costs associated with translation and to improve the speed of the translation process. Large LSPs have integrated MTPE into their workflow, outsourcing post-editing tasks to their resources instead of the usual translation. MTPE tasks are being paid at lower rates than traditional translation – this is where perceived cost reduction lies.
However, cost savings for MTPE tasks are generally less than 10 percent in professional scenarios with a very good translation memory in place. The majority of cost savings comes from the TM – deploying a good TM makes a difference. The better the TM, the fewer cost savings associated with deploying MT. In situations where a TM may not be available – either because new strings are being released or new products are being added – trainable MT engines are the way to go. In such cases you can potentially save up to 25-30 percent.
MTPE is especially interesting for buyers because it can result in substantial productivity gains. These vary from language to language, but generally speaking, a translator can process more words per hour post-editing than translating from scratch.
This is the second use case group for MT. Use for raw MT is specific to large chunks of content which would be too expensive to translate or even post-edit. By extension, the requisite quality benchmarks on the client side may be lower too, although this does not mean raw MT content cannot be further post-edited if certain criteria are met.
Let us list a few concrete examples of raw MT in use across products:
E-commerce platforms: sites such as eBay use raw MT output to translate seller product listings and user reviews.
Forums & user reviews: for a lot of companies, user-generated content is what drives spending and growth. Companies such as TripAdvisor and Booking.com user raw MT output to translate large amounts of text. In cases like these, the goal is to get information across to users. Meaning, rather than quality, is the driving force. Because of the amount of content generated by users, raw MT is a viable option.
Support pages & documentation: raw MT is frequently used for support and help articles. For instance, Microsoft uses raw MT for its support pages. Additionally, Microsoft monitors the performance of support pages, tracking traffic and relevance to determine which pages will benefit from post-editing in order to improve their quality. Also, their localized support pages contain links to the original post, often written in English.
Communication apps & platforms: apps and software used for communication now integrate built-in MT solutions. Such is the case of the Chinese messaging app WeChat.
MT for search and information retrieval
MT content has historically been destined for human use. However, we are seeing more and more uses cases where MT output powers other tools and software which is then being used for information search.
Let’s take a look at a few examples where machines annotate MT output, enabling quick information retrieval:
- Multilingual search: allows to look for information in multilingual e-commerce product catalogs, as well as support pages and forums.
Machine interpreting: used in conjunction with speech-to-text and speech synthesis, MT allows for fully-automated interpreting used in wearable devices.
Entity recognition: MT can translate texts from multiple languages into one, allowing for subsequent keyword identification. This is in use in various fields such as identifying patents, drug and legislation changes across the globe, and data-driven journalism.
eDiscovery: in multilingual litigation or corporate investigation scenarios, MT helps giving users access to terabytes of electronic data in different languages, such as emails, documentation, video assets, voicemail and many more.
Today, there are more than 50 different brands of MT technology – the question is, how to choose the right one for your localization process? We are here to help you make your decision.
To help you navigate the waters of MT technology, we have classified the available MT solutions based on three the number of different CAT tools they integrate with. Perhaps unsurprisingly, Google and Microsoft’s MT tools top this chart. However, generally speaking, MT and TMS are fairly easy to integrate – this is something buyer-side or LSP-side engineers should be able to accomplish.
MT brands classified by number of CAT integrations 17 Microsoft 12 DeepL 8 Systran 8 Omniscien MT 6 Globalese 5 Kantan MT 5 tauyou 5 Moses 4 SDL BeGlobal 4 Capita SmartMATE 3 Yandex 3
Want to take a closer look? Grab all tables in Google Sheets!
We have mapped out a list of requirements you may want to consider when selecting MT. Some of these are hard requirements, others are considerations which may be less important at the start but could be useful, depending on the tool you are evaluating.
|Hard requirements||Good to have||Legal requirements|
|1. Language combination support||8. Terminology & glossary support (i.e. IBM, Amazon, MS glossaries)||17. How does the provider collect and store my data? What do they do with it?|
|2. Security:||9. Formatting fidelity||18. Can I upload my data and my customer data to their servers and comply with GDPR?|
|- Deployment model||10. Connectors to user systems (Outlook, Skype, CMS, PIM)||19. Are they a potential competitor to my company in any areas? Can the MT provider be acquired by my competitor? If yes, what are the implications?|
|- Server locations||11. User portal||20. What happens to my business if the MT provider's system goes down?|
|- No trace availability (yes, no, opt-in)||12. Management dashboard with reporting on quality|
|- SSL encryption strength||13. Support 24/7|
|3. Total cost of ownership||14. Can I train via API (re-upload content right back in)?|
|4. Volume requirements||15. Can I train multiple engines and select the best inside?|
|5. Robust API/Infrastructure||16. What does it do for source text optimization?|
|6. File format support|
|7. Anonymization/redactions for sensitive data|
Want to take a closer look? Grab all tables in Google Sheets!
MT selection process
Step one is selecting your relevant languages and content type.
Step two will be organizing and preparing your translation data. You ought to make sure you have a full accounting of all translation assets – TM, term bases, monolingual documents, etc. – before you start. Organize them by work stream, where possible, making sure the diversity of your text content will be reflected. Our recommendation would be to use between 1000-2000 high quality segments with correct terminology.
After you’ve tested multiple engines, it is important to evaluate the output. This is usually done in two steps:
- The first evaluation round makes use of automated testing metrics, such as the BLEU, TER and LEPOR scores or Memsource MTQE.
- A second evaluation step is performed by humans – usually on the LSP side. Some of the best practices to apply during this step is having multiple evaluators or doing blind tests.
In testing, evaluating, and piloting an MT tool, you can start to gather the necessary data to track post-launch metrics. Edit distance, the number of keystrokes per set number of words or symbols, or the time spent editing the output are good indicators of the productivity of the selected engine. Another potentially insightful tool you can use is polling customer satisfaction – either via surveys online or by monitoring your website analytics such as traffic or time spent per page.
Something to keep in mind, however, is that problems may not necessarily appear during the piloting phase. It is, therefore, advisable for buyers to take one or two months to see how the solution is performing before committing to it long-term.
A good place to start for businesses would be taking a good look at the use cases and where they can use MT as part of their strategy of going global. Over the past few years, we have noticed a shift from MTPE scenarios to situations where raw MT is being used by buyers – part of this shift can be attributed to new use cases being uncovered. It is understandable too, in a world where business growth is driven by generating content for and by users. At some point, there is too much content to translate (or post-edit, for that matter).
Let’s take a look at a few additional things to keep in mind when considering MT:
- Pilot MT in a low-profile work stream. Take your time pitting multiple providers against each other and evaluate the different solutions they offer.
- If possible, verify whether your provider has not been using MT already as part of the existing workflows – you think you will be saving up cost, but your provider may not have any more cost to save.
- Evaluate expected cost and benefits – building relationships with MT specialist and solution architects may give you the insights you’ve been missing.
- Try different MT solutions for different types of content. Marketing content may have different requirements than product or support content.
- Use one language or a representative set of languages. Go for the languages which matter to you. Sprinkle in one or two languages notoriously difficult for MT. Those will give you the most information on the performance of the MT you are piloting.
- Level up your in-house competence using MT, evaluating it and integrating it into your workflow.
- Stay flexible. Chances are a new, and perhaps better MT engine for you will appear. What’s more, your product may undergo changes too. Be prepared to chart a different course with your MT solution.
That being said, deploying MT may not entirely be smooth sailing. Here is what to expect in case things go haywire:
- Integration issues
- Suitability of the MT solution for the work stream
- MT quality issues
Future-proof your choice
Buyers often already have in-house knowledge which MT to choose – they just haven’t realized it yet. At Nimdzi we believe no one has a better knowledge of their business than the buyers themselves. We would therefore advise in-house MT specialists to build on what they already know of their product. Some of our recommendations then would be:
- Map out the use cases for MT.
- Segment your content types and languages. MT may not be applicable for all of them. Consider using different MT engines for different products or platforms.
- Build the key competences of your internal team to understand what MT suppliers are telling you: evaluate quality and security, track metrics, and measure ROI.
Finally, don’t shy away from reaching out to MT specialists. We’d be glad to help.
Stay up to date as Nimdzi publishes new insights. We will keep you posted as each new report is published so that you are sure not to miss anything.