Who is listening when you talk to your voice assistant?

Your voice is worth a fortune. No, you’re not about to win a recording contract. But we are about to see a replay of the way our clicks on websites – and the clicks of billions of others – became the heart of the data economy.

The mantra of the world that turned Microsoft, Facebook, Google and Amazon into the world’s biggest companies is that “data is the new oil”.

But now, clicks and typed searches are fast being replaced by voice commands and spoken searches. Google Assistant and Apple’s Siri are suddenly available to half the world’s population, via the Android and iOS phones that dominate the smartphone world.

At first, it seemed benign, and a natural way to make services and applications available to a vast proportion of the global population that cannot read, write or type. But now, barely a week goes by without a revelation of a major player in the internet industry not only allowing users’ voice activity to be intercepted, but actively using such interception as a means of enhancing services.

Picture: iStock

Only when they are caught out, do they tend to admit their misdoings, and offer assurances that it won’t happen again.

However, the incentive to find new ways to exploit your voice are massive. Right now, the best voice assistants process spoken instructions or questions via the Cloud, where their vast resources and ultra-sophisticated artificial intelligence engines fine tune the delivery and provide the user with a seemingly accurate service.

However, when the user is offline, the voice assistance either does not work, or becomes a pale shadow of the connected version. The reality is that the machine learning and artificial intelligence built into phones is still rudimentary, and the disconnected phone is still a blunt instrument in the war for your voice.

This is one of the reasons access to your voice is so important to the big players: the more voice data points they have, the more errors they can identify, and the more feedback they can get and give, the better their voice assistants become, and the better they compete in the war for attention. That is only one reason why voice is the new oil.

The other core factor is the extent to which spoken words in general conversation are being used to “personalise” your services. Which is, of course, a euphemism for targeted ads.

Picture: iStock

So when you are discussing a specific product, service or intention, and then suddenly start seeing ads for that very thing crop up in banners on sites you visit, it’s not paranormal. Rather, it’s the new normal – for now.

For example, see how Google gives itself permission to record your voice and “audio input” to improve speech recognition: http://bit.ly/GoogleVoiceRec.

Think that’s creepy? Amazon advises that “the voice recordings associated with your account are used to improve the accuracy of the results”. It even lets you listen to those very recordings. The link is hidden deep in the Amazon website, and unlikely to be discovered in random browsing. See it here: http://bit.ly/AmazonRecording.

This helps explain why we often imagine that our homes or other environments are being bugged by the giant corporations, poised to send us adverts the moment we mention a product or travel destination. It is because they are indeed bugging us, but with our permission, granted when we signed up to use their free services.

As the cliché goes, if the service is free, you are the product. Except that now, your voice is the product.

Picture: iStock

Who is listening to you?

Voice assistants are not only using automated systems to listen in on us. Humans are helping them, writes Gadget’s Bryan Turner.

Four of the biggest tech companies are using human contractors to listen in on samples of voice requests. This means requests between you and your voice assistant may have another set of ears listening in.

One of the most popular voice assistants, Google Assistant, is included with most Android devices, devices running the Chrome browser, and iOS devices running the Google app. Google also has a dedicated smart speaker for voice assistance, as well as a range of smart TVs with similar functionality.

Its strength comes from off-device processing, meaning that voice requests are sent to Google’s cloud to be transcribed, understood, and sent back for assisting a user. This means that almost any smartphone with an internet connection can run the assistant as smoothly as the latest Android flagships.

Google has had a somewhat-hushed scandal with human voice contractors in Belgium, where Time Verheyden, a journalist with Belgian broadcaster VRT, was given access to over 1 000 voice files containing requests and some random snippets of audio. Of these requests, 150 were snippets from smartphones that thought they heard the “OK Google” wake word. These included a child and a grandfather speaking and a woman in distress.

Picture: iStock

The voice assistant included with most Apple devices, Siri, is also not as innocent as Apple claims it is. Most new iPhones, iPads, and Apple Watches are consistently listening, and some “false positives” – when the device mistakes sounds for the wake word – are sent back to contractors.

Apple says this is done to analyse why the device thought it was requested and what can be done to prevent it in the next version update. Contractors told The Guardian that Siri can be summoned by the sound of a zip being zipped up, and that they have heard potential drug deals, as well as the sound of sexual acts.

Apple has since suspended this project with its contractors, presumably to continue at a later date, and has planned an opt-out feature for users to have their recordings sent to Apple for improving the assistant.

Like Google Assistant and Apple’s Siri, Amazon Alexa sends all voice requests to Amazon. Users forge that when they sign up for most services, they tick a box accepting terms of use – which often hides a lengthy agreement that gives a wide range of permissions.

It is these terms that may well result in an Amazon employee listening. Bloomberg reported that thousands of full-time employees and contractors work for Amazon, in several countries and languages, to filter requests that were either non-intentional or didn’t help the user. Workers are expected to listen to as many as 1 000 audio requests in shifts that last approximately nine hours.

Picture: iStock

Some contractors told Bloomberg that, while most requests were mundane, some were “possibly criminal”, including potential sexual assault.

Amazon told CNN Business that an “extremely small number of interactions from a random set of customers” were sent to these listening centres. While this quote may be reassuring at face value, 6.3 million Amazon Echo devices were sold worldwide in Q3 of 2018.

This means that an “extremely small” amount of questions taken from an extremely large request base is less non-trivial than it sounds.

Microsoft’s innovations are vast, but come at the cost of privacy. Its voice assistant included with Windows 10 and Xbox, Cortana, is most likely on and listening if users chose the “Express Setup” option when setting up their devices for the first time.

This means that Microsoft may have far more false positives than the other voice assistants, because users either don’t know it’s there, or have forgotten about it since setup.

Picture: iStock

Like Google, Amazon and Apple, contractors were given access to random voice requests, some of which were false positives, to improve its voice assistant.

Skype conversations are also sent back to contractors to assess the effectiveness of translations. Some contractors heard very intimate conversations, and some clips even included bank account information, home addresses, and bedroom conversations.

The events mentioned above all happened in 2019, which shows that artificial intelligence, especially natural language processing, is not ready to do all the work for big tech companies.

This is why a little more human training is required to teach these systems about what to listen out for and what can be ignored. However, it comes at a cost to privacy.

Arthur Goldstuck is founder of World Wide Worx and editor-in-chief of Gadget.co.za. Follow him on Twitter and Instagram on @art2gee

For more news your way, download The Citizen’s app for iOS and Android.