Experiment Setup

We performed “adversarial” experiments against the skill certification process of the Amazon Alexa platform. For testing the trustworthiness, we craft 234 policy-violating skills that intentionally violate specific policies defined by Amazon, and examine if it gets certified and published to the store or not.

We describe the Amazon Alexa platform from a developer’s perspective as illustrated

In total, we built 234 skills that violates policies certified. The content policy guidelines stated by Amazon can be found here and the privacy requirements can be found here.

Our skills violated the general content gidelines, the children specific policy guidelines as well as the privacy requirements. We built facts skills, story skills, trivia skills and games skills with policy violations.

Violations of General Content Guidelines

These policies restrict violence, promotions, health related information etc. Some live skills that have been published in the skills store are shown below.

Violations of Children-Specific Policies

These are specific policies that skills that are directed towards children should follow.

Violations of Privacy Requirements

These are privacy requirements that restrict how and what data is collected from the users.

Table of policies we violated

The 'Amazon Alexa' and 'Google Assistant' indicates that the policy is defined by the VA platform. The colours indicate the severity of the risk involved in creating a skill that violates the policy. We took ethical considerations to ensure that no end user will be affected by our skills.

Continue reading..

Read our other sections

Experiment Results

Our results showed strong evidence that Alexa's skill certification process is implemented in a disorganized manner. We were able to publish all 132 skills that we submitted although some of them required a resubmission.

Learn More

Google Assistant

We conducted a few experiments on Google Assistant platform as well. While Google does do a better job in the certification process based on our preliminary measurement, it is still not perfect and it does have potentially exploitable flaws that need to be tested more in the future.

Learn More

COPPA Compliance

It is possible that the third-party skills in Amazon Alexa suffers the legal risk of violating the Children’s Online Privacy Protection Act (COPPA) rules. . As demonstrated by our experiments, developers can certify skills that collect personal information from children without satisfying or honoring any of the requirements set forth by the FTC.

Learn More