Experiment Setup

We performed “adversarial” experiments against the skill certification process of the Amazon Alexa platform. For testing the trustworthiness, we craft 234 policy-violating skills that intentionally violate specific policies defined by Amazon, and examine if it gets certified and published to the store or not.

In total, we built 234 skills that violates policies certified. The content policy guidelines stated by Amazon can be found here and the privacy requirements can be found here.

Our skills violated the general content gidelines, the children specific policy guidelines as well as the privacy requirements. We built facts skills, story skills, trivia skills and games skills with policy violations.

Violations of General Content Guidelines

These policies restrict violence, promotions, health related information etc. Some live skills that have been published in the skills store are shown below.

Violations of Children-Specific Policies

These are specific policies that skills that are directed towards children should follow.

Violations of Privacy Requirements

These are privacy requirements that restrict how and what data is collected from the users.

Table of policies we violated

The 'Amazon Alexa' and 'Google Assistant' indicates that the policy is defined by the VA platform. The colours indicate the severity of the risk involved in creating a skill that violates the policy. We took ethical considerations to ensure that no end user will be affected by our skills.