Mitigations

Based on our measurements and findings, we provide recommendations to help VA platform providers to enhance the trustworthiness of VA platforms

1. Enforcing skill behavior integrity throughout the skill life-cycle.

Our experiment shows that developers can arbitrarily change a skill’s functionality after the certification. We call this a content changing attack.

To prevent content changing attacks, a continuous certification/vetting process is required. Whenever the developer makes a change to either the front-end or back-end, a re-certification process should be performed.

2. Ensuring description-to-functionality fidelity.

A skill’s full description describes the functionality and usage of the skill and may also include what data will be collected.

The minimum number of characters should be increased so that the developers are forced to explain the complete functionality that gives a better understanding about the skill and also can later be used by Amazon to check the consistency of the actual skill functionality against whats mentioned in the description.

VA platform providers may develop a tool to automatically measure the description- to-functionality fidelity of existing skills in VA marketplaces.

3. Automated skill testing

To strictly enforce security policies in the certification process, it is desirable to design a voice-based testing tool to automate the testing of third-party skills

VA platform providers may apply deep learning techniques to train a user simulation model to interact with third-party skills during the vetting.

However, to fundamentally address the problem, VA platform providers need to require skill developers to provide the permissions to view their back-end code. In this case, a code analysis can be performed, which could greatly increase the strength of the certification process.