Virtual assistants and bots are everywhere now, performing a multitude of tasks to help users. A bot (short for “robot”) is a program that operates as an agent for a user or another program, or simulates a human activity. A chatbot takes this concept further. This is an artificial intelligence (AI) software that can simulate a conversation with a user through messaging applications, websites, mobile apps, or even in telephone conversation.
From Siri and Cortana fetching you the weather forecast, to Nissan’s Micra chatbot helping you choose the colour of your next car, chatbots and the intelligence that underpins them have taken great leaps in recent years. Chatbots have been around in some form since the 60s, but it is only in recent years that they have begun to hit their stride, and evolve into a truly effective tool.
This is no mean feat – the sheer complexity of natural language processing, and the difficulty of constructing artificial learning and logical reasoning out of algorithms was until recently thought of as science fiction. And yet here we are. Virtual assistants like Alexa allow you to use natural language to get tasks done – many people appreciate the feeling of being able to talk to their devices, rather than spend time typing in what they want. The chatbot market is growing all the time, as developers find new ways to use AI to carry out jobs that would traditionally require high labour investment.
The growth in the capabilities of chatbots comes with new sets of challenges, not least for testers. Indeed, according to Jason Arbon, writer of the book “How Google Tests Software”, testing the software is even more difficult than writing it.
Bots need to be tested for:
- Conversational flow
- Understanding/Intent
- Error management
- Compatibility
- User-friendliness
- Speed
- Dialogue accuracy
- Navigation of hyperlinks
- Collecting and interpreting data correctly and securely
- Avoiding cyclic loop
And many different types of testing are typically deployed to achieve this:
- Usability
- Developer testing
- User testing
- Speech
- Behaviour
- Functional
- Precision
- Security
- Load
- Localisation and multi-lingual
- NLP capabilities
There are three main techniques for bot testing:
- Pyramid Approach, comprising three broad steps:
General Test: The funnel for testing should follow an inverse pyramid shape. Question and answer testing then follows that structure. Chatbots are expected to keep the conversation following, because if they fail at any stage then the user is likely to leave the conversation.
Domain Specific Test: Testing for the specific product or service group. The language and expressions related to the product will be the main driving force for the test.
Limit Test: Testing the limits of our chatbot. Will our bot be able to handle the irrelevant parts of the user’s dialogue?
- Chatbottest’s Standardised Test:
This test mechanism broadly follows three categories: Expected scenarios, Possible scenarios, and Almost Impossible Scenarios.
- Human Testing of Chatbots
Developers can leverage this type of testing to build human intelligence directly into their applications, and it can be used to reach for a higher confidence interval. Examples include Amazon’s Mechanical Turk, which a form of crowdsourced outsourcing network.
Other chatbot testing tools on the market include Testyourbot, Botium, Qbox.ai, as well as AI bots testing with AI driven automation tools.
Testing must be rigorous, as bots must be able to adapt to different users, different environments and different languages. The potential rewards are great though:
- Chatbots can work 24/7, is quicker and more efficient, and much less prone to error
- Chatbot applications streamline interactions between people and services
- They enhance customer experience
- They improve the customer’s engagement process and operational efficiency by reducing the typical cost of customer service, both in money and time
- They make it easier for firms to reach customers’ expectations
- They can solve the problems customers traditionally associate with the online experience
- They provide standardisation, making it possible to maintain the same service quality, no matter what the circumstances are
- The data they collect can be analysed to inform better decision making and system efficiency
If you’re thinking about implementing a chatbot for your organisation, there are three key components to consider before making a decision.
- Decide on the value you are expected to gain by implementing a chatbot
- Will the bot be voice-based or text-based? The medium is important, and depends on the context in which the chatbot will operate.
- What data is available to train the chatbot? It only takes one bad experience to put customers off engaging with chatbots, and this is why bots not only have to be tested, but tested in the right way to ensure that they can handle anything thrown at them.
You can learn more about how NTT DATA can help you with the development, testing and implementation of chatbots by contacting us.