Optimizing intents with no K-Fold

So as I blogged about earlier, to help find conflicting training questions in your intents you would normally use K-Fold. There are issues in using this though.

  • It removes some of your training data that can weaken intents.
  • You have to balance number of folds for accuracy over speed.
  • Requires to make multiple workspaces.
  • Once you have created the results, you have to figure them out.

Looking back at an old blog post on compound questions, I realized that this already sees where a question can be confused with different intents.

So the next step was to determine to equate this to intent testing. Removing a question and retraining is not an option. It takes too long, and offers nothing over the existing K-Fold testing.

Sending the question back as-is will always return a 100% match. Every other intent gets a 0.0 score, so you can’t see confusion. But what if you changed the question?

First up, I took the workspace from the compound questions. Despite being a tiny workspace it works quite well. So I had to manufacture some confusion. I did this by taking a question from one intent and pasting it into another (from #ALLOW_VET to #DOG_HEALTH).

  • Can my vet examine the puppies?

Next up for the code we define a “safe word”. This is prepended to any question when talking to Watson Assistant. In this case I selected “SIO”. ¬†What was interesting when testing this is that even if the safe word doesn’t make sense, it can still impact the results on an enterprise level (ie. 1000’s of questions).

We end up with this response:

  • SIO Can my vet examine the puppies?
  • ALLOW_VET  = 0.952616596221924
    DOG_HEALTH = 0.9269126892089845

Great! We can clearly see that these intents are confused with each other. So using the K-Means code from before we can document actual confusion between intents.

I’ve created some Sample Code you can play with to recreate.

Just in case you are not aware, there is also a Watson Assistant premium feature which will do something similar. It’s called “conflict resolution” (CR). Here is a good video explaining the feature.

Compared to that and K-Fold…

PRO

  • This sample code is faster and cleaner than K-Fold.
  • Narrows down to an individual question.
  • (with customizations) Can show multiple intent confusion.
  • Unlike K-Fold, later tests only require you to test new & updated intents.

CON

  • Considerably slower than CR. When using CR, it only takes a few seconds to test everything. The sample can take up to a second per question to test.
  • Costs an API call for each question test.
  • Can’t test the model where the question would not be in the training. (What K-Fold does)
  • CR is slightly more accurate in detection. (based on my tests)
  • CR allows you to swap/edit questions within the same UI. The example I posted requires manual work.

 

To see the world in a grain of sand…

Just a short blog update to inform everyone that I have moved over to a new role in IBM. I now work in IBM Dubai as a Technical Solutions Manager, assisting with the somewhat recently announced ai Lab.

For me in a sense it means more (relative) free time, as a lot of my time was devoted to travelling previously. But I am just getting ramped up, so apologies again if I still maintain my one post a month. ūüôā

On the plus side, I will be playing with a more wider spectrum of Watson + AI related technologies, and will discuss here when I get the chance.

In the meantime here are some blogs I recommend.

I would also recommend to check out the Top 100 people to watch list.

 

 

Treat me like a human.

So one of the main pitfalls in creating a conversational system is assuming that you have to answer everything, no matter how bad it is. In a real life conversation we only go so far.

If you attempt to answer every question the user stops talking and treats the conversation system more like a search engine. So it’s good to force the user to at least give enough context.

One Word

Watson intents generally don’t work great with a single word. To that end create a node with the following condition.

input.text.matches(‘^\S+$’)

This will capture the one word responses. You can then say something like: “Can you explain in more detail what it is you want?“.

Of course you can have single word domain terms. They don’t give you enough context to answer the user, but enough to help the user. For example, lets say you are making a chat bot for a Vet office. So you may set up a node like this:

conv-0919-1

You can then list 2-3 questions the user can click and respond to. For example:

1. Why does my fish call me Bob?
2. How do fish sleep?

If you do need to create a list, try to keep it to four items or under.

Two to Three Words

Two to three words can be enough for Watson to understand. But it’s possible that the person is not asking a question. They could be trying to use it like a search engine, or a statement. You may also want to capture this.

To that end you can use the following condition.

input.text.matches(‘^\S+ \S+$|^\S+ \S+ \S+$’)
AND input.text.matches(‘^((?!\?).)*$’)

This will only capture 2-3 words that do not contain a question mark.

Here is the sample conversation script.

Handling Process Flows

While stepping a user through a process flow, don’t assume that the user will ask random questions. Even if they do, you don’t have to answer them. In real life, we wouldn’t try to answer everything if we are in the middle of something.

We may answer something in context, but we are more likely to get impatient, or ask to stop doing the flow and go back to Q&A.

So when creating your flow, try to keep this in mind. If the user asks something outside of what you expect, ask again only make the answers linkable (as long as there is a limit of answers). For example:

Watson: Do you want to a cat or a dog?
User: How do fishes sleep?
Watson: I don’t understand what you mean. Did you want a cat or a dog?

If the user persists, you can create a break out function. As you do a first pass through your user testing, you can see where you need to expand out. Don’t start off coding as if you expect them to break out everywhere.

 

Do you even need a chat bot?

With everyone rushing out to create bots, I am reminded of the Jurassic Park quote: “Your¬†scientists¬†were so preoccupied with whether or not they could, they didn’t stop to think if they should”.¬†

Despite what people will tell you, conversation or chat bots for that matter are not needed for everything.

For example where a simple web form will do, it will out perform against a chat bot. But if that same form could have most of it answered by one question, then a chat bot may be a better solution (or maybe you just need a better form).

Or simply changing your internal processes may negate the need for a chat bot. For example, imagine after analysing your support calls for training the bot you find that over 90% of calls are due to one printer. Do you create the bot, or just replace the printer?

Or if your customer knows your domain language then a search engine or Retrieve and Rank may be a better solution.

IBM normally does all this checking through what we call a Cognitive Value Assessment (CVA). A well done CVA reduces all head aches on projects. Even if you don’t go with IBM, you should realistically examine your business process to determine if you even need a chat bot, and not just jumping on the bandwagon.

Why Conversation?

Most of the chat bots out there are just messaging frameworks, that allow you as a developer to interact with existing messaging systems. How you interpret, talk, react is all handled through code.

Out of the box, where Watson Conversation excels is the ability to take a knowledge domain (ie. Customer) that doesn’t directly match your Domain knowledge.¬†With a handful of questions you can have your conversational bot answering questions it has never seen before.

Conversation also makes it relatively easy to write out your conversational flow (also known as chat-flow & process flow, depending on how long you worked in Watson).

Conversation is meant to piggy back on existing messaging frameworks to build intelligence into them with ease.

The danger of making things easy, is that people skip the theory and go straight to the development. The older people in the audience will remember when Visual Basic (or Lotus Notes for that matter) came out. It became easy to create applications, but most were travesties in UI, maintainability and functionality.

My focus going forward is to cover more the theory end to resolve this. As I am already competing with a number of blogs + videos in relation to Conversation.

Clean Slate

It’s been a couple of years since I last touched the blog. I thought I would start from scratch. I have everything backed up in case someone is looking for something (stats say otherwise).

I thought I would focus on conversational bots, as most of my work has been in this area. Mainly Watson related though.