So I will be switching between, newbie to expert in no particular order. This post assumes you already know how to use conversation.
In earlier versions of Watson, it was designed to handle high, medium and low confidence answers in different ways. With conversation this all works a little bit differently.
For this example, I am using the NLC demo intents. The first thing you have to do is find your low confidence.
Unlike earlier versions of WEA, the confidence is relative to the number of intents you have. So the quickest way to find the lowest confidence is to send a really ambiguous word.
These are the results I get for determining temperature or conditions.
treehouse = conditions / 0.5940327076534431
goldfish = conditions / 0.5940327076534431
music = conditions / 0.5940327076534431
See a pattern? 🙂 So the low confidence level I will set at 0.6. Next is to determine the higher confidence range. You can do this by mixing intents within the same question text. It may take a few goes to get a reasonable result.
These are results from trying this (C = Conditions, T = Temperature).
hot rain = T/0.7710267712183176, C/0.22897322878168241
windy desert = C/0.8597747113239446, T/0.14022528867605547
ice wind = C/0.5940327076534431, T/0.405967292346557
I purposely left out high confidence ones. In this I am going to go with 0.8 as the high confidence level.
Once you have those, you can create your conversation. Your first node is to check that later nodes won’t fail.
The next nodes check to see if a low confidence or medium confidence is hit. For low, it won’t respond. For medium, it will display text, then continue from the next condition to find the actual answer.
When you test it, you get the following results.
You may still need to test with users to tweak the upper value.
Using this fall through method makes it easier to maintain with your intents. You can use a nested option if only certain intents need to be hit.
Here is the Sample file.
Gotchas: When writing numbers in conditions, always start it with a numerical value like “0.8”, not “.8”. Otherwise you will get a Dialog node error: EL1049E:(pos 24): Unexpected data after ‘.’: ‘8’
One thought on “Handling low confidence answers in Conversation.”
Just an update. There is undocumented feature that if your top intent is less then 20% confident, it will jump straight to the “Anything Else” node. This can’t be modified at this time. However, if your top confidence is 20% it is probably best to train it to a higher level.