Month: October 2021

Visualising Coverage in Conversation Logs.

One of the most important parts of a conversational system is to ensure that your end users are getting the most benefit out of it. To do this requires looking at patterns in your conversation logs. It can be time consuming.

A common approach is to put markers into your nodes then look for those entry/exit point markers. But a user question can hit multiple nodes + slots across multiple log lines. Making it tricker to see. Here is a couple of approaches to try and easily get information on your complex flows.

For this demo I am using the default demo skill in Watson Assistant to generate logs. I have created a number of simple conversations. A couple demonstrate a issue with how the user may interact. I have also supplied the example notebook and files for you to try out.

Creating the graph.

For generating I first need to convert the log to a graph format. The easiest way to is look at the nodes_visited column in the logs. Here is an example of a user making a reservation. 

['Opening']
['Reservation using slots', 'handler_104_1498132501942', 'slot_102_1498132501942', 'handler_103_1498132501942', 'handler_6_1509695999145', 'handler_104_1498132501942', 'slot_102_1498132501942', 'handler_103_1498132501942', 'handler_107_1498132552870', 'slot_105_1498132552870']
['slot_105_1498132552870', 'handler_106_1498132552870', 'handler_10_1509132875735', 'slot_8_1509132875735', 'handler_9_1509132875735', 'handler_17_1509135162089', 'handler_104_1498132501942', 'slot_102_1498132501942']
['slot_102_1498132501942', 'handler_103_1498132501942', 'handler_107_1498132552870', 'slot_105_1498132552870', 'handler_106_1498132552870', 'handler_10_1509132875735', 'slot_8_1509132875735']
['slot_8_1509132875735', 'handler_9_1509132875735', 'handler_14_1509133469904', 'handler_24_1522444583114', 'slot_22_1522444583114', 'handler_23_1522444583114', 'handler_22_1522598191131', 'node_3_1519173961259', 'Reservation using slots']

Although each line is an interaction you can see that it is in fact a chain of events. When joining the chains you end up with. 

['Opening'] ['Reservation using slots', 'handler_104_1498132501942', 'slot_102_1498132501942', 'handler_103_1498132501942', 'handler_6_1509695999145', 'handler_104_1498132501942', 'slot_102_1498132501942', 'handler_103_1498132501942', 'handler_107_1498132552870', 'slot_105_1498132552870', 'handler_106_1498132552870', 'handler_10_1509132875735', 'slot_8_1509132875735', 'handler_9_1509132875735', 'handler_17_1509135162089', 'handler_104_1498132501942', 'slot_102_1498132501942', 'handler_103_1498132501942', 'handler_107_1498132552870', 'slot_105_1498132552870', 'handler_106_1498132552870', 'handler_10_1509132875735', 'slot_8_1509132875735', 'handler_9_1509132875735', 'handler_14_1509133469904', 'handler_24_1522444583114', 'slot_22_1522444583114', 'handler_23_1522444583114', 'handler_22_1522598191131', 'node_3_1519173961259', 'Reservation using slots']

The second part is the whole interaction the user had in trying to book an appointment. It’s still not that readable. So I converted these over to make a little more readable.

  • slot_ = Take the variable that the slot object depends on.
  • node_ = Take the condition for the node in the skill.
  • frame = Top level slot node (not shown above, it’s part of the skill node attributes). Took the condition of the node.
  • response = This is the node that responds to the end user, or part of the slot. Added “response to: <parent node name>”
  • handler = Left the same.

Once this is done I started by converting the chain to Graph nodes and edges. For each time an edge is repeated a count is incremented to the edge object. You end up with this.

Red nodes are entry points to a single flow. Orange is a flow which could have been entered though other parts of the conversation. Blue are the slot values. Pink is a final response to the user from the flow.

As you can see it’s still a mess!

By selecting the entry point node you can delete all other nodes that do not have a path to it. In this case I selected “frame: #Customer_Care_Appointments”. This was generated.

Still a bit of a mess and not easy to see how the paths are flowing through the booking appointment. NetworkX was designed more for analysing graphs than visualising them.

Graph to Sankey

So using the generated graph data I moved it over to a Sankey. The nice thing with plotly is you can easily move the flows to see what is going on. Here is what is generated using the graph information from the last image.

Edge colors are red where there is more output from a node than there is input. In a normal conversational flow it should be fairly static if well trained. Not all red is an issue though. Taking the two biggest we can use these to drill down to a root cause.

#1

This is showing a lot of users are not progressing through the phone section of the flow and are going into a loop. As the second part is much smaller it would suggest that people are giving up on the flow. Looking through the logs shows the following pattern.

Clearly the end users are having problems trying to enter in a valid phone number. So this is something that should be looked at in resolving.

#2

You can see three inputs into the handler before it passes over to the “Ask for date” slot. This isn’t an issue as there are three conditions this could happen.

  • User supplies a date when asking for the appointment.
  • System asks the user for the date.
  • User asks to redo the appointment at final confirmation.

The handler is doing what it should be doing.

Conclusion

So this example is showing just one way to approach this problem. I’d be interested to hear how others are dealing with this.