Let me start this article with a warning: Manufacturing questions causes more problems than it solves.
Sure the documentation, and many videos say the reverse. But they tend to give examples that have a narrow scope.
Take the car demo for example. It works because there is a common domain language that everyone who uses a car knows. For someone who has never seen a car before, won’t understand what a “window wiper” is, but they may say something like “I can’t see out the window, because it is raining”.
This is why when building your conversation system, it is important to get questions from people who actually use the system, but don’t know the content. They tend not to know how to ask a question to get the answer.
But there are times when it can’t be avoided. For example, you might be creating a system that has no end users yet. In this case, manufacturing questions can help bootstrap the system.
There are some things to be aware of.
Manual creation.
This is actually very hard, even for the experienced. Here are the things you need to be aware of.
Your education and culture will shape what you write.
You can’t avoid it. Even if you are aware of this, you will fall back into the same patterns as you progress through creating questions. It’s not easy to see until you have a large sample. Sorting can give you a quick glance, while a bag of words makes it more evident.
If you know the content, you will write what you know.
Again, having knowledge of the systems answers will have you writing domain language in the questions. You will use terms that define the system, rather than describing what it does.
If you don’t know the content, use user stories.
If you manage to get someone who could be a representative user, be careful in how you ask them to write questions. If they don’t fully understand what you ask, they will use terms as keywords, rather than their underlying meaning.
Let’s compare two user stories:
- “Ask questions about using the window wipers.”
- “It is raining outside while you are driving, and it is getting harder to see. How might you ask the car to help you?”
With the first example, you will find that people will use “window wipers”, “wipers”, “window” frequently. Most of the questions will be about switching it on/off.
With the second example, you may end up seeing questions like this.
- Switch on the windshield wipers.
- Activate the windscreen wipers.
- Is my rain sensor activated?
- Please clear the windows.
Your final application will shape the questions as well.
If you have your question creation team working on a desktop machine, they are going to create questions which won’t be the same as someone typing on mobile, or talking to a robot.
The internet can be your friend.
Looking for similar questions online in forums can help you in seeing terms that people may use. For example, all these mean the same thing: “NCT”, “MOT”, “Smog Test”, “RWC”, “WoF”, “COF”.
But those are meaningless to people if they are in different countries.
Automated Creation
A lot of what I have seen in automation tends not to fare much better. If it did, we wouldn’t need people to write questions. 🙂
One technique is to try and create a few questions from existing questions. Again I should stress, this is generally a bad idea, and this example doesn’t work well, but might give you something to build on.
Take this example.
- Can my child purchase a puppy?
- Are children allowed to buy dogs?
From a manual view we can see the intent is to allow minors to buy. Going over to the code.
For this I am using Spacy, which can check the similarity of each word against each other. For example.
import spacy
nlp = spacy.load('en')
dog = nlp(u'dog')
puppy = nlp(u'puppy')
print( dog.similarity(puppy))
Will output: 0.760806754875
The higher the number, the closer the words to each other. By setting a threshold on the value, you can reduce it to the important words. Setting a threshold of 0.7, we get:
Playing with larger questions you will find that certain parts of speech (POS) are more noise. So you can drop the following to remove possible noise.
- DET = determiner
- PART = particle
- CCONJ = conjunction
- ADP = adposition
Now that you have reduced it to the main terms, you can build a synonym off of these, like so:
dog = nlp('dog')
word = nlp.vocab[dog.text]
sym = sorted(word.vocab,
key=lambda w: word.similarity(w),
reverse=True
)
print('\n'.join([w.orth_ for w in sym[:10]]))
Which will print out the following:
- dog
- DOG
- Dog
- dogs
- DOGS
- Dogs
- puppy
- Puppy
- PUPPY
- pet
As you can see, a lot of repetition. You can remove duplicates. Also be wary to set the upper bound of the sym object when reading.
So after you generate a group of sample synonyms, you end up with something like this.
Now it’s a simple matter of just generating a random set of questions from this table. You end up with something like:
- need children purchasing dogs
- can kids buying puppy
- may child purchases dog
- will children purchase dogs
- will kids buy pets
- make children cheap puppies
- need children purchase puppies
As you can see, they are pretty bad. Not because of the word salad, but that you have a very narrow scope of what can be answered. But it can give you enough to have your intent trigger.
You can also mitigate this by creating a tensor of a number of similar questions, n-grams instead of single words, a custom domain dictionary and increasing your dictionary terms.
At the end of the day though, they are still going to be manufactured.
