tips / watson-conversation

Tools of the trade – UltraEdit

sodoherty2 Comments

So I use a number of applications to help in building and testing Watson. But the main ones I use are as follows.

  • Ultraedit
  • SPSS Modeller
  • Excel
  • Languages: Java and Python
  • iPad (no seriously :))

I want to touch on UltraEdit for this one. I primarily use it for the following.

  • Making mass fixes to data that SPSS can’t easily handle, and languages take too long.
  • Making the JSON readable
  • Extracting meaningful data from JSON to work with.
  • Putting that data back easily.

It is a pretty powerful text editor which allows you to do very complex search/replace/modifications to data. I am of the opinion if you prefer a different text editor, then go with what you know. But here are the main scripts I use for UltraEdit.

When you open the conversation JSON file it will look a little bit like this.

conv-0907-1

Just a mess. There is two scripts I picked up from UltraEdit that allow you to decompress it and compress it back to what you see above.

JSON – readable

if (!UltraEdit.columnMode) {
if (!UltraEdit.columnMode) {if (!UltraEdit.activeDocument.isSel()) {
UltraEdit.activeDocument.selectAll();
}
var jsonText = UltraEdit.activeDocument.selection;
var json = JSON.parse(jsonText);
var jsonTextFormatted = JSON.stringify(json, null, 2);
UltraEdit.activeDocument.write(jsonTextFormatted);
}

Will turn it into this.

conv-0907-2

To turn it back again.

JSON – compress

if (!UltraEdit.columnMode) {
if (!UltraEdit.activeDocument.isSel()) {
UltraEdit.activeDocument.selectAll();
}
var jsonText = UltraEdit.activeDocument.selection;
var json = JSON.parse(jsonText);
var jsonTextFormatted = JSON.stringify(json, null, null);
UltraEdit.activeDocument.write(jsonTextFormatted);
}

Now there are purists out there that can live with reading JSON. I’m sure all they can see is blond, brunette, redheads. For me I prefer to have my data with a lot less noise.

To that end I created two macros for extracting the entities and intents from the conversation file, into a format you can open in excel/SPSS.

Conversation – get intents

if (!UltraEdit.columnMode) {
if (!UltraEdit.activeDocument.isSel()) {
UltraEdit.activeDocument.selectAll();
}
var jsonText = UltraEdit.activeDocument.selection;
var json = JSON.parse(jsonText);UltraEdit.newFile();
var intents = json.intents;UltraEdit.activeDocument.write(‘”Question”,”Intent”\n’);
for (var i = 0; i < intents.length; i++) {      var intent = intents[i].intent
var examples = intents[i].examples      for (var j = 0; j < examples.length; j++) {
UltraEdit.activeDocument.write(‘”‘ + examples[j].text + ‘”,”‘ + intent + ‘”\n’);
}}}

Creates a CSV file you can work with.

conv-0907-3

Conversation – get entities

if (!UltraEdit.columnMode) {
if (!UltraEdit.activeDocument.isSel()) {
UltraEdit.activeDocument.selectAll();
}var jsonText = UltraEdit.activeDocument.selection;
var json = JSON.parse(jsonText);UltraEdit.newFile();var entities = json.entities;
UltraEdit.activeDocument.write(‘Entity\tValue\tSynonyms\n’)
for (var i = 0; i < entities.length; i++) {var entity = entities[i].entity
var values = entities[i].valuesfor (var j = 0; j < values.length; j++) {
UltraEdit.activeDocument.write(entity + ‘\t’ + values[j].value + ‘\t’);var synonyms = values[j].synonyms
var syn = ‘ ‘
for (var k = 0; k < synonyms.length; k++) {
syn += ‘”‘ + synonyms[k] + ‘”,’
}
UltraEdit.activeDocument.write(syn + ‘\n’)
}}}

This one creates a Tabbed separated file (TSV) with the entities in a nice readable format.

conv-0907-4

I also have scripts to put the intents and entities back into the conversation script, but I don’t plan to release at this time. As it is so easy to make a mess, and I won’t be supporting these.

You can pick up the scripts here.

2 thoughts on “Tools of the trade – UltraEdit”

  1. Thanks for sharing. I’ve been wondering what tools power users use when working with Watson Conversation… the GUI is nice, but I want to upgrade to something more “industrial”. I use vim in my day-to-today, and was wondering, do you primarily use the GUI tool, or code directly in JSON? I find other chatbot scripting langauges like rivescript/chatscript are easy to write in a text editor (they don’t use JSON), but the Watson learning curve is a bit steeper… (and more to type b/c of the JSON format)

    Reply

Leave a Reply to sodohertyCancel reply