Using LoRA with Granite 3.3

Abstract geometric design featuring a hexagonal shape with layered colors in green, blue, and light blue.

IBM just announced Granite 3.3 with lots of new features!

I am going to focus on the LoRA adapter feature for this post and how to use it.

LoRA stands for “Low rank adaptation” which is a way to fine tune LLMs fast and cheap, while mitigating “Model collapse” that can happen with normal transfer learning. It does this by locking down the model except for a few parts that you can tweak.

Because of this you can actually train a model on your laptop and get decent results in a relatively short amount of time.

For testing this I used the granite3.3-8b-instruct model. My laptop spec is M4 Max Pro 128GB. If your Mac Mx laptop has less than 40GB of memory you would be better to use the granite3.3-2b-instruct model instead. For PC you will need CUDA.

I used the following steps.

Notebook 1:

  1. For my dataset file I took the watsonx Assistant manual. [888 Pages]
  2. Using docling I converted this to markdown.
  3. The markdown I converted into tokens and then chunked those tokens by size 512 with an overlap of 50. This was then saved to a JSON list file.

Total Time: 17 minutes.

Notebook 2:

Using that chunked list I trained a LoRA adapter and saved it.

Total Time: 1 hour 42 minutes

Notebook 3:

At this point it was ready to go. I created a third notebook to test the results using the base model and the model with the LoRA adapter.


You can download the full project from github if you want to reproduce.

The results below are as-is from the LLM (No RAG)

Results

Question

What is the “Trigger Words” feature in watsonx Assistant? [ref]

LoRA Answer

✅ The “Trigger Words” feature in watsonx Assistant is designed to detect and prevent user inputs from triggering sensitive actions or responses that might be harmful or inappropriate.

Base Model Answer

❌ The “Trigger Words” feature in watsonx Assistant allows users to specify certain words or phrases that, when present in a user’s input, will activate a predefined response or action. This enables the assistant to perform specific tasks or provide particular information in response to cues from the user’s input.

How do I stop my action being used as a clarifying question? [ref]

✅ You can stop an action from being used as a clarifying question in the following ways:

  1. If you want to stop all actions from being used as clarifying questions, go to Home -> Actions -> Settings -> Clarifying questions and switch the toggle to Off.
  1. If you want to stop a specific action from being used as a clarifying question, open the action and go to Action settings > Clarifying question and switch the toggle to Off.

❌ To prevent your action from being interpreted as a clarifying question, be clear and direct in your communication. State your intentions explicitly and avoid ambiguous language. If you’re providing information, present it as a statement rather than a query. For instance, instead of saying “Isn’t it true that…?”, say “The fact is that…”.

What are the response types available to use? [ref]

✅ The response types available to use are text, image, video, audio, iframe, and options. (note: correct in relation to UI)

❌ The response types available to use are:

  • Plain text
  • Numbered or bulleted lists
  • Yes/No responses
  • Short phrases or words

Note!

The results are somewhat cherry picked. The model can still fail to give a correct answer without more context. But in conjunction with RAG, your answers are less likely to hallucinate as it understands your domain language/topic.

👻 Spirit Mansion 👻

So for the halloween I thought I would try making a text adventure using wastonx Assistant. Overall it was easy enough and you can download my attempt to see it working.

I wanted to make this just as much a learning exercise as well. So here is my approach.

Plot

The game involved the player trapped in a mansion and has to escape. To do so they need to get certain items and use them in certain ways.

At the same time I would have a ghost that randomly moves through the house and the player has to avoid the ghost or it would be game over.

Creating the map

I started with sketching out a basic map of the mansion as shown below.

The arrows denote up and down. Other lines were north, south, east and west. Location 0 was my inventory and location -1 was if something was removed from the game.

I created an action called “Where am I?” and created a step for each location. The location numbers lined up nicely.

For each step I also set the values of session variables that map to the direction the player go from that location. If the direction was 0 then they could not go that way.

Movement

I created an action “Go” which checks to see what direction to go in by checking each session variables value against the direction. Once a move takes place I update a current_location session variable and call “Where am I?” as a subaction to detail the new location.

Writing up the locations

Rather than write up for each location, instead I used Granite3 with Ollama to create the descriptions. As the model was just recently released I wanted to see how well it could do.

Overall the descriptions were ok except it had a thing for the air getting colder everytime you entered a room. I ran the descriptions a few times and chopped the bits I liked.

Creating the ghost

At first I planned to have the ghost randomly move around the house with it’s own ghost_location session variable. This would be updated every time an action took place (eg. get, drop, move, look, etc)

What I noticed almost straight away is that you ended up dying without even realising it. So I had to code in a way to give a player the chance.

With this I set up this flow.

  • If the ghost is in the same location as player, then game over.
  • Move the ghost.
  • If the ghost is in the same location as player then warn them.
  • If the ghost location matches any of the location directions for where the player is, then warn them of that direction.

This for the most part worked, but there was still incidents of the player getting jumped on without warning. Or forced the player to move to another location and back, sometimes that isn’t an option.

So I added a “hide”, “unhide” command to protect against the ghost if it was in the room and you didn’t want to move.

Hiding disabled the ghost but I forgot to disable doing actions. So you could hide and move around the mansion. So hiding was changed to lock you in the room and prevent taking actions.

Testing this was painful as I desperately roamed the house looking for the ghost. In hindsight locking the ghost to two rooms would have sped up debugging.

Coding game over

With the ghost causing game over I had to create a Game over and Game start action.

The reason for the game start is you cannot call to the Greeting built in action.

Without it the only way to restart was to restart the conversation through the web client UI.

Dealing with items

For these I created an array of objects in the following format.

{
"type": "synonyms",
"value": "Car keys",
"synonyms": ["car keys", "keychain", "keys"],
"location": 0,
"description": "Your car keys. Not much use until you get your car fixed."
}

The type of “synonyms”, value and synonyms fields allow you to construct the object as an entity.

The reason for an array instead of a dictionary is that there are two commands that make life easy.

That is filter() and joinToArray().

Because of how assistant handles variants, just using a normal dictionary can sometimes have side effects. Using those commands also negates the need for loops or complex formatting.

With this array the following commands need to work against it.

  • inventory
  • list items in area
  • get an item
  • drop an item
  • examine
  • extra commands (eg. read/light)

The get and drop used the technique of:

  • Do they mention the item beforehand?
  • Is the item in their pocket or location?

After that I swap out the location by the following:

  • Filter the item in question to its own object.
  • Filter out the item from the overall items.
  • Update the object and add it to the list again.

Inventory and listing items in the room again was just a filter based on location and using joinToArray() to format the output.

Win condition

To win the game certain items have to be in certain locations and other items used in the right sequence. It was fairly straightforward. Only checking the conditions as needed.

Play testing

My son volunteered to test it. It was at this point you realise your peception of the game of the player isn’t yours. For example if you have “You sense an evil presence to the north” you don’t expect them to do it more than once. :/ I added in some clues and hints into the game to make it possible.

Speed running it took around 3 minutes to complete. It should take around 10 minutes.

Basic enough game but a good diversion to see how assistant can handle certain use cases. Hope you enjoy!

Working with hex in watsonx assistant

Converting to/from hex in watsonx Assistant is easy if you use an extension. If you don’t have that choice then you need to get creative in how to do it. In the video I am going to cover the following.

  • Converting Text/Integers to and from hex code.
  • Show how to create temporary variables within your action.
  • Sub-action notation to help your content creator fill out correctly.

Here is the sample assistant used in the video.

Converting flowcharts to Actions

I’ve had to present this so often that I finally bit the bullet and created a generic how-to video.

In this video I show how you can quickly create an action from a flowchart. With the following points.

  • Showing a simple easy to follow flowchart structure.
  • The process of chunking which allows you to easily see the steps.
  • When to use conditions versus fall through to steps.
  • How to present the correct information across multiple steps without having to use session variables.

As always, I have supplied the sample files to the video. The flowchart software is Draw.io.

Enjoy!

Building an Action handler

Revisiting the earlier post on non-blocking options. One of the issues that comes up is that it can be messy to have multiple actions for a single function. The other issue is that user inputs can accidentally trigger such survey events. So you want to prevent that.

So the first part is simple. You just move them into one single action. You can even have the sub-action be the handler for the responses.

Within the json the label is what is shown to the end user. The text piece is what gets sent silently to Assistant. Swapping out the thumbs for UUIDs allows you to create training data that can never be triggered by the end user. To make it easier I put a word tag on the end so I remember which is which like so:

Now in the training data of your sub-action you add the UUIDs you created.

Lastly you want to have the steps to look for these in your first steps. Those steps should have a condition statement using the expression editor like so:

Once captured, ensure you finish on that step. Lastly you want to switch off “Ask clarifying question” and “Change conversation topic”.

You now have a poor mans callback. 🙂 I’ve attached the sample skill for you to review.

Time for an update

I keep meaning to get back to this blog but various things continue to block. So it’s 4am and I’ve nothing to do, so time to put in some updates.

Blockers

The main reasons to why I am so quiet.

  • I am much closer to the magic of how watsonx Assistant is made. So it’s harder to find stuff to blog about that I know will end up as a feature later (case in point, my earlier blog post).
  • My current work carries more potential commercial value, adding another layer of complexity to what I can publicly share.
  • The latest version of watsonx Assistant is designed for non-technical people to do much cooler stuff easier than before. Trying to find anything useful rarely warrants a whole blog post. So I’ve been posting to Stackoverflow as needed.

Highlights

Earlier in the year I received the IBM Tech 2023 award. It’s awarded to the top talent in IBM. I got a chance to go back to Dubai and meet many exceptional people in IBM across the world. That event really puts in perspective of how many moving parts there are in IBM and how there are numerous people excelling in those parts.

More recently I was humbled to receive the Culture Catalyst Award for my work on watsonx.ai and watsonx Assistant. Helping others to build solutions that matter. The award is given to culture leaders and role models who exemplify our purpose, values, and growth behaviours, working to activate our culture and drive growth.

I’ve also helped build a worldwide program focused on helping existing customers fix gaps in their Assistant and migrate easier to the latest version. Helping to oversee numerous experts in different Geos execute this for customers has been very rewarding.

On the personal side I got to visit Japan for my son. A fantastic country, but I was more impressed with how fluently and relaxed my son was there. His self study of Japanese and culture paid off for his first visit.

What now?

I’ll try to add updates as I can. If there is something in particular you want let me know (or just post on Stackoverflow or TechExchange. I can also be found on Bluesky.

Debugging your extension

When working with extensions in Watson Assistant just using the standard UI can be cumbersome to do deep dive analysis on why your extension is not working as expected.

You can use the browsers inspector to look at what is sent and received. You do this by going to the network tab, select “Response” then filter by “callout”. Once you get to the line where callout is mentioned remove the filter and then you can see all the parts.

For the video demo below I created a sample extension that pulls jokes from “I Can Haz Dad Joke” via their API. The sample extension is attached.

Non-Blocking option buttons

This is a common request that comes up often with the new Watson Assistant.

This is where you offer options at the end of an action which can link off to another action. Using normal options forces the user to respond to the buttons and prevents them leaving that action.

Another use case is asking the user if they were happy with the response. In this example I’m going to show how to do a thumbs up/down as a follow up.

First we need to create a Thumbs up and Thumbs down intent. Both are exactly the same. It is one training example, the emoticon 👍 or 👎. One step that says if they had a positive or negative response.

Next click the options on the action and make sure to switch off “Ask clarifying questions”.

Same for thumbs up/down and survey later.

Once you have created those two actions. Next is the survey action. This should contain no training questions and the settings above switched off. This is to prevent it triggering except called from another action.

The step you are going to select the </> button to show the json and replace with the following:

{
  "generic": [
    {
      "title": "How was my response?",
      "options": [
        {
          "label": "👍",
          "value": {
            "input": {
              "text": "👍"
            }
          }
        },
        {
          "label": "👎",
          "value": {
            "input": {
              "text": "👎"
            }
          }
        }
      ],
      "response_type": "option"
    }
  ]
}

This creates custom option buttons which are non-blocking.

Lastly in your action you want the response to trigger the survey. You can do this by calling the action as a sub-action.

Now when you run it, you end up with something like below.

As always, I’ve included the sample skill for you to try.

Important:
In a real world production instance it is not good practise to have a thumbs/up down after every response. As this reduces the users confidence in the system.

Imagine having a customer support person asking for validation after every answer given. You wouldn’t trust their response either.

The other part of it is that often end users will make their thumbs up/down from an emotional response, and not a logical one.

If you really need to do this, then I recommend to put a confidence level into survey. So it doesn’t trigger unless the confidence level is very low. A better option is to use the custom survey demonstrated in the Lendyr demo.