How to Unit Test an AI Prompt

Using Jest and Typescript

I was on a Twitter/X Space talking about that I really didn’t have a mechanism for testing AI Prompts. However, after the Twitter Space really got me thinking and I developed this relatively bonafide way of testing AI prompts.

Note this method is used within eval systems. It’s just extended to unit testing of prompt itself. So it’s a natural extension.

Jest Unit Testing Graphic

Sample AI Prompt Function

Let’s imagine that you are using the following AI prompt to create a bagel recipe.

const response = await openai.chat.completions.create({
    model: 'gpt-3.5-turbo',
    messages: [
      {"role": "system", "content": `You are a bagel recipe creator. It should include type of bread, if cream cheese and or if protein`},
      {role: "user", "content": "Create for me an everything bagel"},
    ],
    response_format: { type: "json_object" },
    temperature: 0,
    top_p: 1,
    frequency_penalty: 0.0,
    presence_penalty: 0.0
});

Well, the above is text only, so you might think there really isn’t a way to test this. However, there actually is. Let’s do the following.

Let’s Create Typescript Type

What we are going to do is create a Typescript Type. So, our new code will look something like this:

export interface Bagel {
  type: string;
  creamCheese: boolean;
  protein: string;
}

const response = await openai.chat.completions.create({
    model: 'gpt-3.5-turbo',
    messages: [
      {"role": "system", "content": `You are a bagel recipe creator. It should include type of bread, if creamCheese which will be a boolean and or if protein`},
      {role: "user", "content": "Create for me an everything bagel"},
    ],
    response_format: { type: "json_object" },
    temperature: 0,
    top_p: 1,
    frequency_penalty: 0.0,
    presence_penalty: 0.0
});

Lets Use The Typescript Type In Our Unit Test

Now tying it together we can use the typescript type in our Jest unit test.

// we import openai to spyOn import * as openAi from 'openai';
import {Bagel} from './my-function';

cont myBagel: Bagel = {
  type: 'plain',
  creamCheese: true,
  protein: 'none'
};

// myBagel is object, string is: 
// `You are a bagel recipe creator. It should include type of bread, if creamCheese which will be a boolean and or if protein`
function areObjectKeysInString(obj, str) {
  const keys = Object.keys(obj);
  return keys.every(key => str.includes(key));
}

// confirm all js keys are located in the prompt string
expect(areObjectKeysInString(openAi.openai.chat.completions.create.messages[0].content, myBagel)

What we are doing here is confirming that all keys in the js object are located in the prompt. This way, we have some sort of logical way of testing that our AI prompts are working as expected.

Popped into my head and thought I would share.