What it does
gpt-prompt-engineer is a set of notebooks for finding a good prompt by experiment. You describe the task and supply test cases, and the system generates a number of candidate prompts with GPT-4, GPT-3.5-Turbo or Claude 3 Opus, runs each one against the test cases and ranks the results. Ranking uses an ELO system in which every prompt starts at 1200 and moves up or down as prompts compete on the test cases. A classification notebook checks outputs against expected true or false answers and prints a score table. The Claude 3 notebook can generate its own test cases and accept several input variables, and a conversion notebook uses Claude 3 Opus examples to guide Claude 3 Haiku, aiming for similar quality at lower latency and cost. To use it, open a notebook in Google Colab or a local Jupyter setup, add your OpenAI or Anthropic API key, and call the generate function with your description and test cases. The README warns that generating many prompts can get expensive and suggests starting with 10. Optional logging to Weights & Biases and Portkey is available, and the code is MIT licensed.
Capabilities
- Generates many candidate prompts from a task description and test cases
- Ranks prompts with an ELO rating after head-to-head tests
- Classification notebook with a score table per prompt
- Claude 3 notebook that auto-generates test cases and takes several input variables
- Opus-to-Haiku conversion notebook aimed at lower cost and latency
- Optional Weights & Biases and Portkey logging
When to use it
- Comparing prompt wording for a task against a set of test inputs
- Choosing between prompts for a classification task with known true or false answers
- Trying to carry a larger model's output quality over to a smaller, cheaper model
