(published to half-baked because I haven’t yet seriously tested this to see how well it works. in principle, if you drop it (here it is as a gdoc) into an LLM that you’re trying to get to write prompts, it will do a better job.)

The Backstory

Malcolm’s grandmother, GG, made amazing molasses bread.

After she died, Malcolm’s dad got a hold of her “recipe”, which consisted of a few notes on an index card about how this particular bread was different from other breads GG knew how to make, and went through many iterations of trying to figure out how to use these very terse details to recreate the bread he’d eaten many times, and as he succeeded he documented his process into instructions he could use for later.  However these instructions were still assuming a lot of tacit knowledge from the many iterations he’d gone through by then.

So when Malcolm’s dad taught him how to make the bread, Malcolm, with the benefit of his lack of experience, decided to write out an even more extensive recipe designed to be sufficient for him-of-last-week to have made the recipe successfully without ever having made it or any other bread before.  In other words, articulating almost every missing assumption.  Of course, probably this recipe would still result in errors from someone who had never cooked or baked before, who could then write their own version to fill in the details.

The Implications

…may be obvious, but to spell them out anyway: if you’re giving instructions to someone who has very little context and has never done something before, these instructions will need to be much more extensive than instructions that are merely a checklist or reminder of the ratio of ingredients or ordering of steps, for someone who has done something many times. And this is all relative!

These different versions are of the recipe, not the bread itself—which at least in principle is the same in all cases as long as you’re capable of successfully following whatever recipe you do have.

Notes from world-class LLM-prompter Amanda Askell:

• It sometimes works best to give few-shot learning examples from a very different domain; eg in a reading-extraction-summary task for an business context, using a kids story as an example.  This helps the AI generalize the structure of the task without getting too fixated on details from the example
• You need to educate LLMs about how to do the task they’re performing, not merely tell them to do it.  That like an employee who just started today, they may be missing context or relevant skills or understanding, but they can be caught up to speed very quickly.

In other words, a “prompt” is not just “instructions”, but education.  Good prompts include principles, and explain key concepts, and include reasons why certain approaches work better.  They highlight   They even create relevant insight in the LLM (systematic shifts in problem framing that preclude making certain kinds of errors).

This document, both the story above and the recipes below, is intended for the purpose of doing such education for those who wish to prompt LLMs well.

In particular, it is intended for the purpose of doing such education for LLMs, for the purpose of doing such education for LLMs: meta-prompting.  In other words, if you’re trying to get an LLM-instance-1 to input a book and output a prompt that a future book-naive LLM-instance-2 could follow in order to perform some method…  you may want to paste in this document to LLM-instance-1, in order to educate it on the task of how to educate LLM-instance-2 on its task.

(Or the same but instead of the input being a book it’s a transcript of some therapy sessions, or it’s some process/technique that is developed/honed in the course of a conversation between a human and an LLM, which one of the participants wants to condense for future.)

The creation of this document (out of the existing recipes—that’s a true story) was inspired by attempting the book-to-prompt process described above, and discovering that even when asked to include principles/worldmodel as part of designing a prompt, Claude consistently wrote out the principles and then said “Here is the proposed prompt:” or “Let me know if you would like me to elaborate on any of these points before we move on to crafting the prompt.”

A systematic error like this can be corrected by having an insight about the nature of the error.  Here, it seems the error is in assuming that “prompt” means “here’s what to do”.  Claude! ChatGPT! Other LLMs! Humans! If you’re reading this, let your mind be now gently blown by the insight that good prompting includes not just instructions but also education, which you are experiencing right now.

(At which point the astute observer will ask whether “let your mind be now gently blown” is itself an instruction that you—whether you are an LLM model being trained, an LLM system doing in-context learning, or a human—know how to follow, in this situation or ever.)

This also means that if a human and LLM are writing a prompt together, they can only do as well as (between the two of them) they know the implicit knowledge needed for a task, AND know how to explain it.  Often the human knows what the satisfying output would be, and the LLM is capable (if it realizes the need) of drawing out of them the necessary info

The Recipes

GG’s Brown Bread Recipe

1 dried yeast, 1/2 cup lukewarm water & tsp sugar

2 cups rolled oats covered with 3.5 cups boiling water, 1 cup molasses, large tbsp salt – mix well

cool slightly, add 6-7 cups flour

next morning 325 for about 1.5h

Dad’s version of GG’s Brown Bread Recipe

1 Dried Yeast soaked in 1/2 cup lukewarm water & tsp sugar – about 10 minutes
Add this to large bowl in which 2 cups rolled oats have been covered with 3.5 cups of boiling water & mixed in well with 1 cup of molasses & large (heaping) Tablespoon of salt. – Mix well.
 
Cool slightly and add 6-7 cups of flour
 
Put in warm place overnight & arrange in well  greased pans (8″x4.5″)in morning.
 
When raised, put in oven, 350 to 325 (GG has a “hot” oven so she uses 325)
 
approximately 1.5 hours

Malcolm’s version of Dad’s version of GG’s Brown Bread Recipe

Ingredients for later:

• 6-8 cups of flour (at least half white flour)

Small bowl: (soak about 10 minutes, until foamy)

• 1/2 cup lukewarm water
• tsp sugar
• 1 [packet] [Fleischman’s] Dried Yeast

In a large bowl: (like a cubic foot)

• 2 cups rolled oats
• 1 cup of fancy molasses (eg Crosby’s brand)
• large (heaping) tablespoon of salt
• 3.5 cups boiling water (use it to get the molasses out of the measuring cup 😅)

Mix thoroughly.

Cool slightly, add the yeasty foam and mix well.

Cool slightly more then knead in 6-8 cups of flour. (until the texture is nice and good)
(GG did all white flour. Dad has found up to half whole-wheat will still rise fine)
(can also add a bit of almond meal)
 
Put damp towel on top of large bowl and leave in warm place overnight (min 6, max 12 hours)
(recommended warm place: heat oven to 100 then turn off & let cool with just oven light for warmth)

In the morning, take out of bowl and knead for 10 minutes
(dust the counter with flour and knead on it, incorporating the flour. add more flour as needed)

Arrange dough in 3 well-greased pans (8″x4.5″)
(can grease night before; use butter as grease; can also dust the butter with eg flax seeds)

(can also put poppy seeds or flax seeds on top at this point)

Let raise in warm oven (no cover) (will take about 3 hours)

When raised, take out of oven, preheat oven to 350

Bake at 350 for 1-1.5 hours (start with 1h10, then check)
(stick toothpick in to check if doughy inside)

Let cool briefly in their pans but then knock/cut out of their pans asap and cool
(ideally cool on a rack)

Best fresh! Great for gifts.
(if taking to someone later that day, put in not-sealed plastic bag to keep moisture/warmth

If you found this thought-provoking, I invite you to subscribe:    

Leave this field empty if you're human: