Predictable Inference output

 


 

 Recently, there was a blog from thinking machines on Defeating Nondeterminism in LLM inference. First, let me say from a boots on the ground prospective. I understand this need. Inference engines are not 'standardized' like SQL. They take different inputs and have optional features. The major models families Qwen, LLama, etc have different Tokenizers. On different hardware an optimization might produce a different result. The inference engine may have a cache or be doing simultaneous requests which also can shape the result of a prompt.

I have seen a few reactions to the thinking machines article and many of them reference "temperature". I would not attribute temperature to be the largest factor in non-determinism, Temperature behind the scenes causes more "iteration", and if the "iteration" is random a higher temperature will make a wider range of results more possible. I wanted to discuss some other causes of non-determinism.

Model Strength


Often, I use a very small model like tiny-lama, for speed of development. Small models have not been trained on a large corpus of data and tend to make "false" "hallucination" answers. If you ask a 70B param model 'Answer only with the single letter: What is the first letter of the alphabet?' and ask the same to a ,3 b param model the tiny model might even get lost in the wording of the question.

Seed

Interestingly, openai has deprecated the seed parameter. The rough idea of the parameter is that Computer random numbers are pseudo random, If you start with the same "seed" and generate N numbers twice, you get the same set for both runs.

Quantization

As you may know the GPU is doing tons of floating point math, If you quantize, you are rounding, if you are rounding you will get less precise results. 

Floating point stuff

We all learn that you cant compare floats and they aren't stored exactly. With hardware optimization floats suffer more: 

a*b*c != a*c*b

Nothing in "the code" other then the floating math is not associative.

What to do?

I have my own inference engine deliverance and fhe code I forked from was using ThreadLocalRandom.  I was playing with a small model, for all the reasons I explained above the results were all over the place. 

Even though openai is deprecating "seed". I brought it back Here is the PR. With it results are repeatable, which surely helps testing :) Even if the answers are a "hallucination" it is the same predicable one based on the seed! 


Now, will Java Panama optimize the matrix operations the same way on another machine? Maybe not :)







Comments

Post a Comment

Popular posts from this blog

For the love of Java

 My old blog is lost, I think I have a mysql DB on a SSD disk on its last leg. Anyway the old Ed, the old blog, might go on a rant. These days I try to take the high road. I say try, I'm not all the way there. A few weeks back I found a linked in post that was discussing Java 23 added "var" and that it was "too little too late".  In case you didn't know the story, I was born to be a Java fanboy. When the applets made 3d shapes in my browser, I was in love. This was a long time ago, I had windows 3.11 and you needed a 32 bit OS to run Java, upgrades were required. My Packard bell PC only had 8MB RAM, upgrades were required.    Everyone is free to like and dislike what they want. I'm going to say my piece about "too little too late". Over the years Java improved drastically, both in terms of performance and features. Even features I didn't like or understand initially, I now use like second nature. Most impressive to me is the 'old wa...
Read more

Large Language Models Termperature and relative performance

 When you read about performance and tuning for large language models the term Temperature comes up often. I find a lot of interesting description of it from "It controls creativity", "How hard it thinks". What I like to say, "temperature at 0.0 is prone to over-fit".  I avoid "high temperature is  more likely to hallucinate," as I feel the 0.0 'over fit' looks a lot like a hallucination as well.   I rebuilt some of the deliverance code for sampling to give it an Object Oriented Design style face-lift, and prepare for adding repetition penalty.  As always feel free to look at the code here:  https://github.com/edwardcapriolo/deliverance/commit/0162c1daa07cba6b43d4e4f75cb95f50b0ff11b2 I was never overjoyed with was the size of the AbstractModel class . Many of the features are tightly coupled, I wanted to get the sampling into it's own class. It is not 100% clean as we still walk back into the AbstractModel to get config values and ot...
Read more

Guided Choice: Get only the answer you want with no fluff

 In my last blog we refactored the text sampling code in deliverance . I did that to prepare the code to take on something clever. You know LLM sometimes give you the answer inside a sea of text and never stop talking (like me). A solution to that is "structured outputs", which helps you control what the inference engine will produce. There are several forms of structured outputs. A simple one is guided choice . Effectively you give the inference engine a prompt and a list of choices. It should only answer with one of them.  Something like:  prompt= "What is the best month for vacation",  choices = ["January", "February"]  So lets code it up! As always here is the code   Guided Choice commit One thing to think about ahead of time. Call it a "trick". LLMs don't answer in words, they answer in tokens.  If you ask an LLM a question like "Who is the best NFL team". It might have "Giants" in the vocabulary, or it migh...
Read more