Exclude Top Choice takes VS repetition penalty

 A few days ago I hacked top_logprobs into deliverance:

 


Logprobs shows you what other tokens were close to being chosen and the logprobs (logarithm of the probability) of how close they were. 

When I had never done something like this before and the math turned out to be not so bad. Most of what I learned I got from this excellent article and this one. 

public static void logSumExpTensor(AbstractTensor result, AbstractTensor input) {
    float logsumexp = (float) logSumExp(input);
    for (int i = 0; i < input.size(); i++) {
        float v = input.get(0, i);
        result.set(v - logsumexp, 0, i);
    }
}


public static double logSumExp(AbstractTensor x){
    float sum = 0.0f;
    for (int i = 0; i < x.size(); i++) {
        sum += (float) FastMath.exp(x.get(0, i));
    }
    return (float) FastMath.log(sum);
}


All this math stuff had me a little bit more confident editing the Sampler code. So I decided to keep going. One thing I have found when using the smaller models is they tend to get into loops frequently. The class solution for looping is to use Frequency Penalty and Presence Penalty. 

I started doing a lot of reading about the two options and I found a lot of articles explaining their pitfalls. "If the word comes up often stop saying it", seems on the surface like a great idea. You can not go in a loop, go in a loop, go in a loop. If the word "loop" gets penalized the repetition is broken. However the problem is....well words like "is" comes up often and they get penalized. The comma "," token will get penalized, and what can happen is the model with starting using ":" in place of "," . That seems bad too. Now you need a "black list" of common words to not penalize.

Enter XTC: Exclude Top Choice

 Well someone came up with a solution I really like XTC. The idea is pretty cool, randomly when generating a token, pick a different one. Which token to pick? The lowest value that is over a threshold.

If the already generated string is "I am hungry, I would like to go to []". The existing sampler might believe "eat" has the highest score. What XTC does is generates a list of possibilities [ 'shopping : 50 ', 'to : 60', 'home: 90'], then if the threshold was 57 it would pick "to". 

 I implemented here.  It is a little bit to annotate, but Ill show you what it does. So here is your normal prompt:

@Test
public void logProbs() throws JsonProcessingException {
    AbstractModel m = Gemma2Suite.getOrCreate();
    String prompt = "Pick a random number between 1 and 9. Replay with only the pick.";
    PromptSupport.Builder g = m.promptSupport().get().builder()
            .addUserMessage(prompt);
    var uuid = UUID.randomUUID();

    Response response = m.generate(uuid, g.build(), new GeneratorParameters()
                    .withTemperature(0.0f)
                    .withMaxTokens(300)
                    .withLogProbs(true)
                    .withTopLogProbs(10)
            , new DoNothingGenerateEvent());
    assertEquals("I picked the number **5**! \uD83C\uDFB2 \n" +
            "<end_of_turn>".trim(), response.responseText.trim());

}

We can set probability of turning on the feature.

Response k = m.generate(uuid, g.build(), new GeneratorParameters()
                .withTemperature(0.0f)
                .withMaxTokens(300)
                .withLogProbs(true)
                .withTopLogProbs(10)
                .withXtcThreshold(0.1f)
                .withXtcProbability(0.5f)
        , new DoNothingGenerateEvent());

The effect is this:

[main] ERROR io.teknek.deliverance.CausualWhisperer - xtc:  picked maxi: '
[main] ERROR io.teknek.deliverance.CausualWhisperer - xtc: 4 maxi: 5
[main] ERROR io.teknek.deliverance.CausualWhisperer - xtc: **. maxi: ** 

assertEquals("I picked the number **4**. <end_of_turn>".trim(), k.responseText.trim());

 Now this should hopefully break the loops without putting penalties on words. 

 

Comments

Post a Comment

Popular posts from this blog

For the love of Java

 My old blog is lost, I think I have a mysql DB on a SSD disk on its last leg. Anyway the old Ed, the old blog, might go on a rant. These days I try to take the high road. I say try, I'm not all the way there. A few weeks back I found a linked in post that was discussing Java 23 added "var" and that it was "too little too late".  In case you didn't know the story, I was born to be a Java fanboy. When the applets made 3d shapes in my browser, I was in love. This was a long time ago, I had windows 3.11 and you needed a 32 bit OS to run Java, upgrades were required. My Packard bell PC only had 8MB RAM, upgrades were required.    Everyone is free to like and dislike what they want. I'm going to say my piece about "too little too late". Over the years Java improved drastically, both in terms of performance and features. Even features I didn't like or understand initially, I now use like second nature. Most impressive to me is the 'old wa...
Read more

Large Language Models Termperature and relative performance

 When you read about performance and tuning for large language models the term Temperature comes up often. I find a lot of interesting description of it from "It controls creativity", "How hard it thinks". What I like to say, "temperature at 0.0 is prone to over-fit".  I avoid "high temperature is  more likely to hallucinate," as I feel the 0.0 'over fit' looks a lot like a hallucination as well.   I rebuilt some of the deliverance code for sampling to give it an Object Oriented Design style face-lift, and prepare for adding repetition penalty.  As always feel free to look at the code here:  https://github.com/edwardcapriolo/deliverance/commit/0162c1daa07cba6b43d4e4f75cb95f50b0ff11b2 I was never overjoyed with was the size of the AbstractModel class . Many of the features are tightly coupled, I wanted to get the sampling into it's own class. It is not 100% clean as we still walk back into the AbstractModel to get config values and ot...
Read more

Guided Choice: Get only the answer you want with no fluff

 In my last blog we refactored the text sampling code in deliverance . I did that to prepare the code to take on something clever. You know LLM sometimes give you the answer inside a sea of text and never stop talking (like me). A solution to that is "structured outputs", which helps you control what the inference engine will produce. There are several forms of structured outputs. A simple one is guided choice . Effectively you give the inference engine a prompt and a list of choices. It should only answer with one of them.  Something like:  prompt= "What is the best month for vacation",  choices = ["January", "February"]  So lets code it up! As always here is the code   Guided Choice commit One thing to think about ahead of time. Call it a "trick". LLMs don't answer in words, they answer in tokens.  If you ask an LLM a question like "Who is the best NFL team". It might have "Giants" in the vocabulary, or it migh...
Read more