So you want to build a NoSql Huh? Virtual Threads

 A long time ago (10 years) in a galaxy far far away, I was 'internet famous'. Every once in a while they would quote me on highscalability.com. I was on my way. 

 Recently I was reading some science news, apparently they are looking for another planet on orbit outside Pluto. This reminds me of ancient aliens, where I first heard of the Nibiru Cataclysm. Omg Ed! Are you on drugs? What are you talking about?....Drum role.....Yes! I have my own NoSQL database Nibiru!

 Nibiru a distributed NoSQL database inspired by the Column Family data store. Besides all the cool code, I even have a power point. My blog is back, and it can't be a one-and-done. What better way to get on track then rekindle the "So you want to build a NoSQL? Huh?" Blog. 

I went over the code for Nibiru and I decided to convert some bits to virtual threads. Virtual threads were introduced in Java 21. The idea behind virtual threads is they are lighter than "platform threads" as they are not handcuffed to a system thread. When they are blocked by IO they can yield.

Nibiru is a distributed database. On the write path the "coordinator" is a piece of the code that decides where data should go and sends it there. On the read path it looks in all the places for data and merges the results.

 

Lets look at the PR:

//look here 
try (ExecutorService service = Executors.newVirtualThreadPerTaskExecutor()) {

  ExecutorCompletionService<Response> completionService = new ExecutorCompletionService<>(service);
  List<RemoteMessageCallable> remote = new ArrayList<>();
  List<Future<Response>> remoteFutures = new ArrayList<>();
  for (final Destination destination : destinations) {
    if (destination.equals(destinationLocal)) {
      completionService.submit(new LocalActionCallable(action));
    } else {
      RemoteMessageCallable r = new RemoteMessageCallable(clientForDestination(destination), message, destination);
      remote.add(r);
      remoteFutures.add(completionService.submit(r));
    }
  }
  long start = System.currentTimeMillis();
  long deadline = start + timeoutInMs;
  if (c.getLevel() == ConsistencyLevel.ALL) {
    Response response = handleAll(start, deadline, completionService, destinations, merger, message);
    if (hinter != null) {
      maybeSendHints(remote, remoteFutures, start, deadline, hinter);
    }
    return response;
  } else if (c.getLevel() == ConsistencyLevel.N) {
    Response response = handleN(start, deadline, completionService, destinations, merger, message, c);
    if (hinter != null) {
      maybeSendHints(remote, remoteFutures, start, deadline, hinter);
    }
    return response;
  } else {
    return new Response().withProperty("exception", "unsupported consistency level");
  }

 

The code didn't change much, but it is important to note virtual threads are used differently than platform threads. First, you no longer have to create an executor service and close it down.

public void case(){
   executor = Executors.newFixedThreadPool(1024);
   Future<Object> fut1 = executor.submit(...);
   Future<Object> fut2 = executor.submit(...);
   executor.shutdown()
}

With Virtual Threads the executors are lightweight

try (ExecutorService service = Executors.newVirtualThreadPerTaskExecutor()) {
  Future<Object> fut1 = service.submit(...);
}

One of the nice things about executor services is they can be bounded sizes. There is no limit on the lightweight threads one can create but eventually if you make to many you will run out of memory. A nice trick to limit the number of tasks in a similar way to a bounded queue is to use a semaphore:

Semaphore sem = new Semaphore(10);
...
Result foo() {
    sem.acquire();
    try {
        return callLimitedService();
    } finally {
        sem.release();
    }
}

Since I took bounded queues of the coordinator, I likely will add the semaphore type thing back. 

But anyway there you have it, the return of Nibiru! 

 




Comments

Post a Comment

Popular posts from this blog

For the love of Java

 My old blog is lost, I think I have a mysql DB on a SSD disk on its last leg. Anyway the old Ed, the old blog, might go on a rant. These days I try to take the high road. I say try, I'm not all the way there. A few weeks back I found a linked in post that was discussing Java 23 added "var" and that it was "too little too late".  In case you didn't know the story, I was born to be a Java fanboy. When the applets made 3d shapes in my browser, I was in love. This was a long time ago, I had windows 3.11 and you needed a 32 bit OS to run Java, upgrades were required. My Packard bell PC only had 8MB RAM, upgrades were required.    Everyone is free to like and dislike what they want. I'm going to say my piece about "too little too late". Over the years Java improved drastically, both in terms of performance and features. Even features I didn't like or understand initially, I now use like second nature. Most impressive to me is the 'old wa...
Read more

Large Language Models Termperature and relative performance

 When you read about performance and tuning for large language models the term Temperature comes up often. I find a lot of interesting description of it from "It controls creativity", "How hard it thinks". What I like to say, "temperature at 0.0 is prone to over-fit".  I avoid "high temperature is  more likely to hallucinate," as I feel the 0.0 'over fit' looks a lot like a hallucination as well.   I rebuilt some of the deliverance code for sampling to give it an Object Oriented Design style face-lift, and prepare for adding repetition penalty.  As always feel free to look at the code here:  https://github.com/edwardcapriolo/deliverance/commit/0162c1daa07cba6b43d4e4f75cb95f50b0ff11b2 I was never overjoyed with was the size of the AbstractModel class . Many of the features are tightly coupled, I wanted to get the sampling into it's own class. It is not 100% clean as we still walk back into the AbstractModel to get config values and ot...
Read more

Guided Choice: Get only the answer you want with no fluff

 In my last blog we refactored the text sampling code in deliverance . I did that to prepare the code to take on something clever. You know LLM sometimes give you the answer inside a sea of text and never stop talking (like me). A solution to that is "structured outputs", which helps you control what the inference engine will produce. There are several forms of structured outputs. A simple one is guided choice . Effectively you give the inference engine a prompt and a list of choices. It should only answer with one of them.  Something like:  prompt= "What is the best month for vacation",  choices = ["January", "February"]  So lets code it up! As always here is the code   Guided Choice commit One thing to think about ahead of time. Call it a "trick". LLMs don't answer in words, they answer in tokens.  If you ask an LLM a question like "Who is the best NFL team". It might have "Giants" in the vocabulary, or it migh...
Read more