All demos LLM Router One front door for every app — local GPU when it's free, cloud when it isn't Demo · simulated traffic

The piece with no user interface

Every app asks one service. It decides where the work runs.

Nine applications in this house need a language model. None of them hold an API key, and none of them know whether their request is about to run on a GPU twenty feet away or in a data centre. They all ask the same front door. It checks a flag, checks whether the GPU box is awake and whether its one slot is free, and routes accordingly — falling back to the cloud in under a second rather than making anything wait.

The only visible surface is a tray icon. Right-click it, and you decide whether the machine is yours for the next hour or the automations'. Everything below is simulated — flip things and watch the traffic re-route.

The tray icon right-click it

the desktop, 20 feet from the server

07:41
Tue

LLM Router — GPU

send — the GPU takes the work whenever it's awake and free.

Conditions
GET /status

      

    


    
    

      

        

          
            
              
                
              
            

            
            

            
            

            
            
            
            LLM Router
            one front door · port 8742
            
            send flag + online → GPU
            queue 0 · slot 0/1

            
            
            
            local
            fallback

            
            
            GPU · next room
            32 GiB · one 32B model resident
            idle · 0/1 slot
            0 served

            
            
            Cloud model
            metered · always available
            standby
            0 served

            
            
          
          

            ran locally
            went to the cloud
            refused (embeddings have no cloud equivalent)
            click a rule below to force that path
          

        

      


      

        
GET /history every decision, with its reason

        

          
timecallermodelroutemswhy

          

        

      

    

  


  

    
0%
Stayed on the local GPU

    
0%
Fell back to the cloud

    
—
Median local latency

    
—
Median cloud latency

    
0
Requests in this session

  







  
The whole decision

  
Six rules, evaluated in order.

  

    Click any rule to force a request down that path and watch it happen.
  

  


  

    Why a queue was the wrong answer. The first version of this held requests in a queue when the GPU
    was busy and returned 202 Accepted. That broke every caller — automation tools
    block waiting for a response, and a 202 with no body isn't one. It now degrades instead of queues: if the
    single GPU slot isn't free within fifteen seconds, the request goes to the cloud and comes back in about a
    second. Slower than local, infinitely faster than never.
    


    Why one slot. A 32-billion-parameter model at full context leaves about 5 GiB of headroom on a
    32 GiB card. A second concurrent request needs its own ~8 GiB of cache, so it doesn't fit — it spills to
    system memory and everything crawls. Measuring that was what turned "why is the machine so slow" into a
    hard concurrency limit of one.
    


    What's simulated. The routing rules, the tray states, the single-slot limit and the failure modes
    are the real ones. The requests, timings and caller names on this page are generated in your browser.