# Building a Private GLM-5.2 Inference Service on Eight B200s

For a few weeks, I had access to a rented server with eight NVIDIA B200 GPUs. The server had enough memory to load GLM-5.2, but loading the model was never the goal. I wanted three or four colleagues to run primary agents that could fan out to subagents and parallel tool calls without the endpoint falling below roughly 30 tok/s per request. That fan-out meant four users could generate dozens of simultaneous requests. During the rental period, I tried different quantizations and replica layouts, benchmarked how each one handled those bursts, and fixed the parts that failed while my colleagues used the endpoint.

> *Implementation attribution: I directed the diagnosis, configuration, validation, and deployment decisions. GPT-5.5 generated the capacity planning and first compose deployments under my direction. Claude Opus 4.8 generated the crash diagnosis, probes, and gateway change under my direction.*

## GLM-5.2 FP8 on one replica

GLM-5.2 is a 753-billion-parameter mixture-of-experts model with 78 layers and 256 routed experts. The model ships in FP8, and the [zai-org/GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) checkpoint occupies roughly 756 GB. vLLM v0.23.0 used tensor parallelism to split each layer's weight and compute across all eight GPUs, then synchronize the partial results as the model ran. I kept its native 1,048,576-token context window.

The model loaded, passed its health check, and answered requests. It also exposed the first problem in its startup log:

```text
(EngineCore pid=1004) INFO 06-17 23:04:32 [kv_cache_utils.py:1744] GPU KV cache size: 1,117,696 tokens
(EngineCore pid=1004) INFO 06-17 23:04:32 [kv_cache_utils.py:1745] Maximum concurrency for 1,048,576 tokens per request: 1.07x
(Worker_TP0 pid=1204) INFO 06-17 23:06:10 [gpu_model_runner.py:6585] Graph capturing finished in 86 secs, took 3.02 GiB
(EngineCore pid=1004) INFO 06-17 23:06:11 [core.py:306] init engine (profile, create kv cache, warmup model) took 263.43 s (compilation: 102.99 s)
```

vLLM can batch and advance multiple requests together as long as their combined KV-cache allocations fit inside the available pool. At maximum context, however, one request could consume almost the entire pool. vLLM would then have to queue the others until enough KV-cache space became available. That was a poor match for several people running agents in parallel, even before I measured speed.

My first configuration also enabled five-token MTP speculative decoding and limited vLLM to 32 active sequences. I tested it with approximate 32K prompts at concurrency levels of 8, 16, 32, 64, and 128. At 128 concurrent requests, the endpoint completed all 128 without an error in 50.985 seconds and reported 450 aggregate tok/s. Spread across those requests, that was only about 3.5 tok/s per stream on average. My service-level objective was roughly 30 tok/s per active request, which I considered the floor for usable agent work. The first configuration missed that target by almost an order of magnitude.

I redeployed without MTP and without the 32-sequence cap, then ran the same sweep so I could compare the two configurations:

![Per-request FP8 throughput under concurrency, comparing MTP5 against the no-MTP configuration and a 30 tok/s service-level objective](how-i-built-a-long-context-inference-service-on-eight-nvidia-b200-gpus-fp8-slo.svg)

*The benchmark estimated prompt size from characters and did not read tokenizer usage from the server, so its reported `tok/s` figures are approximate.*

Removing MTP improved the FP8 deployment substantially, including at 8 and 16 concurrent requests where the 32-sequence limit was not binding. Above 32 requests, removing that limit also helped vLLM admit more work at once. Even with both changes, the deployment exceeded my 30 tok/s SLO only at 16 concurrent requests, then fell to 24.5 at 32, 16.7 at 64, and 13.0 at 128.

The FP8 trial gave me a working model and an important negative result. One copy spread across all eight GPUs could expose the full context window, but it gave the group only one maximum-length request slot and could not sustain the per-request speed I wanted under larger bursts. Before compressing GLM-5.2 further, I wanted to see what the same hardware could do with a smaller model and more replicas.

## MiniMax-M2.7 on four replicas

Since GLM-5.2 FP8 was too slow under the parallel load I wanted to support, I looked at [MiniMax-M2.7 FP8](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) next. I realized that its roughly 215–229 GB of weights could fit on two B200s, which meant I could run four copies in parallel across the server. I deployed four tensor-parallel-two replicas behind LiteLLM, replacing one model process across eight GPUs with four independently schedulable backends.

The downside was that vLLM kept its prefix cache inside each replica. If LiteLLM sent the next request to another replica, that replica had to process the same long system prompt again and regenerate its KV cache, so adding replicas could also duplicate the most expensive prompt work. That was when I learned LMCache could move the KV blocks into a shared host-memory store. I connected all four replicas to one LMCache instance so any replica could reuse a prefix prepared by another.

That gave the four-replica layout both pieces I needed: LiteLLM could distribute concurrent requests across four vLLM schedulers, while LMCache reduced the penalty when related requests landed on different replicas. The complete Compose file shows how I assigned each replica a GPU pair, connected all four to LMCache, and held LiteLLM until every backend passed its health check:

```yaml
networks:
  m27-fp8-lmcache-net:
    driver: bridge

x-replica: &replica
  image: vllm/vllm-openai:latest
  restart: unless-stopped
  ipc: host
  pid: host
  shm_size: 32g
  networks: [m27-fp8-lmcache-net]
  depends_on:
    minimax-m27-fp8-lmc-server:
      condition: service_started
  healthcheck:
    test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=2).read()\""]
    interval: 30s
    timeout: 5s
    retries: 120
    start_period: 60s
  environment:
    PYTHONHASHSEED: "0"
    PYTORCH_CUDA_ALLOC_CONF: expandable_segments:False
    HF_HOME: /root/.cache/huggingface
    HF_HUB_ENABLE_HF_TRANSFER: "1"
    TMPDIR: /tmp/m27-fp8
    VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1200"
    SAFETENSORS_FAST_GPU: "1"
  command: &replica_cmd
    - MiniMaxAI/MiniMax-M2.7
    - --served-model-name
    - MiniMax-M3
    - --host
    - 0.0.0.0
    - --port
    - "8000"
    - --tensor-parallel-size
    - "2"
    - --gpu-memory-utilization
    - "0.90"
    - --kv-cache-dtype
    - fp8
    - --enable-prefix-caching
    - --enable-chunked-prefill
    - --tool-call-parser
    - minimax_m2
    - --reasoning-parser
    - minimax_m2
    - --enable-auto-tool-choice
    - --kv-transfer-config
    - '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.host":"tcp://minimax-m27-fp8-lmc-server","lmcache.mp.port":5555}}'

services:
  minimax-m27-fp8-lmc-server:
    image: vllm/vllm-openai:latest
    container_name: minimax-m27-fp8-lmc-server
    restart: unless-stopped
    init: true
    ipc: host
    shm_size: 32g
    networks: [m27-fp8-lmcache-net]
    devices:
      - nvidia.com/gpu=all
    entrypoint:
      - lmcache
      - server
      - --host
      - 0.0.0.0
      - --port
      - "5555"
      - --http-host
      - 0.0.0.0
      - --http-port
      - "8080"
      - --l1-size-gb
      - "1024"
      - --eviction-policy
      - LRU
      - --eviction-trigger-watermark
      - "0.8"
      - --eviction-ratio
      - "0.2"
      - --max-workers
      - "16"
    environment:
      PYTHONHASHSEED: "0"
    healthcheck:
      test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/healthcheck', timeout=2).read()\""]
      interval: 5s
      timeout: 5s
      retries: 120
      start_period: 15s
    ports:
      - "18080:8080"
    ulimits:
      memlock:
        soft: -1
        hard: -1

  minimax-m27-fp8-r0:
    <<: *replica
    container_name: minimax-m27-fp8-r0
    cpuset: "0-31,64-95"
    ports:
      - "8001:8000"
    devices:
      - nvidia.com/gpu=0
      - nvidia.com/gpu=1
    volumes:
      - /home/ubuntu/minimax-m27-fp8/hf-cache:/root/.cache/huggingface
      - /home/ubuntu/minimax-m27-fp8/tmp-r0:/tmp/m27-fp8
      - /home/ubuntu/minimax-m27-fp8/vllm-cache-r0:/root/.cache/vllm

  minimax-m27-fp8-r1:
    <<: *replica
    container_name: minimax-m27-fp8-r1
    cpuset: "0-31,64-95"
    ports:
      - "8002:8000"
    devices:
      - nvidia.com/gpu=2
      - nvidia.com/gpu=3
    volumes:
      - /home/ubuntu/minimax-m27-fp8/hf-cache:/root/.cache/huggingface
      - /home/ubuntu/minimax-m27-fp8/tmp-r1:/tmp/m27-fp8
      - /home/ubuntu/minimax-m27-fp8/vllm-cache-r1:/root/.cache/vllm

  minimax-m27-fp8-r2:
    <<: *replica
    container_name: minimax-m27-fp8-r2
    cpuset: "32-63,96-127"
    ports:
      - "8003:8000"
    devices:
      - nvidia.com/gpu=4
      - nvidia.com/gpu=5
    volumes:
      - /home/ubuntu/minimax-m27-fp8/hf-cache:/root/.cache/huggingface
      - /home/ubuntu/minimax-m27-fp8/tmp-r2:/tmp/m27-fp8
      - /home/ubuntu/minimax-m27-fp8/vllm-cache-r2:/root/.cache/vllm

  minimax-m27-fp8-r3:
    <<: *replica
    container_name: minimax-m27-fp8-r3
    cpuset: "32-63,96-127"
    ports:
      - "8004:8000"
    devices:
      - nvidia.com/gpu=6
      - nvidia.com/gpu=7
    volumes:
      - /home/ubuntu/minimax-m27-fp8/hf-cache:/root/.cache/huggingface
      - /home/ubuntu/minimax-m27-fp8/tmp-r3:/tmp/m27-fp8
      - /home/ubuntu/minimax-m27-fp8/vllm-cache-r3:/root/.cache/vllm

  minimax-m27-fp8-litellm:
    image: docker.litellm.ai/berriai/litellm:v1.88.1
    container_name: minimax-m27-fp8-litellm
    restart: unless-stopped
    networks: [m27-fp8-lmcache-net]
    depends_on:
      minimax-m27-fp8-r0:
        condition: service_healthy
      minimax-m27-fp8-r1:
        condition: service_healthy
      minimax-m27-fp8-r2:
        condition: service_healthy
      minimax-m27-fp8-r3:
        condition: service_healthy
    ports:
      - "4000:4000"
    command:
      - --config
      - /etc/litellm/config.yaml
      - --host
      - 0.0.0.0
      - --port
      - "4000"
      - --num_workers
      - "8"
    volumes:
      - /home/ubuntu/minimax-m27-fp8/litellm-minimax-m27-fp8-4xtp2-config.yaml:/etc/litellm/config.yaml:ro
```

I ran the same approximate-32K concurrency sweep and plotted it against the best GLM-5.2 FP8 result:

![Per-request throughput under concurrency for four MiniMax-M2.7 replicas versus the best GLM-5.2 FP8 configuration, with a 30 tok/s service-level objective](how-i-built-a-long-context-inference-service-on-eight-nvidia-b200-gpus-m27-slo.svg)

Four M2.7 replicas outpaced the tuned GLM-5.2 FP8 deployment at every tested level. M2.7 also stayed above my 30 tok/s-per-request SLO throughout the sweep. At 128 concurrent requests, it delivered 30.7 tok/s per request against GLM's 13.0, completed all 128 requests without an error in 15.8 seconds, and began returning output after a median 2.8 seconds.

The four-replica deployment met the serving-speed target, but after we used M2.7 for a day, we wanted GLM-5.2 back for its stronger capabilities. I found M2.7 lacking on long-horizon tasks and instruction following. Its four fast replicas gave me a useful capacity reference, but not the service we wanted to keep. I returned to GLM with a clearer requirement: fit at least two replicas while preserving enough per-request speed for parallel agents.

## GLM-5.2 on two replicas

That led to the next experiment: how much could the replica approach help GLM-5.2? I carried the M2.7 design's useful parts back to GLM: independently schedulable backends, LiteLLM in front, and shared prefixes through LMCache. Since the FP8 checkpoint could not fit on four GPUs, I tried [lukealonso/GLM-5.2-NVFP4](https://huggingface.co/lukealonso/GLM-5.2-NVFP4). Its smaller weights let me split the server into two tensor-parallel-four replicas, with r0 on GPUs 0–3 and r1 on GPUs 4–7. I pinned each replica to the CPU cores on the same socket and placed one LiteLLM route in front of both.

Even with the smaller NVFP4 weights, each four-GPU replica lacked enough KV-cache memory for GLM-5.2's full 1,048,576-token context. vLLM estimated maximum lengths of 744,896 and 750,336 tokens, so I capped both replicas at 700,000. That reduced the maximum context by 348,576 tokens but gave me two independently routable GLM-5.2 replicas instead of one. I connected both replicas to a 1 TiB LMCache host-memory store so r0 could reuse KV blocks prepared by r1 and vice versa, preserving the cross-replica caching design I had introduced for M2.7.

I then ran the same approximate-32K concurrency sweep and compared the two-replica deployment with the tuned FP8 baseline:

![Per-request throughput under concurrency for two GLM-5.2 NVFP4 replicas versus the best GLM-5.2 FP8 configuration, with a 30 tok/s service-level objective](how-i-built-a-long-context-inference-service-on-eight-nvidia-b200-gpus-nvfp4-slo.svg)

The trade worked. Even with 128 requests running at once, the two replicas held just above my target at 30.5 tok/s per request and finished the entire burst in 15.875 seconds without an error. The single FP8 deployment had managed only 13.0 tok/s per request at the same load. Running two replicas had bought back nearly all the capacity I saw with M2.7, but this time with the GLM-5.2 model we actually wanted to use. The cost of making those replicas fit was moving from GLM-5.2's native FP8 weights to NVFP4 and reducing the context limit to 700,000 tokens.

## Recovering a failed replica

When r1 crashed, Docker automatically restarted the replica's container, but the vLLM service failed to start. With LMCache still occupying about 51 GiB on each of r1's four GPUs, only 122.61 GiB remained free. vLLM needed 169.43 GiB to start the replacement worker, so it exited with an insufficient-memory error. Although the vLLM process had exited, LMCache's open CUDA IPC handles kept its KV-cache memory allocated.

In order to recover the replica, I had to stop LMCache as well. Once the cache service released those allocations, I could start both replicas again. I also configured LiteLLM to retry a failed request on the other replica and remove a backend from rotation for 60 seconds after three failures. That kept traffic on the working replica instead of repeatedly routing requests to the one stuck restarting.

The incident left me with two recovery paths. If a failed replica released its GPU memory, I could restart it alone. If LMCache kept the allocation alive, I had to restart the cache and replicas together.

I kept LMCache in the deployment despite the more involved recovery process because its shared cache saved substantial work on long prompts. In a direct test, r0 took about 89 seconds to prepare a roughly 690,000-token prefix. Sending the same prompt to r1 immediately afterward took about six seconds because it could reuse the cached KV blocks. Changing the prefix returned the preparation time to roughly 89 seconds.

## The deployment file

The complete Compose file below comes from the later AWQ deployment, but it implements the layout I established during the NVFP4 trial: two tensor-parallel-four replicas, a 700,000-token context limit, FP8 KV cache, shared LMCache, and LiteLLM in front. I used a shared anchor for the common vLLM settings, then assigned each replica four GPUs and the CPU cores on the same socket. I gave LMCache access to all eight GPUs so it could open CUDA IPC mappings into both replicas.

Here is the complete deployment rather than a shortened example:

```yaml
networks:
  glm52-awq-lmcache-net:
    driver: bridge

x-replica: &replica
  image: vllm/vllm-openai:v0.23.0
  restart: unless-stopped
  ipc: host
  pid: host
  shm_size: 32g
  networks: [glm52-awq-lmcache-net]
  depends_on:
    glm52-awq-int4-lmc-server:
      condition: service_started
  healthcheck:
    test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=2).read()\""]
    interval: 30s
    timeout: 5s
    retries: 120
    start_period: 60s
  environment:
    PYTHONHASHSEED: "0"
    PYTORCH_CUDA_ALLOC_CONF: expandable_segments:False
    HF_HOME: /root/.cache/huggingface
    HF_HUB_ENABLE_HF_TRANSFER: "1"
    TMPDIR: /tmp/glm52-awq-int4
    VLLM_DEEP_GEMM_WARMUP: skip
    VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1200"
    SAFETENSORS_FAST_GPU: "1"
  command: &replica_cmd
    - cyankiwi/GLM-5.2-AWQ-INT4
    - --served-model-name
    - GLM-5.2-AWQ-INT4
    - --host
    - 0.0.0.0
    - --port
    - "8000"
    - --tensor-parallel-size
    - "4"
    - --gpu-memory-utilization
    - "0.95"
    - --max-model-len
    - "700000"
    - --kv-cache-dtype
    - fp8_e4m3
    - --enable-prefix-caching
    - --enable-chunked-prefill
    - --tool-call-parser
    - glm47
    - --reasoning-parser
    - glm45
    - --enable-auto-tool-choice
    - --kv-transfer-config
    - '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.host":"tcp://glm52-awq-int4-lmc-server","lmcache.mp.port":5555}}'

services:
  glm52-awq-int4-lmc-server:
    image: vllm/vllm-openai:v0.23.0
    container_name: glm52-awq-int4-lmc-server
    restart: unless-stopped
    init: true
    ipc: host
    shm_size: 32g
    networks: [glm52-awq-lmcache-net]
    devices:
      - nvidia.com/gpu=all
    entrypoint:
      - lmcache
      - server
      - --host
      - 0.0.0.0
      - --port
      - "5555"
      - --http-host
      - 0.0.0.0
      - --http-port
      - "8080"
      - --l1-size-gb
      - "1024"
      - --eviction-policy
      - LRU
      - --eviction-trigger-watermark
      - "0.8"
      - --eviction-ratio
      - "0.2"
      - --max-workers
      - "16"
    environment:
      PYTHONHASHSEED: "0"
    healthcheck:
      test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/healthcheck', timeout=2).read()\""]
      interval: 5s
      timeout: 5s
      retries: 120
      start_period: 15s
    ports:
      - "18080:8080"
    ulimits:
      memlock:
        soft: -1
        hard: -1

  glm52-awq-int4-r0:
    <<: *replica
    container_name: glm52-awq-int4-r0
    cpuset: "0-31,64-95"
    ports:
      - "8001:8000"
    devices:
      - nvidia.com/gpu=0
      - nvidia.com/gpu=1
      - nvidia.com/gpu=2
      - nvidia.com/gpu=3
    volumes:
      - /home/ubuntu/glm52-awq-int4/hf-cache:/root/.cache/huggingface
      - /home/ubuntu/glm52-awq-int4/tmp-r0:/tmp/glm52-awq-int4
      - /home/ubuntu/glm52-awq-int4/vllm-cache-r0:/root/.cache/vllm

  glm52-awq-int4-r1:
    <<: *replica
    container_name: glm52-awq-int4-r1
    cpuset: "32-63,96-127"
    ports:
      - "8002:8000"
    devices:
      - nvidia.com/gpu=4
      - nvidia.com/gpu=5
      - nvidia.com/gpu=6
      - nvidia.com/gpu=7
    volumes:
      - /home/ubuntu/glm52-awq-int4/hf-cache:/root/.cache/huggingface
      - /home/ubuntu/glm52-awq-int4/tmp-r1:/tmp/glm52-awq-int4
      - /home/ubuntu/glm52-awq-int4/vllm-cache-r1:/root/.cache/vllm

  glm52-awq-int4-litellm:
    image: docker.litellm.ai/berriai/litellm:v1.88.1
    container_name: glm52-awq-int4-litellm
    restart: unless-stopped
    networks: [glm52-awq-lmcache-net]
    depends_on:
      glm52-awq-int4-r0:
        condition: service_healthy
      glm52-awq-int4-r1:
        condition: service_healthy
    ports:
      - "4000:4000"
    command:
      - --config
      - /etc/litellm/config.yaml
      - --host
      - 0.0.0.0
      - --port
      - "4000"
      - --num_workers
      - "8"
    volumes:
      - /home/ubuntu/glm52-awq-int4/litellm-glm52-awq-int4-2xtp4-config.yaml:/etc/litellm/config.yaml:ro
```

The two-replica design met my 30 tok/s target through every benchmark level, including 128 simultaneous requests. Getting there meant trading some context for a second replica, sharing repeated prefixes, routing around failed backends, and learning when recovery required restarting LMCache as well as vLLM. The final Compose file was the result of that work, not the starting point.