My local LLM setup for coding#

I’m using tailscale to connect to my devices, when you see scruffy or bender these are the MagicDNS names applied.

dev machine#

Below script starts a opencode server ocs & I’m using opencode attach oca to use that server. That gives me the possibility to continue my coding session on any device in my browser at http://bender:1234

echo '
export EDITOR="subl --wait"
opencode serve --hostname=0.0.0.0 --port=1234
' > ~/bin/ocs

echo '
export EDITOR="subl --wait"
opencode attach http://0.0.0.0:1234 --dir $(PWD)
' > ~/bin/oca

Opencode needs a few adjustments to connect to my LLM machine on startup.

~/.config/opencode/opencode.json

{
  "$schema": "https://opencode.ai/config.json",
  "disabled_providers": ["opencode"],
  "plugin": [],
  "lsp": true,
  "provider": {
    "scruffy": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "scruffy",
      "options": {
        "baseURL": "http://scruffy:8000/v1",
        "timeout": false,
        "includeUsage": true
      },
      "models": {
        "qwen3.6-27b": {
          "id": "qwen3.6-27b",
          "name": "qwen3.6-27b",
          "reasoning": true,
          "tool_call": true,
          "cost": {
            "input": 0.000000039,
            "output": 0.000000778
          },
          "limit": {
            "context": 200000,
            "output": 0
          },
          "modalities": {
            "input": ["text", "image", "video"],
            "output": ["text"]
          }
        }
      }
    }
  },
  "model": "scruffy/qwen3.6-27b",
  "agent": {
    "plan": {}
  },
  "permission": {
    "lsp": "allow",
    "skill": "allow",
    "webfetch": "allow",
    "websearch": "allow",
    "question": "allow",
    "external_directory": {
      "*": "ask",
      "~/.config/opencode/skills/**": "allow",
      "/tmp": "allow",
      "/tmp/*": "allow",
      "/private/tmp": "allow",
      "/private/tmp/*": "allow",
      "/var/folders/*": "allow",
      "env:$TMPDIR": "allow",
      "env:$TMPDIR/*": "allow"
    }
  },
  "mcp": {
    "context7": {
      "type": "remote",
      "url": "https://mcp.context7.com/mcp",
      "headers": {
        "CONTEXT7_API_KEY": "very secret key!"
      }
    },
    "website-spec": {
      "type": "remote",
      "url": "https://mcp.specification.website/mcp"
    },
    "playwright": {
      "type": "local",
      "command": ["npx", "@playwright/mcp@latest"],
      "enabled": true
    },
    "chrome-devtools": {
      "type": "local",
      "command": ["npx", "-y", "chrome-devtools-mcp@latest"]
    },
    "searxng-mcp": {
      "type": "remote",
      "url": "http://scruffy:9002/mcp",
      "headers": {
        "Authorization": "Bearer secret"
      }
    },
    "openzim-mcp": {
      "type": "remote",
      "url": "http://scruffy:9001/mcp",
      "headers": {
        "Authorization": "Bearer secret"
      }
    }
  }
}

llm-server#

below is the full setup script for my llm machine, from a fresh installation of fedora. I’m using this image https://hub.docker.com/r/stilldeadcode/vllm-radiance/ because it is a vllm build for my GPU that works stable & fast out of the box. Usually ROCm on vllm is really hard to setup and crashes a lot.
I also included a llama.cpp build script that includes flags for faster prompt processing on RDNA3 & RDNA4 cards, pre built llama.cpp builds do not include them. This build also enables RCCL for P2P communication, however latest ROCm build contain broken RCCL. Therefore env var HSA_ENABLE_IPC_MODE_LEGACY=1 ensures graceful fallback instead of crash.

Hardware

I have 2 GPUs (AMD R9700 AI Pro) installed in a mainboard supporting PCIe x8/x8 bifurication at gen 4 speed. For proper peer to peer (P2P) GPU communication support, ensure the Mainboard BIOS got these settings:

• ReBar Enabled (also called Above 4G decoding)
• ACS Disabled (Access Control Service)
  • also disable_acs.sh & disable_acs.service startup script ensure this is disabled
• SR-IOV Disabled

To test P2P support on AMD GPUs, run: rocm-bandwidth-test. It should print out that Inter-Device Access block:

Inter-Device Access

          D/D       0         1         2

          0         1         1         1

          1         1         1         1

          2         1         1         1

X & Y Axis are the devices, 0 = CPU, 1 = GPU1, etc… CPU can always access GPU & always prints 1.

GPU tuning

AMD R9700 cards are by default loud and do constant thermal throttling. So I adjusted these GPU settings using LACT: power limit 210W, undervolted -50mV, clock -200Mhz. This does not slow down inference speed that much (~5% slowdown)

Benchmarks

setup script

sudo dnf install vim htop nvtop -y
sudo dnf install vim-default-editor --allowerasing

# prevent pesky energy saver regulations from turning off my server!
sudo systemctl disable suspend
sudo systemctl mask suspend
sudo systemctl mask hibernate
sudo systemctl mask sleep
sudo systemctl mask hybrid-sleep

# install tailscale to access the machine from everywhere
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up
sudo systemctl restart tailscaled.service

# disable ipv6 because it's only 2026 & github/huggigface are still not able to communicate over it...
sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

# LACT allows tuning & limiting power on GPUs
sudo dnf copr enable ilyaz/LACT -y
sudo dnf install lact -y
sudo rm /usr/lib/systemd/system/lactd.service
echo '[Unit]
Description=LACT GPU Control Daemon
After=network-online.target

[Service]
ExecStart=lact daemon
Nice=-10
User=root
Group=root
Restart=always
RestartSec=3

[Install]
WantedBy=default.target' | sudo tee -a /etc/systemd/system/lactd.service
sudo systemctl daemon-reload
sudo systemctl enable lactd
sudo systemctl start lactd

# Cooler Control allows changing the fan speed. nct6687d is a common chip on mainboards & below github project adds support to Cooler Control.
sudo dnf install dnf-plugins-core -y
sudo dnf copr enable codifryed/CoolerControl -y
sudo dnf install coolercontrol -y
sudo systemctl enable --now coolercontrold
sudo systemctl daemon-reload
sudo dnf install make automake gcc gcc-c++ kernel-devel kernel-headers dkms -y
cd ~/Downloads
git clone https://github.com/Fred78290/nct6687d
cd nct6687d
make akmod
sudo sh -c 'echo "nct6687" >> /etc/modules'
sudo coolercontrold detect --load

# Add the current user to the render and video groups
sudo usermod -a -G render,video $LOGNAME
sudo dnf install -y rocm

# MCP server setup & OpenwebUI. It will create a ~/mcp-server/update.sh script to update all componets easily
sudo dnf install -y yq
sudo systemctl disable --now docker.service docker.socket
sudo rm /var/run/docker.sock
dockerd-rootless-setuptool.sh install

mkdir -p /home/chack/mcp-server/
cd /home/chack/mcp-server/
tee mcp-openzim.yml << END
services:
  mcp-openzim:
    image: ghcr.io/cameronrye/openzim-mcp:latest
    container_name: mcp-openzim
    restart: unless-stopped
    ports:
      - "9001:8000"
    volumes:
      - /home/chack/mcp-server/zim-data:/data:ro
      - openzim-cache:/home/appuser/.cache/openzim-mcp
    environment:
      OPENZIM_MCP_TRANSPORT: "http"
      OPENZIM_MCP_HOST: "0.0.0.0"
      OPENZIM_MCP_ALLOWED_HOSTS: "[\"scruffy:*\"]"
      OPENZIM_MCP_CORS_ORIGINS: "[\"http://scruffy\"]"
      OPENZIM_MCP_AUTH_TOKEN: "secret"
      OPENZIM_MCP_CACHE__PERSISTENCE_ENABLED: "true"
volumes:
  openzim-cache:
END

echo 'version: "3.8"

services:
  # 1. The SearXNG Search Engine Service
  searxng:
    image: docker.io/searxng/searxng:latest
    container_name: searxng-backend
    restart: unless-stopped
    volumes:
      - ./searxng:/etc/searxng:rw
    environment:
      - SEARXNG_BASE_URL=http://localhost:8080/
    ports:
      - "9000:8080"
    expose:
      - "8080"
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

  # 2. The MCP Server Layer
  mcp-searxng:
    image: isokoliuk/mcp-searxng:latest
    container_name: mcp-searxng-server
    restart: unless-stopped
    depends_on:
      - searxng
    ports:
      - "9002:9002"
    environment:
      # Directs the MCP server to communicate with the SearXNG service inside the network
      - SEARXNG_URL=http://searxng:8080
      - MCP_HTTP_HOST=0.0.0.0
      - MCP_HTTP_PORT=9002
      - AUTH_USERNAME=admin
      - AUTH_PASSWORD=secret
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
' > mcp-searxng.yml
echo '
services:
  postgres:
    image: postgres:15-alpine
    container_name: openwebui-postgres
    restart: unless-stopped
    environment:
      POSTGRES_DB: openwebui
      POSTGRES_USER: openwebui
      POSTGRES_PASSWORD: secret
      POSTGRES_INITDB_ARGS: "--encoding=UTF8"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "127.0.0.1:5432:5432"  # Only localhost for security
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U openwebui -d openwebui"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s
    networks:
      - openwebui-network
  openwebui:
    image: ghcr.io/open-webui/open-webui:main-slim
    container_name: openwebui
    restart: unless-stopped
    ports:
      - "3000:8080"
    depends_on:
      postgres:
        condition: service_healthy
    environment:
      - DATABASE_URL=postgresql://openwebui:secret@openwebui-postgres:5432/openwebui
    volumes:
      - openwebui_data:/app/backend/data
    ports:
      - "3000:8080"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    networks:
      - openwebui-network
    volumes:
      - openwebui_data:/app/backend/data
volumes:
  postgres_data:
    driver: local
    name: openwebui-postgres-data
  openwebui_data:
    driver: local
    name: open-webui
networks:
  openwebui-network:
    driver: bridge
' > openwebui.yml
echo 'name: mcp-server
' > name.yml
echo '
SEARXNG_VERSION=latest
SEARXNG_HOST=[::]
SEARXNG_PORT=9000
' > .env
tee update.sh << END
docker compose down
curl -fsSL -o searxng.yml https://raw.githubusercontent.com/searxng/searxng/master/container/docker-compose.yml
echo "name: mcp-server" > name.yml
yq eval-all '. as \$item ireduce ({}; . * \$item)' mcp-openzim.yml mcp-searxng.yml openwebui.yml name.yml > docker-compose.yml
docker compose pull
docker compose up -d
END
chmod +x update.sh
./update.sh
mkdir -p /home/chack/mcp-server/zim-data
cd /home/chack/mcp-server/zim-data
curl -OL https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/wikipedia_en_all_mini_2026-03.zim
curl -OL https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/wikipedia_de_all_mini_2026-04.zim

# something did not work without it, I don't know anymore & I forgot to write it down at that time.
echo '
label  ::1/128       0
label  ::/0          1
label  2002::/16     2
label ::/96          3
label ::ffff:0:0/96  4
precedence  ::1/128       50
precedence  ::/0          40
precedence  2002::/16     30
precedence ::/96          20
precedence ::ffff:0:0/96  100
' | sudo tee /etc/gai.conf

# GPU P2P kernel parameter changes that will apply after next reboot.
# iommu & pci=realloc changes allow P2P
# pcie_aspm disable PCIe power saver stuff
# amdgpu.ras_enable=0 disable GPU ECC
# amdgpu.runpm=0 disable GPU power saver stuff
# amdgpu.cwsr_enable=0 disabling this prevents common AMD GPU crashes
# amdgpu.ppfeaturemask allows for tuning the GPU
sudo grubby --update-kernel=ALL --args="quiet splash amd_iommu=on iommu=pt pcie_aspm=off pci=realloc amdgpu.ras_enable=0 amdgpu.runpm=0 amdgpu.cwsr_enable=0 amdgpu.ppfeaturemask=0xffffffff"
sudo tee -a /etc/default/grub << END
GRUB_CMDLINE_LINUX="quiet splash amd_iommu=on iommu=pt pcie_aspm=off pci=realloc amdgpu.ras_enable=0 amdgpu.runpm=0 amdgpu.cwsr_enable=0 amdgpu.ppfeaturemask=0xffffffff"
END
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# needs to run on every boot & disables ACS. Again for proper P2P support
echo '
#!/bin/bash
#
# Disable ACS on every device that supports it
#
PLATFORM=$(dmidecode --string system-product-name)
logger "PLATFORM=${PLATFORM}"
# Enforce platform check here.
#case "${PLATFORM}" in
#"OAM"*)
#logger "INFO: Disabling ACS is no longer necessary for ${PLATFORM}"
#exit 0
#;;
#*)
#;;
#esac

# must be root to access ext PCI config space
if [ "$EUID" -ne 0 ]; then
echo "ERROR: $0 must be run as root"
exit 1
fi

for BDF in $(lspci -d "*:*:*" | awk "{print $1}"); do
    # skip if it doesnt support ACS
    setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
    if [ $? -ne 0 ]; then
        #echo "${BDF} does not support ACS, skipping"
        continue
    fi

    logger "Disabling ACS on $(lspci -s ${BDF})"
    setpci -v -s ${BDF} ECAP_ACS+0x6.w=0000
    if [ $? -ne 0 ]; then
        logger "Error disabling directTrans ACS on ${BDF}"
        continue
    fi

    NEW_VAL=$(setpci -v -s ${BDF} ECAP_ACS+0x6.w | awk "{print $NF}")
    if [ "${NEW_VAL}" != "0000" ]; then
        logger "Failed to disable directTrans ACS on ${BDF}"
        continue
    fi
done
exit 0
' | sudo tee /usr/local/bin/disable_acs.sh
chmod +x /usr/local/bin/disable_acs.sh
sudo tee /etc/systemd/system/disable_acs.service << END
[Unit]
Description=Run disable_acs.sh Script on Startup
After=network.target

[Service]
Type=simple
User=root
ExecStart=/bin/bash /usr/local/bin/disable_acs.sh

[Install]
WantedBy=multi-user.target
END
sudo systemctl daemon-reload
sudo systemctl enable disable_acs.service
sudo systemctl start disable_acs.service


install latest rocm tools
cd $(mktemp -d)
wget https://repo.amd.com/rocm/tarball-multi-arch/therock-dist-linux-gfx120X-all-7.14.0.tar.gz
sudo mkdir -p /opt/rocm
sudo tar -xf *.tar.gz -C /opt/rocm
ROCM_INSTALL_PATH=/opt/rocm
sudo tee /etc/profile.d/set-rocm-env.sh << EOF
export ROCM_PATH=$ROCM_INSTALL_PATH
export PATH=\$PATH:\$ROCM_PATH/bin
export LD_LIBRARY_PATH=\$ROCM_PATH/lib
EOF
sudo chmod +x /etc/profile.d/set-rocm-env.sh
source /etc/profile.d/set-rocm-env.sh
# uninstall again:
# sudo rm -rf /opt/rocm
# sudo rm -f /etc/profile.d/set-rocm-env.sh

# below is a full llama.cpp build script run in a container. It copies the latest rocm tarball & clonse llama.cpp. It enables RCCL & ROCMWMMA for faster inference. When finished, it copies to directory on host
mkdir -p ~/llamabuild
cd ~/llamabuild
tee build.sh << 'END'

#!/usr/bin/env bash
set -euo pipefail

cd /home/chack/llamabuild
rm -rf llama Dockerfile.llamacpp-rocm
docker builder prune -f
docker image pull ubuntu:26.04

IMAGE_NAME="llamacpp-rocm-gfx120x"
docker image rm ${IMAGE_NAME}:latest -f
OUTPUT_DIR="./llama"
TARGET_DIR="/home/chack/.local/bin"
DOCKERFILE="Dockerfile.llamacpp-rocm"

echo "=== Llama.cpp ROCm Build (Ubuntu amd64 + gfx120X) ==="
echo ""

# ──────────────────────────────────────────────
# 1. Dockerfile erstellen
# ──────────────────────────────────────────────
echo "[1/4] Dockerfile erstellen …"

cat > "${DOCKERFILE}" << 'DOCKERFILE_EOF'
FROM ubuntu:26.04

ENV DEBIAN_FRONTEND=noninteractive

# g++ nötig für libstdc++ (ROCm clang++ Linker)
RUN apt-get update && apt-get install -y \
    cmake \
    ninja-build \
    git \
    curl \
    patchelf \
    unzip \
    jq \
    g++ \
    libvulkan-dev \
    glslc \
    spirv-headers \
    && rm -rf /var/lib/apt/lists/*

RUN set -eux; \
    S3_TARGET="gfx120X-all"; \
    BASE_URL="https://rocm.nightlies.amd.com/tarball-multi-arch"; \
    #BASE_URL="https://repo.amd.com/rocm/tarball-multi-arch"; \
    PREFIX="therock-dist-linux-${S3_TARGET}-"; \
    INDEX=$(curl -sL "${BASE_URL}/"); \
    FILES_JSON=$(echo "$INDEX" | tr '\n' ' ' | grep -oP 'const files = \K\[.*?\](?=\s*;)'); \
    LATEST_FILE=$(echo $FILES_JSON | jq -r '[.[] | select(.name | startswith("therock-dist-linux-gfx120X-all") and (contains("-test") | not))] | sort_by(.name) | last | .name'); \
    if [ -z "$LATEST_FILE" ]; then \
      echo "ERROR: No ROCm tarball found for prefix '${PREFIX}'"; \
      echo "Available files:"; \
      echo "$FILES_JSON" | jq -r '.[].name' | grep -i "${S3_TARGET}" || true; \
      exit 1; \
    fi; \
    echo "Downloading ROCm: ${LATEST_FILE}"; \
    mkdir -p /opt/rocm; \
    curl -sL "${BASE_URL}/${LATEST_FILE}" \
      | tar --use-compress-program=gzip -xf - -C /opt/rocm --strip-components=1

ENV HIP_PATH=/opt/rocm \
    ROCM_PATH=/opt/rocm \
    HIP_PLATFORM=amd \
    HIP_CLANG_PATH=/opt/rocm/llvm/bin \
    HIP_INCLUDE_PATH=/opt/rocm/include \
    HIP_LIB_PATH=/opt/rocm/lib \
    HIP_DEVICE_LIB_PATH=/opt/rocm/lib/llvm/amdgcn/bitcode \
    PATH="/opt/rocm/bin:/opt/rocm/llvm/bin:$PATH" \
    LD_LIBRARY_PATH="/opt/rocm/lib:/opt/rocm/lib64:/opt/rocm/llvm/lib" \
    LIBRARY_PATH="/opt/rocm/lib:/opt/rocm/lib64" \
    CPATH="/opt/rocm/include" \
    PKG_CONFIG_PATH="/opt/rocm/lib/pkgconfig"

WORKDIR /build
RUN git clone --depth 1 --single-branch --branch master https://github.com/ggml-org/llama.cpp.git
WORKDIR /build/llama.cpp

RUN set -eux; \
    MAPPED_TARGET="gfx1200;gfx1201"; \
    BUILD_JOBS=$(($(nproc) - 12)); \
    if [ "$BUILD_JOBS" -lt 1 ]; then BUILD_JOBS=1; fi; \
    echo "Building with ${BUILD_JOBS} parallel jobs (CPUs: $(nproc))"; \
    mkdir build && cd build; \
    cmake .. -G Ninja \
      -DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang \
      -DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++ \
      -DCMAKE_CXX_FLAGS="-I/opt/rocm/include" \
      -DCMAKE_CROSSCOMPILING=ON \
      -DCMAKE_BUILD_TYPE=Release \
      -DGPU_TARGETS="${MAPPED_TARGET}" \
      -DBUILD_SHARED_LIBS=ON \
      -DLLAMA_BUILD_TESTS=OFF \
      -DGGML_HIP=ON \
      -DGGML_HIP_RCCL=ON \
      -DGGML_HIP_ROCWMMA_FATTN=ON \
      -DGGML_VULKAN=1 \
      -DGGML_OPENMP=OFF \
      -DGGML_CUDA_FORCE_CUBLAS=OFF \
      -DGGML_RPC=ON \
      -DLLAMA_BUILD_BORINGSSL=ON \
      -DGGML_NATIVE=OFF \
      -DGGML_STATIC=OFF \
      -DCMAKE_SYSTEM_NAME=Linux; \
    cmake --build . -j "${BUILD_JOBS}"

RUN set -eux; \
    BIN="/build/llama.cpp/build/bin"; \
    ROCM="/opt/rocm"; \
    \
    # 1. Copy mandatory runtime directories for compiled modules
    for subdir in rocblas hipblaslt rccl; do \
      if [ -d "${ROCM}/lib/${subdir}/library" ]; then \
        mkdir -p "${BIN}/${subdir}"; \
        cp -r "${ROCM}/lib/${subdir}/library" "${BIN}/${subdir}/"; \
      fi; \
    done; \
    \
    # 2. Trace and copy all dynamically linked ROCm dependencies of the llama binaries
    find "${BIN}" -type f -executable -name "llama-*" | while read -r binary; do \
      ldd "$binary" | grep -oP '/opt/rocm/\S+' | while read -r libpath; do \
        real_lib=$(readlink -f "$libpath"); \
        cp -v "$real_lib" "${BIN}/" 2>/dev/null || true; \
        if [ "$real_lib" != "$libpath" ]; then \
          cp -vd "$libpath" "${BIN}/" 2>/dev/null || true; \
        fi; \
      done; \
    done; \
    \
    # 3. Secondary scan on the copied .so files to catch nested dependencies (libs loading other libs)
    find "${BIN}" -name "*.so*" -type f | while read -r lib; do \
      ldd "$lib" | grep -oP '/opt/rocm/\S+' | while read -r nested_lib; do \
        real_nested=$(readlink -f "$nested_lib"); \
        cp -nv "$real_nested" "${BIN}/" 2>/dev/null || true; \
        if [ "$real_nested" != "$nested_lib" ]; then \
          cp -nvd "$nested_lib" "${BIN}/" 2>/dev/null || true; \
        fi; \
      done; \
    done

RUN cd /build/llama.cpp/build/bin && \
    for file in *.so* llama-*; do \
      [ -f "$file" ] && [ ! -L "$file" ] && \
        patchelf --set-rpath '$ORIGIN' "$file" 2>/dev/null || true; \
    done

ENV OUTPUT_DIR=/build/llama.cpp/build/bin
DOCKERFILE_EOF

echo "  → ${DOCKERFILE} erstellt"
echo ""

# ──────────────────────────────────────────────
# 2. Docker-Image bauen
# ──────────────────────────────────────────────
echo "[2/4] Docker-Image bauen …"
docker build --shm-size=4g -t "${IMAGE_NAME}" -f "${DOCKERFILE}" .
echo ""

# ──────────────────────────────────────────────
# 3. Build-Ergebnis extrahieren
# ──────────────────────────────────────────────
echo "[3/4] Build-Artefakte extrahieren …"
rm -rf "${OUTPUT_DIR}"
mkdir -p "${OUTPUT_DIR}"

CONTAINER_ID=$(docker create "${IMAGE_NAME}" true)
docker cp "${CONTAINER_ID}:/build/llama.cpp/build/bin/" "${OUTPUT_DIR}/"
docker rm "${CONTAINER_ID}"
echo ""

# ──────────────────────────────────────────────
# 4. Ergebnis anzeigen
# ──────────────────────────────────────────────
echo "[4/4] Fertig!"
echo ""
echo "Build-Artefakte liegen in: ${OUTPUT_DIR}/"
echo ""
ls -lh "${OUTPUT_DIR}/"
echo ""
echo "Dockerfile '${DOCKERFILE}' wurde beibehalten (falls erneut gebaut werden soll)."
chmod +x llama/bin/llama*
rm -rf ${TARGET_DIR}/*
mkdir -p ${TARGET_DIR}
rsync -Pa llama/bin/* ${TARGET_DIR}/.


END

chmod +x ~/llamabuild/build.sh
~/llamabuild/build.sh

# llama.cpp systemd setup
echo '[Unit]
Description=Llama.cpp server
After=network.target

[Service]
Type=simple
User=chack
Group=chack
Environment="HSA_ENABLE_IPC_MODE_LEGACY=1"
Environment="HSA_FORCE_FINE_GRAIN_PCIE=1"
Environment="NCCL_IGNORE_CPU_AFFINITY=1"
Environment="GGML_CUDA_P2P=1"
#Environment="NCCL_PROTO=Simple"
#Environment="RCCL_PROTO=Simple"

#Environment="NCCL_DEBUG_SUBSYS=ALL"
#Environment="NCCL_DEBUG=INFO"

ExecStart=/bin/bash -c "/home/chack/.local/bin/llama-server --host 0.0.0.0 --port 8000 --models-max 1 --models-preset /home/chack/llama-models.ini"
Restart=on-failure
RestartSec=3

[Install]
WantedBy=default.target
' | sudo tee /usr/lib/systemd/system/llama.service
sudo systemctl daemon-reload

echo '

version = 1

[*]
n-gpu-layers = all
flash-attn = on
kv-unified = true
batch-size = 8192
ubatch-size = 2048
cache-ram = 16386
ctx-checkpoints = 64
cache-idle-slots = true
sleep-idle-seconds = 30000
timeout = 2400
metrics = true

[qwen3.6-27b]
load-on-startup = true
alias = qwen3.6-27b
temperature = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0
chat-template-kwargs = {"preserve_thinking": true}
image-min-tokens = 1024
reasoning-budget = 4096
reasoning-budget-message = " -- Reasoning budget exceeded, proceed to final answer."
hf = unsloth/Qwen3.6-27B-MTP-GGUF:Q8_0
dev = ROCm0,ROCm1
spec-draft-device = ROCm0,ROCm1
main-gpu = 0
split-mode = tensor
tensor-split = 1,1
no-mmap = true
mmproj-auto = true
no-mmproj-offload = true
ctx-size = 380000
parallel = 2
# f16 is faster than bf16 or q8
ctk = f16
ctv = f16
ctkd = f16
ctvd = f16
spec-type = draft-mtp,ngram-mod
spec-draft-n-max = 4

' > /home/chack/llama-models.ini


echo '# This config file enables a /dev/zram0 device with the default settings:
# — size — same as available RAM or 8GB, whichever is less
# — compression — most likely lzo-rle
#
# To disable, uninstall zram-generator-defaults or create empty
# /etc/systemd/zram-generator.conf file.
[zram0]
zram-size = min(ram, 32000)' | sudo tee /usr/lib/systemd/zram-generator.conf
sudo systemctl daemon-reload
sudo systemctl restart systemd-zram-setup@zram0.service

# vllm setup
sudo tee /etc/systemd/system/vllm.service << END
[Unit]
Description=vLLM Container Service
After=docker.service
Requires=docker.service

[Service]
Restart=always
RestartSec=10
TimeoutStartSec=0
User=chack
Group=chack
ExecStartPre=-/usr/bin/docker stop vllm
ExecStartPre=-/usr/bin/docker rm vllm
ExecStart=/usr/bin/docker run \
  --name vllm \
  --device /dev/kfd --device /dev/dri \
  --group-add video \
  --shm-size 4g --cap-add SYS_PTRACE --security-opt seccomp=unconfined \
  -v "/home/chack/.cache/vllm-cache:/cache" \
  -v /home/chack/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  -e HIP_VISIBLE_DEVICES=0,1 \
  -e VLLM_ROCM_USE_AITER=1 -e VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1 \
  -e VLLM_ROCM_USE_AITER_MHA=0 -e VLLM_ROCM_USE_AITER_MLA=0 -e VLLM_ROCM_USE_AITER_MOE=0 \
  -e VLLM_ROCM_USE_AITER_LINEAR=0 -e VLLM_ROCM_USE_AITER_FP8BMM=0 \
  -e VLLM_ROCM_USE_AITER_FP4BMM=0 -e VLLM_ROCM_USE_AITER_RMSNORM=0 \
  -e NCCL_PROTO=Simple \
  -e RADIANCE_PRESHUFFLE=1 \
  -e RADIANCE_ATTN_TUNE=1 \
  -e RADIANCE_GDN_WMMA=1 \
  -e RADIANCE_VIT_FLASH=1 \
  -e RADIANCE_FAST_REDUCE=1 \
  -e RADIANCE_AR_MAX_KB=32768 \
  -e RADIANCE_AR_QUANT=1 \
  -e RADIANCE_FUSE_RMS_QUANT=1 \
  -e RADIANCE_DYNAMIC_DRAFT=1 \
  -e VLLM_CACHE_ROOT=/cache/vllm -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor \
  -e TRITON_CACHE_DIR=/cache/triton -e AITER_ROOT_DIR=/cache/aiter \
  -e TRITON_CACHE_AUTOTUNING=1 \
  stilldeadcode/vllm-radiance:0.3.0 \
    Qwen/Qwen3.6-27B-FP8 \
    --served-model-name qwen3.6-27b \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --override-generation-config '{"temperature":0.6, "top_p":0.95, "top_k":20, "min_p":0.0, "presence_penalty":0.0, "repetition_penalty":1.0}' \
    --default-chat-template-kwargs '{"enable_thinking": true, "preserve_thinking": true}' \
    --quantization fp8 --kv-cache-dtype auto \
    --max-model-len 220000 \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.92 \
    --attention-backend ROCM_AITER_UNIFIED_ATTN \
    --enable-prefix-caching --mamba-cache-mode align \
    --speculative-config '{"method":"mtp","num_speculative_tokens":8,"attention_backend":"ROCM_AITER_UNIFIED_ATTN","disable_padded_drafter_batch":true}' \
    --no-async-scheduling \
    --host 0.0.0.0 --port 8000

ExecStop=/usr/bin/docker stop vllm

[Install]
WantedBy=multi-user.target

END

sudo systemctl daemon-reload
sudo systemctl enable vllm.service
sudo systemctl restart vllm.service

sudo systemctl reboot