Skip to main content

Run local LLMs with Faheem Code

Use this guide when you want a local model, rather than a local Faheem Code backend or local project files. Local LLMs can have limited functionality; use a capable model and GPU-backed server for the best experience.

News​

  • 2026/05/21: We now recommend Qwen3.6-35B-A3B as the first local model to try with Faheem Code. It is an open-weight MoE model built for agentic coding, supports a large context window, and is available through LM Studio, Ollama, vLLM, and SGLang.

Quickstart: running Faheem Code with a local LLM using LM Studio​

This guide explains how to serve a local LLM using LM Studio and have Faheem Code connect to it.

We recommend:

  • LM Studio as the local model server, which handles metadata downloads automatically and offers a simple, user-friendly interface for configuration.
  • Qwen3.6-35B-A3B as the LLM for software development. This model is optimized for agentic coding and works well with tool-heavy workflows like Faheem Code.

Hardware requirements​

Running Qwen3.6-35B-A3B requires:

  • A recent GPU with at least 24GB of VRAM for quantized variants, or multiple GPUs for full precision and larger context windows, or
  • A Mac with Apple Silicon with at least 64GB of unified memory for quantized variants

1. Install LM Studio​

Download and install the LM Studio desktop app from lmstudio.ai.

2. Download the model​

  1. Make sure to set the User Interface Complexity Level to "Power User", by clicking on the appropriate label at the bottom of the window.
  2. Click the "Discover" button (Magnifying Glass icon) on the left navigation bar to open the Models download page.

Screenshot: 01 lm studio open model hub.

  1. Search for "Qwen3.6-35B-A3B", confirm you're downloading from the official Qwen publisher, then proceed to download.

Screenshot: 02 lm studio download devstral.

  1. Wait for the download to finish.

3. Load the model​

  1. Click the "Developer" button (Console icon) on the left navigation bar to open the Developer Console.
  2. Click the "Select a model to load" dropdown at the top of the application window.

Screenshot: 03 lm studio open load model.

  1. Enable the "Manually choose model load parameters" switch.
  2. Select Qwen3.6-35B-A3B from the model list.

Screenshot: 04 lm studio setup devstral part 1.

  1. Enable the "Show advanced settings" switch at the bottom of the Model settings flyout to show all the available settings.
  2. Set "Context Length" to at least 22000 (for lower VRAM systems) or 32768 (recommended for better performance) and enable Flash Attention.
  3. Click "Load Model" to start loading the model.

Screenshot: 05 lm studio setup devstral part 2.

4. Start the LLM server​

  1. Enable the switch next to "Status" at the top-left of the Window.
  2. Take note of the Model API Identifier shown on the sidebar on the right.

Screenshot: 06 lm studio start server.

5. Start Faheem Code​

  1. Check the installation guide and ensure all prerequisites are met before running Faheem Code, then run:
docker run -it --rm --pull=always \
    -e AGENT_SERVER_IMAGE_REPOSITORY=ghcr.io/alsairy/faheem-code-agent-server \
    -e AGENT_SERVER_IMAGE_TAG=1.26.0-python \
    -e LOG_ALL_EVENTS=true \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v ~/.faheem-code:/.faheem-code \
    -p 3000:3000 \
    --add-host host.docker.internal:host-gateway \
    --name faheem-code-app \
    ghcr.io/alsairy/faheem-code-app:1.8
  1. Wait until the server is running (see log below):
Digest: sha256:e72f9baecb458aedb9afc2cd5bc935118d1868719e55d50da73190d3a85c674f
Status: Image is up to date for ghcr.io/alsairy/faheem-code-app:1.8
Starting Faheem Code...
Running Faheem Code as root
14:22:13 - faheemcode:INFO: server_config.py:50 - Using config class None
INFO:     Started server process [8]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:3000 (Press CTRL+C to quit)
  1. Visit http://localhost:3000 in your browser.

6. Configure Faheem Code to use the LLM server​

Once you open Faheem Code in your browser, you'll need to configure it to use the local LLM server you just started.

When started for the first time, Faheem Code will prompt you to set up the LLM provider.

  1. Click "see advanced settings" to open the LLM Settings page.

Screenshot: 07 Faheem Code open advanced settings.

  1. Enable the "Advanced" switch at the top of the page to show all the available settings.

  2. Set the following values:

    • Custom Model: openai/qwen/qwen3.6-35b-a3b (the Model API identifier from LM Studio, prefixed with "openai/")
    • Base URL: http://host.docker.internal:1234/v1
    • API Key: local-llm

  3. Click "Save Settings" to save the configuration.

Screenshot: 08 Faheem Code configure local llm parameters.

That's it. You can now start using Faheem Code with the local LLM server.

If you encounter any issues, let us know on support.

Community-reported notes and troubleshooting​

If Faheem Code behaves like a plain chatbot, refuses to use tools or files, or has constant failed tool calls with a local model, the issue may be with the model itself rather than your setup. Even with a large context window, some local models may struggle with reliable tool use.

Community-reported working models:

  • qwen2.5-coder-14b-instruct — reported to resolve chatbot-like behavior
  • qwopus3.5-27b-v3 Q8_0 (and similar retrained qwopus variants) — reported to work well with tool calls

If you're experiencing issues, try switching to one of these models before assuming the setup is broken.

Advanced: alternative LLM backends​

This section describes how to run local LLMs with Faheem Code using alternative backends like Ollama, Atomic Chat, SGLang, or vLLM — without relying on LM Studio.

Create an OpenAI-compatible endpoint with Ollama​

# ⚠️ WARNING: Faheem Code requires a large context size to work properly.
# When using Ollama, set OLLAMA_CONTEXT_LENGTH to at least 22000.
# The default (4096) is way too small — not even the system prompt will fit, and the agent will not behave correctly.
OLLAMA_CONTEXT_LENGTH=32768 OLLAMA_HOST=0.0.0.0:11434 OLLAMA_KEEP_ALIVE=-1 nohup ollama serve &
ollama pull qwen3.6:35b-a3b

Create an OpenAI-compatible endpoint with Atomic Chat​

Atomic Chat is an open-source desktop app for running local models (and optional cloud providers). It exposes a single OpenAI-compatible HTTP API on your machine, typically at http://127.0.0.1:1337/v1. See the upstream README for downloads, system requirements, and release notes.

1. Install and start Atomic Chat​

  1. Download Atomic Chat from atomic.chat or GitHub Releases.
  2. Open Atomic Chat and enable the local API server in the app settings (defaults may vary by version; the API is usually served on port 1337).
  3. Download and load a coding-capable model with a large context window. Faheem Code needs enough context for the system prompt and tools — use at least ~22k tokens, and 32k+ when your hardware allows (same guidance as LM Studio on this page).

2. Discover the model id Faheem Code must use​

Atomic Chat lists served models via the OpenAI-compatible GET /v1/models endpoint. From the same machine:

curl -s http://127.0.0.1:1337/v1/models | head

Use the id field of the model you have loaded as the suffix after openai/ in Faheem Code (see Configure Faheem Code (Alternative Backends) below).

3. Point Faheem Code at Atomic Chat​

Follow Run Faheem Code (Alternative Backends) and Configure Faheem Code (Alternative Backends) below. When Faheem Code runs inside Docker and Atomic Chat runs on the host, use:

  • Base URL: http://host.docker.internal:1337/v1
  • Custom Model: openai/<model-id-from-/v1/models> (prefix required, same convention as LM Studio on this page)
  • API Key: any placeholder string (for example local-llm) unless your Atomic Chat build requires a real key

If Faheem Code and Atomic Chat run on the same host without Docker for the web UI, you can use http://127.0.0.1:1337/v1 instead.

Atomic Chat also ships a Launch → Faheem Code integration that can configure LLM_BASE_URL, LLM_MODEL, and LLM_API_KEY for the Faheem Code CLI automatically.

Troubleshooting​

  • Connection refused from Docker: confirm Atomic Chat is running, the local server is enabled, and your docker run includes --add-host host.docker.internal:host-gateway as in local setup.
  • Wrong model errors: the Custom Model string must match an id returned by GET /v1/models after the openai/ prefix.
  • Agent ignores tools or acts like a chatbot: try a stronger coding model or a larger context window; see Community-Reported Notes and Troubleshooting on this page.

Create an OpenAI-compatible endpoint with vLLM or SGLang​

First, download the model checkpoint:

huggingface-cli download Qwen/Qwen3.6-35B-A3B --local-dir Qwen/Qwen3.6-35B-A3B

Serving the model using SGLang​

SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python3 -m sglang.launch_server \
    --model Qwen/Qwen3.6-35B-A3B \
    --served-model-name Qwen3.6-35B-A3B \
    --port 8000 \
    --tp 2 --dp 1 \
    --host 0.0.0.0 \
    --api-key mykey --context-length 131072

Serving the model using vLLM​

vllm serve Qwen/Qwen3.6-35B-A3B \
    --host 0.0.0.0 --port 8000 \
    --api-key mykey \
    --tensor-parallel-size 2 \
    --served-model-name Qwen3.6-35B-A3B \
    --enable-prefix-caching

If you are interested in further improved inference speed, you can also try Snowflake's version of vLLM, ArcticInference, which can achieve up to 2x speedup in some cases.

  1. Install the Arctic Inference library that automatically patches vLLM:
pip install git+https://github.com/snowflakedb/ArcticInference.git
  1. Run the launch command with speculative decoding enabled:
vllm serve Qwen/Qwen3.6-35B-A3B \
    --host 0.0.0.0 --port 8000 \
    --api-key mykey \
    --tensor-parallel-size 2 \
    --served-model-name Qwen3.6-35B-A3B \
    --speculative-config '{"method": "suffix"}'

Run Faheem Code (alternative backends)​

Using Docker​

Run Faheem Code using the official docker run command.

Using development mode​

Use the instructions in Development.md to build Faheem Code.

Start Faheem Code using make run.

Configure Faheem Code (alternative backends)​

Once Faheem Code is running, open the Settings page in the UI and go to the LLM tab.

  1. Click "see advanced settings" to access the full configuration panel.
  2. Enable the Advanced toggle at the top of the page.
  3. Set the following parameters, if you followed the examples above:
    • Custom Model: openai/<served-model-name>
      • For Ollama: openai/qwen3.6:35b-a3b
      • For SGLang/vLLM: openai/Qwen3.6-35B-A3B
      • For Atomic Chat: openai/<model-id-from-/v1/models> (see Atomic Chat above)
    • Base URL: http://host.docker.internal:<port>/v1 Use port 11434 for Ollama, 1337 for Atomic Chat (default), or 8000 for SGLang and vLLM.
    • API Key:
      • For Ollama or Atomic Chat: any placeholder value (e.g. dummy, local-llm) unless your server requires a real key
      • For SGLang or vLLM: use the same key provided when starting the server (e.g. mykey)