How to Merge a LoRA Into a Model Checkpoint in 2026

Merging a LoRA into a base checkpoint means taking two separate files, the full model weights and a small trained adapter, and combining them into one permanent set of weights. The technique is old news to anyone who has trained or collected LoRAs for a while, and it has not gone anywhere in 2026. It is still the same weighted merge under the hood across Flux, SDXL, and Stable Diffusion 1.5 and 2, and it still solves a real problem: turning a base model plus an adapter into a single file you can hand off or build further training on top of.

What has changed is the surrounding context. Stacking LoRAs at inference, loading several adapters into one generation with individual strength sliders, has become the default way most people work day to day, since it is reversible and fast to iterate on. Permanent merging has not disappeared, it is now a tool you reach for in specific situations. This guide covers both: what the merge does mathematically, when it beats stacking, a complete Python script to run it, and a precision issue that trips people up on newer quantized checkpoints.

What weighted merging actually does to the weights

A LoRA does not store a full copy of the model. It stores two small matrices per targeted layer, the down projection and the up projection, plus a scaling value called alpha. Multiplying those two matrices together reconstructs a delta, a change in weight, that is much smaller in file size than the full layer it modifies but produces the same effect once added back in.

The standard formula for baking that delta permanently into a checkpoint is:

W_merged = W_base + (alpha / rank) * (up @ down)

W_base is the original weight tensor from the checkpoint. up and down are the two LoRA matrices for that same layer. rank is the size of the LoRA's bottleneck, usually a small number like 8, 16, or 32. alpha is a scaling value baked in at training time that controls how strongly the effect is applied relative to its rank. Multiply the two matrices together, scale by alpha over rank, add it to the base weight, and you get a tensor that behaves as if the LoRA had been active the whole time, without needing the LoRA file present going forward.

This is exactly what community LoRA merger scripts for ComfyUI and Hugging Face's diffusers library do under the hood. Nothing about this baseline has been replaced. What has been added on top, mostly for combining three or more LoRAs or models at once, are mergekit style methods such as SLERP, TIES, and DARE, which interpolate or selectively combine weights instead of a straight sum. For the common case this guide covers, one LoRA folded into one checkpoint, the plain weighted merge above is still correct.

Merging permanently versus stacking at inference

Before running any script, it is worth being honest about whether merging is actually the right move, because the two approaches solve different problems.

Stacking LoRAs at inference, using a node or extension that loads multiple adapters with independent strength values, wins when you are still iterating. You can turn a LoRA off, dial its strength down, or swap it for a different one between generations without touching a file on disk, which is why stacking has become the default habit for people still experimenting rather than shipping something fixed.

Permanent merging earns its place in a narrower set of situations:

• Single file distribution. You want to hand someone, or a pipeline, one checkpoint file rather than a base model plus a LoRA file plus instructions on what strength to load it at.
• Baking in a speed LoRA. Lightning and similar step reduction LoRAs are almost always meant to be permanent, so merging one in once and reusing the result saves a load step every run.
• Fixing a base before further training. If you are about to train a new LoRA on top of a model that already has a style or correction LoRA applied, merging that correction in first gives the new training a stable base instead of two adapters interacting at once.

Outside of those cases, think twice before merging. Community sentiment among people who train and share LoRAs, on Civitai and similar communities, leans skeptical of merging as a default habit, and for good reason: it is a one way operation. Once alpha is baked into the weights, you cannot dial the strength back down or remove the effect without downloading the original base checkpoint again. Keep an unmerged copy of your base model around regardless of which path you choose.

Where this applies: Flux, SDXL, and beyond

Everything above applies regardless of which diffusion model family you are working with. Flux, a common target for hosted generation today, follows the same down projection, up projection, alpha structure as everything before it. SDXL and the older Stable Diffusion 1.5 and 2 lines work identically, the only difference being which layers a given architecture exposes for a LoRA to target, not the merge math itself.

The one place this guide's scope ends is combining more than one LoRA into a single checkpoint at once. A weighted merge run twice works for two LoRAs that do not conflict, but stacking three or more is where mergekit style techniques, SLERP for interpolation, TIES for resolving conflicting updates, DARE for dropping redundant ones, start to outperform a repeated sum. If your goal is combining several models or LoRAs together, those are worth researching next. This guide focuses on the simpler, far more common single LoRA case.

The precision trap on quantized checkpoints

One nuance that did not exist a few years ago and now catches people regularly: low bit checkpoints. Flux is large enough that a lot of people run it in NF4 or FP8 quantized form to fit consumer GPU memory, rather than the full 16 bit weights it was originally released in.

Quantized formats are not just a smaller floating point type you can add a delta to directly. NF4 is a packed, block scaled representation, not a plain tensor of numbers, so adding a LoRA delta straight onto a quantized weight throws a shape or dtype error, or worse, silently produces a corrupted result. The fix: dequantize the base weight back to fp16 or bf16 first, matching whatever precision the LoRA's matrices are stored in, do the merge math there, and only quantize again afterward if you actually need the smaller footprint. Libraries like bitsandbytes handle NF4 dequantization, and torchao or optimum quanto cover FP8. If a merge script fails with a dtype mismatch you cannot explain, check whether the base checkpoint is quantized before assuming the LoRA file is broken.

Setting up the Python environment

With the theory covered, here is the actual tutorial. Start with a clean virtual environment so this does not collide with whatever else is installed on your machine.

python -m venv lora-merge-env
source lora-merge-env/bin/activate      # Windows: lora-merge-env\Scripts\activate

pip install torch --index-url https://download.pytorch.org/whl/cu121
pip install safetensors accelerate diffusers huggingface_hub

Swap the PyTorch command for the CPU only wheel if you do not have an NVIDIA GPU, though merging on CPU is noticeably slower for large checkpoints like Flux and SDXL. safetensors does the actual loading and saving below; diffusers and accelerate are optional, worth having if you want to load the merged result straight into a pipeline afterward to test it.

Getting your files in order

You need two files: the base checkpoint, and the LoRA safetensors file trained against that same base. Both need to actually match, a LoRA trained against SDXL will not merge meaningfully into a Flux checkpoint, since the layer names and shapes will not line up.

It is worth peeking at the LoRA's internal key names before writing a merge script blind, since not every file uses the same naming convention:

from safetensors import safe_open

with safe_open("my_lora.safetensors", framework="pt") as f:
    for key in list(f.keys())[:10]:
        print(key)

Two naming conventions cover almost everything you will encounter. Kohya style, common for community trained LoRAs shared for ComfyUI or Automatic1111, uses .lora_down.weight and .lora_up.weight suffixes, often with a lora_unet_ or lora_te_ prefix. PEFT style, common for LoRAs trained directly with diffusers, uses .lora_A.weight and .lora_B.weight instead. The script below handles both. One caveat: Kohya prefix underscores do not map cleanly back to dots in the base checkpoint's key names, since module names themselves can contain underscores, like proj_in. Naively replacing every underscore with a dot corrupts those names, so convert it with your training tool's own conversion script first rather than guessing at the mapping.

Writing the merge script

Save the following as merge_lora.py. It loads both safetensors files, walks every LoRA matched pair, applies the weighted merge formula from earlier, and writes out a new checkpoint.

import argparse
import torch
from safetensors.torch import load_file, save_file

def merge_lora_into_checkpoint(base_path, lora_path, output_path, strength=1.0, dtype=torch.float16):
    base_state = load_file(base_path)
    lora_state = load_file(lora_path)

    down_keys = [
        k for k in lora_state
        if k.endswith(".lora_down.weight") or k.endswith(".lora_A.weight")
    ]

    merged = dict(base_state)
    applied, skipped = 0, 0

    for down_key in down_keys:
        if down_key.endswith(".lora_down.weight"):
            root = down_key[: -len(".lora_down.weight")]
            up_key = root + ".lora_up.weight"
        else:
            root = down_key[: -len(".lora_A.weight")]
            up_key = root + ".lora_B.weight"

        alpha_key = root + ".alpha"
        base_key = root + ".weight"

        if base_key not in merged or up_key not in lora_state:
            skipped += 1
            continue

        down = lora_state[down_key].to(torch.float32)
        up = lora_state[up_key].to(torch.float32)
        rank = down.shape[0]
        lora_alpha = lora_state[alpha_key].item() if alpha_key in lora_state else rank
        scale = (lora_alpha / rank) * strength

        delta = (up @ down) * scale
        target = merged[base_key].to(torch.float32)
        merged[base_key] = (target + delta.reshape(target.shape)).to(dtype)
        applied += 1

    print(f"Applied {applied} tensors, skipped {skipped} with no matching base key")
    save_file(merged, output_path, metadata={"format": "pt"})

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--base", required=True, help="Path to the base checkpoint safetensors file")
    parser.add_argument("--lora", required=True, help="Path to the LoRA safetensors file")
    parser.add_argument("--output", required=True, help="Path to write the merged checkpoint")
    parser.add_argument("--strength", type=float, default=1.0, help="Extra multiplier on top of the LoRA's own alpha scaling")
    args = parser.parse_args()

    merge_lora_into_checkpoint(args.base, args.lora, args.output, args.strength)

A few details here matter more than they look. Everything is upcast to float32 for the addition, then cast back to the target dtype, since doing the math directly in fp16 loses precision and produces a noisier merge. The strength argument is separate from alpha on purpose. Alpha is baked into the LoRA at training time; strength is an extra multiplier you control at merge time if you want the effect applied at, say, seventy percent rather than full intensity. Linear layers merge cleanly with the matrix multiply shown here. Convolutional layers store their rank dimension with extra kernel dimensions attached, so the shapes need an extra reshape first, handled here by the final .reshape(target.shape) call.

Running the merge and checking the output

Run it from the command line, pointing at your actual files:

python merge_lora.py \
  --base flux1_dev_fp16.safetensors \
  --lora my_style_lora.safetensors \
  --output flux1_dev_merged.safetensors \
  --strength 0.8

The printed count of applied versus skipped tensors is your first sanity check. If skipped is high relative to applied, or applied is zero, the key names are not lining up, which almost always means the naming convention mismatch described earlier rather than anything wrong with the merge math.

Once written, load the merged file into whatever you normally generate with, ComfyUI or a diffusers pipeline, and run the same prompts you would normally use with the base model plus the LoRA loaded separately at the same strength. The two results should look the same. If they clearly do not, check the strength value first, then the alpha handling, before assuming the formula itself is wrong.

Common errors and what they mean

A shape mismatch on the matrix multiply almost always means the LoRA was trained against a different architecture or version of the base model, even if the file names look similar. A dtype error usually points to the quantization issue covered earlier, check whether the base checkpoint is NF4 or FP8. Zero tensors applied means the key names never matched, go back to listing the LoRA's keys against the base checkpoint's. A merge that runs cleanly but shows no visible difference usually means the strength was effectively zero, or alpha was missing and silently defaulted to the rank, canceling the scaling out entirely.

If you would rather not manage local checkpoints

None of the above is complicated once you have done it a couple of times, but it is still real infrastructure: a Python environment, multiple gigabyte checkpoint files, and enough GPU memory to load and merge them. If what you actually want most days is to generate images with Flux or a similar model rather than own and merge the weights yourself, Enhance AI runs a hosted playground with 250+ models, including GPT Image 2, Nano Banana 2, Seedream, Qwen Image, Recraft V4, and Flux, all in one account with no local setup. It will not merge or manage LoRA checkpoints for you, that stays a local tooling job, but it is free to start, runs on one time payments from $19, and gives new accounts free credits with no card required.

Frequently asked questions

Is merging a LoRA into a checkpoint still the right technique in 2026?

Yes, for one LoRA folded into one checkpoint. The weighted merge formula here is still what most merge tools use under the hood. It has been extended rather than replaced, mergekit style methods like SLERP, TIES, and DARE matter more once you are combining three or more LoRAs or models at once.

Should I merge permanently or just stack LoRAs at inference?

Stack while you are still iterating, since it is reversible and lets you adjust strength between runs. Merge permanently for a single distributable file, for baking in a speed or Lightning LoRA you always use at full strength, or when stabilizing a base checkpoint before further training on top of it.

Does this work for SDXL and Stable Diffusion 1.5, or only Flux?

The merge math is identical across Flux, SDXL, and Stable Diffusion 1.5 and 2. Only which layers a LoRA targets changes between architectures, not the formula used to fold it into the base weights.

Why did my merge script apply zero tensors?

Almost always a key naming mismatch. Kohya style files use a prefix like lora_unet_ with underscores standing in for dots, and naively converting them back breaks module names that legitimately contain underscores, like proj_in. List the keys in both files and compare them directly.

Why does merging fail on an NF4 or FP8 checkpoint?

Quantized weights are not plain floating point tensors you can add a delta to directly. Dequantize the base weight to fp16 or bf16 first, matching the LoRA's own precision, merge in that shared precision, and quantize again afterward only if you need the smaller file size.

Can I merge more than one LoRA into the same checkpoint?

A weighted merge run twice works for two LoRAs that do not meaningfully conflict. Beyond that, mergekit style approaches built for combining several sets of weights at once tend to produce cleaner results than repeating a simple sum several times over.

Try it yourself

The formula has not changed, and neither has the reason to reach for it: a single portable checkpoint file instead of a base model plus a LoRA plus instructions on what strength to load it at. Set up the environment, check your key names before you run anything, and merge with confidence once the applied tensor count matches what you expect.