lora-ortho

Tools for manipulating the direction of LoRA weight deltas, independent of their magnitude.

Both tools operate directly on LoRA .safetensors files (lora_up/lora_down/alpha factors) without ever materializing the full weight matrices, and are non-destructive — they always write to a new file via --save_to.

The problem: banding

When stacking multiple LoRAs, "banding" happens when two LoRAs' deltas point in a similar direction in weight space: dW = (alpha/r) * lora_up @ lora_down. If <dW_a, dW_b> is large, the deltas add constructively and reinforce each other instead of contributing independent information.

Scripts

orthogonalize.py

Rotates a target LoRA's deltas so they're orthogonal to a reference LoRA's dominant direction, killing the inner product between them without changing the target's magnitude or rank.

python orthogonalize.py \
    --model target.safetensors \
    --orthogonalize-to reference.safetensors \
    --save_to target-ortho.safetensors \
    [--k full] [--side left] [--dry-run]

• --k full (default) projects out the reference's entire column space for exact orthogonality; pass an integer to project out only the top-N singular directions (less invasive, partial).
• --side chooses which factor to project: left (lora_up, least invasive), right (lora_down), or both.
• --dry-run prints the planned self-check stats without writing anything.

Only lora_up/lora_down (and matching modules) are touched; alpha is left untouched. Per-module self-checks report the resulting cos(new, ref) and retained ||dW|| so you can confirm the rotation worked as expected.

random_repoint.py

A causal control for the banding question: re-points a LoRA's deltas into random orthonormal directions while preserving each module's exact Frobenius norm and singular-value spectrum. Same magnitude and "strength," different direction.

python random_repoint.py \
    --model source.safetensors \
    --save_to source-repointed.safetensors \
    --seed 777 [--dry-run]

zero_modules.py

Zero out lora_up/lora_down of selected modules by block, role (the non-block part of the name, e.g. mlp_layer1), and/or exact module name. Selections are combined as a union, so you can cut precisely what you don't want before shrinking or stacking.

A module name looks like lora_unet_blocks_<block>_<role>, e.g. lora_unet_blocks_27_mlp_layer1 (block 27, role mlp_layer1).

python zero_modules.py \
    --model gigi-goat-AB10-01.safetensors \
    --blocks 0,1,2,3,4,5,6,7,8,20,21,22,23,24,25,26,27 \
    --save_to gigi-zeroed-0-8_20-27.safetensors

• --blocks N,M,... — zero every module in those blocks.
• --roles p1,p2,... — zero a role across all blocks (prefix match), e.g. mlp_layer1, self_attn, cross_attn_q_proj.
• Combine --blocks + --roles to zero a role only in those blocks.
• --modules name1,name2,... — zero specific modules (exact name or unique substring), e.g. lora_unet_blocks_27_self_attn_q_proj.
• --dry-run / --list — preview the matched modules without writing.

alpha keys are left untouched; the metadata comment and hashes are updated.

style_debias.py and style_aware_resize.py

De-bias a LoRA against a set of same-reg sibling LoRAs (--style-subspace DIR...) and/or shrink it. style_debias.py --mode pca|reachable removes the shared style direction (pca keeps rank, reachable removes the sibling column/row-space union); style_aware_resize.py does de-style + re-rank in one pass. See RESIZING.md for the size/character/style tradeoffs.

Setup

This project imports the library package from sd-scripts:

# assumes sd-scripts is installed in the same parent directory as this repo
ln -s ../sd-scripts/library library
pip install -r requirements.txt

Functional tests

End-to-end tests are written in Gherkin (features/*.feature) and run with behave. Each scenario generates tiny synthetic LoRAs in a temp directory, runs a script as a subprocess, and checks the output file: which modules changed, delta orthogonality/spectra, alpha, metadata and hashes.

pip install -r requirements-test.txt   # after the Setup steps above (CPU torch is enough)
behave                                 # whole suite, ~2 minutes
behave features/zero_modules.feature   # one feature
behave --tags=known_bug                # scenarios that reproduce known defects (excluded by default)
BEHAVE_KEEP_TMP=1 behave -n "Zero whole blocks"   # keep a scenario's files for inspection