Z-Image ncnn Vulkan

❗ ❗ ❗ This software is in the early development stage, it may bite your cat

ncnn implementation of Z-Image image generator, with LanPaint inpaint/outpaint and ControlNet support.

zimage-ncnn-vulkan uses ncnn project as the universal neural network inference framework.

Download

Download Windows/Linux/macOS Executable for Intel/AMD/NVIDIA/Apple-Silicon GPU

https://github.com/nihui/zimage-ncnn-vulkan/releases

This package includes all the binaries required. It is portable, so no CUDA or PyTorch runtime environment is needed :)

prepare model files

Download the z-image-turbo and z-image model folders to the same directory as the executable file

https://huggingface.co/nihui-szyl/z-image-ncnn/tree/main

For ControlNet, download the z-image-control model folder to the same directory as z-image-turbo. For Tile ControlNet upscale, download the z-image-control-tile model folder too.

zimage-ncnn-vulkan
z-image-turbo/
z-image-control/
z-image-control-tile/

About Z-Image

Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer

https://github.com/Tongyi-MAI/Z-Image

Usages

requirements

• Minimum (Linux / macOS): 16GB RAM, any Vulkan capable GPU

• Minimum (Windows):

  Due to WDDM limitations: Vulkan applications can only use half of the system RAM.

  The following condition shall be met: (Half of system RAM) + (GPU memory) >= 16GB

  Examples of valid combinations:

  • Any amount RAM, 16GB dedicated GPU
  • 16GB RAM, 8GB dedicated GPU
  • 24GB RAM, 4GB dedicated GPU
  • 32GB RAM, any Vulkan capable GPU

• Recommended: 32GB RAM, 16GB dedicated GPU with tensorcore/matrix hardware

Example Command

Text to image

zimage-ncnn-vulkan.exe -p "風的彷徨." -o output.png

LanPaint inpaint

zimage-ncnn-vulkan.exe -i input.png -k mask.png -p "a cute girl" -o output.png

LanPaint outpaint

zimage-ncnn-vulkan.exe -i input.png -x 128,128,128,128 -p "a cute girl" -o output.png

ControlNet

zimage-ncnn-vulkan.exe -c pose.png -p "a cute girl, full body" -o output.png

Tile ControlNet upscale

zimage-ncnn-vulkan.exe -c lowres.png -t -s 2048,2048 -p "a cute girl" -o output.png

Full Usages

Usage: zimage-ncnn-vulkan -p prompt -o outfile [options]...

  -h                   show this help
  -p prompt            prompt (default=rand)
  -n negative-prompt   negative prompt (optional)
  -o output-path       output image path (default=out.png)
  -i input-image       input image for inpaint (optional)
  -k mask-image        inpaint mask, white=paint black=preserve (optional)
  -x l,t,r,b           outpaint by expanding input canvas (optional)
  -c control-image     control image for ControlNet (optional)
  -w control-scale     ControlNet scale (default=1.0)
  -t                   Tile ControlNet upscale mode (optional)
  -s image-size        image resolution (default=1024,1024)
  -l steps             denoise steps (default=auto)
  -r random-seed       random seed (default=rand)
  -m model-path        z-image model path (default=z-image-turbo)
  -g gpu-id            gpu device to use (-1=cpu, default=auto)
  -b batch-size        batched generation (default=1)

LanPaint inpaint/outpaint requires -i input-image. Inpaint uses -k mask-image, where white means repaint and black means preserve. Outpaint uses -x left,top,right,bottom.

ControlNet uses -c control-image. The control image size must match the output image size. ControlNet currently requires the z-image-turbo model and the z-image-control model folder.

Tile ControlNet upscale uses -c lowres-image -t -s target-width,target-height. The low resolution image is resized internally to the target size and used as tile control. It loads ControlNet weights from z-image-control-tile.

If you encounter a crash or error, try upgrading your GPU driver:

• Intel: https://downloadcenter.intel.com/product/80939/Graphics-Drivers
• AMD: https://www.amd.com/en/support
• NVIDIA: https://www.nvidia.com/Download/index.aspx

Build from Source

1. Clone this project with all submodules

git clone https://github.com/nihui/zimage-ncnn-vulkan.git
cd zimage-ncnn-vulkan
git submodule update --init --recursive --depth 1

2. Build with CMake

mkdir build
cd build
cmake ../src
cmake --build . -j 4

Sample Images

expand for full command zimage-ncnn-vulkan.exe -p "A half-length portrait in the warm light of a convenience store late at night. An East Asian beauty, holding milk, meets your gaze in front of the freezer." -r 2065820720

expand for full command zimage-ncnn-vulkan.exe -p "A Suzhou garden at night under a bright autumn moon, silvery light bathing the pavilions and rocks, paper lanterns glowing warmly along the corridors, the pond reflecting the moon and lantern light, cool color palette with accents of warmth, mysterious and poetic atmosphere." -r 1593388185

expand for full command zimage-ncnn-vulkan.exe -p "Cinnamoroll, Sanrio character, white puppy with long floppy ears, blue eyes, chubby cheeks, plump tail like a cinnamon roll, simple blue bow on right ear, high-quality anime figure, prize figure style, seamless joints, glossy paint finish, detailed sculpting, sitting pose with head tilted, innocent expression, standing on a fluffy cloud-shaped display base, soft gradient pastel background, studio lighting, product photography, sharp focus, shallow depth of field" -r 1362178531

expand for full command zimage-ncnn-vulkan.exe -p "a cute girl taking a mirror selfie, full-body view in a full-length mirror, holding a large smartphone in front of her face, the phone screen is visible and glowing, slightly obscuring her eyes or lower face, long twin tails, soft makeup, blush, wearing a trendy and cute outfit, in a cozy, clean bedroom with soft lighting, mirror may have a decorative frame, shallow depth of field, digital photo style, cinematic lighting, sharp focus on the phone and reflection, natural skin texture, Instagram aesthetic, candid feeling" -r 2029158043

expand for full command zimage-ncnn-vulkan.exe -p "風的彷徨." -r 77	expand for full command zimage-ncnn-vulkan.exe -p "風的彷徨." -r 777	expand for full command zimage-ncnn-vulkan.exe -p "Anya Forger hugging a large white dog (Bond), both looking happy, in a park, sunny day, grass field, action shot, from Spy x Family" -r 499008670
expand for full command zimage-ncnn-vulkan.exe -p "traditional Chinese handscroll painting in the style of “清明上河图”, horizontal panorama, bustling riverside cityscape in Northern Song Dynasty, Bianliang city, with Rainbow Bridge at the center, boats on the river, shops along the street, teahouses, inns, hundreds of diverse people engaged in daily activities, travelers, merchants, scholars, entertainers, children playing, ink and color on silk, fine brushwork for architecture and figures, subtle expressive brushwork for trees and landscape, soft natural colors, high aerial viewpoint, continuous narrative, lively, prosperous, detailed, harmonious, capturing a complete day of city life, extremely wide aspect ratio, vanishing perspective, Chinese seals and calligraphy on the margins, slight aged paper texture" -r 1321390901 -s 2048,512

LanPaint inpaint

expand for full command zimage-ncnn-vulkan.exe -i 2065820720.jpg -p "Holding a Hachiware plushie, the friend of both Chiikawa and Usagi" -k mask.jpg
	➡️

LanPaint outpaint

expand for full command zimage-ncnn-vulkan.exe -i 2065820720.jpg -p "A half-length portrait in the warm light of a convenience store late at night. An East Asian beauty, holding milk, meets your gaze in front of the freezer." -x 128,128,128,128
	➡️

ControlNet pose

expand for full command zimage-ncnn-vulkan.exe -p "A half-length portrait in the warm light of a convenience store late at night. An East Asian beauty, holding milk, meets your gaze in front of the freezer." -c pose.jpg
	➡️

ControlNet canny

expand for full command zimage-ncnn-vulkan.exe -p "A half-length portrait in the warm light of a convenience store late at night. An East Asian beauty, holding milk, meets your gaze in front of the freezer." -c canny.jpg
	➡️

ControlNet gray

expand for full command zimage-ncnn-vulkan.exe -p "A photorealistic square image of a traditional Japanese street in Kyoto at dusk after a light rain." -c gray.jpg
	➡️

ControlNet tile low resolution

expand for full command zimage-ncnn-vulkan.exe -p "Aerial view of Arashiyama in autumn, a river winding through the center, flanked by mountains ablaze with vibrant, multicolored foliage." -c lowres.jpg -t -s 1024,1024
	➡️

Original Z-Image Project

• https://github.com/Tongyi-MAI/Z-Image

Other Open-Source Projects Referenced

• https://github.com/scraed/LanPaint for inpaint and outpaint pipeline
• https://huggingface.co/alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1 for Z-Image ControlNet model

Other Open-Source Code Used

• https://github.com/Tencent/ncnn for fast neural network inference on ALL PLATFORMS
• https://github.com/futz12/ncnn_llm for BPE tokenizer
• https://github.com/webmproject/libwebp for encoding and decoding Webp images on ALL PLATFORMS
• https://github.com/libjpeg-turbo/libjpeg-turbo for encoding and decoding JPEG images on ALL PLATFORMS
• https://github.com/pnggroup/libpng for encoding and decoding PNG images on ALL PLATFORMS
• https://github.com/zlib-ng/zlib-ng for encoding and decoding PNG images on ALL PLATFORMS
• https://github.com/tronkko/dirent for listing files in directory on Windows