Control Layers

Control Layers give you control over a generation's composition and style. Its innovative feature set is integrated in a new canvas on the Generation tab. This article describes the each feature, how they affect generation, and how to navigate and use Control Layers.

Access Control Layers by clicking the button on the Generation tab. Switch between the Image Viewer and Control Layers by clicking the navigation button in the top right of the screen, or using the Z hotkey.

Different layer types serve different purposes. Some globally affect the whole image, while others give you control over specific regions of the image.

Global Control Layers

There are three global layer types that apply to the entire image.

Global Initial Image

Use an initial image - either an upload, or a previously generated image - to provide a starting point for the denoising process. This process is commonly known as Image to Image.

During generation, the initial image's structure and color will be used. Lower denoising strengths will result in outputs that closely resemble the initial image, while higher denoising strengths bias the output towards your prompts and controls.

You may only have one initial image at a time.

Learn more about Initial Images here.

Global Control Adapters

You may add any number of control adapter layers.

Control images are overlaid on the canvas with adjustable opacity and a transparency filter. For control adapters like Canny or Depth, which directly impact the structure of the image, this preview can be used to visualize how the control will be applied.

By default, control images will resize themselves to fit the canvas. Click the Use Size button on the control image to instead have the canvas resize itself to fit that image.

Learn more about Control Adapters here. 

Global IP Adapters

Global IP Adapter layers apply an image prompt to the whole image. You can add any number of global image prompts, in addition to regional image prompts, but the interaction may become unpredictable as more are added and are utilized together.

Learn more about IP Adapters here. 

Regional Guidance Layers

Regional guidance allows guidance to be applied selectively to certain areas of the image, drawn by the user. The different types of guidance work the same way as their global counterparts, but their effect is limited to the specified region.

• Positive Text Prompt: Biases generation for the specified region towards the content of the text prompt. A regional guidance layer has a positive prompt by default, but you can delete it.
• Negative Text Prompt: Biases generation for the specified region away from the content of the text prompt.
• IP Adapter: Apply an image prompt to the specified region. You may add any number of image prompts for a region, but more than one or two often leads to unpredictable outputs.
  
  

Keep in mind that while regional guidance allows for significant influence over the generation, it is interacting with the global guidance (text prompts and IP adapters). The global guidance may overpower regional guidance, especially if they conflict. Image prompts tend to overpower text prompts. Use text prompt and image prompt weighting to find the best balance of guidance.

Canvas Tools

Each regional guidance layer has a mask, drawn onto the canvas, representing the region in which the layer's guidance will be applied.

Begin by clicking on the Regional Guidance layer you want to edit, then select a tool to edit the mask for the region.

• Brush: Add to the region's mask by drawing with a brush.
• Eraser: Erase parts of the region's mask with an eraser.
• Rectangle: Drag to add a rectangular to the region's mask. Like brush strokes, rectangles can be erased with the eraser.
• Move: Reposition the region's mask.
• Undo/Redo: Undo or redo the last action.
  
  

Regional guidance layers have a mask preview color, but this does not affect generation - it's there to help you differentiate between regions.

Auto Negative

Enabled by default, this feature helps to isolate regional guidance, preventing prompts from "bleeding" into other areas of the image. For example, if you used "dog" as the positive prompt for a region, Auto Negative would add "dog" to the negative prompt for everything outside that region.

Troubleshooting

If your regional guidance layer isn't having the intended effect, you can try a few things to improve the result.

As an example, let's say you wanted an image of a cat on a windowsill, with a motorcycle seen through the window. The global positive prompt is "a cat on a windowsill", and a regional guidance layer with "a motorcycle" as the positive prompt.

• Global Prompts vs Regional Prompts: The global prompts can overpower regional prompts, especially when the prompts are unrelated. In this example, "a cat on a windowsill" and "a motorcycle" are unrelated, and the model may have a hard time finding a region with "a motorcycle" in "a cat on a windowsill". Try adding "a motorcycle" to the global prompt (e.g. "a cat on a windowsill, a motorcycle seen through the window"). You could also use upweighting syntax in the region (e.g. "a motorcycle+++").
• Control Layers: Regional guidance works best when there are clear composition elements to which the regional prompts can be applied. In this example, a control adapter layer with a sketch of the whole scene, including cat, window and motorcycle, will help the process to better realize the desired output.
• Negative Prompting: In addition to the auto negative feature, you may find it useful to add explicit negative prompt regions. For example, if you were getting cat on a motorcycle, you could add "cat" to the negative prompt for the motorcycle region.
• LoRAs: LoRAs work with regional prompts, but you may need to use upweighting syntax on the trigger phrases to get the desired effect.