Cuda Out of Memory Training Dreambooth w/ Stable Diffusion 2.1

This is a common error:

CUDA out of memory. Tried to allocate 12.66 GiB (GPU 0; 23.69 GiB total capacity; 15.57 GiB already allocated;

You’ve read all the blogs and Reddit that tell you to set something like this:

export PYTORCH_CUDA_ALLOC_CONF="garbage_collection_threshold:0.6,max_split_size_mb:64"

But you still get out of memory errors, particularly when trying to use Stable Diffusion 2.1.

Are you trying to use the following:

  --with_prior_preservation --prior_loss_weight=1.0 

Because that could very well be your problem.  It was for me.