I just discovered a stylegan surprise. In stylegan2, the discriminator learning rate can't be set. G_lrate is used for both the generator *and* the discriminator's rate. Everyone's been fooling themselves. It's currently impossible to set the discriminator's learning rate.
My tips are, try the two-time update rule - set the generator learning rate to half of the discriminator learning rate - try increasing the minibatch size - and, yes, if all else fails, also try lowering the learning rate.

6:54 AM · Feb 19, 2020

4
4
2
29
To confirm this, you can grep the stylegan2 codebase for D_lrate and see that it's not used anywhere.
2
0
0
4
It was possible in stylegan1. Looks like stylegan2 broke the ability to set the discriminator's learning rate. Here's stylegan1 (left) vs stylegan2 (right). Notice that in stylegan2, G_lrate is incorrectly used as the learning rate for D_train_op:
3
0
1
6
Replying to @theshawwn
In all the Colab notebooks I used for training, both run-training and training-loop are either %writefile’ed after being edited or heavily edited in the fork (like @pbaylies’s WikiArt fork). Will certainly have a double check
0
0
0
5
Replying to @theshawwn
Good catch. Thought I was going insane the other day trying to tweak my learning rates.
0
0
0
1
Replying to @theshawwn
0
0
0
2
GIF