Dropout

08/21/2023

• Dropout
• https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html
• https://medium.com/@amarbudhiraja/https-medium-com-amarbudhiraja-learning-less-to-learn-better-dropout-in-deep-machine-learning-74334da4bfc5


• torch.nn.Dropout(p=0.5, inplace=False)
• During training, randomly zeroes some of the elements of the input tensor with probability p.
• If inplace is set to True, will do this operation in-place


(a) before applying Dropout (b) after applying Dropout


• The concept for killing some nodes and learning (different nodes die for each batch)
  • No zeroing when evaluating, testing
  • Thing to prevent overfitting
• Furthermore, the outputs are scaled by a factor of 1 / (1 - p) during training
  • The purpose of scaling is to maintain the expected value of each node
  • When p=0.5, each node has a 50% probability of being disabled
  • To reward for this, the values of the surviving nodes are scaled by a factor of 2.

    import torch
    torch.random.manual_seed(777)

    rand_3x3 = torch.rand(size=(3, 3))
    print(rand_3x3)
    # > tensor([[0.0819, 0.4911, 0.4033],
    #           [0.3859, 0.8813, 0.8811],
    #           [0.7242, 0.5033, 0.8249]])

    dropout = torch.nn.Dropout(p=0.5, inplace=False)
    rand_3x3_dropped = dropout(rand_3x3)
    print(rand_3x3_dropped)
    # > tensor([[0.1638, 0.0000, 0.0000],
    #           [0.7718, 1.7626, 0.0000],
    #           [1.4485, 0.0000, 0.0000]])

    dropout.eval()
    print(dropout(rand_3x3))
    # > tensor([[0.0819, 0.4911, 0.4033],
    #           [0.3859, 0.8813, 0.8811],
    #           [0.7242, 0.5033, 0.8249]])

---