I’ve just been playing around with learning rates on a data set, using a validation set to give me ‘loss’ and ‘val_loss’ outputs. I will include different learning rates and the resultant loss and val_loss graphs, each of which was run for 100 epochs.
Learning rate of 0.1 (100 epochs)
model.compile(Adam(lr=.1), loss='mean_squared_error')
Learning rate of 0.01 (100 epochs)
model.compile(Adam(lr=.01), loss='mean_squared_error')
Learning rate of 0.001 (100 epochs)
model.compile(Adam(lr=.001), loss='mean_squared_error')
On the above graph, with a learning rate of 0.001 it certainly looks like the lines are trying to converge, so will run this learning rate again with 300 epochs.
Learning rate of 0.001 (300 epochs)
Learning rate of 0.0001 (300 epochs)
model.compile(Adam(lr=.0001), loss='mean_squared_error')
