I’ve just been playing around with learning rates on a data set, using a validation set to give me ‘loss’ and ‘val_loss’ outputs. I will include different learning rates and the resultant loss and val_loss graphs, each of which was run for 100 epochs.

### Learning rate of 0.1 (100 epochs)

model.compile(Adam(lr=.1), loss='mean_squared_error')

### Learning rate of 0.01 (100 epochs)

model.compile(Adam(lr=.01), loss='mean_squared_error')

### Learning rate of 0.001 (100 epochs)

model.compile(Adam(lr=.001), loss='mean_squared_error')

On the above graph, with a learning rate of 0.001 it certainly looks like the lines are trying to converge, so will run this learning rate again with 300 epochs.

### Learning rate of 0.001 (300 epochs)

### Learning rate of 0.0001 (300 epochs)

model.compile(Adam(lr=.0001), loss='mean_squared_error')