7118202: G1: eden size unnecessarily drops to a minimum

An integer underflow can cause the RSet lengths to be massively overpredicted which forces the eden size to the minimum.

Reviewed-by: brutisso, johnc