bignum: fix boundary condition in montgomery logic
It's not clear whether this inconsistency could lead to an actual
computation error, but it involved a BIGNUM being passed around the
montgomery logic in an inconsistent state. This was found using flags
-DBN_DEBUG -DBN_DEBUG_RAND, and working backwards from this assertion
in 'ectest';
ectest: bn_mul.c:960: BN_mul: Assertion `(_bnum2->top == 0) ||
(_bnum2->d[_bnum2->top - 1] != 0)' failed
(Imported from upstream's 3cc546a3bbcbf26cd14fc45fb133d36820ed0a75)
diff --git a/crypto/bn/exponentiation.c b/crypto/bn/exponentiation.c
index ac47e4b..83f1667 100644
--- a/crypto/bn/exponentiation.c
+++ b/crypto/bn/exponentiation.c
@@ -692,6 +692,9 @@
for (i = 1; i < j; i++)
r->d[i] = (~m->d[i]) & BN_MASK2;
r->top = j;
+ /* Upper words will be zero if the corresponding words of 'm'
+ * were 0xfff[...], so decrement r->top accordingly. */
+ bn_correct_top(r);
} else if (!BN_to_montgomery(r, BN_value_one(), mont, ctx)) {
goto err;
}
