encoder: Return gracefully from entropy encoding errors

Recent error propagation change in encoder introduced a DoS when
output buffer is not large enough to hold the frame encoded.
This commit fixes the DoS seen.

Test: avcenc -c enc.cfg, avcdec -c dec.cfg
Test: revert fix for b/144928581 to reduce the output buffer size
Test: atest android.media.cts.VideoEncoderTest#testGoogH264FlexMaxMax
Test: atest android.media.cts.VideoEncoderTest#testGoogH264Flex1080p

Bug:145019703

Change-Id: I3b3408153a2cb8541a87547d51dd8bde25a33d6c
2 files changed