Downmix: Fix setting an invalid channel mask has side effects

Attempting to set an invalid channel mask has the side effect
of actually setting it.  A legacy bug that should be fixed
because it can cause an invalid state of the downmixer.

Test: atest downmix_tests
Bug: 202162797
Change-Id: I6cc03c5bf68492e8442b5d1ae62b89c4a77c5916