ultrahdr: tune lut tables qstep for quality
ultrahdr input is either 8 and/or 10 bit. This would correspond to
256/1024 distinct float values. During generateGainMap, the yuv
pixel is converted to rgb and then *InvOetf is applied. If *InvOetf
were to be applied on a yuv pel, then 256/1024 lut would suffice.
However the *InvOetf is applied on an rgb transformed value.
If the csc was integral, then 256/1024 yuv sample will map to
256/1024 rgb sample and 256/1024 lut size will be sufficient.
As csc operation is not integral, it is possible that rgb pel
values not necessarily map to 256/1024 values always. It may be
beneficial to have a slightly higher precision than input depth
for *InvOetf luts.
*Oetfs require a much larger precision because the inputs to function
although in range 0-1, but can be of any value not just the 256/1024
states. Further, if the slope of a curve is small, then larger qstep
will not have a huge effect on the error of f'(x). But as slope
increases small changes in x can greatly effect f'(x). So if qstep is
not large enough then error can be pronounced.
The current luts are quantizing the region 0-1 uniformly so we need
to increase the lut size.
TODO: segment luts basing on slope.
Bug: 288383792
Test: ./libultrahdr_app -p inp_1920x1080_p010.yuv \
-y inp_1920x1080_420p.yuv -w 1920 -h 1080 -o 2 -t 2 -c 0
Change-Id: I7517340a2c1e32d9e3c602a67397e17e48fa62b3
2 files changed
