tree 74b83972fd2ffadd99b0c2b877a0112fb86aa9b4
parent bca869836195ff9d22cb001a6d085f39ecb27b23
author Ram Mohan <ram.mohan@ittiam.com> 1687429565 +0530
committer Dichen Zhang <dichenzhang@google.com> 1687886152 +0000

ultrahdr: tune lut tables qstep for quality

ultrahdr input is either 8 and/or 10 bit. This would correspond to
256/1024 distinct float values. During generateGainMap, the yuv
pixel is converted to rgb and then *InvOetf is applied. If *InvOetf
were to be applied on a yuv pel, then 256/1024 lut would suffice.
However the *InvOetf is applied on an rgb transformed value.
If the csc was integral, then 256/1024 yuv sample will map to
256/1024 rgb sample and 256/1024 lut size will be sufficient.
As csc operation is not integral, it is possible that rgb pel
values not necessarily map to 256/1024 values always. It may be
beneficial to have a slightly higher precision than input depth
for *InvOetf luts.

*Oetfs require a much larger precision because the inputs to function
although in range 0-1, but can be of any value not just the 256/1024
states. Further, if the slope of a curve is small, then larger qstep
will not have a huge effect on the error of f'(x). But as slope
increases small changes in x can greatly effect f'(x). So if qstep is
not large enough then error can be pronounced.

The current luts are quantizing the region 0-1 uniformly so we need
to increase the lut size.

TODO: segment luts basing on slope.

Bug: 288383792
Test: ./libultrahdr_app -p inp_1920x1080_p010.yuv \
-y inp_1920x1080_420p.yuv -w 1920 -h 1080 -o 2 -t 2 -c 0

Change-Id: I7517340a2c1e32d9e3c602a67397e17e48fa62b3