ultrahdr: tune lut tables qstep for quality
ultrahdr input is either 8 and/or 10 bit. This would correspond to
256/1024 distinct float values. During generateGainMap, the yuv
pixel is converted to rgb and then *InvOetf is applied. If *InvOetf
were to be applied on a yuv pel, then 256/1024 lut would suffice.
However the *InvOetf is applied on an rgb transformed value.
If the csc was integral, then 256/1024 yuv sample will map to
256/1024 rgb sample and 256/1024 lut size will be sufficient.
As csc operation is not integral, it is possible that rgb pel
values not necessarily map to 256/1024 values always. It may be
beneficial to have a slightly higher precision than input depth
for *InvOetf luts.
*Oetfs require a much larger precision because the inputs to function
although in range 0-1, but can be of any value not just the 256/1024
states. Further, if the slope of a curve is small, then larger qstep
will not have a huge effect on the error of f'(x). But as slope
increases small changes in x can greatly effect f'(x). So if qstep is
not large enough then error can be pronounced.
The current luts are quantizing the region 0-1 uniformly so we need
to increase the lut size.
TODO: segment luts basing on slope.
Bug: 288383792
Test: ./libultrahdr_app -p inp_1920x1080_p010.yuv \
-y inp_1920x1080_420p.yuv -w 1920 -h 1080 -o 2 -t 2 -c 0
Change-Id: I7517340a2c1e32d9e3c602a67397e17e48fa62b3
diff --git a/libs/ultrahdr/include/ultrahdr/gainmapmath.h b/libs/ultrahdr/include/ultrahdr/gainmapmath.h
index edf152d..d594a0d 100644
--- a/libs/ultrahdr/include/ultrahdr/gainmapmath.h
+++ b/libs/ultrahdr/include/ultrahdr/gainmapmath.h
@@ -312,7 +312,7 @@
float hlgOetfLUT(float e);
Color hlgOetfLUT(Color e);
-constexpr size_t kHlgOETFPrecision = 10;
+constexpr size_t kHlgOETFPrecision = 16;
constexpr size_t kHlgOETFNumEntries = 1 << kHlgOETFPrecision;
/*
@@ -325,7 +325,7 @@
float hlgInvOetfLUT(float e_gamma);
Color hlgInvOetfLUT(Color e_gamma);
-constexpr size_t kHlgInvOETFPrecision = 10;
+constexpr size_t kHlgInvOETFPrecision = 12;
constexpr size_t kHlgInvOETFNumEntries = 1 << kHlgInvOETFPrecision;
/*
@@ -338,7 +338,7 @@
float pqOetfLUT(float e);
Color pqOetfLUT(Color e);
-constexpr size_t kPqOETFPrecision = 10;
+constexpr size_t kPqOETFPrecision = 16;
constexpr size_t kPqOETFNumEntries = 1 << kPqOETFPrecision;
/*
@@ -351,7 +351,7 @@
float pqInvOetfLUT(float e_gamma);
Color pqInvOetfLUT(Color e_gamma);
-constexpr size_t kPqInvOETFPrecision = 10;
+constexpr size_t kPqInvOETFPrecision = 12;
constexpr size_t kPqInvOETFNumEntries = 1 << kPqInvOETFPrecision;
diff --git a/libs/ultrahdr/jpegr.cpp b/libs/ultrahdr/jpegr.cpp
index 9c57f34..af95891 100644
--- a/libs/ultrahdr/jpegr.cpp
+++ b/libs/ultrahdr/jpegr.cpp
@@ -1070,7 +1070,7 @@
}
case ULTRAHDR_OUTPUT_HDR_PQ:
{
-#if USE_HLG_OETF_LUT
+#if USE_PQ_OETF_LUT
ColorTransformFn hdrOetf = pqOetfLUT;
#else
ColorTransformFn hdrOetf = pqOetf;
