Change priority of thread allocation.

Now, threads are given the following priority:
a). the joining thread
b). the 'spawn thread on start thread'
c). kernel spawned threads

This reworks thread pool configuration to change two details:
- allows forcing 0 threads in a threadpool (if this is ever needed
  again).
- fixes issue where no threads can be used after configureRpcThreadpool
  is called but before joinRpcThreadpool is called.

(a) automatically gives us a thread, so that must contribute to the
thread count. If the threadpool is of size >= 1, then (b) can also
contribute to the threadcount. Before, we only spawned (b) if not
willJoin. However, this causes a problem where you can't use HIDL
services after you call configureRpcThreadpool and before you call
joinRpcThreadpool because you need the (a) thread to join before it can
be used by the kernel to start more threads. Now, when maxThreads is
greater or equal to 2, we go ahead and spawn a thread on threadpool
start here. The upside of this is that you can use HIDL services at
this time. However, the downside is that two threads will be spawned
by default rather than one.

In the future, we could fix this downside by changing joinRpcThreadpool
to join (wait instead of becoming part of the threadpool) and always
send maxThreads - mSpawnThreadOnStart to the kernel. However, this
would mean that multiple threads calling joinRpcThreadpool would not
contribute any thread count (possibly breaking), and it means those
extra threads are incurring a cost without being able to do any work.

Fixes: 117098530
Fixes: 118344782

Test: hidl_test, boot, use service after configureRpcThreadpool and
    before joinRpcThreadpool.

Change-Id: I4a39bd71c01907d96f0b3be39099273507982362
diff --git a/ProcessState.cpp b/ProcessState.cpp
index 00217f7..84fb8f7 100644
--- a/ProcessState.cpp
+++ b/ProcessState.cpp
@@ -336,24 +336,37 @@
 }
 
 status_t ProcessState::setThreadPoolConfiguration(size_t maxThreads, bool callerJoinsPool) {
-    LOG_ALWAYS_FATAL_IF(maxThreads < 1, "Binder threadpool must have a minimum of one thread.");
-    status_t result = NO_ERROR;
+    // if the caller joins the pool, then there will be one thread which is impossible.
+    LOG_ALWAYS_FATAL_IF(maxThreads == 0 && callerJoinsPool,
+           "Binder threadpool must have a minimum of one thread if caller joins pool.");
+
+    size_t threadsToAllocate = maxThreads;
+
+    // If the caller is going to join the pool it will contribute one thread to the threadpool.
+    // This is part of the API's contract.
+    if (callerJoinsPool) threadsToAllocate--;
+
+    // If we can, spawn one thread from userspace when the threadpool is started. This ensures
+    // that there is always a thread available to start more threads as soon as the threadpool
+    // is started.
+    bool spawnThreadOnStart = threadsToAllocate > 0;
+    if (spawnThreadOnStart) threadsToAllocate--;
+
     // the BINDER_SET_MAX_THREADS ioctl really tells the kernel how many threads
     // it's allowed to spawn, *in addition* to any threads we may have already
-    // spawned locally. If 'callerJoinsPool' is true, it means that the caller
-    // will join the threadpool, and so the kernel needs to create one less thread.
-    // If 'callerJoinsPool' is false, we will still spawn a thread locally, and we should
-    // also tell the kernel to create one less thread than what was requested here.
-    size_t kernelMaxThreads = maxThreads - 1;
-    if (ioctl(mDriverFD, BINDER_SET_MAX_THREADS, &kernelMaxThreads) != -1) {
-        AutoMutex _l(mLock);
-        mMaxThreads = maxThreads;
-        mSpawnThreadOnStart = !callerJoinsPool;
-    } else {
-        result = -errno;
-        ALOGE("Binder ioctl to set max threads failed: %s", strerror(-result));
+    // spawned locally.
+    size_t kernelMaxThreads = threadsToAllocate;
+
+    AutoMutex _l(mLock);
+    if (ioctl(mDriverFD, BINDER_SET_MAX_THREADS, &kernelMaxThreads) == -1) {
+        ALOGE("Binder ioctl to set max threads failed: %s", strerror(errno));
+        return -errno;
     }
-    return result;
+
+    mMaxThreads = maxThreads;
+    mSpawnThreadOnStart = spawnThreadOnStart;
+
+    return NO_ERROR;
 }
 
 size_t ProcessState::getMaxThreads() {