commit 3de283c665387a569b139d589c2409d51ceafee8
author Jared Hance <jhance@dropbox.com> Wed Mar 20 10:23:22 2019 -0700
committer Jared Hance <jhance@dropbox.com> Wed Mar 20 12:01:11 2019 -0700
tree 53519f8870c377470749b5518569025938a159f1
parent ba7da20f1d51a49b8d8063bdb163a8c33bf924e5

Make gil handling in completion queue more robust

It turns out that the code generation for "with gil" is a bit more
complicated than the logic for re-obtaining the gil at the end of
"with nogil." This is because PyGILState_Ensure seems to, during
interpreter finalization, think it needs to call a new thread
(resulting in a call to cpython new_threadstate) which then segfaults.

Because "with nogil" knows that, prior to executing, it already had
the gil, it doesn't need to set up as much state, and thus the segfault
does not occur.

To avoid this, we just only use "with nogil" within the infinite loop,
and then end the "nogil" block before we check signals. This avoids
needing any "with gil" call at all.

I was able to reliably reproduce the segfault within a few minutes
before the patch by running a binary in a loop (with py3) while
maxing out my machines cpu usage. After the patch, I have not
been able to reproduce the segfault after two hours.

Note that this race can only occur when the user does not properly
clean up all their channels, and is relying on garbage collection to
do so (which isn't guaranteed). However, we want to avoid a segfault
on failure to close because this isn't a good user error and makes it
hard to debug.