Problems with file descriptors being inherited by default in Python

Have you ever run into a traceback that ends with something like this?

  File "C:\Python27\lib\logging\handlers.py", line 141, in doRollover
    os.rename(self.baseFilename, dfn)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process

I certainly have, in a few places. The basic problem is that when python creates file objects on Windows (and I think on *nix as well), by default Python will mark the handle as being inheritable (I’m sure there’s a reason why… but, doesn’t make a whole lot of sense for this to be the default behavior to me). So if your script spawns a new process, that new process will inherit all the file handles from your script — and of course since it doesn’t realize that it even has those handles, it’ll never close them. A great example of this is launching a process, then exiting. When you launch your script again and try to open that handle… the other process still has it open, and depending on how the file was opened, you may not be able to open it due to a sharing violation.

It looks like they’re trying to provide ways to fix the problem in PEP 433 for Python 3.3, but that doesn’t help those of us still using Python 2.7. Here’s a snippet that you can put at the very beginning of your script to fix this problem on Windows:

import sys

if sys.platform == 'win32':
    from ctypes import *
    import msvcrt
    
    __builtins__open = __builtins__.open
    
    def __open_inheritance_hack(*args, **kwargs):
        result = __builtins__open(*args, **kwargs)
        handle = msvcrt.get_osfhandle(result.fileno())
        windll.kernel32.SetHandleInformation(handle, 1, 0)
        return result
        
    __builtins__.open = __open_inheritance_hack

Now, I admit, this is a bit of a hack… but it solves the problem for me. Hope you find this useful!

One Response to “Problems with file descriptors being inherited by default in Python”

  1. azalea says:

    Thank you for sharing this. Although it did not solve my problem, it inspired me along the way!
    btw, I think you should

    import __builtin__

    and call “__builtin__.open” rather than “__builtins__.open”

Leave a Reply