class_parser.py

#!/usr/bin/env python3
#
# Copyright 2010, Michael Cohen <scudette@gmail.com>.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Documentation regarding the Python bounded code.

This code originally released as part of the AFF4 project
(http://code.google.com/p/aff4/).

Memory Management
=================

AFF4 uses a reference count system for memory management similar in
many ways to the native Python system. The basic idea is that memory
returned by the library always carries a new reference. When the
caller is done with the memory, they must call aff4_free() on the
memory, afterwhich the memory is considered invalid. The memory may
still not be freed at this point depending on its total reference
count.

New references may be taken to the same memory at any time using the
aff4_incref() function. This increases the reference count of the
object, and prevents it from being really freed until the correct
number of aff4_free() calls are made to it.

This idea is important for example in the following sequence:

FileLikeObject fd = resolver->create(resolver, "w");
RDFURN uri = fd->urn;

Now uri hold a reference to the urn attribute of fd, but that
attribute is actually owned by fd. If fd is freed in future, e.g. (the
close method actually frees the fd implicitely):

fd->close(fd);

Now the uri object is dangling. To prevent fd->urn from disappearing
when fd is freed, we need to take another reference to it:

FileLikeObject fd = resolver->create(resolver, "w");
RDFURN uri = fd->urn;
aff4_incref(uri);

fd->close(fd);

Now uri is valid (but fd is no longer valid). When we are finished
with uri we just call:

aff4_free(uri);


Python Integration
------------------

For every AFF4 object, we create a Python wrapper object of the
corresponding type. The wrapper object contains Python wrapper methods
to allow access to the AFF4 object methods, as well as getattr methods
for attributes. It is very important to allow Python to inherit from C
classes directly - this requires every internal C method call to be
diverted to the Python object.

The C object looks like this normally:

struct obj {
    __class__ pointer to static struct initialised with C method pointers

... Some private members
... Attributes;

/* Following are the methods */
    int (*method)(struct obj *self, ....);
};

I.e. when the method is called the struct.method member is
dereferenced to find the location of the function handling it, the
object is stuffed into the first arg, and the parameters are stuffed
into following args.

Directing Python calls
----------------------

The Python object which is created is a proxy for the c object. When
Python methods are called in the Python object, they need to be
directed into the C structure and a C call must be made, then the
return value must be reconverted into Python objects and returned into
Python. This occurs automatically by the wrapper:

struct PythonWrapper {
      PyObject_HEAD
      void *base;
};

When a Python method is called on this new Python type this is what happens:

 1) The method name is looked up in the PyMethodDef struct as per normal.

 2) If the method is recognised as a valid method the Python wrapper
    function is called (pyCLASSNAME_method)

 3) This method is broken into the general steps:

PyObject *pyCLASSNAME_method(PythonWrapper self, PyObject *args, PyObject *kwds) {
    set up c declerations for all args - call .definition() on all the args and return type

    parse argument using PyArg_ParseTupleAndKeywords

    Precall preparations

    Make the C call

    Post call processing of the returned value (check for errors etc)

    Convert the return value to a Python object using:
    return_type.to_Python_object()

    return the Python object or raise an exception
};

So the aim of the wrapper function is to convert Python args to C
args, find the C method corresponding to the method name by
dereferencing the c object and then call it.


The problem now is what happens when a C method internally calls
another method. This is a problem because the C method has no idea its
running within Python and so will just call the regular C method that
was there already. This makes it impossible to subclass the C class and
update the C method with a Python method. What we really want is when
a C method is called internally, we want to end up calling the Python
object instead to allow a purely Python implementation to override the
C method.

This happens by way of a ProxiedMethod - A proxied method is in a
sense the reverse of the wrapper method:

return_type ProxyCLASSNAME_method(CCLASSNAME self, ....) {
   Take all C args and create Python objects from them

   Dereference the object extension ((Object) self)->extension to
   obtain the Python object which wraps this C class.

   If an extension does not exist, just call the method as normal,
   otherwise make a Python call on the wrapper object.

   Convert the returned Python object to a C type and return it.
};

To make all this work we have the following structures:
struct PythonWrapper {
  PyObject_HEAD
  struct CCLASSNAME *base

       - This is a copy of the item, with all function pointer
         pointing at proxy functions. We can always get the original C
         function pointers through base->__class__

       - We also set the base object extension to be the Python
         object: ((Object) base)->extension = PythonWrapper. This
         allows us to get back the Python object from base.
};


When a Python method is invoked, we use cbase to find the C method
pointer, but we pass to it base:

self->base->__class__->method(self->base, ....)

base is a proper C object which had its methods dynamically replaced
with proxies. Now if an internal C method is called, the method will
dereference base and retrieve the proxied method. Calling the
proxied method will retreive the original Python object from the
object extension and make a Python call.

In the case where a method is not overridden by Python, internal C
method calls will generate an unnecessary conversion from C to Python
and then back to C.

Memory management in Python extension
-------------------------------------

When calling a method which returns a new reference, we just store the
reference in the "base" member of the Python object. When Python
garbage collects our Python object, we call aff4_free() on it.

The getattr method creates a new Python wrapper object of the correct
type, and sets its base attribute to point at the target AFF4
object. We then aff4_incref() the target to ensure that it does not
get freed until we are finished with it.


   Python Object
  -----
 |  P1 |    C Object
 | Base|-->+------+
 |     |   |  C1  |
 |     |   |      |
  -----    |Member|--------------+-->+----+
           +------+              |   | C2 |
                                 |   |    |
              Getattr  -------   |   |    |
              Member  |  P2   |  |   +----+
                      | Base  |--+ New reference
                       -------
                        Python Object

   Figure 1: Python object 1 owns C1's memory (when P1 is GC'ed C1 is
             freed). A reference to a member of C1 is made via P1's
             getattr method. The getattr method creates P2 to provide
             access to C2 by setting base to C2's address. We need to
             guarantee however, that C2 will not be freed suddenly
             (e.g. if C1 is freed). We therefore increase C2's
             reference count using aff4_incref();
"""

import io
import os
import pdb
import re
import sys

import lexer

DEBUG = 0

# The pytsk3 version.
VERSION = "20260502"

# These functions are used to manage library memory.
FREE = "aff4_free"
INCREF = "aff4_incref"
CURRENT_ERROR_FUNCTION = "aff4_get_current_error"
CONSTANTS_BLACKLIST = ["TSK3_H_"]

# Some constants.
DOCSTRING_RE = re.compile("[ ]*\n[ \t]+[*][ ]?")


def dispatch(name, type, *args, **kwargs):
    if not type:
        return PVoid(name)

    m = re.match("struct ([a-zA-Z0-9]+)_t *", type)
    if m:
        type = m.group(1)

    type_components = type.split()
    attributes = set()

    if type_components[0] in method_attributes:
        attributes.add(type_components.pop(0))

    type = " ".join(type_components)
    result = type_dispatcher[type](name, type, *args, **kwargs)

    result.attributes = attributes

    return result


def log(msg):
    if DEBUG > 0:
        sys.stderr.write("{0:s}\n".format(msg))


def format_as_docstring(string):
    # Remove C/C++ comment code statements.
    string = DOCSTRING_RE.sub("\n", string)
    byte_string = string.encode("unicode-escape")
    # Escapes double quoted string. We need to run this after unicode-escape to
    # prevent this operation to escape the escape character (\). In Python 3
    # the replace method requires the arguments to be byte strings.
    byte_string = byte_string.replace(b"\"", b"\\\"")
    # Make sure to return the string a Unicode otherwise in Python 3 the string
    # is prefixed with b when written or printed.
    return byte_string.decode("utf-8")


class Module:
    public_api = None
    public_header = None

    def __init__(self, name):
        self.name = name
        self.constants = set()
        self.constants_blacklist = CONSTANTS_BLACKLIST
        self.classes = {}
        self.headers = ""
        self.files = []
        self.active_structs = set()
        self.function_definitions = set()

    init_string = ""

    def initialization(self):
        result = self.init_string + (
            "\n"
            "talloc_set_log_fn((void (*)(const char *)) printf);\n"
            "// DEBUG: talloc_enable_leak_report();\n"
            "// DEBUG: talloc_enable_leak_report_full();\n")

        for cls in self.classes.values():
            if cls.is_active():
                result += cls.initialise()

        return result

    def add_constant(self, constant, type="numeric"):
        """This will be called to add #define constant macros."""
        self.constants.add((constant, type))

    def add_class(self, cls, handler):
        self.classes[cls.class_name] = cls

        # Make a wrapper in the type dispatcher so we can handle
        # passing this class from/to Python
        type_dispatcher[cls.class_name] = handler

    def get_string(self):
        """Retrieves a string representation."""
        result = "Module {0:s}\n".format(self.name)
        classes_list = list(self.classes.values())
        classes_list.sort(key=lambda cls: cls.class_name)
        for cls in classes_list:
            if cls.is_active():
                result += "    {0:s}\n".format(cls.get_string())

        constants_list = list(self.constants)
        constants_list.sort()
        result += "Constants:\n"
        for name, _ in constants_list:
            result += " {0:s}\n".format(name)

        return result

    def private_functions(self):
        """Emits hard coded private functions for doing various things"""
        values_dict = {
            "classes_length": len(self.classes) + 1,
            "get_current_error": CURRENT_ERROR_FUNCTION}

        return """
/* The following is a static array mapping CCLASS() pointers to their
 * Python wrappers. This is used to allow the correct wrapper to be
 * chosen depending on the object type found - regardless of the
 * prototype.
 *
 * This is basically a safer way for us to cast the correct Python type
 * depending on context rather than assuming a type based on the .h
 * definition. For example consider the function
 *
 * AFFObject Resolver.open(uri, mode)
 *
 * The .h file implies that an AFFObject object is returned, but this is
 * not true as most of the time an object of a derived C class will be
 * returned. In C we cast the returned value to the correct type. In the
 * Python wrapper we just instantiate the correct Python object wrapper
 * at runtime depending on the actual returned type. We use this lookup
 * table to do so.
 */
static int TOTAL_CCLASSES=0;

#define CONSTRUCT_INITIALIZE(cclass, virt_cclass, constructor, object, ...) \\
    (cclass)(((virt_cclass) (&__ ## cclass))->constructor(object, ## __VA_ARGS__))

#undef BUFF_SIZE
#define BUFF_SIZE 10240

/* Python compatibility macros
 */
#if !defined( PyMODINIT_FUNC )
#if PY_MAJOR_VERSION >= 3
#define PyMODINIT_FUNC PyObject *
#else
#define PyMODINIT_FUNC void
#endif
#endif /* !defined( PyMODINIT_FUNC ) */

#if !defined( PyVarObject_HEAD_INIT )
#define PyVarObject_HEAD_INIT( type, size ) \\
    PyObject_HEAD_INIT( type ) \\
    size,

#endif /* !defined( PyVarObject_HEAD_INIT ) */

#if PY_MAJOR_VERSION >= 3
#define Py_TPFLAGS_HAVE_ITER		0
#endif

#if !defined( Py_TYPE )
#define Py_TYPE( object ) \\
    ( ( (PyObject *) object )->ob_type )

#endif /* !defined( Py_TYPE ) */

/* Generic wrapper type
 */
typedef struct Gen_wrapper_t *Gen_wrapper;
struct Gen_wrapper_t {{
    PyObject_HEAD

    void *base;

    /* Value to indicate the base is a Python object.
     */
    int base_is_python_object;

    /* Value to indicate the base is managed internal.
     */
    int base_is_internal;

    PyObject *python_object1;
    PyObject *python_object2;
}};

static struct python_wrapper_map_t {{
    Object class_ref;
    PyTypeObject *python_type;
    void (*initialize_proxies)(Gen_wrapper self, void *item);
}} python_wrappers[{classes_length:d}];

/* Create the relevant wrapper from the item based on the lookup table.
 */
Gen_wrapper new_class_wrapper(Object item, int item_is_python_object) {{
    Gen_wrapper result = NULL;
    Object cls = NULL;
    struct python_wrapper_map_t *python_wrapper = NULL;
    int cls_index = 0;

    // Return a Py_None object for a NULL pointer
    if(item == NULL) {{
        Py_IncRef((PyObject *) Py_None);
        return (Gen_wrapper) Py_None;
    }}
    // Search for subclasses
    for(cls = (Object) item->__class__; cls != cls->__super__; cls = cls->__super__) {{
        for(cls_index = 0; cls_index < TOTAL_CCLASSES; cls_index++) {{
            python_wrapper = &(python_wrappers[cls_index]);

            if(python_wrapper->class_ref == cls) {{
                PyErr_Clear();

                result = (Gen_wrapper) _PyObject_New(python_wrapper->python_type);
                if(result == NULL) {{
                    return NULL;
                }}
                result->base = item;
                result->base_is_python_object = item_is_python_object;
                result->base_is_internal = 1;
                result->python_object1 = NULL;
                result->python_object2 = NULL;

                python_wrapper->initialize_proxies(result, (void *) item);

                return result;
            }}
        }}
    }}
    PyErr_Format(PyExc_RuntimeError, "Unable to find a wrapper for object %s", NAMEOF(item));

    return NULL;
}}

typedef void (*function_initialize_Gen_wrapper_t)(Gen_wrapper, void*);

static PyObject *resolve_exception(char **error_buff) {{
    int *type = (int *){get_current_error:s}(error_buff);

    switch(*type) {{
        case EProgrammingError:
            return PyExc_SystemError;
        case EKeyError:
            return PyExc_KeyError;
        case ERuntimeError:
            return PyExc_RuntimeError;
        case EInvalidParameter:
            return PyExc_TypeError;
        case EWarning:
            return PyExc_AssertionError;
        case EIOError:
            return PyExc_IOError;
        default:
            return PyExc_RuntimeError;
    }}
}}

static int type_check(PyObject *obj, PyTypeObject *type) {{
    PyTypeObject *tmp = NULL;

    // Recurse through the inheritance tree and check if the types are expected
    if(obj) {{
        for(tmp = Py_TYPE(obj);
            tmp && tmp != &PyBaseObject_Type;
            tmp = tmp->tp_base) {{
            if(tmp == type) return 1;
        }}
    }}
    return 0;
}}

static int check_error() {{
   char *buffer = NULL;
   int *error_type = (int *)aff4_get_current_error(&buffer);

   if(*error_type != EZero) {{
         PyObject *exception = resolve_exception(&buffer);

         if(buffer != NULL) {{
           PyErr_Format(exception, "%s", buffer);
         }} else {{
           PyErr_Format(exception, "Unable to retrieve exception reason.");
         }}
         ClearError();
         return 1;
   }}
   return 0;
}}

/* This function checks if a method was overridden in self over a
 * method defined in type. This is used to determine if a Python class is
 * extending this C type. If not, a proxy function is not written and C
 * calls are made directly.
 *
 * This is an optimization to eliminate the need for a call into Python
 * in the case where Python objects do not actually extend any methods.
 *
 * We basically just iterate over the MRO and determine if a method is
 * defined in each level until we reach the base class.
 */
static int check_method_override(PyObject *self, PyTypeObject *type, const char *method) {{
    struct _typeobject *ob_type = NULL;
    PyObject *mro = NULL;
    PyObject *py_method = NULL;
    PyObject *item_object = NULL;
    PyObject *dict = NULL;
    Py_ssize_t item_index = 0;
    Py_ssize_t number_of_items = 0;
    int found = 0;

    ob_type = Py_TYPE(self);
    if(ob_type == NULL ) {{
      return 0;
    }}
    mro = ob_type->tp_mro;

#if PY_MAJOR_VERSION >= 3
    py_method = PyUnicode_FromString(method);
#else
    py_method = PyString_FromString(method);
#endif
    if(py_method == NULL) {{
        return 0;
    }}
    number_of_items = PySequence_Size(mro);

    for(item_index = 0; item_index < number_of_items; item_index++) {{
        int contains_result = 0;
        item_object = PySequence_GetItem(mro, item_index);
        if(item_object == NULL) {{
            break;
        }}

        // Ok - we got to the base class - finish up
        if(item_object == (PyObject *) type) {{
            Py_DecRef(item_object);
            break;
        }}
        /* Extract the dict and check if it contains the method (the
         * dict is not a real dictionary so we can not use
         * PyDict_Contains).
         */
        dict = PyObject_GetAttrString(item_object, "__dict__");
        if(dict != NULL) {{
            contains_result = PySequence_Contains(dict, py_method);
            if(contains_result > 0) {{
                found = 1;
            }}
        }}
        Py_DecRef(dict);
        Py_DecRef(item_object);

        if(found != 0) {{
            break;
        }}
    }}
    Py_DecRef(py_method);
    PyErr_Clear();

    return found;
}}

/* Fetches the Python error (exception)
 */
void pytsk_fetch_error(void) {{
    PyObject *exception_traceback = NULL;
    PyObject *exception_type = NULL;
    PyObject *exception_value = NULL;
    PyObject *string_object = NULL;
    char *str_c = NULL;
    char *error_str = NULL;
    int *error_type = (int *) {get_current_error:s}(&error_str);

#if PY_MAJOR_VERSION >= 3
    PyObject *utf8_string_object  = NULL;
#endif

    // Fetch the exception state and convert it to a string:
    PyErr_Fetch(&exception_type, &exception_value, &exception_traceback);

    /* exception_value can be NULL when the original exception was
     * raised via PyErr_SetNone(type) (e.g. KeyboardInterrupt).
     */
    if(exception_value == NULL) {{
        if(error_str != NULL) {{
            const char *placeholder = "Python exception raised without value";
            size_t placeholder_len = strlen(placeholder);
            if(placeholder_len > (size_t)(BUFF_SIZE - 1)) {{
                placeholder_len = (size_t)(BUFF_SIZE - 1);
            }}
            memcpy(error_str, placeholder, placeholder_len);
            error_str[placeholder_len] = 0;
        }}
        *error_type = ERuntimeError;
        PyErr_Restore(exception_type, exception_value, exception_traceback);
        return;
    }}

    string_object = PyObject_Repr(exception_value);

#if PY_MAJOR_VERSION >= 3
    if(string_object != NULL) {{
        utf8_string_object = PyUnicode_AsUTF8String(string_object);
    }}

    if(utf8_string_object != NULL) {{
        str_c = PyBytes_AsString(utf8_string_object);
    }}
#else
    if(string_object != NULL) {{
        str_c = PyString_AsString(string_object);
    }}
#endif

    if(str_c != NULL) {{
        strncpy(error_str, str_c, BUFF_SIZE-1);
        error_str[BUFF_SIZE - 1] = 0;
        *error_type = ERuntimeError;
    }} else {{
        /* Repr/encode failed; record a generic message so callers
         * still observe ERuntimeError instead of EZero (which would
         * make check_error() report success).
         */
        if(error_str != NULL) {{
            const char *placeholder = "Python exception (repr failed)";
            size_t placeholder_len = strlen(placeholder);
            if(placeholder_len > (size_t)(BUFF_SIZE - 1)) {{
                placeholder_len = (size_t)(BUFF_SIZE - 1);
            }}
            memcpy(error_str, placeholder, placeholder_len);
            error_str[placeholder_len] = 0;
        }}
        *error_type = ERuntimeError;
    }}
    PyErr_Restore(exception_type, exception_value, exception_traceback);

#if PY_MAJOR_VERSION >= 3
    if( utf8_string_object != NULL ) {{
        Py_DecRef(utf8_string_object);
    }}
#endif
    if(string_object != NULL) {{
        Py_DecRef(string_object);
    }}

    return;
}}

/* Copies a Python int or long object to an unsigned 64-bit value
 */
uint64_t integer_object_copy_to_uint64(PyObject *integer_object) {{
#if defined( HAVE_LONG_LONG )
    PY_LONG_LONG long_value = 0;
#else
    long long_value = 0;
#endif
    int result = 0;

    if(integer_object == NULL) {{
        PyErr_Format(PyExc_ValueError, "Missing integer object");

        return (uint64_t) -1;
    }}
    PyErr_Clear();

    result = PyObject_IsInstance(integer_object, (PyObject *) &PyLong_Type);

    if(result == -1) {{
        pytsk_fetch_error();

        return (uint64_t) -1;

    }} else if(result != 0) {{
        PyErr_Clear();

#if defined( HAVE_LONG_LONG )
    long_value = PyLong_AsUnsignedLongLong(integer_object);
#else
    long_value = PyLong_AsUnsignedLong(integer_object);
#endif
        /* PyLong_AsUnsignedLong / PyLong_AsUnsignedLongLong returns
         * (unsigned)-1 and sets OverflowError when the value does
         * not fit. Surfacing the original exception is more useful
         * than continuing into the generic "out of bounds" path
         * below, which would clobber the OverflowError with a fresh
         * ValueError.
         */
        if(PyErr_Occurred()) {{
            pytsk_fetch_error();

            return (uint64_t) -1;
        }}
    }}
#if PY_MAJOR_VERSION < 3
    if(result == 0) {{
        PyErr_Clear();

        result = PyObject_IsInstance(integer_object, (PyObject *) &PyInt_Type);

        if(result == -1) {{
            pytsk_fetch_error();

            return (uint64_t) -1;

        }} else if(result != 0) {{
            PyErr_Clear();

#if defined( HAVE_LONG_LONG )
            long_value = PyInt_AsUnsignedLongLongMask(integer_object);
#else
            long_value = PyInt_AsUnsignedLongMask(integer_object);
#endif
            if(PyErr_Occurred()) {{
                pytsk_fetch_error();

                return (uint64_t) -1;
            }}
        }}
    }}
#endif /* PY_MAJOR_VERSION < 3 */
    if(result == 0) {{
        if(PyErr_Occurred()) {{
            pytsk_fetch_error();

            return (uint64_t) -1;
        }}
    }}
#if defined( HAVE_LONG_LONG )
#if ( SIZEOF_LONG_LONG > 8 )
    if((long_value < (PY_LONG_LONG) 0) || (long_value > (PY_LONG_LONG) UINT64_MAX)) {{
#else
    if(long_value < (PY_LONG_LONG) 0) {{
#endif
        PyErr_Format(PyExc_ValueError, "Integer object value out of bounds");

        return (uint64_t) -1;
    }}
#else
#if ( SIZEOF_LONG > 8 )
    if((long_value < (long) 0) || (long_value > (long) UINT64_MAX)) {{
#else
    if(long_value < (PY_LONG_LONG) 0) {{
#endif
        PyErr_Format(PyExc_ValueError, "Integer object value out of bounds");

        return (uint64_t) -1;
    }}
#endif
    return (uint64_t) long_value;
}}

""".format(**values_dict)

    def initialise_class(self, class_name, out, done=None):
        if done and class_name in done:
            return

        done.add(class_name)

        cls = self.classes[class_name]
        """Write out class initialisation code into the main init function."""
        if cls.is_active():
            base_class = self.classes.get(cls.base_class_name)

            if base_class and base_class.is_active():
                # We have a base class - ensure it gets written out
                # first:
                self.initialise_class(cls.base_class_name, out, done)

                # Now assign ourselves as derived from them
                out.write(
                    "    {0:s}_Type.tp_base = &{1:s}_Type;".format(
                        cls.class_name, cls.base_class_name))

            values_dict = {
                "name": cls.class_name}

            out.write((
                "\n"
                "    /* Initialize: {name:s} */\n"
                "    {name:s}_Type.tp_new = PyType_GenericNew;\n").format(
                    **values_dict))

            if isinstance(cls, Enum):
                out.write((
                    "    if ({name:s}_init_type(&{name:s}_Type) != 1) {{\n"
                    "        goto on_error;\n"
                    "    }}\n").format(**values_dict))

            out.write((
                "    if (PyType_Ready(&{name:s}_Type) < 0) {{\n"
                "        goto on_error;\n"
                "    }}\n"
                "    if (PyModule_AddType(module, &{name:s}_Type) < 0) {{\n"
                "        goto on_error;\n"
                "    }}\n").format(
                    **values_dict))

    def write(self, out):
        # Write the headers
        if self.public_api:
            self.public_api.write(
                "#ifdef BUILDING_DLL\n"
                "#include \"misc.h\"\n"
                "#else\n"
                "#include \"aff4_public.h\"\n"
                "#endif\n")

        # Prepare all classes
        for cls in self.classes.values():
            cls.prepare()

        out.write((
            "/*************************************************************\n"
            " * Autogenerated module {0:s}\n"
            " *\n"
            " * This module was autogenerated from the following files:\n").format(
                self.name))

        for filename in self.files:
            out.write(" * {0:s}\n".format(filename))

        out.write(
            " *\n"
            " * This module implements the following classes:\n")
        out.write(self.get_string())
        out.write(
            " ************************************************************/\n"
            "\n")
        out.write(self.headers)
        out.write(
            "\n"
            "#ifdef __cplusplus\n"
            "extern \"C\" {\n"
            "#endif\n"
            "\n"
            "#include <Python.h>\n")

        out.write(self.private_functions())

        for cls in self.classes.values():
            if cls.is_active():
                out.write(
                    "/******************** {0:s} ***********************/".format(
                        cls.class_name))
                cls.struct(out)
                cls.prototypes(out)

        out.write(
            "/*****************************************************\n"
            " *           Implementation\n"
            " ****************************************************/\n"
            "\n")

        for cls in self.classes.values():
            if cls.is_active():
                cls.PyMethodDef(out)
                cls.PyGetSetDef(out)
                cls.PyTypeObject(out)

        for cls in self.classes.values():
            if cls.is_active():
                cls.code(out)

        # Write the module initializer
        values_dict = {
            "module": self.name,
            "version": VERSION,
            "version_length": len(VERSION)}

        out.write((
            "/* Retrieves the {module:s} version\n"
            " * Returns a Python object if successful or NULL on error\n"
            " */\n"
            "PyObject *{module:s}_get_version(PyObject *self, PyObject *arguments) {{\n"
            "    const char *errors = NULL;\n"
            "    return(PyUnicode_DecodeUTF8(\"{version:s}\", (Py_ssize_t) {version_length:d}, errors));\n"
            "}}\n"
            "\n"
            "static PyMethodDef {module:s}_module_methods[] = {{\n"
            "    {{ \"get_version\",\n"
            "        (PyCFunction) {module:s}_get_version,\n"
            "        METH_NOARGS,\n"
            "        \"get_version() -> String\\n\"\n"
            "        \"\\n\"\n"
            "        \"Retrieves the version.\" }},\n"
            "\n"
            "    {{NULL, NULL, 0, NULL}}  /* Sentinel */\n"
            "}};\n"
            "\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "\n"
            "/* The {module:s} module definition\n"
            " */\n"
            "PyModuleDef {module:s}_module_definition = {{\n"
            "	PyModuleDef_HEAD_INIT,\n"
            "\n"
            "	/* m_name */\n"
            "	\"{module:s}\",\n"
            "	/* m_doc */\n"
            "	\"Python {module:s} module.\",\n"
            "	/* m_size */\n"
            "	-1,\n"
            "	/* m_methods */\n"
            "	{module:s}_module_methods,\n"
            "	/* m_reload */\n"
            "	NULL,\n"
            "	/* m_traverse */\n"
            "	NULL,\n"
            "	/* m_clear */\n"
            "	NULL,\n"
            "	/* m_free */\n"
            "	NULL,\n"
            "}};\n"
            "\n"
            "#endif /* PY_MAJOR_VERSION >= 3 */\n"
            "\n"
            "/* Initializes the {module:s} module\n"
            " */\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "PyMODINIT_FUNC PyInit_{module:s}(void) {{\n"
            "#else\n"
            "PyMODINIT_FUNC init{module:s}(void) {{\n"
            "#endif\n"
            "    PyGILState_STATE gil_state;\n"
            "\n"
            "    PyObject *module = NULL;\n"
            "    PyObject *d = NULL;\n"
            "    PyObject *tmp = NULL;\n"
            "\n"
            "    /* Create the module\n"
            "     * This function must be called before grabbing the GIL\n"
            "     * otherwise the module will segfault on a version mismatch\n"
            "     */\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    module = PyModule_Create(\n"
            "        &{module:s}_module_definition );\n"
            "#else\n"
            "    module = Py_InitModule3(\n"
            "        \"{module:s}\",\n"
            "        {module:s}_module_methods,\n"
            "        \"Python {module:s} module.\" );\n"
            "#endif\n"
            "    if (module == NULL) {{\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "        return(NULL);\n"
            "#else\n"
            "        return;\n"
            "#endif\n"
            "    }}\n"
            "\n"
            "#ifdef Py_GIL_DISABLED\n"
            "    /* Declare this module safe for free-threaded Python\n"
            "     * Without this call, CPython force-enables\n"
            "     * the GIL for our module at import time on\n"
            "     * free-threaded builds, which would serialize every\n"
            "     * pytsk3 call and defeat the point of free-threading.\n"
            "     * The symbol is only declared when Py_GIL_DISABLED is set\n"
            "     */\n"
            "    PyUnstable_Module_SetGIL(module, Py_MOD_GIL_NOT_USED);\n"
            "#endif\n"
            "\n"
            "    d = PyModule_GetDict(module);\n"
            "\n"
            "    gil_state = PyGILState_Ensure();\n"
            "\n"
            "    /* Reset class registry so reimport or subinterpreter import\n"
            "     * repopulates from scratch and never overruns python_wrappers[].\n"
            "     */\n"
            "    TOTAL_CCLASSES = 0;\n").format(**values_dict))

        # The trick is to initialise the classes in order of their
        # inheritance. The following code will order initializations
        # according to their inheritance tree
        done = set()
        for class_name in self.classes.keys():
            self.initialise_class(class_name, out, done)

        # Add the constants here. Make sure they are sorted so builds
        # of pytsk3.c are reproducible.
        for constant, type in sorted(self.constants):
            if type == "integer":
                out.write(
                    "    tmp = PyLong_FromUnsignedLongLong((uint64_t) {0:s});\n".format(constant))
            elif type == "string":
                if constant == "TSK_VERSION_STR":
                    out.write((
                        "#if PY_MAJOR_VERSION >= 3\n"
                        "    tmp = PyUnicode_FromString((char *){0:s});\n"
                        "#else\n"
                        "    tmp = PyString_FromString((char *){0:s});\n"
                        "#endif\n").format(constant))

                else:
                    out.write((
                        "#if PY_MAJOR_VERSION >= 3\n"
                        "    tmp = PyBytes_FromString((char *){0:s});\n"
                        "#else\n"
                        "    tmp = PyString_FromString((char *){0:s});\n"
                        "#endif\n").format(constant))
            else:
                out.write(
                    "    /* I dont know how to convert {0:s} type {1:s} */\n".format(
                        constant, type))
                continue

            out.write((
                "    PyDict_SetItemString(d, \"{0:s}\", tmp);\n"
                "    Py_DecRef(tmp);\n").format(constant))

        out.write(self.initialization())
        out.write(
            "    PyGILState_Release(gil_state);\n"
            "\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "	return module;\n"
            "#else\n"
            "	return;\n"
            "#endif\n"
            "\n"
            "on_error:\n"
            "	PyGILState_Release(gil_state);\n"
            "\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "	return NULL;\n"
            "#else\n"
            "	return;\n"
            "#endif\n"
            "}\n"
            "\n"
            "#ifdef __cplusplus\n"
            "}\n"
            "#endif\n")


class Type:
    interface = None
    buildstr = "O"
    sense = "IN"
    error_value = "return 0;"
    active = True

    def __init__(self, name, type, *args, **kwargs):
        super().__init__()
        self.name = name
        self.type = type
        self.attributes = set()
        self.additional_args = kwargs

    def comment(self):
        return "{0:s} {1:s} ".format(self.type, self.name)

    def get_string(self):
        """Retrieves a string representation."""
        if self.name == "func_return":
            return self.type
        if "void" in self.type:
            return ""

        return "{0:s} : {1:s}".format(self.type, self.name)

    def python_name(self):
        return self.name

    def python_proxy_post_call(self):
        """This is called after a proxy call"""
        return ""

    def returned_python_definition(self, *arg, **kwargs):
        return self.definition(*arg, **kwargs)

    def definition(self, default=None, **kwargs):
        if default:
            return "    {0:s} {1:s} = {2:s};\n".format(
                self.type, self.name, default)
        elif "array_size" in self.additional_args:
            return (
                "    int array_index = 0;\n"
                "    {0:s} UNUSED *{1:s};\n").format(
                    self.type, self.name)
        else:
            return "    {0:s} UNUSED {1:s};\n".format(
                self.type, self.name)

    def local_definition(self, default=None, **kwargs):
        return ""

    def byref(self):
        return "&{0:s}".format(self.name)

    def call_arg(self):
        return self.name

    def passthru_call(self):
        """Returns how we should call the function when simply passing args directly"""
        return self.call_arg()

    def pre_call(self, method, **kwargs):
        return ""

    def assign(self, call, method, target=None, **kwargs):
        return (
            "Py_BEGIN_ALLOW_THREADS\n"
            "{0:s} = {1:s};\n"
            "Py_END_ALLOW_THREADS\n").format(
                target or self.name, call)

    def post_call(self, method):
        # Check for errors
        result = (
            "if(check_error()) {\n"
            "    goto on_error;\n"
            "}\n")

        if "DESTRUCTOR" in self.attributes:
            result += "self->base = NULL;  //DESTRUCTOR - C object no longer valid\n"

        return result

    def from_python_object(self, source, destination, method, **kwargs):
        return ""

    def return_value(self, value):
        return "return {0!s};".format(value)


class String(Type):
    interface = "string"
    buildstr = "s"
    error_value = "return NULL;"

    def __init__(self, name, type, *args, **kwargs):
        super().__init__(name, type, *args, **kwargs)
        self.length = "strlen({0:s})".format(name)

    def byref(self):
        return "&{0:s}".format(self.name)

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        values_dict = {
            "length": self.length,
            "name": name or self.name,
            "result": result}

        result = (
            "    PyErr_Clear();\n"
            "\n"
            "    if(!{name:s}) {{\n"
            "        Py_IncRef(Py_None);\n"
            "        {result:s} = Py_None;\n"
            "    }} else {{\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "        {result:s} = PyBytes_FromStringAndSize((char *){name:s}, {length:s});\n"
            "#else\n"
            "        {result:s} = PyString_FromStringAndSize((char *){name:s}, {length:s});\n"
            "#endif\n"
            "        if(!{result:s}) {{\n"
            "            goto on_error;\n"
            "        }}\n"
            "    }}\n").format(**values_dict)

        if "BORROWED" not in self.attributes and "BORROWED" not in kwargs:
            result += "talloc_unlink(NULL, {0:s});\n".format(name)

        return result

    def from_python_object(self, source, destination, method, context="NULL"):
        method.error_set = True

        values_dict = {
            "context": context,
            "destination": destination,
            "source": source}

        return (
            "{{\n"
            "    char *buff = NULL;\n"
            "    Py_ssize_t length = 0;\n"
            "\n"
            "    PyErr_Clear();\n"
            "\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    if(PyBytes_AsStringAndSize({source:s}, &buff, &length) == -1) {{\n"
            "#else\n"
            "    if(PyString_AsStringAndSize({source:s}, &buff, &length) == -1) {{\n"
            "#endif\n"
            "        goto on_error;\n"
            "    }}\n"
            "    {destination:s} = (char *) talloc_size({context:s}, length + 1);\n"
            "    if({destination:s} == NULL) {{\n"
            "        PyErr_NoMemory();\n"
            "        goto on_error;\n"
            "    }}\n"
            "    memcpy({destination:s}, buff, length);\n"
            "    {destination:s}[length] = 0;\n"
            "}};\n").format(**values_dict)


class ZString(String):
    interface = "null_terminated_string"

    def definition(self, default=None, **kwargs):
        if default == "\"\"":
            default = "(char *) \"\""

        return super().definition(default=default, **kwargs)


class BorrowedString(String):
    def to_python_object(self, name=None, result="Py_result", **kwargs):
        values_dict = {
            "length": self.length,
            "name": name or self.name,
            "result": result}

        return (
            "    PyErr_Clear();\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    {result:s} = PyBytes_FromStringAndSize((char *){name:s}, {length:s});\n"
            "#else\n"
            "    {result:s} = PyString_FromStringAndSize((char *){name:s}, {length:s});\n"
            "#endif\n").format(**values_dict)


class Char_and_Length(Type):
    interface = "char_and_length"
    buildstr = "s#"
    error_value = "return NULL;"

    def __init__(self, data, data_type, length, length_type, *args, **kwargs):
        super().__init__(data, data_type, *args, **kwargs)

        self.name = data
        self.data_type = data_type
        self.length = length
        self.length_type = length_type

    def comment(self):
        return "{0:s} {1:s}, {2:s} {3:s}".format(
            self.data_type, self.name, self.length_type, self.length)

    def definition(self, default="\"\"", **kwargs):
        return (
            "    char *{0:s} = {1:s};\n"
            "    Py_ssize_t {2:s} = strlen({3:s});\n").format(
                self.name, default, self.length, default)

    def byref(self):
        return "&{0:s}, &{1:s}".format(self.name, self.length)

    def call_arg(self):
        return "({0:s}){1:s}, ({2:s}){3:s}".format(
            self.data_type, self.name, self.length_type, self.length)

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        values_dict = {
            "length": self.length,
            "name": self.name,
            "result": result}

        return (
            "    PyErr_Clear();\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    {result:s} = PyBytes_FromStringAndSize((char *){name:s}, {length:s});\n"
            "#else\n"
            "    {result:s} = PyString_FromStringAndSize((char *){name:s}, {length:s});\n"
            "#endif\n"
            "\n"
            "    if(!{result:s}) {{\n"
            "        goto on_error;\n"
            "    }}\n").format(**values_dict)


class Integer(Type):
    interface = "integer"
    buildstr = "i"
    int_type = "int"

    def __init__(self, name, type, *args, **kwargs):
        super().__init__(name, type, *args, **kwargs)
        self.type = self.int_type
        self.original_type = type

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        values_dict = {
            "name": name or self.name,
            "result": result}

        code = (
            "    PyErr_Clear();\n"
            "    {result:s} = PyLong_FromLong({name:s});\n").format(**values_dict)
        if kwargs.get('sense') == 'proxied':
            code += (
                "    if({result:s} == NULL) {{\n"
                "        goto on_error;\n"
                "    }}\n").format(**values_dict)
        return code

    def from_python_object(self, source, destination, method, **kwargs):
        values_dict = {
            "destination": destination,
            "source": source}

        return (
            "    PyErr_Clear();\n"
            "    {destination:s} = PyLong_AsLongMask({source:s});\n").format(**values_dict)

    def comment(self):
        return "{0:s} {1:s} ".format(self.original_type, self.name)


class IntegerUnsigned(Integer):
    buildstr = "I"
    int_type = "unsigned int"

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        if "array_size" in self.additional_args:
            values_dict = {
                "name": name or self.name,
                "result": result,
                "array_size": self.additional_args["array_size"]
            }
            return (
                "    PyErr_Clear();\n"
                "    {result:s} = PyList_New(0);\n"
                "    for(array_index = 0; array_index < {array_size:s}; array_index++) {{\n"
                "#if PY_MAJOR_VERSION >= 3\n"
                "       PyList_Append({result:s}, PyLong_FromLong((long) {name:s}[array_index]));\n"
                "#else\n"
                "       PyList_Append({result:s}, PyInt_FromLong((long) {name:s}[array_index]));\n"
                "#endif\n"
                "    }}\n"
            ).format(**values_dict)
        else:
            values_dict = {
                "name": name or self.name,
                "result": result}
            code = (
                "    PyErr_Clear();\n"
                "    {result:s} = PyLong_FromLong((long) {name:s});\n").format(**values_dict)
            if kwargs.get('sense') == 'proxied':
                code += (
                    "    if({result:s} == NULL) {{\n"
                    "        goto on_error;\n"
                    "    }}\n").format(**values_dict)
            return code

    def from_python_object(self, source, destination, method, **kwargs):
        values_dict = {
            "destination": destination,
            "source": source}

        return (
            "    PyErr_Clear();\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    {destination:s} = PyLong_AsUnsignedLongMask({source:s});\n"
            "#else\n"
            "    {destination:s} = PyInt_AsUnsignedLongMask({source:s});\n"
            "#endif\n").format(**values_dict)


class Integer8(Integer):
    int_type = "int8_t"


class Integer8Unsigned(IntegerUnsigned):
    int_type = "uint8_t"


class Integer16(Integer):
    int_type = "int16_t"


class Integer16Unsigned(IntegerUnsigned):
    int_type = "uint16_t"


class Integer32(Integer):
    int_type = "int32_t"


class Integer32Unsigned(IntegerUnsigned):
    int_type = "uint32_t"


class Integer64(Integer):
    buildstr = "L"
    int_type = "int64_t"

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        values_dict = {
            "name": name or self.name,
            "result": result}

        code = (
            "    PyErr_Clear();\n"
            "#if defined( HAVE_LONG_LONG )\n"
            "    {result:s} = PyLong_FromLongLong({name:s});\n"
            "#else\n"
            "    {result:s} = PyLong_FromLong({name:s});\n"
            "#endif\n").format(**values_dict)
        if kwargs.get('sense') == 'proxied':
            code += (
                "    if({result:s} == NULL) {{\n"
                "        goto on_error;\n"
                "    }}\n").format(**values_dict)
        return code

    def from_python_object(self, source, destination, method, **kwargs):
        values_dict = {
            "destination": destination,
            "source": source}

        return (
            "    PyErr_Clear();\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "#if defined( HAVE_LONG_LONG )\n"
            "    {destination:s} = PyLong_AsLongLongMask({source:s});\n"
            "#else\n"
            "    {destination:s} = PyLong_AsLongMask({source:s});\n"
            "#endif\n"
            "#else\n"
            "#if defined( HAVE_LONG_LONG )\n"
            "    {destination:s} = PyInt_AsLongLongMask({source:s});\n"
            "#else\n"
            "    {destination:s} = PyInt_AsLongMask({source:s});\n"
            "#endif\n"
            "#endif /* PY_MAJOR_VERSION >= 3 */\n").format(**values_dict)


class Integer64Unsigned(Integer):
    buildstr = "K"
    int_type = "uint64_t"

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        values_dict = {
            "name": name or self.name,
            "result": result}

        code = (
            "    PyErr_Clear();\n"
            "#if defined( HAVE_LONG_LONG )\n"
            "    {result:s} = PyLong_FromUnsignedLongLong({name:s});\n"
            "#else\n"
            "    {result:s} = PyLong_FromUnsignedLong({name:s});\n"
            "#endif\n").format(**values_dict)
        if kwargs.get('sense') == 'proxied':
            code += (
                "    if({result:s} == NULL) {{\n"
                "        goto on_error;\n"
                "    }}\n").format(**values_dict)
        return code

    def from_python_object(self, source, destination, method, **kwargs):
        values_dict = {
            "destination": destination,
            "source": source}

        # TODO: use integer_object_copy_to_uint64 instead to support both
        # long and int objects.
        return (
            "    PyErr_Clear();\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "#if defined( HAVE_LONG_LONG )\n"
            "    {destination:s} = PyLong_AsUnsignedLongLongMask({source:s});\n"
            "#else\n"
            "    {destination:s} = PyLong_AsUnsignedLongMask({source:s});\n"
            "#endif\n"
            "#else\n"
            "#if defined( HAVE_LONG_LONG )\n"
            "    {destination:s} = PyInt_AsUnsignedLongLongMask({source:s});\n"
            "#else\n"
            "    {destination:s} = PyInt_AsUnsignedLongMask({source:s});\n"
            "#endif\n"
            "#endif /* PY_MAJOR_VERSION >= 3 */\n").format(**values_dict)


class Long(Integer):
    buildstr = "l"
    int_type = "long"

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        values_dict = {
            "name": name or self.name,
            "result": result}

        code = (
            "PyErr_Clear();\n"
            "{result:s} = PyLong_FromLongLong({name:s});\n").format(**values_dict)
        if kwargs.get('sense') == 'proxied':
            code += (
                "if({result:s} == NULL) {{\n"
                "    goto on_error;\n"
                "}}\n").format(**values_dict)
        return code

    def from_python_object(self, source, destination, method, **kwargs):
        values_dict = {
            "destination": destination,
            "source": source}

        return (
            "PyErr_Clear();\n"
            "{destination:s} = PyLong_AsLongMask({source:s});\n").format(
                **values_dict)


class LongUnsigned(Integer):
    buildstr = "k"
    int_type = "unsigned long"

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        values_dict = {
            "name": name or self.name,
            "result": result}

        code = (
            "PyErr_Clear();\n"
            "{result:s} = PyLong_FromUnsignedLong({name:s});\n").format(**values_dict)
        if kwargs.get('sense') == 'proxied':
            code += (
                "if({result:s} == NULL) {{\n"
                "    goto on_error;\n"
                "}}\n").format(**values_dict)
        return code

    def from_python_object(self, source, destination, method, **kwargs):
        values_dict = {
            "destination": destination,
            "source": source}

        return (
            "PyErr_Clear();\n"
            "{destination:s} = PyLong_AsUnsignedLongMask({source:s});\n").format(
                **values_dict)


class Char(Integer):
    buildstr = "s"
    interface = "small_integer"

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        # We really want to return a string here
        values_dict = {
            "name": name or self.name,
            "result": result}

        return (
            "{{\n"
            "    char *str_{name:s} = &{name:s};\n"
            "\n"
            "    PyErr_Clear();\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    {result:s} = PyBytes_FromStringAndSize(str_{name:s}, 1);\n"
            "#else\n"
            "    {result:s} = PyString_FromStringAndSize(str_{name:s}, 1);\n"
            "#endif\n"
            "\n"
            "    if(!{result:s}) {{\n"
            "        goto on_error;\n"
            "}}\n").format(**values_dict)

    def definition(self, default="\"\\x0\"", **kwargs):
        # Shut up unused warnings
        return (
            "char {0:s} UNUSED = 0;\n"
            "char *str_{0:s} UNUSED = {1:s};\n").format(
                self.name, default)

    def byref(self):
        return "&str_{0:s}".format(self.name)

    def pre_call(self, method, **kwargs):
        method.error_set = True

        values_dict = {
            "name": self.name}

        return (
            "    if(strlen(str_{name:s}) != 1) {\n"
            "        PyErr_Format(PyExc_RuntimeError, \"You must only provide a single character for arg {name:s}\");\n"
            "        goto on_error;\n"
            "    }\n"
            "\n"
            "    {name:s} = str_{name:s}[0];\n").format(
                **values_dict)


class StringOut(String):
    sense = "OUT"


class IntegerOut(Integer):
    """Handle Integers pushed out through OUT int *result."""
    sense = "OUT_DONE"
    buildstr = ""
    int_type = "int *"

    def definition(self, default=0, **kwargs):
        # We need to make static storage for the pointers
        storage = "storage_{0:s}".format(self.name)
        bare_type = self.type.split()[0]
        type_definition = Type.definition(
            self, default="&{0:s}".format(storage))

        return (
            "{0:s} {1:s} = 0;\n"
            "{2:s}\n").format(
                bare_type, storage, type_definition)

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        values_dict = {
            "name": name or self.name,
            "result": result}

        return (
            "PyErr_Clear();\n"
            "{result:s} = PyLong_FromLongLong(*{name:s});\n").format(
                **values_dict)

    def python_name(self):
        return None

    def byref(self):
        return self.name

    def call_arg(self):
        return "{0:s}".format(self.name)

    def passthru_call(self):
        return self.name


class PInteger32UnsignedOut(IntegerOut):
    buildstr = ""
    int_type = "uint32_t *"


class PInteger64UnsignedOut(IntegerOut):
    buildstr = ""
    int_type = "uint64_t *"


class Char_and_Length_OUT(Char_and_Length):
    sense = "OUT_DONE"
    buildstr = "l"

    def definition(self, default=0, **kwargs):
        values_dict = {
            "default": default,
            "length": self.length,
            "name": self.name}

        return (
            "    char *{name:s} = NULL;\n"
            "    Py_ssize_t {length:s} = {default:d};\n"
            "    PyObject *tmp_{name:s} = NULL;\n").format(
                **values_dict)

    def error_cleanup(self):
        values_dict = {
            "name": self.name}

        return (
            "    if(tmp_{name:s} != NULL) {{\n"
            "        Py_DecRef(tmp_{name:s});\n"
            "    }}\n").format(**values_dict)

    def python_name(self):
        return self.length

    def byref(self):
        return "&{0:s}".format(self.length)

    def pre_call(self, method, **kwargs):
        values_dict = {
            "length": self.length,
            "name": self.name}

        return (
            "    PyErr_Clear();\n"
            "\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    tmp_{name:s} = PyBytes_FromStringAndSize(NULL, {length:s});\n"
            "#else\n"
            "    tmp_{name:s} = PyString_FromStringAndSize(NULL, {length:s});\n"
            "#endif\n"
            "    if(!tmp_{name:s}) {{\n"
            "        goto on_error;\n"
            "    }}\n"
            "\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    PyBytes_AsStringAndSize(tmp_{name:s}, &{name:s}, (Py_ssize_t *)&{length:s});\n"
            "#else\n"
            "    PyString_AsStringAndSize(tmp_{name:s}, &{name:s}, (Py_ssize_t *)&{length:s});\n"
            "#endif\n").format(**values_dict)

    def to_python_object(self, name=None, result="Py_result", sense="in", **kwargs):
        if "results" in kwargs:
            kwargs["results"].pop(0)

        if sense == "proxied":
            return (
                "py_{0:s} = PyLong_FromSize_t((size_t) {1:s});\n"
                "if(py_{0:s} == NULL) {{\n"
                "    goto on_error;\n"
                "}}\n").format(self.name, self.length)

        values_dict = {
            "length": self.length,
            "name": name or self.name,
            "result": result}

        return (
            "    /* NOTE - this should never happen\n"
            "     * it might indicate an overflow condition.\n"
            "     */\n"
            "    if(func_return > (uint64_t) {length:s}) {{\n"
            "        printf(\"Programming Error - possible overflow!!\\n\");\n"
            "        abort();\n"
            "\n"
            "    // Do we need to truncate the buffer for a short read?\n"
            "    }} else if(func_return < (uint64_t) {length:s}) {{\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "        _PyBytes_Resize(&tmp_{name:s}, (Py_ssize_t) func_return);\n"
            "#else\n"
            "        _PyString_Resize(&tmp_{name:s}, (Py_ssize_t) func_return);\n"
            "#endif\n"
            "    }}\n"
            "\n"
            "    {result:s} = tmp_{name:s};\n").format(**values_dict)

    def python_proxy_post_call(self, result="Py_result"):
        values_dict = {
            "length": self.length,
            "name": self.name,
            "result": result}

        return (
            "{{\n"
            "    char *tmp_buff = NULL;\n"
            "    Py_ssize_t tmp_len = 0;\n"
            "\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    if(PyBytes_AsStringAndSize({result:s}, &tmp_buff, &tmp_len) == -1) {{\n"
            "#else\n"
            "    if(PyString_AsStringAndSize({result:s}, &tmp_buff, &tmp_len) == -1) {{\n"
            "#endif\n"
            "        goto on_error;\n"
            "    }}\n"
            "    /* Bound the user-controlled return length to the buffer\n"
            "     * size that libtsk requested; a Python override that\n"
            "     * returns more bytes than asked for would otherwise\n"
            "     * overflow the caller's buffer.\n"
            "     */\n"
            "    if((size_t) tmp_len > (size_t) {length:s}) {{\n"
            "        tmp_len = (Py_ssize_t) {length:s};\n"
            "    }}\n"
            "    memcpy({name:s}, tmp_buff, tmp_len);\n"
            "    Py_DecRef({result:s});\n"
            "    {result:s} = PyLong_FromLong(tmp_len);\n"
            "    if({result:s} == NULL) {{\n"
            "        goto on_error;\n"
            "    }}\n"
            "}}\n").format(**values_dict)


class TDB_DATA_P(Char_and_Length_OUT):
    bare_type = "TDB_DATA"

    def __init__(self, name, type, *args, **kwargs):
        super().__init__(name, type, *args, **kwargs)

    def definition(self, default=None, **kwargs):
        return Type.definition(self)

    def byref(self):
        return "{0:s}.dptr, &{0:s}.dsize".format(self.name)

    def pre_call(self, method, **kwargs):
        return ""

    def call_arg(self):
        return Type.call_arg(self)

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        values_dict = {
            "name": name or self.name,
            "result": result}

        return (
            "    PyErr_Clear();\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    {result:s} = PyBytes_FromStringAndSize((char *){name:s}->dptr, {name:s}->dsize);\n"
            "#else\n"
            "    {result:s} = PyString_FromStringAndSize((char *){name:s}->dptr, {name:s}->dsize);\n"
            "#endif\n"
            "    talloc_free({name:s});\n").format(**values_dict)

    def from_python_object(self, source, destination, method, **kwargs):
        method.error_set = True
        values_dict = {
            "bare_type": self.bare_type,
            "destination": destination,
            "source": source}

        return (
            "{destination:s} = talloc_zero(self, {bare_type:s});\n"
            "{{\n"
            "    char *buf = NULL;\n"
            "    Py_ssize_t tmp = 0;\n"
            "\n"
            "    PyErr_Clear();\n"
            "\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    if(PyBytes_AsStringAndSize({source:s}, &buf, &tmp) == -1) {{\n"
            "#else\n"
            "    if(PyString_AsStringAndSize({source:s}, &buf, &tmp) == -1) {{\n"
            "#endif\n"
            "        goto on_error;\n"
            "    }}\n"
            "\n"
            "    // Take a copy of the Python string\n"
            "    {destination:s}->dptr = talloc_memdup({destination:s}, buf, tmp);\n"
            "    {destination:s}->dsize = tmp;\n"
            "}}\n"
            "// We no longer need the Python object\n"
            "Py_DecRef({source:s});\n").format(**values_dict)


class TDB_DATA(TDB_DATA_P):
    error_value = (
        "{result:s}.dptr = NULL;\n"
        "return {result:s};")

    def from_python_object(self, source, destination, method, **kwargs):
        method.error_set = True
        values_dict = {
            "destination": destination,
            "source": source}

        return (
            "{{\n"
            "    char *buf = NULL;\n"
            "    Py_ssize_t tmp = 0;\n"
            "\n"
            "    PyErr_Clear();\n"
            "\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    if(PyBytes_AsStringAndSize({source:s}, &buf, &tmp) == -1) {{\n"
            "#else\n"
            "    if(PyString_AsStringAndSize({source:s}, &buf, &tmp) == -1) {{\n"
            "#endif\n"
            "        goto on_error;\n"
            "    }}\n"
            "    // Take a copy of the Python string - This leaks - how to fix it?\n"
            "    {destination:s}.dptr = talloc_memdup(NULL, buf, tmp);\n"
            "    {destination:s}.dsize = tmp;\n"
            "}}\n"
            "// We no longer need the Python object\n"
            "Py_DecRef({source:s});\n").format(**values_dict)

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        values_dict = {
            "name": name or self.name,
            "result": result}

        return (
            "    PyErr_Clear();\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    {result:s} = PyBytes_FromStringAndSize((char *){name:s}.dptr, {name:s}.dsize);\n"
            "#else\n"
            "    {result:s} = PyString_FromStringAndSize((char *){name:s}.dptr, {name:s}.dsize);\n"
            "#endif\n").format(**values_dict)


class Void(Type):
    buildstr = ""
    error_value = "return;"
    original_type = ""

    def __init__(self, name, type="void", *args, **kwargs):
        super().__init__(name, type, *args, **kwargs)

    def comment(self):
        return "void *ctx"

    def definition(self, default=None, **kwargs):
        return ""

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        return (
            "Py_IncRef(Py_None);\n"
            "Py_result = Py_None;\n")

    def call_arg(self):
        return "NULL"

    def byref(self):
        return None

    def assign(self, call, method, target=None, **kwargs):
        # We don't assign the result to anything.
        return (
            "    Py_BEGIN_ALLOW_THREADS\n"
            "    (void) {0:s};\n"
            "    Py_END_ALLOW_THREADS\n").format(call)

    def return_value(self, value):
        return "return;"


class PVoid(Void):
    def __init__(self, name, type="void *", *args, **kwargs):
        super().__init__(name, type, *args, **kwargs)


class StringArray(String):
    interface = "array"
    buildstr = "O"

    def definition(self, default="\"\"", **kwargs):
        return (
            "    char **{0:s} = NULL;\n"
            "    PyObject *py_{0:s} = NULL;\n").format(self.name)

    def byref(self):
        return "&py_{0:s}".format(self.name)

    def from_python_object(self, source, destination, method, context="NULL"):
        method.error_set = True
        values_dict = {
            "destination": destination,
            "source": source}

        return (
            "{{\n"
            "    Py_ssize_t i = 0;\n"
            "    Py_ssize_t size = 0;\n"
            "\n"
            "    if({source:s}) {{\n"
            "        if(!PySequence_Check({source:s})) {{\n"
            "            PyErr_Format(PyExc_ValueError, \"{destination:s} must be a sequence\");\n"
            "            goto on_error;\n"
            "        }}\n"
            "        size = PySequence_Size({source:s});\n"
            "    }}\n"
            "    {destination:s} = talloc_zero_array(NULL, char *, size + 1);\n"
            "\n"
            "    for(i = 0; i < size; i++) {{\n"
            "        PyObject *tmp = PySequence_GetItem({source:s}, i);\n"
            "        if(!tmp) {{\n"
            "            goto on_error;\n"
            "        }}\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "        {destination:s}[i] = PyBytes_AsString(tmp);\n"
            "#else\n"
            "        {destination:s}[i] = PyString_AsString(tmp);\n"
            "#endif\n"
            "\n"
            "        if(!{destination:s}[i]) {{\n"
            "            Py_DecRef(tmp);\n"
            "            goto on_error;\n"
            "        }}\n"
            "        Py_DecRef(tmp);\n"
            "    }}\n"
            "}}\n").format(**values_dict)

    def pre_call(self, method, **kwargs):
        return self.from_python_object(
            "py_{0:s}".format(self.name), self.name, method)

    def error_condition(self):
        return (
            "    if({0:s}) {{\n"
            "        talloc_free({0:s});\n"
            "    }}\n").format(self.name)


class Wrapper(Type):
    """This class represents a wrapped C type """
    sense = "IN"
    error_value = "return NULL;"

    def from_python_object(self, source, destination, method, **kwargs):
        values_dict = {
            "destination": destination,
            "source": source,
            "type": self.type}

        return (
            "     /* First check that the returned value is in fact a Wrapper */\n"
            "     if(!type_check({source:s}, &{type:s}_Type)) {{\n"
            "          PyErr_Format(PyExc_RuntimeError, \"function must return an {type:s} instance\");\n"
            "          goto on_error;\n"
            "     }}\n"
            "\n"
            "     {destination:s} = ({type:s}) ((Gen_wrapper) {source:s})->base;\n"
            "\n"
            "     if(!{destination:s}) {{\n"
            "          PyErr_Format(PyExc_RuntimeError, \"{type:s} instance is no longer valid (was it gc'ed?)\");\n"
            "          goto on_error;\n"
            "}}\n"
            "\n").format(**values_dict)

    def to_python_object(self, **kwargs):
        return ""

    def returned_python_definition(self, default="NULL", sense="in", **kwargs):
        return "{0:s} {1:s} = {2:s};\n".format(
            self.type, self.name, default)

    def byref(self):
        return "&wrapped_{0:s}".format(self.name)

    def definition(self, default="NULL", sense="in", **kwargs):
        result = "    Gen_wrapper wrapped_{0:s} UNUSED = {1:s};\n".format(
            self.name, default)

        if sense == "in" and not "OUT" in self.attributes:
            result += "    {0:s} UNUSED {1:s};\n".format(
                self.type, self.name)

        return result

    def call_arg(self):
        return "{0:s}".format(self.name)

    def pre_call(self, method, python_object_index=1, **kwargs):
        if "OUT" in self.attributes or self.sense == "OUT":
            return ""
        self.original_type = self.type.split()[0]

        values_dict = {
            "name": self.name,
            "original_type": self.original_type,
            "python_object_index": python_object_index,
            "type": self.type}

        return (
            "    if(wrapped_{name:s} == NULL || (PyObject *)wrapped_{name:s} == Py_None) {{\n"
            "        {name:s} = NULL;\n"
            "    }} else if(!type_check((PyObject *)wrapped_{name:s},&{original_type:s}_Type)) {{\n"
            "        PyErr_Format(PyExc_RuntimeError, \"{name:s} must be derived from type {original_type:s}\");\n"
            "        goto on_error;\n"
            "    }} else if(wrapped_{name:s}->base == NULL) {{\n"
            "        PyErr_Format(PyExc_RuntimeError, \"{original_type:s} instance is no longer valid (was it gc'ed?)\");\n"
            "        goto on_error;\n"
            "    }} else {{\n"
            "        {name:s} = ({type:s}) wrapped_{name:s}->base;\n"
            "        if(self->python_object{python_object_index:d} == NULL) {{\n"
            "            self->python_object{python_object_index:d} = (PyObject *) wrapped_{name:s};\n"
            "            Py_IncRef(self->python_object{python_object_index:d});\n"
            "        }}\n"
            "    }}\n").format(**values_dict)

    def assign(self, call, method, target=None, **kwargs):
        method.error_set = True;

        values_dict = {
            "call": call.strip(),
            "incref": INCREF,
            "name": target or self.name,
            "type": self.type}

        result = (
            "    {{\n"
            "        Object returned_object = NULL;\n"
            "\n"
            "        ClearError();\n"
            "\n"
            "        Py_BEGIN_ALLOW_THREADS\n"
            "        // This call will return a Python object if the base is a proxied Python object\n"
            "        // or a talloc managed object otherwise.\n"
            "        returned_object = (Object) {call:s};\n"
            "        Py_END_ALLOW_THREADS\n"
            "\n"
            "        if(check_error()) {{\n"
            "            if(returned_object != NULL) {{\n"
            "                if(self->base_is_python_object != 0) {{\n"
            "                    Py_DecRef((PyObject *) returned_object);\n"
            "                }} else if(self->base_is_internal != 0) {{\n"
            "                    talloc_free(returned_object);\n"
            "                }}\n"
            "            }}\n"
            "            goto on_error;\n"
            "        }}\n").format(**values_dict)

        # Is NULL an acceptable return type? In some Python code NULL
        # can be returned (e.g. in iterators) but usually it should
        # be converted to Py_None.
        if "NULL_OK" in self.attributes:
            result += (
                "        if(returned_object == NULL) {\n"
                "            goto on_error;\n"
                "        }\n")

        result += (
            "        wrapped_{name:s} = new_class_wrapper(returned_object, self->base_is_python_object);\n"
            "\n"
            "        if(wrapped_{name:s} == NULL) {{\n"
            "            if(returned_object != NULL) {{\n"
            "                if(self->base_is_python_object != 0) {{\n"
            "                    Py_DecRef((PyObject *) returned_object);\n"
            "                }} else if(self->base_is_internal != 0) {{\n"
            "                    talloc_free(returned_object);\n"
            "                }}\n"
            "            }}\n"
            "            goto on_error;\n"
            "        }}\n").format(**values_dict)

        if "BORROWED" in self.attributes:
            result += (
                "        #error unchecked BORROWED code segment\n"
                "        {incref:s}(wrapped_{name:s}->base);\n"
                "        if(((Object) wrapped_{name:s}->base)->extension) {{\n"
                "            Py_IncRef((PyObject *) ((Object) wrapped_{name:s}->base)->extension);\n"
                "        }}\n").format(**values_dict)

        result += (
            "    }\n")

        return result

    def to_python_object(
            self, name=None, result="Py_result", sense="in", **kwargs):
        values_dict = {
            "name": name or self.name,
            "result": result}

        if sense == "proxied":
            return (
                "{result:s} = (PyObject *) new_class_wrapper((Object){name:s}, 0);\n").format(
                    **values_dict)

        return "{result:s} = (PyObject *) wrapped_{name:s};\n".format(
            **values_dict)


class PointerWrapper(Wrapper):
    """A pointer to a wrapped class """

    def __init__(self, name, type, *args, **kwargs):
        type = type.split()[0]
        super().__init__(name, type, *args, **kwargs)

    def comment(self):
        return "{0:s} *{1:s}".format(self.type, self.name)

    def definition(self, default="NULL", sense="in", **kwargs):
        result = "    Gen_wrapper wrapped_{0:s} = {1:s};\n".format(
            self.name, default)
        if sense == "in" and not "OUT" in self.attributes:
            result += "    {0:s} *{1:s};\n".format(self.type, self.name)

        return result

    def byref(self):
        return "&wrapped_{0:s}".format(self.name)

    def pre_call(self, method, **kwargs):
        if "OUT" in self.attributes or self.sense == "OUT":
            return ""
        self.original_type = self.type.split()[0]
        values_dict = {
            "name": self.name,
            "original_type": self.original_type}

        return (
            "if(!wrapped_{name:s} || (PyObject *)wrapped_{name:s}==Py_None) {{\n"
            "   {name:s} = NULL;\n"
            "}} else if(!type_check((PyObject *)wrapped_{name:s},&{original_type:s}_Type)) {{\n"
            "     PyErr_Format(PyExc_RuntimeError, \"{name:s} must be derived from type {original_type:s}\");\n"
            "     goto on_error;\n"
            "}} else {{\n"
            "   {name:s} = ({original_type:s} *)&wrapped_{name:s}->base;\n"
            "}};\n").format(**values_dict)


class StructWrapper(Wrapper):
    """A wrapper for struct classes """
    active = False

    def __init__(self, name, type, *args, **kwargs):
        super().__init__(name, type, *args, **kwargs)
        self.original_type = type.split()[0]

    def assign(self, call, method, target=None, borrowed=True, **kwargs):
        self.original_type = self.type.split()[0]
        values_dict = {
            "call": call.strip(),
            "name": target or self.name,
            "type": self.original_type}

        result = (
            "\n"
            "        PyErr_Clear();\n"
            "\n"
            "        wrapped_{name:s} = (Gen_wrapper) PyObject_New(py{type:s}, &{type:s}_Type);\n"
            "        if(wrapped_{name:s} == NULL) {{\n"
            "            return NULL;\n"
            "        }}\n"
            "\n").format(**values_dict)

        if borrowed:
            result += (
                "        // Base is borrowed from another object.\n"
                "        wrapped_{name:s}->base = {call:s};\n"
                "        wrapped_{name:s}->base_is_python_object = 0;\n"
                "        wrapped_{name:s}->base_is_internal = 0;\n"
                "        wrapped_{name:s}->python_object1 = NULL;\n"
                "        wrapped_{name:s}->python_object2 = NULL;\n"
                "\n").format(**values_dict)
        else:
            result += (
                "        wrapped_{name:s}->base = {call:s};\n"
                "        wrapped_{name:s}->base_is_python_object = 0;\n"
                "        wrapped_{name:s}->base_is_internal = 1;\n"
                "        wrapped_{name:s}->python_object1 = NULL;\n"
                "        wrapped_{name:s}->python_object2 = NULL;\n"
                "\n").format(**values_dict)

        if "NULL_OK" in self.attributes:
            result += (
                "        if(wrapped_{name:s}->base == NULL) {{\n"
                "             Py_DecRef((PyObject *) wrapped_{name:s});\n"
                "             if(check_error()) {{\n"
                "                 goto on_error;\n"
                "             }}\n"
                "             return NULL;\n"
                "        }}\n").format(**values_dict)

        result += (
            "        // A NULL object gets translated to a None\n"
            "        if(wrapped_{name:s}->base == NULL) {{\n"
            "            Py_DecRef((PyObject *) wrapped_{name:s});\n"
            "            Py_IncRef(Py_None);\n"
            "            wrapped_{name:s} = (Gen_wrapper) Py_None;\n"
            "        }}\n").format(**values_dict)

        # TODO: with the following code commented out is makes no sense to have the else clause here.
        #   "    }} else {{\n").format(**values_dict)

        # if "FOREIGN" in self.attributes:
        #     result += "// Not taking references to foreign memory\n"
        # elif "BORROWED" in self.attributes:
        #     result += "talloc_reference({name:s}->ctx, {name:s}->base);\n".format(**values_dict)
        # else:
        #     result += "talloc_steal({name:s}->ctx, {name:s}->base);\n".format(**values_dict)
        # result += "}}\n"

        return result

    def byref(self):
        return "&{0:s}".format(self.name)

    def definition(self, default="NULL", sense="in", **kwargs):
        result = "    Gen_wrapper wrapped_{0:s} = {1:s};\n".format(
            self.name, default)
        if sense == "in" and not "OUT" in self.attributes:
            result += "    {0:s} *{1:s} = NULL;\n".format(
                self.original_type, self.name)

        return result;


class PointerStructWrapper(StructWrapper):
    def from_python_object(self, source, destination, method, **kwargs):
        values_dict = {
            "destination": destination,
            "source": source,
            "type": self.original_type}
        return (
            "    if({source:s} == NULL || !type_check({source:s}, &{type:s}_Type)) {{\n"
            "        PyErr_Format(PyExc_RuntimeError,\n"
            "            \"proxied {type:s} method returned NULL or wrong type\");\n"
            "        goto on_error;\n"
            "    }}\n"
            "    {destination:s} = ({type:s} *) ((Gen_wrapper) {source:s})->base;\n"
        ).format(**values_dict)

    def byref(self):
        return "&wrapped_{0:s}".format(self.name)


class Timeval(Type):
    """Handle struct timeval values."""
    interface = "numeric"
    buildstr = "f"

    def definition(self, default=None, **kwargs):
        return (
            "struct timeval {0:s};\n".format(self.name) +
            self.local_definition(default=default, **kwargs))

    def local_definition(self, default=None, **kwargs):
        return "float {0:s}_flt;\n".format(self.name)

    def byref(self):
        return "&{0:s}_flt".format(self.name)

    def pre_call(self, method, **kwargs):
        return (
            "{0:s}.tv_sec = (int){0:s}_flt;\n"
            "{0:s}.tv_usec = ({0:s}_flt - {0:s}.tv_sec) * 1e6;\n").format(
                self.name)

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        values_dict = {
            "name": name or self.name,
            "result": result}

        return (
            "{name:s}_flt = (double)({name:s}.tv_sec) + {name:s}.tv_usec;\n"
            "{result:s} = PyFloat_FromDouble({name:s}_flt);\n").format(
                **values_dict)


class PyObject(Type):
    """Accept an opaque Python object."""
    interface = "opaque"
    buildstr = "O"

    def definition(self, default="NULL", **kwargs):
        self.default = default
        values_dict = {
            "default": self.default,
            "name": self.name}

        return (
            "PyObject *{name:s} = {default:s};\n").format(
                **values_dict)

    def byref(self):
        return "&{0:s}".format(self.name)


type_dispatcher = {
    "IN unsigned char *": String,
    "IN char *": String,

    "unsigned char *": String,
    "char *": String,

    "ZString": ZString,

    "OUT unsigned char *": StringOut,
    "OUT char *": StringOut,

    "OUT uint64_t *": PInteger64UnsignedOut,
    "OUT uint32_t *": PInteger32UnsignedOut,

    "void *": PVoid,
    "void": Void,

    "TDB_DATA *": TDB_DATA_P,
    "TDB_DATA": TDB_DATA,
    "TSK_INUM_T": Integer,

    "off_t": Integer64,
    "size_t": Integer64Unsigned,
    "ssize_t": Integer64,
    "time_t": Integer64,

    "unsigned long": LongUnsigned,
    "long": Long,
    "unsigned long int": LongUnsigned,
    "long int": Integer,
    "unsigned int": Integer,
    "int": Integer,

    "uint64_t": Integer64Unsigned,
    "uint32_t": Integer32Unsigned,
    "uint16_t": Integer16Unsigned,
    "uint8_t": Integer8Unsigned,
    "int64_t": Integer64,
    "int32_t": Integer32,
    "int16_t": Integer16,
    "int8_t": Integer8,
    "char": Char,

    "struct timeval": Timeval,
    "char **": StringArray,
    "PyObject *": PyObject,
}

method_attributes = ["BORROWED", "DESTRUCTOR", "IGNORE"]


class ResultException:
    value = 0
    exception = "PyExc_IOError"

    def __init__(self, check, exception, message):
        self.check = check
        self.exception = exception
        self.message = message

    def write(self, out):
        out.write((
            "\n"
            "/* Handle exceptions */\n"
            "if({0:s}) {{\n"
            "    PyErr_Format(PyExc_{1:s}, {2:s});\n"
            "    goto on_error;\n"
            "}}\n"
            "\n").format(self.check, self.exception, self.message))


class Method:
    default_re = re.compile(r"DEFAULT\(([A-Z_a-z0-9]+)\) =(.+);")
    exception_re = re.compile(r"RAISES\(([^,]+),\s*([^\)]+)\) =(.+);")
    typedefed_re = re.compile(r"struct (.+)_t \*")

    def __init__(
        self, class_name, base_class_name, name, args, return_type,
        myclass=None):
        if not isinstance(myclass, ClassGenerator):
            raise RuntimeError("myclass must be a class generator")

        self.args = []
        self.base_class_name = base_class_name
        self.class_name = class_name
        self.defaults = {}
        self.definition_class_name = class_name
        self.docstring = ""
        self.error_set = False
        self.exception = None
        self.name = name
        self.myclass = myclass

        for type, name in args:
            self.add_arg(type, name)

        try:
            self.return_type = dispatch("func_return", return_type)
            self.return_type.attributes.add("OUT")
            self.return_type.original_type = return_type
        except KeyError:
            # Is it a wrapped type?
            if return_type:
                log("Unable to handle return type {0:s}.{1:s} {2:s}".format(
                    self.class_name, self.name, return_type))
                # pdb.set_trace()
            self.return_type = PVoid("func_return")

    def get_string(self):
        """Retrieves a string representation."""
        return "def {0:s} {1:s}({2:s}):".format(
            self.return_type.get_string(), self.name,
            " , ".join([a.get_string() for a in self.args]))

    def clone(self, new_class_name):
        self.find_optional_vars()

        result = self.__class__(
            new_class_name, self.base_class_name, self.name, [], "void *",
            myclass=self.myclass)
        result.args = self.args
        result.return_type = self.return_type
        result.definition_class_name = self.definition_class_name
        result.defaults = self.defaults
        result.exception = self.exception

        return result

    def find_optional_vars(self):
        for line in self.docstring.splitlines():
            m = self.default_re.search(line)
            if m:
                name = m.group(1)
                value = m.group(2)
                log("Setting default value for {0:s} of {1:s}".format(
                    m.group(1), m.group(2)))
                self.defaults[name] = value.strip()

            m = self.exception_re.search(line)
            if m:
                self.exception = ResultException(
                    m.group(1), m.group(2), m.group(3))

    def write_local_vars(self, out):
        self.find_optional_vars()

        # We do it in two passes - first mandatory then optional
        kwlist = "    const char *kwlist[] = {"
        # Mandatory
        for type in self.args:
            python_name = type.python_name()
            if python_name and python_name not in self.defaults:
                kwlist += "\"{0:s}\", ".format(python_name)

        for type in self.args:
            python_name = type.python_name()
            if python_name and python_name in self.defaults:
                kwlist += "\"{0:s}\", ".format(python_name)

        kwlist += " NULL};\n"

        for type in self.args:
            out.write(
                "    // DEBUG: local arg type: {0:s}\n".format(
                    type.__class__.__name__))
            python_name = type.python_name()
            try:
                out.write(type.definition(default=self.defaults[python_name]))
            except KeyError:
                out.write(type.definition())

        # Make up the format string for the parse args in two pases
        parse_line = ""
        for type in self.args:
            python_name = type.python_name()
            if type.buildstr and python_name not in self.defaults:
                parse_line += type.buildstr

        optional_args = ""
        for type in self.args:
            python_name = type.python_name()
            if type.buildstr and python_name in self.defaults:
                optional_args += type.buildstr

        if optional_args:
            parse_line += "|" + optional_args

        # Iterators have a different prototype and do not need to
        # unpack any args
        if not "iternext" in self.name:
            # Now parse the args from Python objects
            out.write("\n")
            out.write(kwlist)
            out.write((
                "\n"
                "    if(!PyArg_ParseTupleAndKeywords(args, kwds, \"{0:s}\", ").format(
                    parse_line))

            tmp = ["(char **) kwlist"]
            for type in self.args:
                ref = type.byref()
                if ref:
                    tmp.append(ref)

            out.write(", ".join(tmp))
            self.error_set = True
            out.write(
                ")) {\n"
                "        goto on_error;\n"
                "    }\n")

    def error_condition(self):
        result = ""
        if "DESTRUCTOR" in self.return_type.attributes:
            result += "self->base = NULL;\n"

        # If a Python wrapper was already allocated but check_error() fired
        # in the postcall, it must be released to avoid a refcount leak.
        if isinstance(self.return_type,
                      (StructWrapper, PointerStructWrapper, Wrapper, PointerWrapper)):
            name = self.return_type.name
            result += (
                "    if(wrapped_{0:s} != NULL) {{\n"
                "        Py_DecRef((PyObject *) wrapped_{0:s});\n"
                "    }}\n").format(name)

        if hasattr(self, "args"):
            for type in self.args:
                if hasattr(type, "error_cleanup"):
                    result += type.error_cleanup()

        result += "    return NULL;\n";
        return result

    def write_definition(self, out):
        out.write(
            "\n"
            "/********************************************************\n"
            "Autogenerated wrapper for function:\n")
        out.write(self.comment())
        out.write("********************************************************/\n")

        self._prototype(out)
        out.write((
            "{{\n"
            "    PyObject *returned_result = NULL;\n"
            "    PyObject *Py_result = NULL;\n"
            "\n"
            "    // DEBUG: return type: {0:s}\n"
            "    ").format(
                self.return_type.__class__.__name__))

        out.write(self.return_type.definition())

        self.write_local_vars(out)

        values_dict = {
            "class_name": self.class_name,
            "method": self.name}

        out.write((
            "\n"
            "    // Make sure that we have something valid to wrap\n"
            "    if(self->base == NULL) {{\n"
            "        return PyErr_Format(PyExc_RuntimeError, \"{class_name:s} object no longer valid\");\n"
            "    }}\n"
            "\n").format(**values_dict))

        # Precall preparations
        out.write("    // Precall preparations\n")
        out.write(self.return_type.pre_call(self))
        for type in self.args:
            out.write(type.pre_call(self))

        values_dict = {
            "class_name": self.class_name,
            "def_class_name": self.definition_class_name,
            "method": self.name}

        out.write((
            "    // Check the function is implemented\n"
            "    {{\n"
            "        void *method = (void *) (({def_class_name:s}) self->base)->{method:s};\n"
            "\n"
            "        if(method == NULL || (void *) unimplemented == (void *) method) {{\n"
            "            PyErr_Format(PyExc_RuntimeError, \"{class_name:s}.{method:s} is not implemented\");\n"
            "            goto on_error;\n"
            "        }}\n"
            "\n"
            "        // Make the call\n"
            "        ClearError();\n").format(**values_dict))

        base = "(({0:s}) self->base)".format(self.definition_class_name)
        call = "        {0:s}->{1:s}({2:s}".format(base, self.name, base)
        tmp = ""
        for argument in self.args:
            call_arg = argument.call_arg()
            if isinstance(argument, EnumType):
              tmp += ", ({0:s}) {1:s}".format(argument.type, call_arg)
            else:
              tmp += ", {0:s}".format(call_arg)

        call += "{0:s})".format(tmp)

        # Now call the wrapped function
        out.write(self.return_type.assign(call, self, borrowed=False))
        if self.exception:
            self.exception.write(out)

        self.error_set = True

        out.write(
            "    };\n"
            "\n"
            "    // Postcall preparations\n")
        # Postcall preparations
        post_calls = []

        post_call = self.return_type.post_call(self)
        post_calls.append(post_call)
        out.write("    {0:s}".format(post_call))

        for type in self.args:
            post_call = type.post_call(self)
            if post_call not in post_calls:
                post_calls.append(post_call)
                out.write("    {0:s}".format(post_call))

        # Now assemble the results
        results = [self.return_type.to_python_object()]
        for type in self.args:
            if type.sense == "OUT_DONE":
                results.append(type.to_python_object(results=results))

        # If all the results are returned by reference we dont need
        # to prepend the void return value at all.
        if isinstance(self.return_type, Void) and len(results) > 1:
            results.pop(0)

        out.write(
            "\n"
            "    // prepare results\n")
        # Make a tuple of results and pass them back
        if len(results) > 1:
            out.write("returned_result = PyList_New(0);\n")
            for result in results:
                out.write(result)
                out.write(
                    "PyList_Append(returned_result, Py_result);\n"
                    "Py_DecRef(Py_result);\n")
            out.write("return returned_result;\n")
        else:
            out.write(results[0])
            # This useless code removes compiler warnings
            out.write(
                "    returned_result = Py_result;\n"
                "    return returned_result;\n")

        # Write the error part of the function
        if self.error_set:
            out.write((
                "\n"
                "on_error:\n"
                "{0:s}").format(self.error_condition()))

        out.write("};\n\n")

    def add_arg(self, type, name):
        try:
            t = type_dispatcher[type](name, type)
        except KeyError:
            # Sometimes types must be typedefed in advance
            try:
                m = self.typedefed_re.match(type)
                type = m.group(1)
                log("Trying {0:s} for {1:s}".format(type, m.group(0)))
                t = type_dispatcher[type](name, type)
            except (KeyError, AttributeError):
                log("Unable to handle type {0:s}.{1:s} {2:s}".format(
                    self.class_name, self.name, type))
                return

        # Here we collapse char * + int type interfaces into a
        # coherent string like interface.
        try:
            previous = self.args[-1]
            if t.interface == "integer" and previous.interface == "string":

                # We make a distinction between IN variables and OUT
                # variables
                if previous.sense == "OUT":
                    cls = Char_and_Length_OUT
                else:
                    cls = Char_and_Length

                cls = cls(
                    previous.name,
                    previous.type,
                    name, type)

                self.args[-1] = cls

                return
        except IndexError:
            pass

        self.args.append(t)

    def comment(self):
        args = []
        for type in self.args:
            args.append(type.comment())

        return "{0:s} {1:s}.{2:s}({3:s});\n".format(
            self.return_type.original_type, self.class_name, self.name,
            ", ".join(args))

    def prototype(self, out):
        self._prototype(out)
        out.write(";\n")

    def _prototype(self, out):
        values_dict = {
            "class_name": self.class_name,
            "method": self.name}

        out.write(
            "static PyObject *py{class_name:s}_{method:s}(py{class_name:s} *self, PyObject *args, PyObject *kwds)".format(
                **values_dict))

    def PyMethodDef(self, out):
        docstring = self.comment() + "\n\n" + self.docstring.strip()
        values_dict = {
            "class_name": self.class_name,
            "docstring": format_as_docstring(docstring),
            "name": self.name}

        out.write((
            "    {{ \"{name:s}\",\n"
            "      (PyCFunction) py{class_name:s}_{name:s},\n"
            "      METH_VARARGS|METH_KEYWORDS,\n"
            "      \"{docstring:s}\" }},\n"
            "\n").format(**values_dict))


class IteratorMethod(Method):
    """A method which implements an iterator."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        # Tell the return type that a NULL Python return is ok
        self.return_type.attributes.add("NULL_OK")

    def get_string(self):
        """Retrieves a string representation."""
        return "Iterator returning {0:s}.".format(self.return_type.get_string())

    def _prototype(self, out):
        values_dict = {
            "class_name": self.class_name,
            "method": self.name}

        out.write(
            "static PyObject *py{class_name:s}_{method:s}(py{class_name:s} *self)".format(
                **values_dict))

    def PyMethodDef(self, out):
        # This method should not go in the method table as its linked
        # in directly.
        pass


class SelfIteratorMethod(IteratorMethod):
    def write_definition(self, out):
        out.write(
            "\n"
            "/********************************************************\n"
            " * Autogenerated wrapper for function:\n")
        out.write(self.comment())
        out.write(
            "********************************************************/\n")

        self._prototype(out)

        values_dict = {
            "class_name": self.class_name,
            "method": self.name}

        out.write((
            "{{\n"
            "    if(self->base == NULL) {{\n"
            "        return PyErr_Format(PyExc_RuntimeError,\n"
            "            \"{class_name:s}.{method:s}: object is not bound to any libtsk handle\");\n"
            "    }}\n"
            "    (({class_name:s}) self->base)->{method:s}(({class_name:s}) self->base);\n"
            "    return PyObject_SelfIter((PyObject *) self);\n"
            "}}\n").format(**values_dict))


class ConstructorMethod(Method):
    # Python constructors are a bit different than regular methods

    def _prototype(self, out):
        values_dict = {
            "class_name": self.class_name,
            "method": self.name}

        out.write(
            "static int py{class_name:s}_init(py{class_name:s} *self, PyObject *args, PyObject *kwds)\n".format(
                **values_dict))

    def prototype(self, out):
        self._prototype(out)

        values_dict = {
            "class_name": self.class_name}

        out.write((
            ";\n"
            "static void py{class_name:s}_initialize_proxies(py{class_name:s} *self, void *item);\n").format(
                **values_dict))

    def write_destructor(self, out):
        values_dict = {
            "class_name": self.class_name,
            "free": FREE}

        out.write((
            "static void {class_name:s}_dealloc(py{class_name:s} *self) {{\n"
            "    struct _typeobject *ob_type = NULL;\n"
            "\n"
            "    if(self != NULL) {{\n"
            "        if(self->base != NULL) {{\n"
            "            if(self->base_is_python_object != 0) {{\n"
            "                Py_DecRef((PyObject*) self->base);\n"
            "            }} else if(self->base_is_internal != 0) {{\n"
            "                {free:s}(self->base);\n"
            "            }}\n"
            "            self->base = NULL;\n"
            "        }}\n"
            "        if(self->python_object2 != NULL) {{\n"
            "            Py_DecRef(self->python_object2);\n"
            "            self->python_object2 = NULL;\n"
            "        }}\n"
            "        if(self->python_object1 != NULL) {{\n"
            "            Py_DecRef(self->python_object1);\n"
            "            self->python_object1 = NULL;\n"
            "        }}\n"
            "        ob_type = Py_TYPE(self);\n"
            "        if(ob_type != NULL && ob_type->tp_free != NULL) {{\n"
            "            ob_type->tp_free((PyObject*) self);\n"
            "        }}\n"
            "    }}\n"
            "}}\n"
            "\n").format(**values_dict))

    def error_condition(self):
        return "    return -1;";

    def initialise_proxies(self, out):
        self.myclass.module.function_definitions.add(
            "py{0:s}_initialize_proxies".format(self.class_name))

        values_dict = {
            "class_name": self.class_name}

        out.write((
            "static void py{class_name:s}_initialize_proxies(py{class_name:s} *self, void *item) {{\n"
            "    {class_name:s} target = ({class_name:s}) item;\n"
            "\n"
            "    /* Maintain a reference to the Python object\n"
            "     * in the C object extension\n"
            "     */\n"
            "    ((Object) item)->extension = self;\n"
            "\n").format(**values_dict))

        # Install proxies for all the method in the current class.
        for method in self.myclass.module.classes[self.class_name].methods:
            if method.name.startswith("_"):
                continue

            # Since the SleuthKit uses close method also for freeing it needs
            # to be handled separately to prevent the C/C++ code calling back
            # into a garbage collected Python object. For close we keep the
            # default implementation and have its destructor deal with
            # correctly closing the SleuthKit object.
            if method.name != "close":
                values_dict = {
                    "class_name": method.class_name,
                    "definition_class_name": method.definition_class_name,
                    "name": method.name,
                    "proxied_name": method.proxied.get_name()}

                out.write((
                    "    if(check_method_override((PyObject *) self, &{class_name:s}_Type, \"{name:s}\")) {{\n"
                    "        // Proxy the {name:s} method\n"
                    "        (({definition_class_name:s}) target)->{name:s} = {proxied_name:s};\n"
                    "    }}\n").format(**values_dict))

        out.write("}\n\n")

    def write_definition(self, out):
        self.initialise_proxies(out)
        self._prototype(out)
        values_dict = {
            "class_name": self.class_name,
            "definition_class_name": self.definition_class_name}

        out.write((
            "{{\n"
            "    {class_name:s} result_constructor = NULL;\n").format(
                **values_dict))

        # pdb.set_trace()
        self.write_local_vars(out)

        # Assign the initialise_proxies handler
        out.write((
            "    /* Release any state from a prior __init__ call so that\n"
            "     * re-initialization does not leak keepalives or libtsk handles.\n"
            "     */\n"
            "    Py_CLEAR(self->python_object1);\n"
            "    Py_CLEAR(self->python_object2);\n"
            "    if(self->base != NULL) {{\n"
            "        if(self->base_is_python_object != 0) {{\n"
            "            Py_DecRef((PyObject *) self->base);\n"
            "        }} else if(self->base_is_internal != 0) {{\n"
            "            talloc_free(self->base);\n"
            "        }}\n"
            "        self->base = NULL;\n"
            "    }}\n"
            "\n"
            "    /* Initialise is used to keep a reference on the object?\n"
            "     * If not called no longer valid warnings have been seen\n"
            "     * on Windows.\n"
            "     */\n"
            "    self->initialise = (function_initialize_Gen_wrapper_t) py{class_name:s}_initialize_proxies;\n"
            "\n").format(**values_dict))

        # Precall preparations
        python_object_index = 1
        for type in self.args:
            out.write(type.pre_call(
                self, python_object_index=python_object_index))
            python_object_index += 1

        # Now call the wrapped function
        out.write((
            "    ClearError();\n"
            "\n"
            "    /* Allocate a new instance */\n"
            "    self->base = ({class_name:s}) alloc_{class_name:s}();\n"
            "    if(self->base == NULL) {{\n"
            "        PyErr_NoMemory();\n"
            "        goto on_error;\n"
            "    }}\n"
            "    self->base_is_python_object = 0;\n"
            "    self->base_is_internal = 1;\n"
            "    self->object_is_proxied = 0;\n"
            "\n"
            "    /* Update the target by replacing its methods with proxies\n"
            "     * to call back into Python\n"
            "     */\n"
            "    py{class_name:s}_initialize_proxies(self, self->base);\n"
            "\n"
            "    /* Now call the constructor */\n"
            "    Py_BEGIN_ALLOW_THREADS\n"
            "    result_constructor = CONSTRUCT_INITIALIZE({class_name:s}, {definition_class_name:s}, Con, self->base").format(
                **values_dict))

        tmp = ""
        for argument in self.args:
            call_arg = argument.call_arg()
            if isinstance(argument, EnumType):
              tmp += ", ({0:s}) {1:s}".format(argument.type, call_arg)
            else:
              tmp += ", {0:s}".format(call_arg)

        self.error_set = True
        out.write(tmp)

        out.write((
            ");\n"
            "    Py_END_ALLOW_THREADS\n"
            "\n"
            "    if(!CheckError(EZero)) {{\n"
            "        char *buffer = NULL;\n"
            "        PyObject *exception = resolve_exception(&buffer);\n"
            "\n"
            "        PyErr_Format(exception, \"%s\", buffer);\n"
            "        ClearError();\n"
            "        goto on_error;\n"
            "    }}\n"
            "    if(result_constructor == NULL) {{\n"
            "        PyErr_Format(PyExc_IOError, \"Unable to construct class {class_name:s}\");\n"
            "        goto on_error;\n"
            "    }}\n"
            "\n"
            "    return 0;\n").format(**values_dict))

        # Write the error part of the function.
        if self.error_set:
            out.write((
                "\n"
                "on_error:\n"
                "    if(self->python_object2 != NULL) {{\n"
                "        Py_DecRef(self->python_object2);\n"
                "        self->python_object2 = NULL;\n"
                "    }}\n"
                "    if(self->python_object1 != NULL) {{\n"
                "        Py_DecRef(self->python_object1);\n"
                "        self->python_object1 = NULL;\n"
                "    }}\n"
                "    if(self->base != NULL) {{\n"
                "        talloc_free(self->base);\n"
                "        self->base = NULL;\n"
                "    }}\n"
                "{0:s}\n").format(self.error_condition()))

        out.write("}\n\n")


class GetattrMethod(Method):
    def __init__(self, class_name, base_class_name, myclass):
        # Cannot use super here due to certain logic in Method.__init__().
        self._attributes = []
        self.base_class_name = base_class_name
        self.class_name = class_name
        self.error_set = True
        self.myclass = myclass
        self.name = ""
        self.return_type = Void("")

        self.rename_class_name(class_name)

    def get_string(self):
        """Retrieves a string representation."""
        result = ""
        for class_name, attr in self.get_attributes():
            result += "    {0:s}\n".format(attr.get_string())

        return result

    def add_attribute(self, attr):
        if attr.name:
            self._attributes.append([self.class_name, attr])

    def rename_class_name(self, new_name):
        """This allows us to rename the class_name at a later stage.
        Required for late initialization of Structs whose name is not
        know until much later on.
        """
        # TODO fix this behavior, new_name can be None but it is unclear what
        # the behavious should be. Python 3 requires the values to be set to
        # string types.
        if not new_name:
            self.class_name = ""
            self.name = ""
        else:
            self.class_name = new_name
            self.name = "py{0:s}_getattr".format(new_name)

        for attribure in self._attributes:
            attribure[0] = new_name

    def get_attributes(self):
        for class_name, attr in self._attributes:
            try:
                # If its not an active struct, skip it
                if (not type_dispatcher[attr.type].active and
                    not attr.type in self.myclass.module.active_structs):
                    continue

            except KeyError:
                pass

            yield class_name, attr

    def clone(self, class_name):
        result = self.__class__(class_name, self.base_class_name, self.myclass)
        result._attributes = self._attributes[:]

        return result

    def prototype(self, out):
        if not self.name:
            return

        values_dict = {
            "class_name": self.class_name,
            "name": self.name}

        # Define getattr.
        out.write(
            "static PyObject *{name:s}(py{class_name:s} *self, PyObject *name);\n".format(
                **values_dict))

        # Define getters.
        for _, attr in self.get_attributes():
            values_dict = {
                "class_name": self.class_name,
                "name": attr.name}

            out.write(
                "PyObject *py{class_name:s}_{name:s}_getter(py{class_name:s} *self, PyObject *arguments);\n".format(
                    **values_dict))

    def built_ins(self, out):
        """Check for some built in attributes we need to support."""
        out.write(
            "    if(strcmp(name, \"__members__\") == 0) {\n"
            "        PyMethodDef *i = NULL;\n"
            "        PyObject *list_object = NULL;\n"
            "        PyObject *string_object = NULL;\n"
            "\n"
            "        list_object = PyList_New(0);\n"
            "        if(list_object == NULL) {\n"
            "            goto on_error;\n"
            "        }\n"
            "\n")

        # Add attributes
        for class_name, attr in self.get_attributes():
            values_dict = {
                "name": attr.name}

            out.write((
                "#if PY_MAJOR_VERSION >= 3\n"
                "        string_object = PyUnicode_FromString(\"{name:s}\");\n"
                "#else\n"
                "        string_object = PyString_FromString(\"{name:s}\");\n"
                "#endif\n"
                "        if(string_object == NULL) {{\n"
                "            Py_DecRef(list_object);\n"
                "            goto on_error;\n"
                "        }}\n"
                "        if(PyList_Append(list_object, string_object) < 0) {{\n"
                "            Py_DecRef(string_object);\n"
                "            Py_DecRef(list_object);\n"
                "            goto on_error;\n"
                "        }}\n"
                "        Py_DecRef(string_object);\n"
                "\n").format(**values_dict))

        # Add methods
        out.write((
            "\n"
            "        for(i = {0:s}_methods; i->ml_name; i++) {{\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "            string_object = PyUnicode_FromString(i->ml_name);\n"
            "#else\n"
            "            string_object = PyString_FromString(i->ml_name);\n"
            "#endif\n"
            "            if(string_object == NULL) {{\n"
            "                Py_DecRef(list_object);\n"
            "                goto on_error;\n"
            "            }}\n"
            "            if(PyList_Append(list_object, string_object) < 0) {{\n"
            "                Py_DecRef(string_object);\n"
            "                Py_DecRef(list_object);\n"
            "                goto on_error;\n"
            "            }}\n"
            "            Py_DecRef(string_object);\n"
            "        }}\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "        if( utf8_string_object != NULL ) {{\n"
            "            Py_DecRef(utf8_string_object);\n"
            "        }}\n"
            "#endif\n"
            "        return list_object;\n"
            "    }}\n").format(self.class_name))

    def write_definition(self, out):
        if not self.name:
            return

        values_dict = {
            "class_name": self.class_name,
            "name": self.name}

        out.write((
            "static PyObject *py{class_name:s}_getattr(py{class_name:s} *self, PyObject *pyname) {{\n"
            "    PyObject *result = NULL;\n"
            "    char *name = NULL;\n"
            "\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    PyObject *utf8_string_object  = NULL;\n"
            "#endif\n"
            "\n"
            "    // Try to hand it off to the Python native handler first\n"
            "    result = PyObject_GenericGetAttr((PyObject*) self, pyname);\n"
            "\n"
            "    if(result) {{\n"
            "        return result;\n"
            "    }}\n"
            "\n"
            "    PyErr_Clear();\n"
            "    // No - nothing interesting was found by python\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    utf8_string_object = PyUnicode_AsUTF8String(pyname);\n"
            "\n"
            "    if(utf8_string_object != NULL) {{\n"
            "        name = PyBytes_AsString(utf8_string_object);\n"
            "    }}\n"
            "#else\n"
            "    name = PyString_AsString(pyname);\n"
            "#endif\n"
            "\n"
            "    if(!self->base) {{\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "        if( utf8_string_object != NULL ) {{\n"
            "            Py_DecRef(utf8_string_object);\n"
            "        }}\n"
            "#endif\n"
            "        return PyErr_Format(PyExc_RuntimeError, \"Wrapped object ({class_name:s}.{name:s}) no longer valid\");\n"
            "    }}\n"
            "    if(!name) {{\n"
            "        goto on_error;\n"
            "    }}\n").format(**values_dict))

        self.built_ins(out)

        out.write(
            "\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    if( utf8_string_object != NULL ) {{\n"
            "        Py_DecRef(utf8_string_object);\n"
            "    }}\n"
            "#endif\n"
            "    return PyObject_GenericGetAttr((PyObject *) self, pyname);\n")

        # Write the error part of the function.
        if self.error_set:
            out.write(
                "on_error:\n"
                "#if PY_MAJOR_VERSION >= 3\n"
                "    if( utf8_string_object != NULL ) {{\n"
                "        Py_DecRef(utf8_string_object);\n"
                "    }}\n"
                "#endif\n" + self.error_condition())

        out.write("}\n\n")

        self.write_definition_getters(out)

    def write_definition_getters(self, out):
        for _, attr in self.get_attributes():
            if self.base_class_name:
                call = "((({0:s}) self->base)->{1:s})".format(
                    self.class_name, attr.name)
            else:
                call = "(self->base->{0:s})".format(attr.name)

            values_dict = {
                "class_name": self.class_name,
                "name": attr.name,
                "python_obj": attr.to_python_object(),
                "python_assign": attr.assign(call, self, borrowed=True),
                "python_def": attr.definition(sense="out")}

            out.write((
                "PyObject *py{class_name:s}_{name:s}_getter(py{class_name:s} *self, PyObject *arguments) {{\n"
                "    PyObject *Py_result = NULL;\n"
                "{python_def:s}\n"
                "\n"
                "    if(self->base == NULL) {{\n"
                "        return PyErr_Format(PyExc_RuntimeError,\n"
                "            \"{class_name:s}.{name:s}: object is not bound \"\n"
                "            \"to any libtsk handle (was it instantiated \"\n"
                "            \"directly?)\");\n"
                "    }}\n"
                "\n"
                "{python_assign:s}\n"
                "{python_obj:s}\n"
                "\n"
                "    return Py_result;\n"
                "\n").format(**values_dict))

            # Work-around for the String class that generates code that contains "goto on_error".
            if isinstance(attr, String):
                out.write((
                    "on_error:\n"
                    "    {0:s}\n").format(attr.error_value))

            out.write("}\n\n")

    def PyGetSetDef(self, out):
        for _, attr in self.get_attributes():
            # TODO: improve docstring.
            docstring = "{0:s}.".format(attr.name)
            values_dict = {
                "class_name": self.class_name,
                "docstring": format_as_docstring(docstring),
                "name": attr.name}

            out.write((
                "    {{ \"{name:s}\",\n"
                "      (getter) py{class_name:s}_{name:s}_getter,\n"
                "      (setter) 0,\n"
                "      \"{docstring:s}\",\n"
                "      NULL }},\n"
                "\n").format(**values_dict))


class ProxiedMethod(Method):
    def __init__(self, method, myclass):
        # Cannot use super here due to certain logic in Method.__init__().
        self.args = method.args
        self.base_class_name = method.base_class_name
        self.class_name = method.class_name
        self.defaults = {}
        self.definition_class_name = method.definition_class_name
        self.docstring = "Proxy for {0:s}".format(method.name)
        self.error_set = False
        self.exception = None
        self.method = method
        self.myclass = myclass
        self.name = method.name
        self.return_type = method.return_type

    def get_name(self):
        return "Proxied{0:s}_{1:s}".format(
            self.myclass.class_name, self.name)

    def _prototype(self, out):
        out.write("static {0:s} {1:s}({2:s} self".format(
            self.return_type.type.strip(), self.get_name(),
            self.definition_class_name))

        for arg in self.args:
            tmp = arg.comment().strip()
            if tmp:
                out.write(", {0:s}".format(tmp))

        out.write(")")

    def prototype(self, out):
        self._prototype(out)
        out.write(";\n")

    def write_definition(self, out):
        name = self.get_name()
        if name in self.myclass.module.function_definitions:
            return
        else:
            self.myclass.module.function_definitions.add(name)

        self._prototype(out)
        self._write_definition(out)

    def _write_definition(self, out):
        out.write(
            " {\n"
            "    PyGILState_STATE gil_state;\n"
            "    PyObject *Py_result = NULL;\n"
            "    PyObject *method_name = NULL;\n")

        out.write(self.return_type.returned_python_definition())

        for arg in self.args:
            out.write(arg.local_definition())
            out.write("PyObject *py_{0:s} = NULL;\n".format(arg.name))

        out.write((
            "\n"
            "    // Grab the GIL so we can do Python stuff\n"
            "    gil_state = PyGILState_Ensure();\n"
            "\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    method_name = PyUnicode_FromString(\"{0:s}\");\n"
            "#else\n"
            "    method_name = PyString_FromString(\"{0:s}\");\n"
            "#endif\n"
            "    /* PyUnicode_FromString sets MemoryError on failure;\n"
            "     * propagate via the proxied error machinery rather than\n"
            "     * passing NULL to PyObject_CallMethodObjArgs (which would\n"
            "     * raise SystemError and lose the original cause).\n"
            "     */\n"
            "    if(method_name == NULL) {{\n"
            "        pytsk_fetch_error();\n"
            "        goto on_error;\n"
            "    }}\n").format(self.name))

        out.write("\n// Obtain Python objects for all the args:\n")
        for arg in self.args:
            out.write(arg.to_python_object(
                result=("py_{0:s}".format(arg.name)), sense="proxied",
                BORROWED=True))

        out.write((
            "    if(((Object) self)->extension == NULL) {{\n"
            "        RaiseError(ERuntimeError, \"No proxied object in {0:s}\");\n"
            "        goto on_error;\n"
            "    }}\n").format(self.myclass.class_name))

        out.write(
            "\n"
            "    // Now call the method\n"
            "    Py_result = PyObject_CallMethodObjArgs((PyObject *) ((Object) self)->extension, method_name, ")

        for arg in self.args:
            out.write("py_{0:s},".format(arg.name))

        # Sentinal
        out.write(
            "NULL);\n"
            "\n")

        self.error_set = True
        out.write((
            "    /* Check for Python errors */\n"
            "    if(PyErr_Occurred()) {{\n"
            "        pytsk_fetch_error();\n"
            "\n"
            "        goto on_error;\n"
            "    }}\n"
            "\n").format(CURRENT_ERROR_FUNCTION))

        for arg in self.args:
            out.write(arg.python_proxy_post_call())

        # Now convert the Python value back to a value
        return_type = self.return_type.from_python_object(
            "Py_result", self.return_type.name, self, context="self")
        out.write("    {0:s}".format(return_type))

        # For Wrapper return types the Python wrapper keeps the C object alive.
        # Transfer ownership to the parent's python_object2 so the C pointer
        # remains valid after this callback returns to libtsk.
        if isinstance(self.return_type, Wrapper) and not isinstance(
                self.return_type, (StructWrapper, PointerStructWrapper)):
            out.write(
                "    if(Py_result != NULL) {\n"
                "        PyObject *old = ((Gen_wrapper) ((Object) self)->extension)->python_object2;\n"
                "        if(old != NULL) Py_DecRef(old);\n"
                "        ((Gen_wrapper) ((Object) self)->extension)->python_object2 = Py_result;\n"
                "        Py_result = NULL;\n"
                "    }\n"
                "    Py_DecRef(method_name);\n"
                "\n")
        else:
            out.write(
                "    if(Py_result != NULL) {\n"
                "        Py_DecRef(Py_result);\n"
                "    }\n"
                "    Py_DecRef(method_name);\n"
                "\n")

        # Decref all our Python objects:
        for arg in self.args:
            out.write((
                "    if(py_{0:s} != NULL) {{\n"
                "        Py_DecRef(py_{0:s});\n"
                "    }}\n").format(arg.name))

        out.write((
            "    PyGILState_Release(gil_state);\n"
            "\n"
            "    {0:s}\n").format(
                self.return_type.return_value("func_return")))

        if self.error_set:
            out.write(
                "\n"
                "on_error:\n"
                "    if(Py_result != NULL) {\n"
                "        Py_DecRef(Py_result);\n"
                "    }\n"
                "    Py_DecRef(method_name);\n"
                "\n")

            # Decref all our Python objects:
            for arg in self.args:
                out.write((
                    "    if(py_{0:s} != NULL) {{\n"
                    "        Py_DecRef(py_{0:s});\n"
                    "    }}\n").format(arg.name))

            out.write((
                "    PyGILState_Release(gil_state);\n"
                "\n"
                "    {0:s}\n").format(
                    self.error_condition()))

        out.write(
            "}\n"
            "\n")

    def error_condition(self):
        values_dict = {
            "result": "func_return"}
        return self.return_type.error_value.format(**values_dict)


class StructConstructor(ConstructorMethod):
    """A constructor for struct wrappers - basically just allocate
    memory for the struct.
    """

    def prototype(self, out):
        return Method.prototype(self, out)

    def write_destructor(self, out):
        """We do not deallocate memory from structs.

        This is a real problem since struct memory is usually
        allocated in some proprietary way and we cant just call free
        on it when done.
        """
        values_dict = {
            "class_name": self.class_name}

        out.write((
            "static void {class_name:s}_dealloc(py{class_name:s} *self) {{\n"
            "    struct _typeobject *ob_type = NULL;\n"
            "\n"
            "    if(self != NULL) {{\n"
            "        if(self->base != NULL) {{\n"
            "            self->base = NULL;\n"
            "        }}\n"
            "        ob_type = Py_TYPE(self);\n"
            "        if(ob_type != NULL && ob_type->tp_free != NULL) {{\n"
            "            ob_type->tp_free((PyObject*) self);\n"
            "        }}\n"
            "    }}\n"
            "}}\n"
            "\n").format(**values_dict))

    def write_definition(self, out):
        values_dict = {
            "class_name": self.class_name}

        out.write((
            "static int py{class_name:s}_init(py{class_name:s} *self, PyObject *args, PyObject *kwds) {{\n"
            "    // Base is borrowed from another object.\n"
            "    self->base = NULL;\n"
            "    return 0;\n"
            "}}\n"
            "\n").format(**values_dict))


class EmptyConstructor(ConstructorMethod):
    def prototype(self, out):
        return Method.prototype(self, out)

    def write_definition(self, out):
        values_dict = {
            "class_name": self.class_name}

        out.write(
            "static int py{class_name:s}_init(py{class_name:s} *self, PyObject *args, PyObject *kwds) {{\n"
            "    return 0;\n"
            "}}\n"
            "\n".format(**values_dict))


class ClassGenerator:
    docstring = ""

    def __init__(self, class_name, base_class_name, module):
        self.class_name = class_name
        self.methods = []
        # self.methods = [DefinitionMethod(
        #     class_name, base_class_name, "_definition", [], "",
        #     myclass=self)]
        self.module = module
        self.constructor = EmptyConstructor(
            class_name, base_class_name, "Con", [], "", myclass=self)

        self.base_class_name = base_class_name
        self.attributes = GetattrMethod(
            self.class_name, self.base_class_name, self)
        self.modifier = set()
        self.active = True
        self.iterator = None

    def get_string(self):
        """Retrieves a string representation."""
        result = (
            "#{0:s}\n"
            "Class {1:s}({2:s}):\n"
            "    Constructor:{3:s}\n"
            "    Attributes:\n{4:s}\n"
            "    Methods:\n").format(
                self.docstring, self.class_name, self.base_class_name,
                self.constructor.get_string(), self.attributes.get_string())

        for method in self.methods:
            result += "        {0:s}\n".format(method.get_string())

        return result

    def prepare(self):
        """This method is called just before we need to write the
        output and allows us to do any last minute fixups.
        """
        pass

    def is_active(self):
        """Returns true if this class is active and should be generated"""
        if self.class_name in self.module.active_structs:
            return True

        if (not self.active or self.modifier and
            ("PRIVATE" in self.modifier or "ABSTRACT" in self.modifier)):
            log("{0:s} is not active {1!s}".format(
                self.class_name, self.modifier))
            return False

        return True

    def clone(self, new_class_name):
        """Creates a clone of this class - usefull when implementing
        class extensions.
        """
        result = ClassGenerator(new_class_name, self.class_name, self.module)
        result.constructor = self.constructor.clone(new_class_name)
        result.methods = [
            method.clone(new_class_name) for method in self.methods]
        result.attributes = self.attributes.clone(new_class_name)

        return result

    def add_attribute(self, attr_name, attr_type, modifier, *args, **kwargs):
        try:
            if not self.module.classes[attr_type].is_active():
                return
        except KeyError:
            pass

        try:
            # All attribute references are always borrowed - that
            # means we dont want to free them after accessing them
            type_class = dispatch(
                attr_name, "BORROWED {0:s}".format(attr_type), *args, **kwargs)
        except KeyError:
            # TODO: fix that self.class_name is None.
            log("Unknown attribute type {0:s} for {1!s}.{2:s}".format(
                attr_type, self.class_name, attr_name))
            return

        type_class.attributes.add(modifier)
        self.attributes.add_attribute(type_class)

    def add_constructor(self, method_name, args, return_type, docstring):
        if method_name.startswith("Con"):
            self.constructor = ConstructorMethod(
                self.class_name, self.base_class_name, method_name, args,
                return_type, myclass=self)
            self.constructor.docstring = docstring

    def struct(self, out):
        values_dict = {
            "class_name": self.class_name}

        out.write((
            "\n"
            "typedef struct {{\n"
            "    PyObject_HEAD\n"
            "    {class_name:s} base;\n"
            "    int base_is_python_object;\n"
            "    int base_is_internal;\n"
            "    PyObject *python_object1;\n"
            "    PyObject *python_object2;\n"
            "    int object_is_proxied;\n"
            "\n"
            "    void (*initialise)(Gen_wrapper self, void *item);\n"
            "}} py{class_name:s};\n").format(**values_dict))

    def code(self, out):
        if not self.constructor:
            raise RuntimeError(
                "No constructor found for class {0:s}".format(self.class_name))

        self.constructor.write_destructor(out)
        self.constructor.write_definition(out)
        if self.attributes:
            self.attributes.write_definition(out)

        for method in self.methods:
            method.write_definition(out)

            if hasattr(method, "proxied"):
                method.proxied.write_definition(out)

    def initialise(self):
        values_dict = {
            "class_name": self.class_name}

        result = (
            "python_wrappers[TOTAL_CCLASSES].class_ref = (Object)&__{class_name:s};\n"
            "python_wrappers[TOTAL_CCLASSES].python_type = &{class_name:s}_Type;\n").format(**values_dict)

        func_name = "py{class_name:s}_initialize_proxies".format(**values_dict)
        if func_name in self.module.function_definitions:
            result += (
                "python_wrappers[TOTAL_CCLASSES].initialize_proxies = (void (*)(Gen_wrapper, void *)) &{0:s};\n").format(
                func_name)

        result += "TOTAL_CCLASSES++;\n"
        return result

    def PyGetSetDef(self, out):
        out.write(
            "static PyGetSetDef {0:s}_get_set_definitions[] = {{\n".format(
                self.class_name))

        if self.attributes:
            self.attributes.PyGetSetDef(out)

        out.write(
            "    {NULL, NULL, NULL, NULL, NULL}  /* Sentinel */\n"
            "};\n"
            "\n")

    def PyMethodDef(self, out):
        out.write("static PyMethodDef {0:s}_methods[] = {{\n".format(
            self.class_name))

        for method in self.methods:
            method.PyMethodDef(out)

        out.write(
            "    {NULL, NULL, 0, NULL}  /* Sentinel */\n"
            "};\n"
            "\n")

    def prototypes(self, out):
        """Write prototype suitable for .h file"""
        out.write("/* static PyTypeObject {0:s}_Type; */\n".format(
            self.class_name))

        self.constructor.prototype(out)

        out.write("static void {0:s}_dealloc(py{0:s} *self);\n".format(
            self.class_name))

        if self.attributes:
            self.attributes.prototype(out)
        for method in self.methods:
            method.prototype(out)

            # Each method, except for close, needs a proxy method that
            # is called when the object is sub typed.
            if method.name == "close":
                continue

            method.proxied = ProxiedMethod(method, method.myclass)
            method.proxied.prototype(out)

    def numeric_protocol_int(self):
        pass

    def numeric_protocol_nonzero(self):
        values_dict = {
            "class_name": self.class_name}

        return (
            "static int {class_name:s}_nonzero(py{class_name:s} *v) {{\n"
            "    return v->base != 0;\n"
            "}}\n").format(**values_dict)

    def numeric_protocol(self, out):
        args = {
            "class": self.class_name}
        for type, func in [
            ("nonzero", self.numeric_protocol_nonzero),
            ("int", self.numeric_protocol_int)]:

            definition = func()
            if definition:
                out.write(definition)
                args[type] = "{0:s}_{1:s}".format(self.class_name, type)
            else:
                args[type] = "0"

        out.write((
            "#if PY_MAJOR_VERSION >= 3\n"
            "static PyNumberMethods {class:s}_as_number = {{\n"
            "    (binaryfunc)    0,             /* nb_add */\n"
            "    (binaryfunc)    0,             /* nb_subtract */\n"
            "    (binaryfunc)    0,             /* nb_multiply */\n"
            "    (binaryfunc)    0,             /* nb_remainder */\n"
            "    (binaryfunc)    0,             /* nb_divmod */\n"
            "    (ternaryfunc)   0,             /* nb_power */\n"
            "    (unaryfunc)     0,             /* nb_negative */\n"
            "    (unaryfunc)     0,             /* nb_positive */\n"
            "    (unaryfunc)     0,             /* nb_absolute */\n"
            "    (inquiry)       {nonzero:s},   /* nb_bool */\n"
            "    (unaryfunc)     0,             /* nb_invert */\n"
            "    (binaryfunc)    0,             /* nb_lshift */\n"
            "    (binaryfunc)    0,             /* nb_rshift */\n"
            "    (binaryfunc)    0,             /* nb_and */\n"
            "    (binaryfunc)    0,             /* nb_xor */\n"
            "    (binaryfunc)    0,             /* nb_or */\n"
            "    (unaryfunc)     {int:s},       /* nb_int */\n"
            "    (void *)        NULL,          /* nb_reserved */\n"
            "    (unaryfunc)     0,             /* nb_float */\n"
            "\n"
            "    (binaryfunc)    0,             /* nb_inplace_add */\n"
            "    (binaryfunc)    0,             /* nb_inplace_subtract */\n"
            "    (binaryfunc)    0,             /* nb_inplace_multiply */\n"
            "    (binaryfunc)    0,             /* nb_inplace_remainder */\n"
            "    (ternaryfunc)   0,             /* nb_inplace_power */\n"
            "    (binaryfunc)    0,             /* nb_inplace_lshift */\n"
            "    (binaryfunc)    0,             /* nb_inplace_rshift */\n"
            "    (binaryfunc)    0,             /* nb_inplace_and */\n"
            "    (binaryfunc)    0,             /* nb_inplace_xor */\n"
            "    (binaryfunc)    0,             /* nb_inplace_or */\n"
            "\n"
            "    (binaryfunc)    0,             /* nb_floor_divide */\n"
            "    (binaryfunc)    0,             /* nb_true_divide */\n"
            "    (binaryfunc)    0,             /* nb_inplace_floor_divide */\n"
            "    (binaryfunc)    0,             /* nb_inplace_true_divide */\n"
            "\n"
            "    (unaryfunc)     0,             /* nb_index */\n"
            "}};\n"
            "#else\n"
            "static PyNumberMethods {class:s}_as_number = {{\n"
            "    (binaryfunc)    0,             /* nb_add */\n"
            "    (binaryfunc)    0,             /* nb_subtract */\n"
            "    (binaryfunc)    0,             /* nb_multiply */\n"
            "    (binaryfunc)    0,             /* nb_divide */\n"
            "    (binaryfunc)    0,             /* nb_remainder */\n"
            "    (binaryfunc)    0,             /* nb_divmod */\n"
            "    (ternaryfunc)   0,             /* nb_power */\n"
            "    (unaryfunc)     0,             /* nb_negative */\n"
            "    (unaryfunc)     0,             /* nb_positive */\n"
            "    (unaryfunc)     0,             /* nb_absolute */\n"
            "    (inquiry)       {nonzero:s},   /* nb_nonzero */\n"
            "    (unaryfunc)     0,             /* nb_invert */\n"
            "    (binaryfunc)    0,             /* nb_lshift */\n"
            "    (binaryfunc)    0,             /* nb_rshift */\n"
            "    (binaryfunc)    0,             /* nb_and */\n"
            "    (binaryfunc)    0,             /* nb_xor */\n"
            "    (binaryfunc)    0,             /* nb_or */\n"
            "    (coercion)      0,             /* nb_coerce */\n"
            "    (unaryfunc)     {int:s},       /* nb_int */\n"
            "    (unaryfunc)     0,             /* nb_long */\n"
            "    (unaryfunc)     0,             /* nb_float */\n"
            "    (unaryfunc)     0,             /* nb_oct */\n"
            "    (unaryfunc)     0,             /* nb_hex */\n"
            "\n"
            "    (binaryfunc)    0,             /* nb_inplace_add */\n"
            "    (binaryfunc)    0,             /* nb_inplace_subtract */\n"
            "    (binaryfunc)    0,             /* nb_inplace_multiply */\n"
            "    (binaryfunc)    0,             /* nb_inplace_divide */\n"
            "    (binaryfunc)    0,             /* nb_inplace_remainder */\n"
            "    (ternaryfunc)   0,             /* nb_inplace_power */\n"
            "    (binaryfunc)    0,             /* nb_inplace_lshift */\n"
            "    (binaryfunc)    0,             /* nb_inplace_rshift */\n"
            "    (binaryfunc)    0,             /* nb_inplace_and */\n"
            "    (binaryfunc)    0,             /* nb_inplace_xor */\n"
            "    (binaryfunc)    0,             /* nb_inplace_or */\n"
            "\n"
            "    (binaryfunc)    0,             /* nb_floor_divide */\n"
            "    (binaryfunc)    0,             /* nb_true_divide */\n"
            "    (binaryfunc)    0,             /* nb_inplace_floor_divide */\n"
            "    (binaryfunc)    0,             /* nb_inplace_true_divide */\n"
            "\n"
            "    (unaryfunc)     0,             /* nb_index */\n"
            "}};\n"
            "#endif /* PY_MAJOR_VERSION >= 3 */\n"
            "\n").format(**args))

        return "&{class:s}_as_number".format(**args)

    def PyTypeObject(self, out):
        docstring = "{0:s}: {1:s}".format(
            self.class_name, format_as_docstring(self.docstring))

        args = {
            "class": self.class_name,
            "module": self.module.name,
            "iterator": 0,
            "iternext": 0,
            "tp_str": 0,
            "tp_eq": 0,
            "getattr_func": 0,
            "docstring": docstring}

        if self.attributes:
            args["getattr_func"] = self.attributes.name

        args["numeric_protocol"] = self.numeric_protocol(out)
        if "ITERATOR" in self.modifier:
            args["iterator"] = "PyObject_SelfIter"
            args["iternext"] = "py{0:s}_iternext".format(self.class_name)

        if "SELF_ITER" in self.modifier:
            args["iterator"] = "py{0:s}___iter__".format(self.class_name)

        if "TP_STR" in self.modifier:
            args["tp_str"] = "py{0:s}___str__".format(self.class_name)

        if "TP_EQUAL" in self.modifier:
            args["tp_eq"] = "{0:s}_eq".format(self.class_name)

        out.write((
            "static PyTypeObject {class:s}_Type = {{\n"
            "    PyVarObject_HEAD_INIT(NULL, 0)\n"
            "    /* tp_name */\n"
            "    \"{module:s}.{class:s}\",\n"
            "    /* tp_basicsize */\n"
            "    sizeof(py{class:s}),\n"
            "    /* tp_itemsize */\n"
            "    0,\n"
            "    /* tp_dealloc */\n"
            "    (destructor) {class:s}_dealloc,\n"
            "    /* tp_print */\n"
            "    0,\n"
            "    /* tp_getattr */\n"
            "    0,\n"
            "    /* tp_setattr */\n"
            "    0,\n"
            "    /* tp_compare */\n"
            "    0,\n"
            "    /* tp_repr */\n"
            "    0,\n"
            "    /* tp_as_number */\n"
            "    {numeric_protocol:s},\n"
            "    /* tp_as_sequence */\n"
            "    0,\n"
            "    /* tp_as_mapping */\n"
            "    0,\n"
            "    /* tp_hash */\n"
            "    0,\n"
            "    /* tp_call */\n"
            "    0,\n"
            "    /* tp_str */\n"
            "    (reprfunc) {tp_str!s},\n"
            "    /* tp_getattro */\n"
            "    (getattrofunc) {getattr_func!s},\n"
            "    /* tp_setattro */\n"
            "    0,\n"
            "    /* tp_as_buffer */\n"
            "    0,\n"
            "    /* tp_flags */\n"
            "    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,\n"
            "    /* tp_doc */\n"
            "    \"{docstring:s}\",\n"
            "    /* tp_traverse */\n"
            "    0,\n"
            "    /* tp_clear */\n"
            "    0,\n"
            "    /* tp_richcompare */\n"
            "    {tp_eq!s},\n"
            "    /* tp_weaklistoffset */\n"
            "    0,\n"
            "    /* tp_iter */\n"
            "    (getiterfunc) {iterator!s},\n"
            "    /* tp_iternext */\n"
            "    (iternextfunc) {iternext!s},\n"
            "    /* tp_methods */\n"
            "    {class:s}_methods,\n"
            "    /* tp_members */\n"
            "    0,\n"
            "    /* tp_getset */\n"
            "    {class:s}_get_set_definitions,\n"
            "    /* tp_base */\n"
            "    0,\n"
            "    /* tp_dict */\n"
            "    0,\n"
            "    /* tp_descr_get */\n"
            "    0,\n"
            "    /* tp_descr_set */\n"
            "    0,\n"
            "    /* tp_dictoffset */\n"
            "    0,\n"
            "    /* tp_init */\n"
            "    (initproc) py{class:s}_init,\n"
            "    /* tp_alloc */\n"
            "    0,\n"
            "    /* tp_new */\n"
            "    0,\n"
            "}};\n"
            "\n").format(**args))


class StructGenerator(ClassGenerator):
    """A wrapper generator for structs."""

    def __init__(self, class_name, module):
        self.class_name = class_name
        self.methods = []
        self.module = module
        self.base_class_name = None
        self.active = False
        self.modifier = set()
        self.constructor = None
        self.attributes = GetattrMethod(
            self.class_name, self.base_class_name, self)

    def get_string(self):
        """Retrieves a string representation."""
        return (
            "# {0:s}\n"
            "Struct {1:s}:\n"
            "{2:s}\n").format(
                self.docstring, self.class_name, self.attributes.get_string())

    def prepare(self):
        # This is needed for late stage initialization - sometimes
        # our class_name is not know until now.
        if not self.constructor:
            self.constructor = StructConstructor(
                self.class_name, self.base_class_name, "Con", [], "void",
                myclass=self)

            self.attributes.rename_class_name(self.class_name)
            for x in self.attributes._attributes:
                x[1].attributes.add("FOREIGN")

    def struct(self, out):
        values_dict = {
            "class_name": self.class_name}

        out.write((
            "\n"
            "typedef struct {{\n"
            "    PyObject_HEAD\n"
            "    {class_name:s} *base;\n"
            "    int base_is_python_object;\n"
            "    int base_is_internal;\n"
            "    PyObject *python_object1;\n"
            "    PyObject *python_object2;\n"
            "    int object_is_proxied;\n"
            "    {class_name:s} *cbase;\n"
            "}} py{class_name:s};\n").format(
                **values_dict))

    def initialise(self):
        return ""


class EnumConstructor(ConstructorMethod):
    def prototype(self, out):
        return Method.prototype(self, out)

    def write_destructor(self, out):
        values_dict = {
            "class_name": self.class_name}

        out.write((
            "static void {class_name:s}_dealloc(py{class_name:s} *self) {{\n"
            "    struct _typeobject *ob_type = NULL;\n"
            "\n"
            "    if(self != NULL) {{\n"
            "        Py_DecRef(self->value);\n"
            "        ob_type = Py_TYPE(self);\n"
            "        if(ob_type != NULL && ob_type->tp_free != NULL) {{\n"
            "            ob_type->tp_free((PyObject*) self);\n"
            "        }}\n"
            "    }}\n"
            "}}\n").format(**values_dict))

    def write_definition(self, out):
        self.myclass.modifier.add("TP_STR")
        self.myclass.modifier.add("TP_EQUAL")
        self._prototype(out)

        values_dict = {
            "class_name": self.class_name}

        out.write((
            "{{\n"
            "    const char *kwlist[] = {{\"value\", NULL}};\n"
            "\n"
            "    if(!PyArg_ParseTupleAndKeywords(args, kwds, \"O\", (char **) kwlist, &self->value)) {{\n"
            "        goto on_error;\n"
            "    }}\n"
            "\n"
            "    Py_IncRef(self->value);\n"
            "\n"
            "    return 0;\n"
            "\n"
            "on_error:\n"
            "    return -1;\n"
            "}}\n"
            "\n").format(**values_dict))


class Enum(StructGenerator):
    def __init__(self, name, module):
        super().__init__(name, module)
        self.values = []
        self.name = name
        self.attributes = None
        self.active = True

    def get_string(self):
        """Retrieves a string representation."""
        result = "Enum {0:s}:\n".format(self.name)
        for attr in self.values:
            result += "    {0:s}\n".format(attr)

        return result

    def prepare(self):
        self.constructor = EnumConstructor(
            self.class_name, self.base_class_name, "Con", [], "void",
            myclass=self)
        StructGenerator.prepare(self)

    def struct(self, out):
        values_dict = {
            "class_name": self.class_name}

        out.write((
            "\n"
            "typedef struct {{\n"
            "    PyObject_HEAD\n"
            "    PyObject *value;\n"
            "}} py{class_name:s};\n"
            "\n"
            "int {class_name:s}_init_type(\n"
            "    PyTypeObject *type_object )\n"
            "{{\n"
            "    type_object->tp_dict = PyDict_New();\n"
            "    if(type_object->tp_dict == NULL) {{\n"
            "        return 0;\n"
            "    }}\n").format(
                **values_dict))

        if self.values:
            out.write("    PyObject *integer_object = NULL;\n")

            for attr in self.values:
                values_dict = {
                    "class_name": self.class_name,
                    "value": attr}

                # Each enum value must succeed; otherwise the type's
                # tp_dict is partially populated and the module loads
                # with broken constants. Bail out cleanly so PyType_Ready
                # never sees a half-initialized enum.
                out.write((
                    "    integer_object = PyLong_FromLong({value:s});\n"
                    "    if(integer_object == NULL) {{\n"
                    "        Py_DecRef(type_object->tp_dict);\n"
                    "        type_object->tp_dict = NULL;\n"
                    "        return 0;\n"
                    "    }}\n"
                    "    if(PyDict_SetItemString(type_object->tp_dict, \"{value:s}\", integer_object) < 0) {{\n"
                    "        Py_DecRef(integer_object);\n"
                    "        Py_DecRef(type_object->tp_dict);\n"
                    "        type_object->tp_dict = NULL;\n"
                    "        return 0;\n"
                    "    }}\n"
                    "    Py_DecRef(integer_object);\n"
                    "\n").format(**values_dict))

        out.write((
            "    return( 1 );\n"
            "}\n"
            "\n"))

    def PyGetSetDef(self, out):
        out.write((
            "static PyGetSetDef {0:s}_get_set_definitions[] = {{\n"
            "    {{NULL, NULL, NULL, NULL, NULL}}  /* Sentinel */\n"
            "}};\n"
            "\n").format(self.class_name))

    def PyMethodDef(self, out):
        out.write((
            "static PyMethodDef {0:s}_methods[] = {{\n"
            "    {{NULL, NULL, 0, NULL}}  /* Sentinel */\n"
            "}};\n"
            "\n").format(self.class_name))

    def numeric_protocol_nonzero(self):
        pass

    def numeric_protocol_int(self):
        values_dict = {
            "class_name": self.class_name}

        return (
            "static PyObject *{class_name:s}_int(py{class_name:s} *self) {{\n"
            "    Py_IncRef(self->value);\n"
            "    return self->value;\n"
            "}}\n").format(**values_dict)

    def initialise(self):
        return "\n"


class EnumType(Integer):
    buildstr = "i"

    def __init__(self, name, type, *args, **kwargs):
        super().__init__(name, type, *args, **kwargs)
        self.type = type

    def definition(self, default=None, **kwargs):
        # Force the enum to be an int just in case the compiler chooses
        # a random size.
        if default:
            return "    int {0:s} = {1:s};\n".format(self.name, default)
        else:
            return "    int UNUSED {0:s} = 0;\n".format(self.name)

    def to_python_object(self, name=None, result="Py_result", **kwargs):
        values_dict = {
            "name": name or self.name,
            "result": result}

        return (
            "    PyErr_Clear();\n"
            "#if PY_MAJOR_VERSION >= 3\n"
            "    {result:s} = PyLong_FromLong({name:s});\n"
            "#else\n"
            "    {result:s} = PyInt_FromLong({name:s});\n"
            "#endif\n").format(**values_dict)

    def pre_call(self, method, **kwargs):
        method.error_set = True

        values_dict = {
            "name": self.name,
            "type": self.type}

        return ""


class HeaderParser(lexer.SelfFeederMixIn):
    tokens = [
        ["INITIAL", r"#define\s+", "PUSH_STATE", "DEFINE"],
        ["DEFINE", r"([A-Za-z_0-9]+)\s+[^\n]+", "DEFINE,POP_STATE", None],
        ["DEFINE", r"\n", "POP_STATE", None],
        # Ignore macros with args
        ["DEFINE", r"\([^\n]+", "POP_STATE", None],

        # Recognize ansi c comments
        [".", r"/\*(.)", "PUSH_STATE", "COMMENT"],
        ["COMMENT", r"(.+?)\*/\s+", "COMMENT_END,POP_STATE", None],
        ["COMMENT", r"(.+)", "COMMENT", None],

        # And c++ comments
        [".", r"//([^\n]+)", "COMMENT", None],

        # An empty line clears the current comment
        [".", r"\r?\n\r?\n", "CLEAR_COMMENT", None],

        # Ignore whitespace
        [".", r"\s+", "SPACE", None],
        [".", r"\\\n", "SPACE", None],

        # Recognize CCLASS() definitions
        ["INITIAL", r"^([A-Z]+)?\s*CCLASS\(([A-Z_a-z0-9]+)\s*,\s*([A-Z_a-z0-9]+)\)",
         "PUSH_STATE,CCLASS_START", "CCLASS"],

        ["CCLASS", r"^\s*(FOREIGN|ABSTRACT|PRIVATE)?([0-9A-Z_a-z ]+( |\*))METHOD\(([A-Z_a-z0-9]+),\s*([A-Z_a-z0-9]+),?",
         "PUSH_STATE,METHOD_START", "METHOD"],
        ["METHOD", r"\s*([0-9A-Z a-z_]+\s+\*?\*?)([0-9A-Za-z_]+),?", "METHOD_ARG", None],
        ["METHOD", r"\);", "POP_STATE,METHOD_END", None],

        ["CCLASS", r"^\s*(FOREIGN|ABSTRACT)?([0-9A-Z_a-z ]+\s+\*?)\s*([A-Z_a-z0-9]+)\s*;",
         "CCLASS_ATTRIBUTE", None],
        ["CCLASS", "END_CCLASS", "END_CCLASS,POP_STATE", None],

        # Recognize struct definitions (With name)
        ["INITIAL", r"([A-Z_a-z0-9 ]+)?struct\s+([A-Z_a-z0-9]+)\s+{",
         "PUSH_STATE,STRUCT_START", "STRUCT"],

        # Without name (using typedef)
        ["INITIAL", r"typedef\s+struct\s+{",
         "PUSH_STATE,TYPEDEF_STRUCT_START", "STRUCT"],

        ["STRUCT", r"^\s*([0-9A-Z_a-z ]+\s+\*?)\s*([A-Z_a-z0-9]+)(?:\[([A-Z_a-z0-9]+)\])?\s*;",
         "STRUCT_ATTRIBUTE", None],

        ["STRUCT", r"^\s*([0-9A-Z_a-z ]+)\*\s+([A-Z_a-z0-9]+)\s*;",
         "STRUCT_ATTRIBUTE_PTR", None],

        # Struct ended with typedef
        ["STRUCT", r"}\s+([0-9A-Za-z_]+);", "POP_STATE,TYPEDEF_STRUCT_END", None],
        ["STRUCT", "}", "POP_STATE,STRUCT_END", None],

        # Handle recursive struct or union definition (At the moment
        # we cant handle them at all)
        ["(RECURSIVE_)?STRUCT", r"(struct|union)\s+([_A-Za-z0-9]+)?\s*{", "PUSH_STATE", "RECURSIVE_STRUCT"],
        ["RECURSIVE_STRUCT", r"}\s+[0-9A-Za-z]+", "POP_STATE", None],
        ["RECURSIVE_STRUCT", "};", "POP_STATE", None],

        # Process enums (2 forms - named and typedefed)
        ["INITIAL", r"enum\s+([0-9A-Za-z_]+)\s+{", "PUSH_STATE,ENUM_START", "ENUM"],
        # Unnamed
        ["INITIAL", r"typedef\s+enum\s+{", "PUSH_STATE,TYPEDEF_ENUM_START", "ENUM"],
        ["ENUM", r"([0-9A-Za-z_]+)\s+=[^\n]+", "ENUM_VALUE", None],

        # Typedefed ending
        ["ENUM", r"}\s+([0-9A-Za-z_]+);", "POP_STATE,TYPEDEFED_ENUM_END", None],
        ["ENUM", r"}", "POP_STATE,ENUM_END", None],

        ["INITIAL", r"BIND_STRUCT\(([0-9A-Za-z_ \*]+)\)", "BIND_STRUCT", None],

        # A simple typedef of one type for another type:
        ["INITIAL", r"typedef ([A-Za-z_0-9]+) +([^;]+);", "SIMPLE_TYPEDEF", None],

        # Handle proxied directives
        ["INITIAL", r"PXXROXY_CCLASS\(([A-Za-z0-9_]+)\)", "PROXY_CCLASS", None],

    ]

    def __init__(self, name, verbose=0, base=""):
        if DEBUG > 0:
          verbose = 1

        self.module = Module(name)
        self.base = base
        super().__init__(verbose=verbose)

        file_object = io.BytesIO(
            b"// Base object\n"
            b"CCLASS(Object, Obj)\n"
            b"END_CCLASS\n")
        self.parse_fd(file_object)

    current_comment = ""

    def COMMENT(self, t, m):
        self.current_comment += m.group(1) + "\n"

    def COMMENT_END(self, t, m):
        self.current_comment += m.group(1)

    def CLEAR_COMMENT(self, t, m):
        self.current_comment = ""

    def DEFINE(self, t, m):
        line = m.group(0)
        line = line.split("/*")[0]
        if "\"" in line:
            type = "string"
        else:
            type = "integer"

        name = m.group(1).strip()
        if (len(name) > 3 and name[0] != "_" and name == name.upper() and
            name not in self.module.constants_blacklist):
            self.module.add_constant(name, type)

    current_class = None

    def CCLASS_START(self, t, m):
        class_name = m.group(2).strip()
        base_class_name = m.group(3).strip()

        try:
            self.current_class = self.module.classes[base_class_name].clone(class_name)
        except (KeyError, AttributeError):
            log("Base class {0:s} is not defined !!!!".format(base_class_name))
            self.current_class = ClassGenerator(class_name, base_class_name, self.module)

        self.current_class.docstring = self.current_comment
        self.current_class.modifier.add(m.group(1))
        self.module.add_class(self.current_class, Wrapper)
        identifier = "{0:s} *".format(class_name)
        type_dispatcher[identifier] = PointerWrapper

    current_method = None

    def METHOD_START(self, t, m):
        return_type = m.group(2).strip()
        method_name = m.group(5).strip()
        modifier = m.group(1) or ""

        if "PRIVATE" in modifier:
            return

        # Is it a regular method or a constructor?
        self.current_method = Method
        if (return_type == self.current_class.class_name and
            method_name.startswith("Con")):
            self.current_method = ConstructorMethod
        elif method_name == "iternext":
            self.current_method = IteratorMethod
            self.current_class.modifier.add("ITERATOR")
        elif method_name == "__iter__":
            self.current_method = SelfIteratorMethod
            self.current_class.modifier.add("SELF_ITER")
        elif method_name == "__str__":
            self.current_class.modifier.add("TP_STR")

        self.current_method = self.current_method(
            self.current_class.class_name,
            self.current_class.base_class_name,
            method_name, [], return_type,
            myclass=self.current_class)
        self.current_method.docstring = self.current_comment
        self.current_method.modifier = modifier

    def METHOD_ARG(self, t, m):
        name = m.group(2).strip()
        type = m.group(1).strip()
        if self.current_method:
            self.current_method.add_arg(type, name)

    def METHOD_END(self, t, m):
        if not self.current_method:
            return

        if isinstance(self.current_method, ConstructorMethod):
            self.current_class.constructor = self.current_method
        else:
            found = False
            for i in range(len(self.current_class.methods)):
                # Try to replace existing methods with this new method
                method = self.current_class.methods[i]
                if method.name == self.current_method.name:
                    self.current_class.methods[i] = self.current_method
                    self.current_method = None
                    return

            # Method does not exist, just add to the end
            self.current_class.methods.append(self.current_method)

        self.current_method = None

    def CCLASS_ATTRIBUTE(self, t, m):
        modifier = m.group(1) or ""
        type = m.group(2).strip()
        name = m.group(3).strip()
        self.current_class.add_attribute(name, type, modifier)

    def END_CCLASS(self, t, m):
        self.current_class = None

    current_struct = None

    def STRUCT_START(self, t, m):
        self.current_struct = StructGenerator(m.group(2).strip(), self.module)
        self.current_struct.docstring = self.current_comment
        self.current_struct.modifier.add(m.group(1))

    def TYPEDEF_STRUCT_START(self, t, m):
        self.current_struct = StructGenerator(None, self.module)
        self.current_struct.docstring = self.current_comment

    def STRUCT_ATTRIBUTE(self, t, m):
        name = m.group(2).strip()
        type = m.group(1).strip()
        array_size = m.group(3)
        if array_size is not None:
            array_size = array_size.strip()
            self.current_struct.add_attribute(name, type, "", array_size=array_size)
        else:
            self.current_struct.add_attribute(name, type, "")

    def STRUCT_ATTRIBUTE_PTR(self, t, m):
        type = "{0:s} *".format(m.group(1).strip())
        name = m.group(2).strip()
        self.current_struct.add_attribute(name, type, "")

    def STRUCT_END(self, t, m):
        self.module.add_class(self.current_struct, StructWrapper)
        identifier = "{0:s} *".format(self.current_struct.class_name)
        type_dispatcher[identifier] = PointerStructWrapper
        self.current_struct = None

    def TYPEDEF_STRUCT_END(self, t, m):
        self.current_struct.class_name = m.group(1).strip()

        self.STRUCT_END(t, m)

    current_enum = None

    def ENUM_START(self, t, m):
        self.current_enum = Enum(m.group(1).strip(), self.module)

    def TYPEDEF_ENUM_START(self, t, m):
        self.current_enum = Enum(None, self.module)

    def ENUM_VALUE(self, t, m):
        self.current_enum.values.append(m.group(1).strip())

    def ENUM_END(self, t, m):
        self.module.classes[self.current_enum.name] = self.current_enum

        # For now we just treat enums as an integer, and also add
        # them to the constant table. In future it would be nice to
        # have them as a proper Python object so we can override
        # __unicode__, __str__ and __int__.
        for attr in self.current_enum.values:
            self.module.add_constant(attr, "integer")

        # type_dispatcher[self.current_enum.name] = Integer
        type_dispatcher[self.current_enum.name] = EnumType
        self.current_enum = None

    def TYPEDEFED_ENUM_END(self, t, m):
        self.current_enum.name = self.current_enum.class_name = m.group(1)
        self.ENUM_END(t, m)

    def BIND_STRUCT(self, t, m):
        self.module.active_structs.add(m.group(1))
        self.module.active_structs.add("{0:s} *".format(m.group(1)))

    def SIMPLE_TYPEDEF(self, t, m):
        # We basically add a new type as a copy of the old
        # type
        old, new = m.group(1).strip(), m.group(2).strip()
        if old in type_dispatcher:
            type_dispatcher[new] = type_dispatcher[old]

    def PROXY_CCLASS(self, t, m):
        base_class_name = m.group(1).strip()
        class_name = "Proxied{0:s}".format(base_class_name)
        try:
            proxied_class = self.module.classes[base_class_name]
        except KeyError:
            raise RuntimeError((
                "Need to create a proxy for {0:s} but it has not been "
                "defined (yet). You must place the PROXIED_CCLASS() "
                "instruction after the class definition").format(
                    base_class_name))
        current_class = ProxyClassGenerator(class_name,
                                            base_class_name, self.module)
        # self.current_class.constructor.args += proxied_class.constructor.args
        current_class.docstring = self.current_comment

        # Create proxies for all these methods
        for method in proxied_class.methods:
            if method.name[0] != "_":
                current_class.methods.append(ProxiedMethod(method, current_class))

        self.module.add_class(current_class, Wrapper)

    def parse_filenames(self, filenames):
        for f in filenames:
            self._parse(f)

        # Second pass
        for f in filenames:
            self._parse(f)

    def _parse(self, filename):
        file_object = open(filename, "rb")
        self.parse_fd(file_object)
        file_object.close()

        if filename not in self.module.files:
            if filename.startswith(self.base):
                filename = filename[len(self.base):]

            self.module.headers += "#include \"{0:s}\"\n".format(filename)
            self.module.files.append(filename)

    def write(self, out):
        try:
            self.module.write(out)
        except:
            # pdb.post_mortem()
            raise

    def write_headers(self):
        pass
        # pdb.set_trace()


if __name__ == "__main__":
    p = HeaderParser("pytsk3", verbose=1)
    for arg in sys.argv[1:]:
        p.parse_fd(open(arg, "rb"))

    log("second parse")
    for arg in sys.argv[1:]:
        p.parse_fd(open(arg, "rb"))

    p.write(sys.stdout)
    p.write_headers()