Python - C interfacing with ctypes

Ctypes is a module in Python for facilitating easy interface between Python and C. Interfacing Python with C can be useful for improving the performance of a Python program. Interfacing C with Python can be useful from a testing point of view. Either way, python-c interfacing is going to save our lives and give us better results(most of the time).

The procedure in using ctypes is very simple.
(a) Create a shared library from c-code
(b) Load the shared library in Python using ctypes
(c) Create equivalent data structures in Python
(d) Call the functions in shared object with equivalent data structures

(a) Creating a shared library

gcc -shared -fPIC exlib.c -o libex.so

Here, I am assuming that we have the c code in a file named exlib.c

(b) Loading shared library in Python

# import ctype library
import ctypes as ct
# load the shared library
exlib = ct.cdll.LoadLibrary("./libex.so")

Once, we have these two working, we can proceed to examples.

C code is given first, followed by Python code.

--------------------------------------------------------------------------------
(1) Simplest One: Takes nothing and Returns nothing

void function_01(void)

{
    printf("Hello World\n");
}

exlib.function_01()

--------------------------------------------------------------------------------
(2) Integer and String

void function_02(char *pString, int value)
{
    printf("%s%d\n", pString, value);
}

exlib.function_02("this year is ", 2012)
a = 1024
print "a =", a
exlib.function_02("a = ", a)

--------------------------------------------------------------------------------
(3) Unsigned int, Long, Unsigned Long

void function_03(   unsigned int    ui,
                    long            l,
                    unsigned long   ul )
{
    printf( "uint  = %u\n"
            "long  = %ld\n"
            "ulong = %lu\n",
            ui, l, ul);
}

ui = ct.c_uint ( 2**31   )
l  = ct.c_long ( 2 ** 62 )
ul = ct.c_ulong( 2 ** 63 )
print "uint  = ",   ui.value
print "long  = ",   l .value
print "ulong = ",   ul.value
exlib.function_03(ui, l, ul)
# Setting value directly
ui.value = 1000
l .value = 2000
ul.value = 3000
exlib.function_03(ui, l, ul)

--------------------------------------------------------------------------------
(4) Two Doubles go in, One Double comes out

double function_04(double a, double b)
{
    return a * b;
}

# specifying input types, this will make the function call safe
exlib.function_04.argtypes = [ct.c_double, ct.c_double]
# specifying return type
exlib.function_04.restype = ct.c_double
a = ct.c_double( 10.0 )
b = ct.c_double( 2.0  )
c = exlib.function_04(a, b)
print c

--------------------------------------------------------------------------------
(5) Sending data by reference

void function_05( double * a)
{
    *a = 2 * (*a);
}

a = ct.c_double( 100 )
print a.value
exlib.function_05( ct.byref(a) )
print a.value

--------------------------------------------------------------------------------
(6) Making things complicated with structures

typedef struct
{
    double real;
    double imag;
} complex_t;

void function_06( complex_t a)
{
    printf("Real = %f, Imag = %f\n", a.real, a.imag);
}


class ComplexType(ct.Structure):
    _fields_ = [("real", ct.c_double), ("imag", ct.c_double) ]

cc = ComplexType(10, 20)
exlib.function_06( cc )
cc.imag = 33
exlib.function_06( cc )

--------------------------------------------------------------------------------
(7) Making things more complicated with pointers and array

void function_07( int* a, int len)
{
    int i;
    for( i = 0 ; i < len; i++)
    {
        printf("%d, ", a[i]);
    }
    printf("\n");
}


a_len = 20
# defining array type
array_type = ct.c_int * a_len
# making an array
a = array_type()
# assigning values
for i in xrange(a_len) :
    a[i] = i*i;
#taking the address
ap = ct.pointer(a)
exlib.function_07(ap, a_len)

--------------------------------------------------------------------------------
(8)  With pointers and arrays inside structure

typedef struct
{
    int a_len;
    complex_t a[10];
    int b_len;
    complex_t *b;
} AB_t;

void function_08( AB_t* inp)
{
    int i;
    for(i = 0; i < inp->a_len; i++)
    {
        printf("%f,",  inp->a[i].real);
        printf("%f\n", inp->a[i].imag);
    }
    printf("\n");
    for(i = 0; i < inp->b_len; i++)
    {
        printf("%f,",  inp->b[i].real);
        printf("%f\n", inp->b[i].imag);
    }
}

ComplexArrayType10 = ComplexType * 10
ComplexArrayType100 = ComplexType * 100
class ABType(ct.Structure):
    _fields_ = [ ("a_len", ct.c_int),
                 ("a",ComplexArrayType10),
                 ("b_len", ct.c_int),
                 ("b", ct.POINTER(ComplexType) ) ]

x = ABType()
x.a_len = 8
for ii in xrange( x.a_len ):
    x.a[ii].real = ii * 3;
    x.a[ii].imag = ii * -3;
x.b_len = 5
b = ComplexArrayType100()
for ii in xrange( x.b_len ):
    b[ii].real = ii * 10
    b[ii].imag = ii * -10
# the following will not work without cast
x.b = ct.cast( ct.pointer(b), ct.POINTER(ComplexType) )
exlib.function_08( ct.pointer(x) )

--------------------------------------------------------------------------------
(9) Playing with Numpy Arrays

double function_09(double* inpVec, int stride, int length)
{
    /*
     * stride - address difference between successive elements
     */
    int i;
    int step = stride / sizeof(double);
    double sum  = 0;
    double elem;
    double* elem_p;
    if(0 != stride % sizeof(double))
    {
        /* not needed if we specify argtypes in python */
        printf("ERROR: Stride should be multiple of sizeof(double)\n");
        return 0.0;
    }

    /*
     * (a) find sum
     * (b) do an inplace value change
     */

     sum = 0;
     printf("stride = %d, length = %d\n", stride, length);
     for( i = 0; i < length; i++)
     {
         /* pointer to i th element */
         elem_p = inpVec + i * step;
         /* value of i th element */
         elem = *elem_p;
         /* calculate sum */
         sum = sum + elem;
         /* inplace value change */
         *elem_p = 1.1 * elem + 0.01;
     }
     return sum;
}

# wrapper function
def wr_function_09( ipvec ):
    # define input argument types
    exlib.function_09.argtypes=[np.ctypeslib.ndpointer(dtype=np.double,
                                                       ndim = 1),
                                ct.c_int,
                                ct.c_int]
    # define retrun type
    exlib.function_09.restype  = ct.c_double
    stride = ipvec.strides[0]
    length = ipvec.size
    sum_r = exlib.function_09( ipvec, stride, length )
    return sum_r

a = np.arange(6.0)
print a
sa = wr_function_09(a)
print sa
print a
sa = wr_function_09(a)
print sa
print a
# our function should work for both rows and columns
b = np.zeros( (6, 6), dtype = np.double )
# sending 4th column
sb = wr_function_09(b[:,4])
print b
# sending 3rd row
sb = wr_function_09(b[3,:])
print b

--------------------------------------------------------------------------------
(10) Handling Numpy Matrices

/* 2 dimensional matrix structure */
typedef struct
{
    double *pData;
    int rstride;
    int cstride;
    int rlen;
    int clen;
} matrix_t;

/* function to get the address of element (i, j) */
double* get_elem_p( matrix_t m, int i, int j)
{
    return (double *) ( (char *)(m.pData) +
                        i * m.rstride +
                        j * m.cstride );
}


void function_10( matrix_t a, matrix_t b)
{
    /*
     * A simple function to do an inplace matrix addition
     */
    /* TODO:
     * check for stride % sizeof(double) has to be done for both a and b
     * check for rlen and clen has to be done for both a and b
     */
    int i, j;
    printf("%d, %d, %d, %d\n" ,
            a.rstride, b.rstride, a.cstride, b.cstride);
    printf("%d, %d, %d, %d\n" ,
            a.rlen, b.rlen, a.clen, b.clen);
    for( i = 0; i < a.rlen; i++ )
    {
        for( j = 0; j < a.clen; j++ )
        {
            *get_elem_p(a, i, j) += *get_elem_p(b, i, j);
        }
    }
}

class MatrixType(ct.Structure):
    _fields_ = [("pData", ct.POINTER(ct.c_double)),
                ("rstride", ct.c_int),
                ("cstride", ct.c_int),
                ("rlen", ct.c_int),
                ("clen", ct.c_int)]

# wrapper function 
def wr_function_10( inpmx1, inpmx2 ):
    m1 = MatrixType()
    m2 = MatrixType()
    m1.pData = inpmx1.ctypes.data_as(ct.POINTER(ct.c_double))
    m2.pData = inpmx2.ctypes.data_as(ct.POINTER(ct.c_double))
    m1.rstride = inpmx1.strides[0]
    m1.cstride = inpmx1.strides[1]
    m2.rstride = inpmx2.strides[0]
    m2.cstride = inpmx2.strides[1]
    m1.rlen    = inpmx1.shape[0]
    m1.clen    = inpmx1.shape[1]
    m2.rlen    = inpmx2.shape[0]
    m2.clen    = inpmx2.shape[1]
    print m1.rstride, m2.rstride, m1.cstride, m2.cstride
    print m1.rlen, m2.rlen, m1.clen, m2.clen
    exlib.function_10(m1, m2)

a1 = np.zeros((5, 4), dtype = np.double)
a2 = a1 + 0.5
print a1
print a2
wr_function_10(a1, a2)
print a1
wr_function_10(a1, a2)
print a1
wr_function_10(a1, a2)
print a1

--------------------------------------------------------------------------------
Notes :
[1] Always make sure that memory allocation is handled by Python. Dynamic allocation inside c function for results required in Python can be problematic. Instead, create the required structures in Python and send their reference to C.

Alternate line colors for terminal

For a long time, I was in search for a Vim plug-in that will highlight the odd lines in a slightly different color. I was not able to find anything that works neat. Later, I realized that the problem should be solved with the help of the terminal program, not with vim.

Most of the terminal programs (terminator, gnome terminal etc) support setting background image. All I had to do was writing a Python script to generate the background I needed. Here is the script I wrote.

import scipy as sp
import Image as im
hight = 768
width = 1366
# 17 for djvu sans mono 10, inconsolata 12
# 18 for djvu sans mono 11, monaco 10
line_width = 18
offset = 1
x = sp.zeros( (hight, width, 3), dtype=sp.uint8 )
# odd line color
[r1, g1, b1] = [230, 230, 230]
# even line color
[r2, g2, b2] = [255, 255, 255]
# color for thin line
[r0, g0, b0] = [200, 200, 200]
ind=sp.arange(hight)
indoff = sp.absolute(ind - offset)
ind1 = ind[sp.where( (indoff / line_width) % 2 == 0 )]
ind2 = ind[sp.where( (indoff / line_width) % 2 == 1 )]
ind0 = ind[sp.where( (indoff % line_width)     == 0 )]
x[ind1, :, 0] = r1
x[ind1, :, 1] = g1
x[ind1, :, 2] = b1
x[ind2, :, 0] = r2
x[ind2, :, 1] = g2
x[ind2, :, 2] = b2
x[ind0, :, 0] = r0
x[ind0, :, 1] = g0
x[ind0, :, 2] = b0
img = im.fromarray(x, "RGB")
img.save("term_wall.png")

Save it as wallgen.py and run it using python wallgen.py. It will generate an image named term_wall.png. (You can add some extra beautification with Gimp if you want)
You can set it as background for your terminal. (Disable scrolling of background, if you feel it as annoying)

one, two, three - some example images which can be directly downloaded.

This is how it is going to look like:

Search And Replace - with Vim and Sed

Here are some examples on search and replace with Vim and Sed.

Please note that the commands starting with : are for vim and the ones starting with $ are for bash. Whatever given in () is explanation.

 
(1) Replace one pattern with another pattern in a file

:%s/windows/ubuntu/g
(replaces "windows" with "ubuntu" in current file)

$sed -e 's/windows/ubuntu/g' -i myfile
(-i makes an in-place replacement in myfile)

For case insensitive matching, use \c in vim and gI in sed.

:%s/windows\c/Ubuntu/g
(matches windows, Windows, winDows..... etc)

$sed -e 's/windows/Ubuntu/gI' -i myfile
(makes an in-place replacement in myfile)


(2) Replace with range

:3,5s/windows/ubuntu/g
(replaces in lines from 3 to 5)

:1,.s/windows/ubuntu/g
(from first line to current line[. represents current line])

:.,$s/windows/ubuntu/g 
(from current line to last line [$ represents last line])

:.,.+15s/windows/ubuntu/g
(from current line to 15 lines below)

$sed -e '10,15 s/windows/ubuntu/g' -i myfile
(replace in lines 10 to 15)

$sed -e "10,$ s/windows/ubuntu/g' -i myfile
(replaces from  10th line to last line)

$sed -e '/begin/,/end/ s/windows/ubuntu/g' -i myfile
( replacing from pattern "begin" to pattern "end")

$sed -e '/begin/,100 s/windows/ubuntu/g' -i myfile
(from pattern "begin" to line 100)

$sed -e '/begin/,+10 s/windows/ubuntu/g' -i myfile
(from pattern "begin" to 10 lines after match)


(3) Remove lines matching or not matching a pattern

:g/windows/d 
(delete all lines having pattern windows)

:10,15g/windows/d
(works with range also)

:g!/freedom/d
(delete lines not having "freedom")

 $sed -e '/windows/d' -i myfile
(deletes all lines matching pattern "windows")

$sed -e '10,15 {/windows/d}' -i myfile
(specifying range restriction)

$sed -e '/freedom/!d' -i myfile
(delete all lines without "freedom")

Here onwards, examples are mostly related to regular expressions. I will be giving examples only from Vim.


(4) Pattern at the beginning or end

Use ^ for matching patterns at the beginning of line.
Use $ for patterns at the end of line.

:%s/^windows/ubuntu/g
:%s/windows$/ubuntu/g


(5) Multiple patterns

 "|" ("OR" operator) can be used to match multiple patterns. Make sure to escape it with a "\"

:%s/windows xp\|windows 7/ubuntu/g 


(6) Patterns with wild character

 . ("dot") matches any characters except newline

:%s/a.c/pqr/g


(7) Pattern repetition

*   - zero or more
+   - one or more
{n}      - exactly matches n repeats
{n,m} - for repeats between n and m
{n,}  - for repeats at least n
{,m} - for repeats at most m

Make sure to escape "+", "{" and "}" with "\" (for both vim and sed)

:%s/px*q/pzq/g 
(matches pq, pxq, pxxq and so on)

:%s/px\+q/pzq/g
(matches pxq, pxxq, pxxxq and so on)

:%s/px\{3\}q/pzq/g 
(for exact 3 x between p and q)

:%s/px\{3,\}q/pzq/g
(for at least 3 x between p and q)


(8) Lazy matching

Usual regex syntax for lazy matching is *? (where ? makes * lazy)
But in Vim, we have to use \{-\} for lazy matching (to consider as few as possible)

(I couldn't find any lazy matching option with sed :( )

Example string:
ABC ooiiooii XYZ ooiiooii XYZ

:%s/ABC.\{-\}XYZ/PQR/g
(gives PQR ooiiooii XYZ )

:%s/ABC.*XYZ/PQR/g 
(gives PQR)


(9) Back reference

Expressions grouped with "( )" can be referenced with \1, \2 etc.
\1, \2 etc can be used in the replace string also.

:%s/\(ab\|xy\)_\1/pq/g 
(replaces "ab_ab" and "xy_xy" with "pq", leaves "ab_xy" and "xy_ab" untouched)

:%s/pq\(.*\)rs/ab\1cd/g
("pq 123 rs" will change to "ab 123 cd")


(10) Character class 

[a-z]    - matches all small letter chars
[^a-z]   - matches anything other than small letter
[a-zA-Z0-9] - will match all alphabets and numbers

:%s/(\([0-9]\+\))/[\1]/g 
(replaces all "(number)" pattern with "[number]" pattern

Some regularly used search and replaces

(a) Remove all trailing white space

:%s/\s\+$//g 
(\s- space , \+ one or more, $ - at the end)

(b) Remove ^M chars

:%s/^M//g
(Getting that ^M is bit tricky, first you need to press Ctrl-v and then press Ctrl-m)

(c) Delete blank lines

:g/^\s*$/d 
(^ - beginning of line, \s* - any number of spaces, $ - end, d - delete)

(d) Remove line numbers from the beginning

:s/^[0-9]\+\s\+//g
(^ - beginning of line, [0-9]\+ - one or more numbers, \s\+ - one or more spaces)

(e) some_file.cpp(line_no) to some_file.cpp:line_no:0

%s/\(\w\+\.cpp\)(\([0-9]\+\))/\1:\2:0/g

Notes:
[1] Be always ready to go for  :help regex or $man sed if you get stuck.
[2] Like all other posts, these aren't things I invented. I am just listing things I collected from net over a long time.