Python - C interfacing with ctypes

Ctypes is a module in Python for facilitating easy interface between Python and C. Interfacing Python with C can be useful for improving the performance of a Python program. Interfacing C with Python can be useful from a testing point of view. Either way, python-c interfacing is going to save our lives and give us better results(most of the time).

The procedure in using ctypes is very simple.
(a) Create a shared library from c-code
(b) Load the shared library in Python using ctypes
(c) Create equivalent data structures in Python
(d) Call the functions in shared object with equivalent data structures

(a) Creating a shared library

gcc -shared -fPIC exlib.c -o libex.so

Here, I am assuming that we have the c code in a file named exlib.c

(b) Loading shared library in Python

# import ctype library
import ctypes as ct
# load the shared library
exlib = ct.cdll.LoadLibrary("./libex.so")

Once, we have these two working, we can proceed to examples.

C code is given first, followed by Python code.

--------------------------------------------------------------------------------
(1) Simplest One: Takes nothing and Returns nothing

void function_01(void)

{
    printf("Hello World\n");
}

exlib.function_01()

--------------------------------------------------------------------------------
(2) Integer and String

void function_02(char *pString, int value)
{
    printf("%s%d\n", pString, value);
}

exlib.function_02("this year is ", 2012)
a = 1024
print "a =", a
exlib.function_02("a = ", a)

--------------------------------------------------------------------------------
(3) Unsigned int, Long, Unsigned Long

void function_03(   unsigned int    ui,
                    long            l,
                    unsigned long   ul )
{
    printf( "uint  = %u\n"
            "long  = %ld\n"
            "ulong = %lu\n",
            ui, l, ul);
}

ui = ct.c_uint ( 2**31   )
l  = ct.c_long ( 2 ** 62 )
ul = ct.c_ulong( 2 ** 63 )
print "uint  = ",   ui.value
print "long  = ",   l .value
print "ulong = ",   ul.value
exlib.function_03(ui, l, ul)
# Setting value directly
ui.value = 1000
l .value = 2000
ul.value = 3000
exlib.function_03(ui, l, ul)

--------------------------------------------------------------------------------
(4) Two Doubles go in, One Double comes out

double function_04(double a, double b)
{
    return a * b;
}

# specifying input types, this will make the function call safe
exlib.function_04.argtypes = [ct.c_double, ct.c_double]
# specifying return type
exlib.function_04.restype = ct.c_double
a = ct.c_double( 10.0 )
b = ct.c_double( 2.0  )
c = exlib.function_04(a, b)
print c

--------------------------------------------------------------------------------
(5) Sending data by reference

void function_05( double * a)
{
    *a = 2 * (*a);
}

a = ct.c_double( 100 )
print a.value
exlib.function_05( ct.byref(a) )
print a.value

--------------------------------------------------------------------------------
(6) Making things complicated with structures

typedef struct
{
    double real;
    double imag;
} complex_t;

void function_06( complex_t a)
{
    printf("Real = %f, Imag = %f\n", a.real, a.imag);
}


class ComplexType(ct.Structure):
    _fields_ = [("real", ct.c_double), ("imag", ct.c_double) ]

cc = ComplexType(10, 20)
exlib.function_06( cc )
cc.imag = 33
exlib.function_06( cc )

--------------------------------------------------------------------------------
(7) Making things more complicated with pointers and array

void function_07( int* a, int len)
{
    int i;
    for( i = 0 ; i < len; i++)
    {
        printf("%d, ", a[i]);
    }
    printf("\n");
}


a_len = 20
# defining array type
array_type = ct.c_int * a_len
# making an array
a = array_type()
# assigning values
for i in xrange(a_len) :
    a[i] = i*i;
#taking the address
ap = ct.pointer(a)
exlib.function_07(ap, a_len)

--------------------------------------------------------------------------------
(8)  With pointers and arrays inside structure

typedef struct
{
    int a_len;
    complex_t a[10];
    int b_len;
    complex_t *b;
} AB_t;

void function_08( AB_t* inp)
{
    int i;
    for(i = 0; i < inp->a_len; i++)
    {
        printf("%f,",  inp->a[i].real);
        printf("%f\n", inp->a[i].imag);
    }
    printf("\n");
    for(i = 0; i < inp->b_len; i++)
    {
        printf("%f,",  inp->b[i].real);
        printf("%f\n", inp->b[i].imag);
    }
}

ComplexArrayType10 = ComplexType * 10
ComplexArrayType100 = ComplexType * 100
class ABType(ct.Structure):
    _fields_ = [ ("a_len", ct.c_int),
                 ("a",ComplexArrayType10),
                 ("b_len", ct.c_int),
                 ("b", ct.POINTER(ComplexType) ) ]

x = ABType()
x.a_len = 8
for ii in xrange( x.a_len ):
    x.a[ii].real = ii * 3;
    x.a[ii].imag = ii * -3;
x.b_len = 5
b = ComplexArrayType100()
for ii in xrange( x.b_len ):
    b[ii].real = ii * 10
    b[ii].imag = ii * -10
# the following will not work without cast
x.b = ct.cast( ct.pointer(b), ct.POINTER(ComplexType) )
exlib.function_08( ct.pointer(x) )

--------------------------------------------------------------------------------
(9) Playing with Numpy Arrays

double function_09(double* inpVec, int stride, int length)
{
    /*
     * stride - address difference between successive elements
     */
    int i;
    int step = stride / sizeof(double);
    double sum  = 0;
    double elem;
    double* elem_p;
    if(0 != stride % sizeof(double))
    {
        /* not needed if we specify argtypes in python */
        printf("ERROR: Stride should be multiple of sizeof(double)\n");
        return 0.0;
    }

    /*
     * (a) find sum
     * (b) do an inplace value change
     */

     sum = 0;
     printf("stride = %d, length = %d\n", stride, length);
     for( i = 0; i < length; i++)
     {
         /* pointer to i th element */
         elem_p = inpVec + i * step;
         /* value of i th element */
         elem = *elem_p;
         /* calculate sum */
         sum = sum + elem;
         /* inplace value change */
         *elem_p = 1.1 * elem + 0.01;
     }
     return sum;
}

# wrapper function
def wr_function_09( ipvec ):
    # define input argument types
    exlib.function_09.argtypes=[np.ctypeslib.ndpointer(dtype=np.double,
                                                       ndim = 1),
                                ct.c_int,
                                ct.c_int]
    # define retrun type
    exlib.function_09.restype  = ct.c_double
    stride = ipvec.strides[0]
    length = ipvec.size
    sum_r = exlib.function_09( ipvec, stride, length )
    return sum_r

a = np.arange(6.0)
print a
sa = wr_function_09(a)
print sa
print a
sa = wr_function_09(a)
print sa
print a
# our function should work for both rows and columns
b = np.zeros( (6, 6), dtype = np.double )
# sending 4th column
sb = wr_function_09(b[:,4])
print b
# sending 3rd row
sb = wr_function_09(b[3,:])
print b

--------------------------------------------------------------------------------
(10) Handling Numpy Matrices

/* 2 dimensional matrix structure */
typedef struct
{
    double *pData;
    int rstride;
    int cstride;
    int rlen;
    int clen;
} matrix_t;

/* function to get the address of element (i, j) */
double* get_elem_p( matrix_t m, int i, int j)
{
    return (double *) ( (char *)(m.pData) +
                        i * m.rstride +
                        j * m.cstride );
}


void function_10( matrix_t a, matrix_t b)
{
    /*
     * A simple function to do an inplace matrix addition
     */
    /* TODO:
     * check for stride % sizeof(double) has to be done for both a and b
     * check for rlen and clen has to be done for both a and b
     */
    int i, j;
    printf("%d, %d, %d, %d\n" ,
            a.rstride, b.rstride, a.cstride, b.cstride);
    printf("%d, %d, %d, %d\n" ,
            a.rlen, b.rlen, a.clen, b.clen);
    for( i = 0; i < a.rlen; i++ )
    {
        for( j = 0; j < a.clen; j++ )
        {
            *get_elem_p(a, i, j) += *get_elem_p(b, i, j);
        }
    }
}

class MatrixType(ct.Structure):
    _fields_ = [("pData", ct.POINTER(ct.c_double)),
                ("rstride", ct.c_int),
                ("cstride", ct.c_int),
                ("rlen", ct.c_int),
                ("clen", ct.c_int)]

# wrapper function 
def wr_function_10( inpmx1, inpmx2 ):
    m1 = MatrixType()
    m2 = MatrixType()
    m1.pData = inpmx1.ctypes.data_as(ct.POINTER(ct.c_double))
    m2.pData = inpmx2.ctypes.data_as(ct.POINTER(ct.c_double))
    m1.rstride = inpmx1.strides[0]
    m1.cstride = inpmx1.strides[1]
    m2.rstride = inpmx2.strides[0]
    m2.cstride = inpmx2.strides[1]
    m1.rlen    = inpmx1.shape[0]
    m1.clen    = inpmx1.shape[1]
    m2.rlen    = inpmx2.shape[0]
    m2.clen    = inpmx2.shape[1]
    print m1.rstride, m2.rstride, m1.cstride, m2.cstride
    print m1.rlen, m2.rlen, m1.clen, m2.clen
    exlib.function_10(m1, m2)

a1 = np.zeros((5, 4), dtype = np.double)
a2 = a1 + 0.5
print a1
print a2
wr_function_10(a1, a2)
print a1
wr_function_10(a1, a2)
print a1
wr_function_10(a1, a2)
print a1

--------------------------------------------------------------------------------
Notes :
[1] Always make sure that memory allocation is handled by Python. Dynamic allocation inside c function for results required in Python can be problematic. Instead, create the required structures in Python and send their reference to C.

Alternate line colors for terminal

For a long time, I was in search for a Vim plug-in that will highlight the odd lines in a slightly different color. I was not able to find anything that works neat. Later, I realized that the problem should be solved with the help of the terminal program, not with vim.

Most of the terminal programs (terminator, gnome terminal etc) support setting background image. All I had to do was writing a Python script to generate the background I needed. Here is the script I wrote.

import scipy as sp
import Image as im
hight = 768
width = 1366
# 17 for djvu sans mono 10, inconsolata 12
# 18 for djvu sans mono 11, monaco 10
line_width = 18
offset = 1
x = sp.zeros( (hight, width, 3), dtype=sp.uint8 )
# odd line color
[r1, g1, b1] = [230, 230, 230]
# even line color
[r2, g2, b2] = [255, 255, 255]
# color for thin line
[r0, g0, b0] = [200, 200, 200]
ind=sp.arange(hight)
indoff = sp.absolute(ind - offset)
ind1 = ind[sp.where( (indoff / line_width) % 2 == 0 )]
ind2 = ind[sp.where( (indoff / line_width) % 2 == 1 )]
ind0 = ind[sp.where( (indoff % line_width)     == 0 )]
x[ind1, :, 0] = r1
x[ind1, :, 1] = g1
x[ind1, :, 2] = b1
x[ind2, :, 0] = r2
x[ind2, :, 1] = g2
x[ind2, :, 2] = b2
x[ind0, :, 0] = r0
x[ind0, :, 1] = g0
x[ind0, :, 2] = b0
img = im.fromarray(x, "RGB")
img.save("term_wall.png")

Save it as wallgen.py and run it using python wallgen.py. It will generate an image named term_wall.png. (You can add some extra beautification with Gimp if you want)
You can set it as background for your terminal. (Disable scrolling of background, if you feel it as annoying)

one, two, three - some example images which can be directly downloaded.

This is how it is going to look like:

Search And Replace - with Vim and Sed

Here are some examples on search and replace with Vim and Sed.

Please note that the commands starting with : are for vim and the ones starting with $ are for bash. Whatever given in () is explanation.

 
(1) Replace one pattern with another pattern in a file

:%s/windows/ubuntu/g
(replaces "windows" with "ubuntu" in current file)

$sed -e 's/windows/ubuntu/g' -i myfile
(-i makes an in-place replacement in myfile)

For case insensitive matching, use \c in vim and gI in sed.

:%s/windows\c/Ubuntu/g
(matches windows, Windows, winDows..... etc)

$sed -e 's/windows/Ubuntu/gI' -i myfile
(makes an in-place replacement in myfile)


(2) Replace with range

:3,5s/windows/ubuntu/g
(replaces in lines from 3 to 5)

:1,.s/windows/ubuntu/g
(from first line to current line[. represents current line])

:.,$s/windows/ubuntu/g 
(from current line to last line [$ represents last line])

:.,.+15s/windows/ubuntu/g
(from current line to 15 lines below)

$sed -e '10,15 s/windows/ubuntu/g' -i myfile
(replace in lines 10 to 15)

$sed -e "10,$ s/windows/ubuntu/g' -i myfile
(replaces from  10th line to last line)

$sed -e '/begin/,/end/ s/windows/ubuntu/g' -i myfile
( replacing from pattern "begin" to pattern "end")

$sed -e '/begin/,100 s/windows/ubuntu/g' -i myfile
(from pattern "begin" to line 100)

$sed -e '/begin/,+10 s/windows/ubuntu/g' -i myfile
(from pattern "begin" to 10 lines after match)


(3) Remove lines matching or not matching a pattern

:g/windows/d 
(delete all lines having pattern windows)

:10,15g/windows/d
(works with range also)

:g!/freedom/d
(delete lines not having "freedom")

 $sed -e '/windows/d' -i myfile
(deletes all lines matching pattern "windows")

$sed -e '10,15 {/windows/d}' -i myfile
(specifying range restriction)

$sed -e '/freedom/!d' -i myfile
(delete all lines without "freedom")

Here onwards, examples are mostly related to regular expressions. I will be giving examples only from Vim.


(4) Pattern at the beginning or end

Use ^ for matching patterns at the beginning of line.
Use $ for patterns at the end of line.

:%s/^windows/ubuntu/g
:%s/windows$/ubuntu/g


(5) Multiple patterns

 "|" ("OR" operator) can be used to match multiple patterns. Make sure to escape it with a "\"

:%s/windows xp\|windows 7/ubuntu/g 


(6) Patterns with wild character

 . ("dot") matches any characters except newline

:%s/a.c/pqr/g


(7) Pattern repetition

*   - zero or more
+   - one or more
{n}      - exactly matches n repeats
{n,m} - for repeats between n and m
{n,}  - for repeats at least n
{,m} - for repeats at most m

Make sure to escape "+", "{" and "}" with "\" (for both vim and sed)

:%s/px*q/pzq/g 
(matches pq, pxq, pxxq and so on)

:%s/px\+q/pzq/g
(matches pxq, pxxq, pxxxq and so on)

:%s/px\{3\}q/pzq/g 
(for exact 3 x between p and q)

:%s/px\{3,\}q/pzq/g
(for at least 3 x between p and q)


(8) Lazy matching

Usual regex syntax for lazy matching is *? (where ? makes * lazy)
But in Vim, we have to use \{-\} for lazy matching (to consider as few as possible)

(I couldn't find any lazy matching option with sed :( )

Example string:
ABC ooiiooii XYZ ooiiooii XYZ

:%s/ABC.\{-\}XYZ/PQR/g
(gives PQR ooiiooii XYZ )

:%s/ABC.*XYZ/PQR/g 
(gives PQR)


(9) Back reference

Expressions grouped with "( )" can be referenced with \1, \2 etc.
\1, \2 etc can be used in the replace string also.

:%s/\(ab\|xy\)_\1/pq/g 
(replaces "ab_ab" and "xy_xy" with "pq", leaves "ab_xy" and "xy_ab" untouched)

:%s/pq\(.*\)rs/ab\1cd/g
("pq 123 rs" will change to "ab 123 cd")


(10) Character class 

[a-z]    - matches all small letter chars
[^a-z]   - matches anything other than small letter
[a-zA-Z0-9] - will match all alphabets and numbers

:%s/(\([0-9]\+\))/[\1]/g 
(replaces all "(number)" pattern with "[number]" pattern

Some regularly used search and replaces

(a) Remove all trailing white space

:%s/\s\+$//g 
(\s- space , \+ one or more, $ - at the end)

(b) Remove ^M chars

:%s/^M//g
(Getting that ^M is bit tricky, first you need to press Ctrl-v and then press Ctrl-m)

(c) Delete blank lines

:g/^\s*$/d 
(^ - beginning of line, \s* - any number of spaces, $ - end, d - delete)

(d) Remove line numbers from the beginning

:s/^[0-9]\+\s\+//g
(^ - beginning of line, [0-9]\+ - one or more numbers, \s\+ - one or more spaces)

(e) some_file.cpp(line_no) to some_file.cpp:line_no:0

%s/\(\w\+\.cpp\)(\([0-9]\+\))/\1:\2:0/g

Notes:
[1] Be always ready to go for  :help regex or $man sed if you get stuck.
[2] Like all other posts, these aren't things I invented. I am just listing things I collected from net over a long time.  

Vim - tips and tweaks for C/C++ coding

Here are some tweaks I found useful while playing with C/C++ code in vim. You can try them by adding them to your .vimrc file. These aren't things I invented, but collected from net over time (Though I have added some minor modifications myself).

I am going to assume that you are familiar with ctags and cscope. Please take some time to make sure that the assumption is correct.

1) Adding cscope and ctags files automatically.

This small function will search for cscope and ctags files (cscope.out, tags) in parent folders and connect them to vim automatically.

function SetTags()

   let curdir = getcwd()

   while !filereadable("tags") && getcwd() != "/"

      cd ..

   endwhile

   if filereadable("tags")

      execute "set tags=" . getcwd() . "/tags"

   endif

   execute "cd " . curdir

   while !filereadable("cscope.out") && getcwd() != "/"

      cd ..

   endwhile

   if filereadable("cscope.out")

      execute "cscope add ".getcwd()."/cscope.out ".getcwd()

   endif

   execute "cd " . curdir

endfunction

call SetTags()

2) Quickfix and cscope

Vim has feature named quickfix window.  Next time when you want to give a make, try giving it from vim and you will see how quickfix window makes things easy.

:make something

Vim will call make and will take you to the places where compiler found errors.

:copen ~ will open the list of errors in quickfix window.

:cn  ~ will take you to the next error.

:cp  ~ well, guess it.

You can also use quickfix window with grep. Try,

:grep -r something *

Here comes the interesting part. You can use quickfix functionality along with cscope by adding the following settings to .vimrc

set cscopequickfix=s-,c-,d-,i-,t-,e-,g-

3) Invoke cscope with split window

nnoremap <C-\>s :vs<CR>:execute  ("cs find s ".expand("<cword>"))<CR>

nnoremap <C-\>g :vs<CR>:execute  ("cs find g ".expand("<cword>"))<CR>

What does it do ?

Whenever  we give a Ctrl-\ s, it will (first) split the window  vertically into two (then) invoke cscope.

We can use :cn and :cp to go forward and backward the cscope result list.

I found it useful to re assign keys[1] for :cn and :cp.

nnoremap ^[n :cn<CR>

nnoremap ^[m :cp<CR>

4) Filtering quickfix list

Try the first result/reply  here.

5) Tracking locations in multiple files

It is sometimes important to keep track of all the function call paths. Simplest option is to use global marks. All the marks with capital letters are global marks.

Try mA in one file (in normal mode) and  give 'A while you are in some other file.

(Try Ctrl-o and Ctrl-i also)

Another option is to add the locations directly to a file. It can be done easily as follows.

nmap ^[8 :call system('echo '.expand('%:p').':'.line('.').':0 >> ~/.loc_vim')<CR>

What does it do ?

It will append current_file_name:line_no:0 to a file named .loc_vim in your home folder, when you press Alt-8 (8 key has a *). [1]

We can load this location file to quickfix window using the command

:cfile ~/.loc_vim.txt

6) Debug prints

We can load the debug log file to quickfix window if we make sure that the debug prints are of the form:

file_name:line_no:debug_prints

(same format in which grep gives results with option -n)

Here is an example debug macro that can be used in C [2].

#define DEBUG_PRINT(...)\
    printf("%s:%d:",__FILE__,__LINE__);\
    printf(__VA_ARGS__);\
    printf("\n");

Collect the logs generated by your program(eg : a.out) to a file (eg: debug.log)

>> ./a.out |tee debug.log

Load the log file in  vim using

:cfile debug.log

Notes:

[1] One interesting thing about Vim is that all the "Alt" key combinations are completely free by default. But the usual Alt key binding <A-f> (or <M-f>) for Alt-f may not work(f is just an example) in non-gui version of vim. To make it work, we need to understand what vim is getting when we give Alt-f. To find that, press Ctrl-v and then press Alt-f in insert mode. It will print a set of key combinations. Use them for defining key combinations. 

For me, it gives something like ^[f (where ^[ is a single character).

[2] Beware. This is an evil, unprotected macro.

You may have to protect it with a do{ }while(0) if you want to use them in any serious code.

A simple Bookmark Script for Bash

Ever since I started using bash, I have felt the need for a bookmark system. One day, I went to omniscient Google and prayed for help. Sadly, I wasn't really happy with the results*. Finally, I wrote a script** and here it is:

fileName="${HOME}/.markedLocations.txt"
flag=0
if [ "${1}" = "." ]; then
   pwd >>${fileName}
   flag=1
fi
if [ "${1}" = "s" ]
then
   touch ${fileName}
   line_no=1
   color1='\e[1;35m'
   color2='\e[1;32m'
   while read line
   do
      color=$color2
      if [[ $(($line_no%2)) -eq 1 ]]; then
         color=$color1
      fi
      echo -e $color"${line_no}\t"$line;
      line_no=$(($line_no+1))
   done < ${fileName}
   echo -en '\e[0m';
   flag=1
fi
if [[ "${1}" =~ ^[0-9]+$ ]]; then
   pathCmd="awk "FNR==${1}" ${fileName}"
   path=`${pathCmd}`
   if [[ "${path}" = "" ]]; then
      echo "NO MARK"
   else
      cd ${path}
   fi
   flag=1
fi
if [ "${flag}" = "0" ]
then
   cat -n ${fileName} 2>/dev/null|grep -i $1
fi

Save this in a file named something.sh(or whatever you like)
Add the following in your ~/.bashrc :

alias mark="source /path-where-you-saved/something.sh"

Open a new bash and try the follwoing:
mark .  (it will add the current directory to bookmark file)
mark s  (it should show the numbered list of bookmarks)
Now cd to some other directory and try:
mark 1 (it should take you to the directory where you gave mark .)
mark pattern (will show you marks matching "pattern" )

The following screenshot will give you a better idea of how it works.

Enjoy !

*  Out of all the results, I found this one  to be simple enough to give a try.
** Thanks to Pravin for all the help related to bash scripting.