@r05: try compiling the older version of the algorithm, which consumes much less memory:
make clean
CFLAGS="-DDEBUG_GPUV3" make -f makefile.unix
and then try several memory amounts from -m 20 (4Mb) to -m 27 (512MB) (default).
If you're still with the same error, I'm lost

Which graphics card are you running it on? How much RAM on your machine?