python - improving Numpy dot performance by removing arrays copy -

given matrix qt:

% ipython python 2.7.3 in [3]: qt.dtype out[3]: dtype('float64')  in [4]: qt.__class__ out[4]: numpy.ndarray  in [5]: qt.flags out[5]:       c_contiguous : true       f_contiguous : false       owndata : true       writeable : true       aligned : true       updateifcopy : false 

i need results of:

qt.t * qt 

problem: whenever try compute these matrices multiplication, memory overflows , code stop running. happen because of matrix copy numpy doing behind.

tried solutions:

first:

q = numpy.array(qt.t, order='c') numpy.dot(q, qt) 

second:

qt = numpy.array(qt, order='f') q = numpy.array(qt.t, order='f') numpy.dot(q, qt) 

third:

qt = numpy.matrix(qt) qt = qt.copy('f') q = numpy.matrix(qt.t) q = q.copy('f') q.dot(qt) 

however, none of them solving.

question

how can operate qt.t * qt without having memory explode?

references

http://numpy-discussion.10968.n7.nabble.com/inplace-matrix-multiplication-td21817.html

is there "enhanced" numpy/scipy dot method?

numpy dot product slow using ints

http://www.scipy.org/performancetips

if result won't fit core memory, can put in memory-mapped array overflow written hard disk:

shape = (qt.shape[2],)*2 result = np.memmap('result.dat', dtype=qt.dtype, mode='w+', shape=shape) np.dot(qt.t, qt, out=result) 

you may want take @ this algorithm performing out-of-core svd on large arrays.