memory usage with gfor

[thomvil]

In reference to the memory considerations with using a gfor-loop (see http://www.accelereyes.com/arrayfire/c/page_gfor.htm#gfor_restrictions), it is suggested to break up the loop into segments.

Code: Select all

// BEFORE
gfor (array k, 400) {
  array B = A(span,k);
  C(span,span,k) = B * B.T();  // outer product expansion runs out of memory
}
// AFTER
for (int kk = 0; kk < 400; kk += 100) {
  gfor (array k, kk, kk+99) { // four batches of 100
         array B = A(span,k);
         C(span,span,k) = B * B.T(); // now several smaller problems fit in card memory
  }
}


Are there some general guidelines in how big the segments can be. Or does this have to be done by trial and error?

More specific, in the example the loop of 400 is broken up into 4 batches of 100. I assume the increment (or size of batch) of 100 is dependant on the size of array B and device memory. Is there some way to deduce from this information that 100 is the ideal (or an acceptable) batch size?

Thank you

[pavanky]

Hi,

The ideal batch size will usually depend on the memory limits of the GPU you are running on.

You may want to fix the memory size the outer product can take and work backwards from there. I am not sure how much memory the rest of the variables take up. But if you know you have atleast 500 MB space for the gfor operation, you will be doing something like this.

Code: Select all

size_t maxmem = 500 * (1 << 20); // 500 MB
size_t mem_per_iter = A.dims(0) * A.dims(0) * sizeof(float); // Change float to double or cfloat or cdouble depending on the type
unsigned batch = maxmem / mem_per_iter;


Now you can use batch instead of 100 which would be ideal for your GPU.