Opened 5 years ago

Last modified 5 years ago

#311 new enhancement

Buffers: Acquisition counted buffers

Reported by: dagss Owned by: somebody
Priority: major Milestone: wishlist
Component: Code Generation Keywords: numerics kurtgsoc
Cc:

Description (last modified by dagss)

Currently when one does

cdef object[int] a = ..., b
b = a

then the second line does a full acquisition of the buffer. This is unecesarry as a has all that is needed already.

To implement this in full generality one needs a reference count on the acquired Py_buffer struct.

While the immediate benefit may seem small, this becomes more crucial if #178 is implemented; where each slice would need a new reference to the buffer. The time taken of the tp_getbuffer of a given object is outside of our control; and we can't even be guaranteed to get back the same buffer each time, making slicing less robust.

Implementation notes

It seems clear that one needs a reference count allocated on the heap, since we want to eventually be able to pass buffers to other functions (#177) and store them in fields and global variables (#301). So something like:

typedef struct {
  size_t refcount;
  Py_buffer bufinfo;
} __Pyx_Buffer;

and then clients would, instead of having Py_buffers on the stack, malloc and free a __Pyx_buffer which would be shared among all users of it.

With regards to #299, this would mean that the structs in #299 would keep a pointer instead. I'll add a comment there.

(One could keep the Py_buffer on the stack and copy it by value, only the refcount absolutely needs to live on the heap, however it is the mallc/free that is expensive and might as well save some hassle and memory. The original must be kept anyway, as we are not allowed to change it before passing it back to releasebuffer)

As a possible optimization, if a buffer is never passed on to another variable/function then one can use a Py_buffer on the stack instead. Not a priority though.

Change History (4)

comment:1 Changed 5 years ago by dagss

Kurt: I tagged this as kurtgsoc only because the details would be relevant for you, not because you have to do it in your GSoC.

comment:2 Changed 5 years ago by dagss

  • Description modified (diff)

comment:3 Changed 5 years ago by dagss

  • Description modified (diff)

comment:4 Changed 5 years ago by dagss

  • Description modified (diff)
Note: See TracTickets for help on using tickets.