Implementing an expanded object in C

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Implementing an expanded object in C

Michel Pelletier
Hello,

Apologies in advance for the long question.  I've made a lot of progress on my GraphBLAS extension and getting close to having most of the API usefully exposed to postgres, but I'm been struggling with an issue related to when i switched to using an expanded representation of matrix types.

I've tried to follow closely how arrays work, but the answer still eludes me.  The situation is slightly different in that where an array's flat representation is useful, a sparse matrix's flat form is an edge-list, so it's not useful unexpanded so I have a macro, PGGRB_GETARG_MATRIX(n) always returns the expanded form by checking VARATT_IS_EXTERNAL_EXPANDED_RW and then DatumGetEOHP if true, otherwise expanding from the flat representation:


I also have a PGGRB_RETURN_MATRIX(m) macro that calls `return EOHPGetRWDatum(&(A)->hdr)`


This chain of calls works for me in some cases, for example an operator function, 'matrix_mxm' which overloads the '*' operator, can be used to multiply two matrices:

postgres=# select '{{0,1,2},{1,2,0},{4,5,6}}'::matrix * '{{0,1,2},{1,2,0},{4,5,6}}'::matrix;
           ?column?
------------------------------
 {{0,1,2},{2,0,1},{20,30,24}}
(1 row)

Works great! Internally this was `matrix_out(matrix_mxm(matrix_in(), matrix_in()))` where the data flow fine both in and out of the functions.  But I have another function, 'matrix_agg', that aggregates edges from a query into a matrix.  It builds and returns the result matrix similarly to matrix_mxm does and returns it using the same macro, but matrix_out's call to get the agregates final value segfaults.

select matrix_agg(i, j, v) from edges;  -- segfaults in matrix_out at PG_GETARG_MATRIX(0)

at 


Afaict matrix_agg and matrix_mxm are both creating and returning matrices the same way, using the same function to build them and the same macro that `return EOHPGetRWDatum(&(A)->hdr)`, but when matrix_out fetches the argument to print the result it bombs on the aggregate's final value.  The only salient different I can see if the agg's final function calls:

  if (!AggCheckCallContext(fcinfo, &resultcxt)) {
    resultcxt = CurrentMemoryContext;
  }
 
  oldcxt = MemoryContextSwitchTo(resultcxt);
  // do matrix creation stuff
  MemoryContextSwitchTo(oldcxt);

 
But even if I remove that and do not switch contexts, it still crashes the same way.

It must be possible to return expanded objects from aggregates so I'm clearly doing something wrong.  The final function actually worked before I was using expanded representation and just using PG_RETURN_POINTER, but despite having all these clues I've been staring at this segfault in gdb for a couple of days now.

Any pointers on this subject would be greatly appreciated!  I know someone else out there recently was working on an expanded object posted on the list, if you don't see this, I may reach out to you. :)

-Michel


Reply | Threaded
Open this post in threaded view
|

Re: Implementing an expanded object in C

Michel Pelletier
Replying to my own problem here, I guess this was a situation where explaining it in detail revealed the problem to me.  By specifying my type is 'passedbyvalue' and 'alignment = double' it now works!

CREATE TYPE matrix (
    internallength = 8,
    input = matrix_in,
    output = matrix_out,
    passedbyvalue,
    alignment = double
);

Thanks for being a sounding board.

-Michel

On Sun, Jan 27, 2019 at 8:59 AM Michel Pelletier <[hidden email]> wrote:
Hello,

Apologies in advance for the long question.  I've made a lot of progress on my GraphBLAS extension and getting close to having most of the API usefully exposed to postgres, but I'm been struggling with an issue related to when i switched to using an expanded representation of matrix types.

I've tried to follow closely how arrays work, but the answer still eludes me.  The situation is slightly different in that where an array's flat representation is useful, a sparse matrix's flat form is an edge-list, so it's not useful unexpanded so I have a macro, PGGRB_GETARG_MATRIX(n) always returns the expanded form by checking VARATT_IS_EXTERNAL_EXPANDED_RW and then DatumGetEOHP if true, otherwise expanding from the flat representation:


I also have a PGGRB_RETURN_MATRIX(m) macro that calls `return EOHPGetRWDatum(&(A)->hdr)`


This chain of calls works for me in some cases, for example an operator function, 'matrix_mxm' which overloads the '*' operator, can be used to multiply two matrices:

postgres=# select '{{0,1,2},{1,2,0},{4,5,6}}'::matrix * '{{0,1,2},{1,2,0},{4,5,6}}'::matrix;
           ?column?
------------------------------
 {{0,1,2},{2,0,1},{20,30,24}}
(1 row)

Works great! Internally this was `matrix_out(matrix_mxm(matrix_in(), matrix_in()))` where the data flow fine both in and out of the functions.  But I have another function, 'matrix_agg', that aggregates edges from a query into a matrix.  It builds and returns the result matrix similarly to matrix_mxm does and returns it using the same macro, but matrix_out's call to get the agregates final value segfaults.

select matrix_agg(i, j, v) from edges;  -- segfaults in matrix_out at PG_GETARG_MATRIX(0)

at 


Afaict matrix_agg and matrix_mxm are both creating and returning matrices the same way, using the same function to build them and the same macro that `return EOHPGetRWDatum(&(A)->hdr)`, but when matrix_out fetches the argument to print the result it bombs on the aggregate's final value.  The only salient different I can see if the agg's final function calls:

  if (!AggCheckCallContext(fcinfo, &resultcxt)) {
    resultcxt = CurrentMemoryContext;
  }
 
  oldcxt = MemoryContextSwitchTo(resultcxt);
  // do matrix creation stuff
  MemoryContextSwitchTo(oldcxt);

 
But even if I remove that and do not switch contexts, it still crashes the same way.

It must be possible to return expanded objects from aggregates so I'm clearly doing something wrong.  The final function actually worked before I was using expanded representation and just using PG_RETURN_POINTER, but despite having all these clues I've been staring at this segfault in gdb for a couple of days now.

Any pointers on this subject would be greatly appreciated!  I know someone else out there recently was working on an expanded object posted on the list, if you don't see this, I may reach out to you. :)

-Michel


Reply | Threaded
Open this post in threaded view
|

Re: Implementing an expanded object in C

Andrew Gierth
>>>>> "Michel" == Michel Pelletier <[hidden email]> writes:

 Michel> Replying to my own problem here, I guess this was a situation
 Michel> where explaining it in detail revealed the problem to me. By
 Michel> specifying my type is 'passedbyvalue'

That cannot possibly be appropriate.

 Michel> CREATE TYPE matrix (
 Michel>     internallength = 8,

Your type is clearly not a fixed-length type, because fixed-length types
cannot have expanded datums. A fixed-length type must contain its entire
representation within the fixed length - it is not allowed to be a
pointer to something else.

--
Andrew (irc:RhodiumToad)

Reply | Threaded
Open this post in threaded view
|

Re: Implementing an expanded object in C

Michel Pelletier
Thanks Andrew, sorry for my late reply it took me a while to get time to analyze the issue.

You are of course correct, I had confusion around the create type statement, but the good news is, I have figured out the final ins and outs of creating an expanded type, and my matrix type can now be used as variables plpgsql functions and works great!  Thanks to you and Tom and everyone else who helped me get unblocked and making progress.

-Michel

On Sun, Jan 27, 2019 at 7:14 PM Andrew Gierth <[hidden email]> wrote:
>>>>> "Michel" == Michel Pelletier <[hidden email]> writes:

 Michel> Replying to my own problem here, I guess this was a situation
 Michel> where explaining it in detail revealed the problem to me. By
 Michel> specifying my type is 'passedbyvalue'

That cannot possibly be appropriate.

 Michel> CREATE TYPE matrix (
 Michel>     internallength = 8,

Your type is clearly not a fixed-length type, because fixed-length types
cannot have expanded datums. A fixed-length type must contain its entire
representation within the fixed length - it is not allowed to be a
pointer to something else.

--
Andrew (irc:RhodiumToad)