Email to hackers for test coverage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Email to hackers for test coverage

movead li
Hello hackers,

One of the area that didn't get much attention in the community
recently is analysing and increasing the code coverage of 
PostgreSQL regession test suite. I have started working on the
code coverage by running the GCOV code coverage analysis tool in
order to analyse the current code coverage and come up with test
cases to increase the code coverage. This is going to be a long
exercise so my plan is do it incrementaly. I will be analysing
some area of untested code and then coming up with test cases to
test those lines of code in regression and then moving on next
area of untested code and so on.

So far I have come up with 3 test cases to increase the code
coverage of PostgreSQL regression test suite.

I have performed the regression run for this exercise on this commit:
(Commit version 75c1921cd6c868c5995b88113b4463a4830b9a27):

The regression is executed with make check-world command and the
results are gathered using  'make coverage-html' command. 

Below are the lines of untested code that i have analysed and the
test cases added to regression to test these as part of regression.

1. src/include/utils/float.h:140

Analyze: 
This is an error report line when converting a big float8 value
which a float4 can not storage to float4.

Test case:
Add a test case as below in file float4.sql:
select float4(1234567890123456789012345678901234567890::float8);

2. src/include/utils/float.h:145

Analyze:
This is an error report line when converting a small float8 value
which a float4 can not storage to float4.

Test case:
Add a test case as below in file float4.sql:
select float4(0.0000000000000000000000000000000000000000000001::float8);

3.src/include/utils/sortsupport.h:264

Analyze:
It is reverse sorting for the data type that has abbreviated for
sort, for example macaddr, uuid, numeric, network and I choose
numeric to do it.

Test cast:
Add a test case as below in file numeric.sql:
INSERT INTO num_input_test(n1) values('99999999999999999999999999.998');
INSERT INTO num_input_test(n1) values('99999999999999999999999999.997');
SELECT * FROM num_input_test ORDER BY n1 DESC;

Result and patch

By adding the test cases, the test coverage of  float.h increased from
97.7% to 100% and sortsupport.h increased from 76.7% to 80.0%.
 
The increase in code coverage can be seen in the before and after
pictures of GCOV test coverage analysis summary.

The attached patch contain the test cases added in regression for
increasing the coverage.


--
Movead Li

before.png (56K) Download Attachment
after.png (21K) Download Attachment
regression_20190820.patch (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Email to hackers for test coverage

Ahsan Hadi-2
The subject of the email may be bit misleading however this is really good exercise to analyse the current code of Postgres regression suite using GCOV and then adding test cases incrementally to increase the test coverage. 

On Thu, 22 Aug 2019 at 2:46 PM, [hidden email] <[hidden email]> wrote:
Hello hackers,

One of the area that didn't get much attention in the community
recently is analysing and increasing the code coverage of 
PostgreSQL regession test suite. I have started working on the
code coverage by running the GCOV code coverage analysis tool in
order to analyse the current code coverage and come up with test
cases to increase the code coverage. This is going to be a long
exercise so my plan is do it incrementaly. I will be analysing
some area of untested code and then coming up with test cases to
test those lines of code in regression and then moving on next
area of untested code and so on.

So far I have come up with 3 test cases to increase the code
coverage of PostgreSQL regression test suite.

I have performed the regression run for this exercise on this commit:
(Commit version 75c1921cd6c868c5995b88113b4463a4830b9a27):

The regression is executed with make check-world command and the
results are gathered using  'make coverage-html' command. 

Below are the lines of untested code that i have analysed and the
test cases added to regression to test these as part of regression.

1. src/include/utils/float.h:140

Analyze: 
This is an error report line when converting a big float8 value
which a float4 can not storage to float4.

Test case:
Add a test case as below in file float4.sql:
select float4(1234567890123456789012345678901234567890::float8);

2. src/include/utils/float.h:145

Analyze:
This is an error report line when converting a small float8 value
which a float4 can not storage to float4.

Test case:
Add a test case as below in file float4.sql:
select float4(0.0000000000000000000000000000000000000000000001::float8);

3.src/include/utils/sortsupport.h:264

Analyze:
It is reverse sorting for the data type that has abbreviated for
sort, for example macaddr, uuid, numeric, network and I choose
numeric to do it.

Test cast:
Add a test case as below in file numeric.sql:
INSERT INTO num_input_test(n1) values('99999999999999999999999999.998');
INSERT INTO num_input_test(n1) values('99999999999999999999999999.997');
SELECT * FROM num_input_test ORDER BY n1 DESC;

Result and patch

By adding the test cases, the test coverage of  float.h increased from
97.7% to 100% and sortsupport.h increased from 76.7% to 80.0%.
 
The increase in code coverage can be seen in the before and after
pictures of GCOV test coverage analysis summary.

The attached patch contain the test cases added in regression for
increasing the coverage.


--
Movead Li
Reply | Threaded
Open this post in threaded view
|

Re: Email to hackers for test coverage

Ibrar Ahmed-4
In reply to this post by movead li

On Thu, Aug 22, 2019 at 2:46 PM [hidden email] <[hidden email]> wrote:
Hello hackers,

One of the area that didn't get much attention in the community
recently is analysing and increasing the code coverage of 
PostgreSQL regession test suite. I have started working on the
code coverage by running the GCOV code coverage analysis tool in
order to analyse the current code coverage and come up with test
cases to increase the code coverage. This is going to be a long
exercise so my plan is do it incrementaly. I will be analysing
some area of untested code and then coming up with test cases to
test those lines of code in regression and then moving on next
area of untested code and so on.

So far I have come up with 3 test cases to increase the code
coverage of PostgreSQL regression test suite.

I have performed the regression run for this exercise on this commit:
(Commit version 75c1921cd6c868c5995b88113b4463a4830b9a27):

The regression is executed with make check-world command and the
results are gathered using  'make coverage-html' command. 

Below are the lines of untested code that i have analysed and the
test cases added to regression to test these as part of regression.

1. src/include/utils/float.h:140

Analyze: 
This is an error report line when converting a big float8 value
which a float4 can not storage to float4.

Test case:
Add a test case as below in file float4.sql:
select float4(1234567890123456789012345678901234567890::float8);

2. src/include/utils/float.h:145

Analyze:
This is an error report line when converting a small float8 value
which a float4 can not storage to float4.

Test case:
Add a test case as below in file float4.sql:
select float4(0.0000000000000000000000000000000000000000000001::float8);

3.src/include/utils/sortsupport.h:264

Analyze:
It is reverse sorting for the data type that has abbreviated for
sort, for example macaddr, uuid, numeric, network and I choose
numeric to do it.

Test cast:
Add a test case as below in file numeric.sql:
INSERT INTO num_input_test(n1) values('99999999999999999999999999.998');
INSERT INTO num_input_test(n1) values('99999999999999999999999999.997');
SELECT * FROM num_input_test ORDER BY n1 DESC;

Result and patch

By adding the test cases, the test coverage of  float.h increased from
97.7% to 100% and sortsupport.h increased from 76.7% to 80.0%.
 
The increase in code coverage can be seen in the before and after
pictures of GCOV test coverage analysis summary.

The attached patch contain the test cases added in regression for
increasing the coverage.


--
Movead Li

Hi Movead,
Please add that to commitfest.
 


--
Ibrar Ahmed
Reply | Threaded
Open this post in threaded view
|

Re: Re: Email to hackers for test coverage

movead li
>Hi Movead,
>Please add that to commitfest.

Thanks and I just do it, it is
https://commitfest.postgresql.org/24/2258/

--
Movead Li
Reply | Threaded
Open this post in threaded view
|

Re: Re: Email to hackers for test coverage

Michael Paquier-2
On Sat, Aug 24, 2019 at 11:23:32PM +0800, [hidden email] wrote:
> Thanks and I just do it, it is
> https://commitfest.postgresql.org/24/2258/

Your patch has forgotten to update the alternate output in
float4-misrounded-input.out.
--
Michael

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Email to hackers for test coverage

movead li
  On Mon, 26 Aug 2019 12:48:40 +0800 [hidden email] wrote
>Your patch has forgotten to update the alternate output in
>float4-misrounded-input.out.

Thanks for your remind, I have modified the patch and now it is 
'regression_20190826.patch' in attachment, and I have done a successful
test on Cygwin.

--
Movead

regression_20190826.patch (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Email to hackers for test coverage

Michael Paquier-2
On Mon, Aug 26, 2019 at 05:10:59PM +0800, [hidden email] wrote:
> Thanks for your remind, I have modified the patch and now it is
> 'regression_20190826.patch' in attachment, and I have done a successful
> test on Cygwin.

There is a section in float4.sql which deals with overflow and
underflow, so wouldn't it be better to move the tests there?  You
could just trigger the failures with that:
=# insert into float4_tbl values ('-10e-70'::float8);
ERROR:  22003: value out of range: underflow
LOCATION:  check_float4_val, float.h:145
=# insert into float4_tbl values ('-10e70'::float8);
ERROR:  22003: value out of range: overflow
LOCATION:  check_float4_val, float.h:140

I would also test all four patterns: 10e70, 10e-70, -10e70, -10e-70.

For the numeric part, this improves the case of
ApplySortAbbrevFullComparator() where both values are not NULL.  Could
things be done so as the other code paths are fully covered?  One
INSERT is fine by the way to add the extra coverage.
--
Michael

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Email to hackers for test coverage

movead li

On Tue, 27 Aug 2019 14:07:48 +0800 [hidden email] wrote:
> There is a section in float4.sql which deals with overflow and
underflow, so wouldn't it be better to move the tests there?  You
could just trigger the failures with that:
=# insert into float4_tbl values ('-10e-70'::float8);
ERROR:  22003: value out of range: underflow
LOCATION:  check_float4_val, float.h:145
=# insert into float4_tbl values ('-10e70'::float8);
ERROR:  22003: value out of range: overflow
LOCATION:  check_float4_val, float.h:140
> I would also test all four patterns: 10e70, 10e-70, -10e70, -10e-70.
 
I think your way is much better, so I change the patch and it is  
'regression_20190827.patch' now.

> For the numeric part, this improves the case of
> ApplySortAbbrevFullComparator() where both values are not NULL.  Could
> things be done so as the other code paths are fully covered?  One
> INSERT is fine by the way to add the extra coverage.
There are code lines related to NULL values in
ApplySortAbbrevFullComparator(), but I think the code lines for
comparing a NULL and a NOT NULL can be never reached, because it is
handled in ApplySortComparator() which is called before
ApplySortAbbrevFullComparator(). So I think it is no use to INSERT
a NULL value.

--
Movead

regression_20190827.patch (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Email to hackers for test coverage

Michael Paquier-2
On Tue, Aug 27, 2019 at 03:57:20PM +0800, [hidden email] wrote:
> I think your way is much better, so I change the patch and it is  
> 'regression_20190827.patch' now.

Thanks for the new patch, I have committed the part for float4.

> There are code lines related to NULL values in
> ApplySortAbbrevFullComparator(), but I think the code lines for
> comparing a NULL and a NOT NULL can be never reached, because it is
> handled in ApplySortComparator() which is called before
> ApplySortAbbrevFullComparator(). So I think it is no use to INSERT
> a NULL value.

But I am not sold to that part yet, for three reasons:
- numeric is not a test suite designed for sorting, and hijacking it
to do so it not a good approach.
- it would be good to get coverage for the two extra code paths while
we are on it with NULL datums.
- There is no need for two INSERT queries, I am getting the same
coverage with only one.

Please note that I have not looked in details where we could put that,
but perhaps Robert and Peter G who worked on 4ea51cd to add support
for abbreviated keys have some ideas, so I am adding them in CC for
input.
--
Michael

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Re: Email to hackers for test coverage

movead li

On Wed, 28 Aug 2019 11:30:23 +0800 [hidden email] wrote
>- numeric is not a test suite designed for sorting, and hijacking it
>to do so it not a good approach.
Absolutely agreement. We can wait for repling from
Robert and Peter G.

>- it would be good to get coverage for the two extra code paths while
>we are on it with NULL datums..
The remained untested line in ApplySortComparator() can be never
reached I think, and I have explained it on last mail. 
Or I don't fully understand what you mean?
 
>- There is no need for two INSERT queries, I am getting the same
>coverage with only one.
Yes It is my mistake,  the key point is 'ORDER BY n1 DESC' not the 
'INSERT'. Infact it can run the code line without any of the two INSERT.

--
Movead
Reply | Threaded
Open this post in threaded view
|

Re: Email to hackers for test coverage

Peter Eisentraut-6
In reply to this post by movead li
On 2019-08-22 11:46, [hidden email] wrote:
> *1. src/include/utils/float.h:140*
>
> Analyze: 
> This is an error report line when converting a big float8 value
> which a float4 can not storage to float4.
>
> Test case:
> Add a test case as below in file float4.sql:
> select float4(1234567890123456789012345678901234567890::float8);

> +-- Add test case for float4() type fonversion function

Check spelling

> *2. src/include/utils/float.h:145*
>
> Analyze:
> This is an error report line when converting a small float8 value
> which a float4 can not storage to float4.
>
> Test case:
> Add a test case as below in file float4.sql:
> select float4(0.0000000000000000000000000000000000000000000001::float8);
>
> *3.src/include/utils/sortsupport.h:264*
>
> Analyze:
> It is reverse sorting for the data type that has abbreviated for
> sort, for example macaddr, uuid, numeric, network and I choose
> numeric to do it.
>
> Test cast:
> Add a test case as below in file numeric.sql:
> INSERT INTO num_input_test(n1) values('99999999999999999999999999.998');
> INSERT INTO num_input_test(n1) values('99999999999999999999999999.997');
> SELECT * FROM num_input_test ORDER BY n1 DESC;

>  INSERT INTO num_input_test(n1) VALUES ('        nan');
> +INSERT INTO num_input_test(n1) values('99999999999999999999999999.998');
> +INSERT INTO num_input_test(n1) values('99999999999999999999999999.997');

Make spaces and capitalization match surrounding code.

> Result and patch
>
> By adding the test cases, the test coverage of  float.h increased from
> 97.7% to 100% and sortsupport.h increased from 76.7% to 80.0%.

That's fine, but I suggest that if you really want to make an impact in
test coverage, look to increase function coverage rather than line
coverage.  Or look for files that are not covered at all.

--
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


Reply | Threaded
Open this post in threaded view
|

Re: Re: Email to hackers for test coverage

Peter Geoghegan-4
In reply to this post by Michael Paquier-2
On Tue, Aug 27, 2019 at 8:30 PM Michael Paquier <[hidden email]> wrote:
> Please note that I have not looked in details where we could put that,
> but perhaps Robert and Peter G who worked on 4ea51cd to add support
> for abbreviated keys have some ideas, so I am adding them in CC for
> input.

Movead is correct -- the NULL handling within
ApplySortAbbrevFullComparator() cannot actually be used currently. I
wouldn't change anything about the code, though, since it's useful to
defensively handle NULLs.

--
Peter Geoghegan


Reply | Threaded
Open this post in threaded view
|

Re: Re: Email to hackers for test coverage

movead li
In reply to this post by Peter Eisentraut-6

On 2019-08-29 00:43, [hidden email] wrote:
 
 
> Make spaces and capitalization match surrounding code..
>That's fine, but I suggest that if you really want to make an impact in
>test coverage, look to increase function coverage rather than line
>coverage.  Or look for files that are not covered at all.
 
Thanks for pointing all the things, I will reconsider my way on
code coverage work.

--
Movead
Reply | Threaded
Open this post in threaded view
|

Re: Email to hackers for test coverage

Ahsan Hadi-2
In reply to this post by Peter Eisentraut-6


On Wed, Aug 28, 2019 at 9:43 PM Peter Eisentraut <[hidden email]> wrote:
On 2019-08-22 11:46, [hidden email] wrote:
> *1. src/include/utils/float.h:140*
>
> Analyze: 
> This is an error report line when converting a big float8 value
> which a float4 can not storage to float4.
>
> Test case:
> Add a test case as below in file float4.sql:
> select float4(1234567890123456789012345678901234567890::float8);

> +-- Add test case for float4() type fonversion function

Check spelling

> *2. src/include/utils/float.h:145*
>
> Analyze:
> This is an error report line when converting a small float8 value
> which a float4 can not storage to float4.
>
> Test case:
> Add a test case as below in file float4.sql:
> select float4(0.0000000000000000000000000000000000000000000001::float8);
>
> *3.src/include/utils/sortsupport.h:264*
>
> Analyze:
> It is reverse sorting for the data type that has abbreviated for
> sort, for example macaddr, uuid, numeric, network and I choose
> numeric to do it.
>
> Test cast:
> Add a test case as below in file numeric.sql:
> INSERT INTO num_input_test(n1) values('99999999999999999999999999.998');
> INSERT INTO num_input_test(n1) values('99999999999999999999999999.997');
> SELECT * FROM num_input_test ORDER BY n1 DESC;

>  INSERT INTO num_input_test(n1) VALUES ('        nan');
> +INSERT INTO num_input_test(n1) values('99999999999999999999999999.998');
> +INSERT INTO num_input_test(n1) values('99999999999999999999999999.997');

Make spaces and capitalization match surrounding code.

> Result and patch
>
> By adding the test cases, the test coverage of  float.h increased from
> 97.7% to 100% and sortsupport.h increased from 76.7% to 80.0%.

That's fine, but I suggest that if you really want to make an impact in
test coverage, look to increase function coverage rather than line
coverage.  Or look for files that are not covered at all.


+1 
 
--
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


Reply | Threaded
Open this post in threaded view
|

Re: Re: Email to hackers for test coverage

Michael Paquier-2
In reply to this post by Peter Geoghegan-4
On Wed, Aug 28, 2019 at 11:27:15AM -0700, Peter Geoghegan wrote:
> Movead is correct -- the NULL handling within
> ApplySortAbbrevFullComparator() cannot actually be used currently. I
> wouldn't change anything about the code, though, since it's useful to
> defensively handle NULLs.

No objections with this line of thoughts.  Thanks, Peter.  Please note
that I have marked the original patch as committed in the CF app.  If
there are tests to improve the coverage, let's do that on a new
thread.  I am still not sure where I would put tests dedicated to
abbreviated keys, but let's sort out that if necessary later.
--
Michael

signature.asc (849 bytes) Download Attachment