how to debug the postgres performance issue

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

how to debug the postgres performance issue

范国腾

Hi,

 

We are doing the performance test in two nodes system(active/standby). Now we find that the disk IO is very high but the CPU is low. Is there any method or tool to help us debug to find which function makes this issue?

 

We try to use the gprof but it does not help us. The result show the LWLockAssignthe shared buffer init is called the most time. The other function is called no more that 33139. That seems no problem.

 

Could you please  give any suggestion how to debug the high IO/low CPU issue?

 

Thanks

Steven

 

===================================

no time accumulated

 

  %   cumulative   self              self     total          

 time   seconds   seconds    calls  Ts/call  Ts/call  name   

  0.00      0.00     0.00  9240928     0.00     0.00  LWLockAssign

  0.00      0.00     0.00    33139     0.00     0.00  ShmemAlloc

  0.00      0.00     0.00    33084     0.00     0.00  seg_alloc

….

Reply | Threaded
Open this post in threaded view
|

Re: how to debug the postgres performance issue

Ron-2
On 09/14/2018 04:34 AM, 范国腾 wrote:

Hi,

 

We are doing the performance test in two nodes system(active/standby). Now we find that the disk IO is very high but the CPU is low.


On both nodes or just one (and if so, which one)?

If you incrementally increase the workload, starting from the quiescent state, when do you see this happening?

Is there any method or tool to help us debug to find which function makes this issue?

 

We try to use the gprof but it does not help us. The result show the LWLockAssignthe shared buffer init is called the most time. The other function is called no more that 33139. That seems no problem.

 

Could you please  give any suggestion how to debug the high IO/low CPU issue?

 

Thanks

Steven

 

===================================

no time accumulated

 

  %   cumulative   self              self     total          

 time   seconds   seconds    calls  Ts/call  Ts/call  name   

  0.00      0.00     0.00  9240928     0.00     0.00  LWLockAssign

  0.00      0.00     0.00    33139     0.00     0.00  ShmemAlloc

  0.00      0.00     0.00    33084     0.00     0.00  seg_alloc

….


--
Angular momentum makes the world go 'round.
Reply | Threaded
Open this post in threaded view
|

Re: how to debug the postgres performance issue

Olivier Gautherot
In reply to this post by 范国腾
On Fri, Sep 14, 2018 at 6:29 AM 范国腾 <[hidden email]> wrote:

Hi,

 

We are doing the performance test in two nodes system(active/standby). Now we find that the disk IO is very high but the CPU is low. Is there any method or tool to help us debug to find which function makes this issue?

 

We try to use the gprof but it does not help us. The result show the LWLockAssignthe shared buffer init is called the most time. The other function is called no more that 33139. That seems no problem.

 

Could you please  give any suggestion how to debug the high IO/low CPU issue?

 

Thanks

Steven

 

===================================

no time accumulated

 

  %   cumulative   self              self     total          

 time   seconds   seconds    calls  Ts/call  Ts/call  name   

  0.00      0.00     0.00  9240928     0.00     0.00  LWLockAssign

  0.00      0.00     0.00    33139     0.00     0.00  ShmemAlloc

  0.00      0.00     0.00    33084     0.00     0.00  seg_alloc

….


There are a few things you should investigate for a start, as you seem to be lifting more data than you probably should.

1) Check the sizes of your tables (in particular the number of rows) and identify the heaviest
2) With pgfouine or pgbadger, check which queries take most time and see if there is a correlation (there probably is)
3) Run EXPLAIN on these queries to see if you have a sequential scan on the whole table that should be optimized
4) Design indexes to relieve these queries (note that the queries will run faster but the INSERT on these tables will be heavier)
5) In postgresql.conf, check the following parameters:
  * shared_buffers (I've set it to 1/4 of the RAM with good success)
  * temp_buffers (you may need to experiment in double increments)
  * work_mem (try to increase it too)
6)  Check how often the autovacuum runs (could be another indicator)

Olivier