Slow application response on lightly loaded server?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Slow application response on lightly loaded server?

Mike Blackwell-2
We're seeing slow application performance on a PostgreSQL 9.1 server which appears to be relatively lightly loaded.  Some graphs from pgstatview are at  http://www2.uptimeforce.com/pgstatview/e35ba4e7db0842a1b9cf2e10a4c03d91/   These cover approximately 40 minutes, during which there was some activity from a web application and two bulk loads in process.  

The machine running the bulk loads (perl scripts) is also running at about 70% idle with very little iowait.  That seems to suggest network latency to me.

Am I missing something in the server stats that would indicate a problem?  If not, where should I look next?  

__________________________________________________________________________________
Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management | RR Donnelley
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
[hidden email]
http://www.rrdonnelley.com


[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Slow application response on lightly loaded server?

Scott Marlowe-2
On Tue, Jul 17, 2012 at 10:27 AM, Mike Blackwell <[hidden email]> wrote:

> We're seeing slow application performance on a PostgreSQL 9.1 server which
> appears to be relatively lightly loaded.  Some graphs from pgstatview are at
> http://www2.uptimeforce.com/pgstatview/e35ba4e7db0842a1b9cf2e10a4c03d91/
> These cover approximately 40 minutes, during which there was some activity
> from a web application and two bulk loads in process.
>
> The machine running the bulk loads (perl scripts) is also running at about
> 70% idle with very little iowait.  That seems to suggest network latency to
> me.
>
> Am I missing something in the server stats that would indicate a problem?
> If not, where should I look next?

I'd run vmstat and look for high cs or int numbers (100k and above) to
see if you're maybe seeing an issue with that.  A lot of times a
"slow" server is just too much process switching.  But yeah, the
graphs you've posted don't seem overly bad.

--
Sent via pgsql-performance mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Reply | Threaded
Open this post in threaded view
|

Re: Slow application response on lightly loaded server?

Mike Blackwell-2



On Tue, Jul 17, 2012 at 11:35 AM, Scott Marlowe <[hidden email]> wrote:

I'd run vmstat and look for high cs or int numbers (100k and above) to
see if you're maybe seeing an issue with that.  A lot of times a
"slow" server is just too much process switching.  But yeah, the
graphs you've posted don't seem overly bad.


Thanks for the tip.  Here's a quick look at those numbers under that same load.  Watching it for a while longer didn't show any spikes.  That doesn't seem to be it, either.  

$ vmstat 5
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 3  0  11868  34500  16048 3931436    0    0     4     2    0     0  6  2 91  1
 2  0  11868  21964  16088 3931396    0    0     0   212 8667  8408 15  3 80  2
 0  0  11868  37772  16112 3932152    0    0     2   249 9109  8811 34  2 62  1
 2  0  11868  34068  16124 3932400    0    0     1   168 9142  9165 12  3 84  1
 1  0  11868  38036  16124 3932920    0    0     8   155 9995 10904 16  4 80  1
 1  0  11868  40212  16124 3933440    0    0     0   146 9586  9825 13  3 83  1

__________________________________________________________________________________
Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management | RR Donnelley
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
[hidden email]
http://www.rrdonnelley.com



Reply | Threaded
Open this post in threaded view
|

Re: Slow application response on lightly loaded server?

Scott Marlowe-2
On Tue, Jul 17, 2012 at 11:37 AM, Mike Blackwell <[hidden email]> wrote:

>
> On Tue, Jul 17, 2012 at 11:35 AM, Scott Marlowe <[hidden email]>
> wrote:
>
>> I'd run vmstat and look for high cs or int numbers (100k and above) to
>> see if you're maybe seeing an issue with that.  A lot of times a
>> "slow" server is just too much process switching.  But yeah, the
>> graphs you've posted don't seem overly bad.
>
>
>
> Thanks for the tip.  Here's a quick look at those numbers under that same
> load.  Watching it for a while longer didn't show any spikes.  That doesn't
> seem to be it, either.

Yep it all looks good to me.  Are you sure you're not getting network
lag or something like that?

--
Sent via pgsql-performance mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Reply | Threaded
Open this post in threaded view
|

Re: Slow application response on lightly loaded server?

Mike Blackwell-2
I'm wondering about that.  However, the database server and the server doing the bulk loads are on the same subnet.  Traceroute shows only a single hop.  Traceroute and ping both show reply times in the area of .25 - .50 ms or so.  Is that reasonable?

__________________________________________________________________________________
Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management | RR Donnelley
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
[hidden email]
http://www.rrdonnelley.com


[hidden email]


On Tue, Jul 17, 2012 at 12:49 PM, Scott Marlowe <[hidden email]> wrote:
On Tue, Jul 17, 2012 at 11:37 AM, Mike Blackwell <[hidden email]> wrote:
>
> On Tue, Jul 17, 2012 at 11:35 AM, Scott Marlowe <[hidden email]>
> wrote:
>
>> I'd run vmstat and look for high cs or int numbers (100k and above) to
>> see if you're maybe seeing an issue with that.  A lot of times a
>> "slow" server is just too much process switching.  But yeah, the
>> graphs you've posted don't seem overly bad.
>
>
>
> Thanks for the tip.  Here's a quick look at those numbers under that same
> load.  Watching it for a while longer didn't show any spikes.  That doesn't
> seem to be it, either.

Yep it all looks good to me.  Are you sure you're not getting network
lag or something like that?

Reply | Threaded
Open this post in threaded view
|

Re: Slow application response on lightly loaded server?

Scott Marlowe-2
Yeah seems reasonable.  The last thing I'd look at is something like
improperly configured dns service.  Are you connecting by IP or by
host name?

On Tue, Jul 17, 2012 at 12:10 PM, Mike Blackwell <[hidden email]> wrote:

> I'm wondering about that.  However, the database server and the server doing
> the bulk loads are on the same subnet.  Traceroute shows only a single hop.
> Traceroute and ping both show reply times in the area of .25 - .50 ms or so.
> Is that reasonable?
>
> __________________________________________________________________________________
> Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management
> | RR Donnelley
> 1750 Wallace Ave | St Charles, IL 60174-3401
> Office: 630.313.7818
> [hidden email]
> http://www.rrdonnelley.com
>
>
>
>
>
> On Tue, Jul 17, 2012 at 12:49 PM, Scott Marlowe <[hidden email]>
> wrote:
>>
>> On Tue, Jul 17, 2012 at 11:37 AM, Mike Blackwell <[hidden email]>
>> wrote:
>> >
>> > On Tue, Jul 17, 2012 at 11:35 AM, Scott Marlowe
>> > <[hidden email]>
>> > wrote:
>> >
>> >> I'd run vmstat and look for high cs or int numbers (100k and above) to
>> >> see if you're maybe seeing an issue with that.  A lot of times a
>> >> "slow" server is just too much process switching.  But yeah, the
>> >> graphs you've posted don't seem overly bad.
>> >
>> >
>> >
>> > Thanks for the tip.  Here's a quick look at those numbers under that
>> > same
>> > load.  Watching it for a while longer didn't show any spikes.  That
>> > doesn't
>> > seem to be it, either.
>>
>> Yep it all looks good to me.  Are you sure you're not getting network
>> lag or something like that?
>
>



--
To understand recursion, one must first understand recursion.

--
Sent via pgsql-performance mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
Reply | Threaded
Open this post in threaded view
|

Re: Slow application response on lightly loaded server?

Mike Blackwell-2
On Tue, Jul 17, 2012 at 2:36 PM, Scott Marlowe <[hidden email]> wrote:
 
Yeah seems reasonable.  The last thing I'd look at is something like
improperly configured dns service.  Are you connecting by IP or by
host name?

 
Interesting possibility.  We're currently connecting by host name.  I could try temporarily using the IP from one of the servers to see if that helps.  I'm not familiar enough with DNS services to do any diagnostics other than using dig to see where something points.

Thanks for your help, BTW!

__________________________________________________________________________________
Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management | RR Donnelley
1750 Wallace Ave | St Charles, IL 60174-3401
Office: 630.313.7818
[hidden email]
http://www.rrdonnelley.com


Reply | Threaded
Open this post in threaded view
|

Re: Slow application response on lightly loaded server?

Scott Marlowe-2
Well if it suddenly gets faster when connecting by IP, you'll know
where your problem lies.  DNS issues are more common in windows
installs, due to Windows having more interesting ways to misconfigure
dns etc.

On Tue, Jul 17, 2012 at 1:48 PM, Mike Blackwell <[hidden email]> wrote:

> On Tue, Jul 17, 2012 at 2:36 PM, Scott Marlowe <[hidden email]>
> wrote:
>
>
>>
>> Yeah seems reasonable.  The last thing I'd look at is something like
>> improperly configured dns service.  Are you connecting by IP or by
>> host name?
>>
>
> Interesting possibility.  We're currently connecting by host name.  I could
> try temporarily using the IP from one of the servers to see if that helps.
> I'm not familiar enough with DNS services to do any diagnostics other than
> using dig to see where something points.
>
> Thanks for your help, BTW!
>
> __________________________________________________________________________________
> Mike Blackwell | Technical Analyst, Distribution Services/Rollout Management
> | RR Donnelley
> 1750 Wallace Ave | St Charles, IL 60174-3401
> Office: 630.313.7818
> [hidden email]
> http://www.rrdonnelley.com
>
>



--
To understand recursion, one must first understand recursion.

--
Sent via pgsql-performance mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance