how to make query

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

how to make query

Sachin Kumar-2
Hi Team,

we are using Python Django and Postgresql. Our requirement is to search 10 Million Card data and bring specific records as a result after checking the record every time.

We have to find data from a 2nd table that is related to the first one which we are checking using the loop.

For Ex:- 
10000 data - loop 10000 times 
to find data from the 2nd table. 

This makes our query too slower and it gets time out.

--

Best Regards,
Sachin Kumar
Reply | Threaded
Open this post in threaded view
|

Re: how to make query

Philip Semanchuk-2


> On Jan 7, 2021, at 5:15 AM, Sachin Kumar <[hidden email]> wrote:
>
> Hi Team,
>
> we are using Python Django and Postgresql. Our requirement is to search 10 Million Card data and bring specific records as a result after checking the record every time.
>
> We have to find data from a 2nd table that is related to the first one which we are checking using the loop.
>
> For Ex:-
> 10000 data - loop 10000 times
> to find data from the 2nd table.
>
> This makes our query too slower and it gets time out.

Hi Sachin,
I’m not sure based on your description, but it sounds like you need to tell Django to select related objects all in one query rather than one at a time. If that’s the case, then your problem is a Django issue, not a Postgres one. Check out select_related() --

https://docs.djangoproject.com/en/3.1/ref/models/querysets/#django.db.models.query.QuerySet.select_related


Cheers
Philip

Reply | Threaded
Open this post in threaded view
|

Re: how to make query

Sachin Kumar-2
Hi Philip,

I am using Python Django and PostgreSQL to build our report app.

I am using below mention query to find some details. the query mention below is running fine in PostgreSQL command line and giving result in 2 sc for 2 million cards but when i i am try the same in Django application it is taking too much of time and the web page is expired while running this query.

can anyone help how to make query faster in Django? Or help me in correcting this query to be used in Django

rows = M_Assign.objects.raw(''' select
C.id,M."merchant_name",C."MERCHANT_ID",S."store_id" Owning_Store_ID ,S."store_name" Owning_Store_Name,F."franchisee_name" Owning_Group,C."ACCOUNT_NUMBER",C."STORE_ID",C."GIFT_LIST_REF" FROM vdaccount_card_assign C INNER JOIN vd_merchant_master M ON C."MERCHANT_ID"=M."merchant_id" AND C."MERCHANT_ID"='003561002966107' INNER JOIN vd_store_master S ON S."store_id"=C."GIFT_LIST_REF" OR C."GIFT_LIST_REF"='' INNER JOIN vd_franchisee F ON S."franchisee_id"=F."franchisee_id" '''    

On Thu, Jan 7, 2021 at 7:28 PM Philip Semanchuk <[hidden email]> wrote:


> On Jan 7, 2021, at 5:15 AM, Sachin Kumar <[hidden email]> wrote:
>
> Hi Team,
>
> we are using Python Django and Postgresql. Our requirement is to search 10 Million Card data and bring specific records as a result after checking the record every time.
>
> We have to find data from a 2nd table that is related to the first one which we are checking using the loop.
>
> For Ex:-
> 10000 data - loop 10000 times
> to find data from the 2nd table.
>
> This makes our query too slower and it gets time out.

Hi Sachin,
I’m not sure based on your description, but it sounds like you need to tell Django to select related objects all in one query rather than one at a time. If that’s the case, then your problem is a Django issue, not a Postgres one. Check out select_related() --

https://docs.djangoproject.com/en/3.1/ref/models/querysets/#django.db.models.query.QuerySet.select_related


Cheers
Philip


--

Best Regards,
Sachin Kumar
Reply | Threaded
Open this post in threaded view
|

Re: how to make query

Steve Midgley-3


On Thu, Jan 7, 2021 at 6:26 AM Sachin Kumar <[hidden email]> wrote:
Hi Philip,

I am using Python Django and PostgreSQL to build our report app.

I am using below mention query to find some details. the query mention below is running fine in PostgreSQL command line and giving result in 2 sc for 2 million cards but when i i am try the same in Django application it is taking too much of time and the web page is expired while running this query.

can anyone help how to make query faster in Django? Or help me in correcting this query to be used in Django

rows = M_Assign.objects.raw(''' select
C.id,M."merchant_name",C."MERCHANT_ID",S."store_id" Owning_Store_ID ,S."store_name" Owning_Store_Name,F."franchisee_name" Owning_Group,C."ACCOUNT_NUMBER",C."STORE_ID",C."GIFT_LIST_REF" FROM vdaccount_card_assign C INNER JOIN vd_merchant_master M ON C."MERCHANT_ID"=M."merchant_id" AND C."MERCHANT_ID"='003561002966107' INNER JOIN vd_store_master S ON S."store_id"=C."GIFT_LIST_REF" OR C."GIFT_LIST_REF"='' INNER JOIN vd_franchisee F ON S."franchisee_id"=F."franchisee_id" '''    

On Thu, Jan 7, 2021 at 7:28 PM Philip Semanchuk <[hidden email]> wrote:


> On Jan 7, 2021, at 5:15 AM, Sachin Kumar <[hidden email]> wrote:
>
> Hi Team,
>
> we are using Python Django and Postgresql. Our requirement is to search 10 Million Card data and bring specific records as a result after checking the record every time.
>
> We have to find data from a 2nd table that is related to the first one which we are checking using the loop.
>
> For Ex:-
> 10000 data - loop 10000 times
> to find data from the 2nd table.
>
> This makes our query too slower and it gets time out.

Hi Sachin,
I’m not sure based on your description, but it sounds like you need to tell Django to select related objects all in one query rather than one at a time. If that’s the case, then your problem is a Django issue, not a Postgres one. Check out select_related() --

https://docs.djangoproject.com/en/3.1/ref/models/querysets/#django.db.models.query.QuerySet.select_related


Cheers
Philip


Hi Sachin,

As a general matter, people on this list will be able to help you more effectively if you provide some DDL, ideally some sample data, and an "explain analyze" report from your current setup, along with the query SQL. If you can simplify your table structures in the DDL to focus only on the place where you are having performance problems, that's even better.

Best,
Steve
Reply | Threaded
Open this post in threaded view
|

Re: how to make query

Philip Semanchuk-2
In reply to this post by Sachin Kumar-2


> On Jan 7, 2021, at 9:25 AM, Sachin Kumar <[hidden email]> wrote:
>
> Hi Philip,
>
> I am using Python Django and PostgreSQL to build our report app.
>
> I am using below mention query to find some details. the query mention below is running fine in PostgreSQL command line and giving result in 2 sc for 2 million cards but when i i am try the same in Django application it is taking too much of time and the web page is expired while running this query.
>
> can anyone help how to make query faster in Django? Or help me in correcting this query to be used in Django


Sachin,
If your query runs well in Postgres, then your problem is with Django, not Postgres. You would be better off asking on a Django mailing list how to get Django to construct the query you want.




>
> On Thu, Jan 7, 2021 at 7:28 PM Philip Semanchuk <[hidden email]> wrote:
>
>
> > On Jan 7, 2021, at 5:15 AM, Sachin Kumar <[hidden email]> wrote:
> >
> > Hi Team,
> >
> > we are using Python Django and Postgresql. Our requirement is to search 10 Million Card data and bring specific records as a result after checking the record every time.
> >
> > We have to find data from a 2nd table that is related to the first one which we are checking using the loop.
> >
> > For Ex:-
> > 10000 data - loop 10000 times
> > to find data from the 2nd table.
> >
> > This makes our query too slower and it gets time out.
>
> Hi Sachin,
> I’m not sure based on your description, but it sounds like you need to tell Django to select related objects all in one query rather than one at a time. If that’s the case, then your problem is a Django issue, not a Postgres one. Check out select_related() --
>
> https://docs.djangoproject.com/en/3.1/ref/models/querysets/#django.db.models.query.QuerySet.select_related
>
>
> Cheers
> Philip
>
>
> --
>
> Best Regards,
> Sachin Kumar