Error in COPY command with files over 1GB

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Error in COPY command with files over 1GB

Andrew Zimmerman

I am getting the following error when importing from a tab-delimited text file longer than 1 GB.

ERROR: could not stat file "C:\ABC Software\Database Export\Data\invoice_detail.data": Unknown error

Splitting the tab-delimited file into multiple files less than 1 GB and COPY’ing them individually works correctly.

I tested on both versions 11.2 and 11.4 on Windows with the same result.

 

Andrew Zimmerman

ABC Softek LLC

[hidden email]

www.SimplifyMyBiz.com

 

Reply | Threaded
Open this post in threaded view
|

Re: Error in COPY command with files over 1GB

Michael Paquier-2
On Thu, Aug 08, 2019 at 04:30:00PM +0000, Andrew Zimmerman wrote:
> I am getting the following error when importing from a tab-delimited text file longer than 1 GB.
> ERROR: could not stat file "C:\ABC Software\Database Export\Data\invoice_detail.data": Unknown error
> Splitting the tab-delimited file into multiple files less than 1 GB and COPY'ing them individually works correctly.
> I tested on both versions 11.2 and 11.4 on Windows with the same result.

This relates to the fact that Windows does not offer an easy way to
stat() correctly files which are more than 2GB, as the struct stat is
built with a 4-byte size field.

There has been some work on that lately (quoting Tom from a recent
mail):
https://www.postgresql.org/message-id/flat/1803D792815FC24D871C00D17AE95905CF5099%40g01jpexmbkw24

However there is no way that a complex patch would get back-patched,
and that's what we were heading to in the latest discussions I recall
on the matter.
--
Michael

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Error in COPY command with files over 1GB

Juan José Santamaría Flecha
On Fri, Aug 9, 2019 at 4:24 AM Michael Paquier <[hidden email]> wrote:
>
> There has been some work on that lately (quoting Tom from a recent
> mail):
> https://www.postgresql.org/message-id/flat/1803D792815FC24D871C00D17AE95905CF5099%40g01jpexmbkw24
>

There is currently an open item in the commitfest:

https://commitfest.postgresql.org/24/2189/


Regards,

Juan José Santamaría Flecha