Simplifying the tsvector format for simple glossaries

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Simplifying the tsvector format for simple glossaries

Marc Mamin-2
Simplifying the tsvector format for simple glossaries

Hello,

We  have a text search on data from error logs, and our application
offer a rather simple search on lexemes only (no weighting, no
neighbouring ...).
This works quite well, except when the applications generating the logs
get mad and we have to handle millions of messages per day :-)
We also have an ETL (perl) tool, that first transform the logs to CSV
files for COPY

My idea is to let perl create a list of single words for each message,
and run the search only on these "glossaries".
Going further, I'd like to import these lists directly as tsvectors to
save a processing step within Postgres.

The standard tsvector representation in CSV would then look like

'lex_1':1 'lex_2':2 'lex_3':3 ...

when casting from text to tsvector, I've notice with 9.1 that this simpler format is valid too:

'lex_1 lex_2 lex_3 ...'

So my questions:
Is it safe to define tsvectors that way, or should I expect problems
with future release being stricter with the tsvector format?

Do I have to respect the lexemes ordering within a tsvector (using which
NLS Format) ?

Is it an issue if some tsvectors contains stop words, or is it just
annoying noise ?

For the case when this simplification is fine, I'd suggest to add a
description on this possible tsvector representation to the doc.

best regards,

Marc Mamin

Reply | Threaded
Open this post in threaded view
|

Re: Simplifying the tsvector format for simple glossaries

Oleg Bartunov
Always use strip(to_tsvector()) and you'll be happy. Stop words affect
tsvector size, so if you don't search them, don't store.

strip() function described in docs.

Oleg

On Sun, 29 Jan 2012, Marc Mamin wrote:

>
> Hello,
>
> We  have a text search on data from error logs, and our application
> offer a rather simple search on lexemes only (no weighting, no
> neighbouring ...).
> This works quite well, except when the applications generating the logs
> get mad and we have to handle millions of messages per day :-)
> We also have an ETL (perl) tool, that first transform the logs to CSV
> files for COPY
>
> My idea is to let perl create a list of single words for each message,
> and run the search only on these "glossaries".
> Going further, I'd like to import these lists directly as tsvectors to
> save a processing step within Postgres.
>
> The standard tsvector representation in CSV would then look like
>
> 'lex_1':1 'lex_2':2 'lex_3':3 ...
>
> when casting from text to tsvector, I've notice with 9.1 that this simpler format is valid too:
>
> 'lex_1 lex_2 lex_3 ...'
>
> So my questions:
> Is it safe to define tsvectors that way, or should I expect problems
> with future release being stricter with the tsvector format?
>
> Do I have to respect the lexemes ordering within a tsvector (using which
> NLS Format) ?
>
> Is it an issue if some tsvectors contains stop words, or is it just
> annoying noise ?
>
> For the case when this simplification is fine, I'd suggest to add a
> description on this possible tsvector representation to the doc.
>
> best regards,
>
> Marc Mamin
>

  Regards,
  Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: [hidden email], http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

--
Sent via pgsql-general mailing list ([hidden email])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general