apt.postgresql.org django app for www.postgresql.org

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

apt.postgresql.org django app for www.postgresql.org

Christoph Berg-2
Hi,

We have implemented a repository browser for apt.postgresql.org to be
run on www.postgresql.org/repos/apt/. This has been in the works for
years, but has somehow never reached this list.

It is mostly feature-complete, i.e. browsing the list of supported
distributions, source and binary packages works. It supports searching
for packages and filenames in packages.

Dependencies:
python-apt, postgresql-*-debversion, pg_trgm (from postgresql-*-contrib)

To test, create a mirror of a (subset of) apt.postgresql.org:
* debmirror -v -h apt.postgresql.org -r pub/repos/apt --method=http -d sid-pgdg --omit-suite-symlinks -s main,12 -a source,amd64 --exclude='\.deb$' --getcontents --no-check-gpg /srv/repo
* set APT_DIR to that directory in settings_local.py
* run import-packagelists.py

TODO:
* The contained homepage at /repos/apt/ is an outdated mockup, the
  list of supported dists should be rendered from the database
* There are no links to /repos/apt/ from anywhere else in the pg.o
  page yet
* import-packagelists.py uses django to open the PG connection, but
  this should be rewritten to use plain psycopg2.
* The idea is that import-packagelists.py writes directly to the
  database, so appropriate GRANTs are needed to allow writing to the
  apt_* tables only
* The "qa" part contains queries that have not yet been ported from an
  older datamodel yet and is hence disabled
* There is no documentation yet

Still, it already works nicely if you point the browser directly at
/repos/apt, so please review :)

Christoph

0001-Add-APT-repository-webpages-at-repos-apt.patch (72K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: apt.postgresql.org django app for www.postgresql.org

Ray O'Donnell
On 22/01/2019 14:00, Christoph Berg wrote:
> Hi,
>
> We have implemented a repository browser for apt.postgresql.org to be
> run on www.postgresql.org/repos/apt/. This has been in the works for
> years, but has somehow never reached this list.

[snip]

> Still, it already works nicely if you point the browser directly at
> /repos/apt, so please review :)

Maybe I was too quick off the mark, but I'm getting a 404 at the above URL.

Ray.

--
Raymond O'Donnell // Galway // Ireland
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: apt.postgresql.org django app for www.postgresql.org

Christoph Berg-2
Re: Ray O'Donnell 2019-01-22 <[hidden email]>
> Maybe I was too quick off the mark, but I'm getting a 404 at the above URL.

That's why there's a patch attached :)

Christoph

Reply | Threaded
Open this post in threaded view
|

Re: apt.postgresql.org django app for www.postgresql.org

Ray O'Donnell
On 22/01/2019 14:16, Christoph Berg wrote:
> Re: Ray O'Donnell 2019-01-22 <[hidden email]>
>> Maybe I was too quick off the mark, but I'm getting a 404 at the above URL.
>
> That's why there's a patch attached :)
>

Ah, OK, I misunderstood - I thought it was up and running. :-)

Ray.


--
Raymond O'Donnell // Galway // Ireland
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: apt.postgresql.org django app for www.postgresql.org

Christoph Berg-2
In reply to this post by Christoph Berg-2
Updated for python3. I had to fix several places where {{package}} was
rendered as "Package Object". The new syntax is {{package.package}}.

Christoph

0001-Add-APT-repository-webpages-at-repos-apt.patch (72K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: apt.postgresql.org django app for www.postgresql.org

Christoph Berg-2
Re: To PostgreSQL WWW 2019-01-30 <[hidden email]>
> Updated for python3. I had to fix several places where {{package}} was
> rendered as "Package Object". The new syntax is {{package.package}}.

Ping?

Christoph

Reply | Threaded
Open this post in threaded view
|

Re: apt.postgresql.org django app for www.postgresql.org

Jonathan S. Katz-3
On 2/27/19 10:09 AM, Christoph Berg wrote:
> Re: To PostgreSQL WWW 2019-01-30 <[hidden email]>
>> Updated for python3. I had to fix several places where {{package}} was
>> rendered as "Package Object". The new syntax is {{package.package}}.
>
> Ping?


tl;dr I do think having a repo browser on pgweb would be nice, but we
need to solve a few fundamental questions first:

1. Will we make the same thing available for the yums?
2. What will be the manual overhead on the pgweb, pginfra, + release team?

And the rest of it:

This will take some involved review. I made a quick pass, and saw a few
things.

First, an off-list comment that Magnus had raised (amongst a few
others): is this be set up to handle RPM/YUM, and if not, what will it
take to get it there? (Will somewhat answer this below).

There are a few things that are immediate nonstarters for me:

1. There is a bunch of DDL hanging out in SQL files that should be in
the Django migration file (and in fact, I don't see a migration file)

2. There appear to be some hardcoded things specific to deb/apt (e.g.
"all" => "amd64") which would not work if we wanted to extend this to
rpm/yum. The question becomes can we / do we want to build a
one-size-fits-all system?

Also:

1. This appears to introduce an extension from PGXN, "debversion" -- I
don't believe we've added any pgxn extensions directly into pgweb (I may
be wrong) but we would have to see what that means from an
administration standpoint.

2. There seem to be some easy improvements to make in some of the logic:

def search(request):
    if 'package' in request.GET and request.GET['package']:
        package = request.GET['package']

becomes:

def search(request):
    if request.GET.get('package'):
        package = request.GET['package']

3. For now, the "active" booleans scare me until I understand how much
of this is automated and how much of this is manual. Adding more manual
steps to each release terrifies me, mostly because we have more than our
fair share at the moment :)

Thanks,

Jonathan


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: apt.postgresql.org django app for www.postgresql.org

Christoph Berg-2
Re: Jonathan S. Katz 2019-02-27 <[hidden email]>
> tl;dr I do think having a repo browser on pgweb would be nice, but we
> need to solve a few fundamental questions first:
>
> 1. Will we make the same thing available for the yums?

I don't think it is fair to burden the extra work of supporting yum on
the initial implementation of this. I don't think it makes much sense
to extend the system to support both via the same because the package
structures are really different (both because of deb vs. rpm and the
actual packaging structure). Of course it makes sense to think about
if we could share some infrastructure with yum, but I don't think that
would fly.

> 2. What will be the manual overhead on the pgweb, pginfra, + release team?

Basically none. I'd handle all the necessary database updates from the
atalia.postgresql.org host by integrating it into the "add new
package" workflow.

> First, an off-list comment that Magnus had raised (amongst a few
> others): is this be set up to handle RPM/YUM, and if not, what will it
> take to get it there? (Will somewhat answer this below).

(See above)

> There are a few things that are immediate nonstarters for me:
>
> 1. There is a bunch of DDL hanging out in SQL files that should be in
> the Django migration file (and in fact, I don't see a migration file)

That's a leftover from when it was first written 5 years ago. Fixable.

> 2. There appear to be some hardcoded things specific to deb/apt (e.g.
> "all" => "amd64") which would not work if we wanted to extend this to
> rpm/yum. The question becomes can we / do we want to build a
> one-size-fits-all system?

"all" is a special snowflake in the Debian architecture list. The
repository solves that by including "all" in each Packages files for
all architectures. If information about an Arch:all package is
requested, we need to pull it from any of the "real" architectures.

> Also:
>
> 1. This appears to introduce an extension from PGXN, "debversion" -- I
> don't believe we've added any pgxn extensions directly into pgweb (I may
> be wrong) but we would have to see what that means from an
> administration standpoint.

It would allow proper comparison of Debian version numbers, but I
think we aren't really using this anywhere, so it could just be
removed.

> def search(request):
>     if 'package' in request.GET and request.GET['package']:
>         package = request.GET['package']
>
> becomes:
>
> def search(request):
>     if request.GET.get('package'):
>         package = request.GET['package']

Will fix.

> 3. For now, the "active" booleans scare me until I understand how much
> of this is automated and how much of this is manual. Adding more manual
> steps to each release terrifies me, mostly because we have more than our
> fair share at the moment :)

There are the markers for "is this Debian release still active". This
would be up to me to handle from the apt.pg.o side. (We can't really
automate this.)

Thanks,
Christoph