Over this last weekend I went to PyCascades and after Python conferences
they have "Sprints", which are like hackathons, but lots of different
Python projects get together to do stuff as well as use it as a time to
attract new contributors. The last few conferences I attended, I took
that time write some perl while to learning more about how the Python
Package Index (PyPI) works and finally this time made some progress on
adding dependency support for PyPI to portgen(1). I think with this
support it works well enough to merge that recent patch mentioning the
support on in the manpage.
Of note, there is some weird new thing they are doing with "Wheels" and
my test runs wanted to download these things into a .eggs directory for
running tests even though the packages were installed. I don't know
what needs to happen to fix that, but it seems unhelpful.
https://pythonwheels.com/
I would appreciate tests and reviews of this, especially from folks who
know about Python packaging or who want to tell me to simplify my perl.
There are three semi-unrelated patches here, as the python port I used
for testing (pwned) is Python 3 only and its license is "GPL-3.0"
1. Add gpl_3_0 as a recognized variant for "GPLv3"
2. Better detect what MODPY_VERSIONs the module supports
3. Find dependencies
OK for the license detection?
Index: infrastructure/lib/OpenBSD/PortGen/License.pm
===================================================================
RCS file: /cvs/ports/infrastructure/lib/OpenBSD/PortGen/License.pm,v
retrieving revision 1.2
diff -u -p -u -r1.2 License.pm
--- infrastructure/lib/OpenBSD/PortGen/License.pm 27 Apr 2016 09:58:35 -0000 1.2
+++ infrastructure/lib/OpenBSD/PortGen/License.pm 27 Feb 2019 19:32:29 -0000
@@ -38,6 +38,7 @@ my %good_licenses = (
gpl_2 => 'GPLv2',
gpl_2_0 => 'GPLv2',
gpl_3 => 'GPLv3',
+ gpl_3_0 => 'GPLv3',
lgpl => 'LGPL',
lgpl_2_1 => 'LGPL v2.1',
'lgpl_2_1+' => 'LGPL v2.1',
OK for the MODPY_VERSION detection? Do I need to do something else so
that these Python 3 only ports work with ${MODPY_FLAVOR}?
Index: infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm
===================================================================
RCS file: /cvs/ports/infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm,v
retrieving revision 1.1.1.1
diff -u -p -u -r1.1.1.1 PyPI.pm
--- infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm 18 Jan 2016 18:08:20 -0000 1.1.1.1
+++ infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm 27 Feb 2019 19:32:29 -0000
@@ -73,12 +74,22 @@ sub fill_in_makefile
$self->set_license( $di->{info}{license} );
$self->set_descr( $di->{info}{summary} );
- for ( @{ $di->{info}{classifiers} } ) {
- if (/^Programming Language :: Python :: 3/) {
- $self->set_other( 'FLAVORS', 'python3' );
- $self->set_other( 'FLAVOR', '' );
- last;
- }
+ my @versions = do {
+ my %seen;
+ sort { $a <=> $b } grep { !$seen{$_}++ } map {
+ /^Programming Language :: Python :: (\d+)/ ? $1 : ()
+ } @{ $di->{info}{classifiers} }
+ };
+
+ if ( @versions > 1 ) {
+ shift @versions; # remove default, lowest
+ $self->set_other( 'FLAVORS', "python$_" ) for @versions;
+ $self->set_other( 'FLAVOR', '' );
+ }
+ elsif ( @versions == 1 && $versions[0] != 2 ) {
+ $self->set_other(
+ MODPY_VERSION => "\${MODPY_DEFAULT_VERSION_$_}" )
+ for @versions;
}
}
I actually first wrote a version of this that extracts the egg_info and
looks through the "requirements" that generates. Then I accidentally
dumped out the JSON that PyPI returns and it turns out it now figures
out those dependencies somehow and lists them for me! So, after parsing
their very odd dependency strings, that is probably subtly wrong, I now
have something that seems to work.
One thing I wasn't sure of was whether the "*_DEPENDS" need a
${MODPY_FLAVOR} attached?
I also let it add the "extra" dependencies I didn't recognize as
"RUN_DEPENDS" although they are probably optional. Would it be
preferred to instead skip those items?
OK for the get_deps change?
(I assume not yet as I am fairly sure there are some needed tweaks)
Index: infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm
===================================================================
RCS file: /cvs/ports/infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm,v
retrieving revision 1.1.1.1
diff -u -p -u -r1.1.1.1 PyPI.pm
--- infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm 18 Jan 2016 18:08:20 -0000 1.1.1.1
+++ infrastructure/lib/OpenBSD/PortGen/Port/PyPI.pm 27 Feb 2019 19:32:29 -0000
@@ -22,6 +22,7 @@ use warnings;
use parent 'OpenBSD::PortGen::Port';
use OpenBSD::PortGen::Dependency;
+use OpenBSD::PortGen::Utils qw( module_in_ports );
sub ecosystem_prefix
{
@@ -51,7 +52,7 @@ sub name_new_port
{
my ( $self, $di ) = @_;
- my $name = $di->{info}{name};
+ my $name = ref $di ? $di->{info}{name} : $di;
$name = "py-$name" unless $name =~ /^py-/;
return "pypi/$name";
@@ -73,12 +74,22 @@ sub fill_in_makefile
$self->set_license( $di->{info}{license} );
$self->set_descr( $di->{info}{summary} );
- for ( @{ $di->{info}{classifiers} } ) {
- if (/^Programming Language :: Python :: 3/) {
- $self->set_other( 'FLAVORS', 'python3' );
- $self->set_other( 'FLAVOR', '' );
- last;
- }
+ my @versions = do {
+ my %seen;
+ sort { $a <=> $b } grep { !$seen{$_}++ } map {
+ /^Programming Language :: Python :: (\d+)/ ? $1 : ()
+ } @{ $di->{info}{classifiers} }
+ };
+
+ if ( @versions > 1 ) {
+ shift @versions; # remove default, lowest
+ $self->set_other( 'FLAVORS', "python$_" ) for @versions;
+ $self->set_other( 'FLAVOR', '' );
+ }
+ elsif ( @versions && $versions[0] != 2 ) {
+ $self->set_other(
+ MODPY_VERSION => "\${MODPY_DEFAULT_VERSION_$_}" )
+ for @versions;
}
}
@@ -94,6 +105,71 @@ sub postextract
sub get_deps
{
+ my ( $self, $di, $wrksrc ) = @_;
+ my $deps = OpenBSD::PortGen::Dependency->new();
+
+ # This, especially the os detection, may need additional
+ # work to be reliable, but I don't really know what PyPI returns.
+ # https://www.python.org/dev/peps/pep-0508/
+ foreach ( @{ $di->{info}->{requires_dist} || [] } ) {
+ my $phase = 'run';
+
+ #https://packaging.python.org/specifications/core-metadata/#name
+ my $name;
+ if ( s/^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])\b//i ) {
+ $name = $1;
+ }
+ next unless $name;
+
+ # remove comma separated "extra" inside []
+ if ( s/^ \s* \[ ( [^]]* ) \] \s*//x ) {
+ $phase = $1;
+ }
+
+ my ( $req, $meta ) = split /\s*;\s*/;
+ $req =~ s/^\s+//;
+ $req ||= ">=0";
+ $req =~ s/^.*( [(] ) (.*?) (?(1) [)] ).*/$2/x;
+
+ $meta ||= '';
+ while ( $meta =~ /\bextra \s* == \s* (['"])? (.+?) \1 /gx ) {
+ $phase = $2;
+ }
+
+ my @plat;
+ while ( $meta =~ /
+ (?:sys_platform|os_name) \s* == \s* (['"]) (.+?) \1
+ /gx ) {
+ push @plat, $2;
+ }
+
+ next if @plat and join( " ", @plat ) !~ /OpenBSD/i;
+
+ my $port = module_in_ports( $name, 'py-' )
+ || $self->name_new_port($name);
+
+ if ( $phase eq 'build' ) {
+ $deps->add_build( $port, $req );
+ } elsif ( $phase eq 'test' ) {
+ $deps->add_test( $port, $req );
+ } elsif ( $phase eq 'dev' ) {
+ # switch this to "ne 'run'" to avoid optional deps
+ warn "Not adding '$phase' dep on $port\n";
+ next;
+ } else {
+ warn "Adding '$phase' dep on $port as run dep\n"
+ unless $phase eq 'run';
+ $deps->add_run( $port, $req );
+ }
+
+ # don't have it in tree, port it
+ if ( $port =~ m{^pypi/} ) {
+ my $o = OpenBSD::PortGen::Port::PyPI->new();
+ $o->port($name);
+ }
+ }
+
+ return $deps->format;
}
sub get_config_style
No comments:
Post a Comment