Thursday, January 09, 2025

better SITE_PYPI

Primary storage for pypi distfiles is at a URL embedding a hash,
but this is inconvenient for packages, so we're using an alternative
via a redirector to make this easier.

Currently we're using pypi.io, but there are some problems:

- it doesn't seem to be officially recommended
cf https://docs.pypi.org/api/#integration-guide "Predictable URLs"

- it results in a chain of redirects, across 3 domains and 2 TLDs:
Redirected to https://pypi.org/packages/source/s/setuptools/setuptools-75.6.0.tar.gz
Redirected to https://files.pythonhosted.org/packages/source/s/setuptools/setuptools-75.6.0.tar.gz
Redirected to https://files.pythonhosted.org/packages/43/54/292f26c208734e9a7f067aea4a7e282c080750c4546559b58e2e45413ca0/setuptools-75.6.0.tar.gz

- .io is pretty distasteful, considering the history
https://en.wikipedia.org/wiki/Expulsion_of_the_Chagossians etc

The recommended redirector using URLs of the form
https://files.pythonhosted.org/packages/source/a/abc/abc-123.tar.gz
only has a single redirect, and it's in the same domain as the original
request, so more efficient all round.

Changing the location for port fetches is straightforward, but portroach
will need a tweak to cope too.

Antoine, if I commit the portroach bit first, how long should I wait
before committing network.conf?

OK?

Index: infrastructure/db/network.conf
===================================================================
RCS file: /cvs/ports/infrastructure/db/network.conf,v
diff -u -p -r1.26 network.conf
--- infrastructure/db/network.conf 29 Apr 2024 05:12:48 -0000 1.26
+++ infrastructure/db/network.conf 9 Jan 2025 11:20:57 -0000
@@ -141,7 +141,7 @@ SITE_APACHE+= \
https://archive.apache.org/dist/

SITE_PYPI += \
- https://pypi.io/packages/source/
+ https://files.pythonhosted.org/packages/source/

SITE_RUBYGEMS += \
https://rubygems.org/downloads/
Index: misc/portroach/Makefile
===================================================================
RCS file: /cvs/ports/misc/portroach/Makefile,v
diff -u -p -r1.40 Makefile
--- misc/portroach/Makefile 11 Sep 2023 14:45:35 -0000 1.40
+++ misc/portroach/Makefile 9 Jan 2025 11:20:57 -0000
@@ -3,7 +3,7 @@ COMMENT= OpenBSD ports distfile version
GH_ACCOUNT= jasperla
GH_PROJECT= portroach
GH_TAGNAME= 2.0.11
-REVISION= 7
+REVISION= 8

CATEGORIES= misc

Index: misc/portroach/patches/patch-Portroach_SiteHandler_PyPI_pm
===================================================================
RCS file: misc/portroach/patches/patch-Portroach_SiteHandler_PyPI_pm
diff -N misc/portroach/patches/patch-Portroach_SiteHandler_PyPI_pm
--- /dev/null 1 Jan 1970 00:00:00 -0000
+++ misc/portroach/patches/patch-Portroach_SiteHandler_PyPI_pm 9 Jan 2025 11:20:57 -0000
@@ -0,0 +1,12 @@
+Index: Portroach/SiteHandler/PyPI.pm
+--- Portroach/SiteHandler/PyPI.pm.orig
++++ Portroach/SiteHandler/PyPI.pm
+@@ -73,7 +73,7 @@ sub CanHandle
+
+ my ($url) = @_;
+
+- return ($url =~ /https:\/\/pypi\.io\//);
++ return ($url =~ /https:\/\/(files\.pythonhosted\.org|pypi\.io)\//);
+ }
+
+

No comments:

Post a Comment