On 01/30/2018 09:50 AM, James E Keenan wrote:
> On 01/28/2018 04:36 PM, Andrew Hewus Fresh wrote:
> [snip]
> >
> > I think this is OK, although I've been led to believe that we usually
> > prefer to enable as many tests as possible so that we don't get
> > different results depending on what folks might have installed or what
> > environment they have set, which would mean making these additions.
> >
> > TEST_ENV = AUTOMATED_TESTING=1
> > TEST_DEPENDS = devel/p5-Test-MockObject \
> > devel/p5-Test-Pod>=1.00 \
> > devel/p5-Test-Pod-Coverage>=0.08 \
> > devel/p5-Test-CPAN-Meta
> >
> > # Additional tests could be enabled by porting
> > # devel/p5-Test-CPAN-Meta-JSON
> >
>
> Andrew,
>
> I agree with you that, in general, we don't want to get different test
> results depending on what particular users have installed on their
> systems. That would lead us to activate tests which the CPAN author
> indicates can be run if certain other Perl extensions are present.
>
> However, I think we should look more closely at the environmental
> variable which you are suggesting to include in the port's Makefile:
>
> #####
> TEST_ENV = AUTOMATED_TESTING=1
> #####
>
> In the context of tests in CPAN distributions 'AUTOMATED_TESTING' has a
> specific meaning. At the 2013 Perl QA Hackathon held in Lancaster,
> U.K., a consensus emerged to define the meaning of certain terms that
> would be used by CPAN authors to activate or not activate tests under
> different circumstances
> (https://github.com/Perl-Toolchain-Gang/toolchain-site/blob/master/lancaster-consensus.md#environment-variables-for-testing-contexts).
> Here's what was recommended for 'AUTOMATED_TESTING':
>
> #####
> AUTOMATED_TESTING: if true, tests are being run by an automated testing
> facility and not as part of the installation of a module; CPAN smokers
> must set this to true; CPAN clients must not set this
> #####
>
> In this context, "automated testing facility" refers to a server where
> someone is testing large parts of the 36,000+ distributions on CPAN
> against various versions of Perl on a specific platform and is sending
> test reports to cpantesters.org. For example, Alceu Rodrigues de
> Freitas Junior runs CPAN smokers for testing distributions against Perl
> 5.26.0, 5.26.1 and 5.27.6 (blead) on OpenBSD 6.0, 6.1 and 6.2. In those
> set-ups, he has indeed turned on an AUTOMATED_TESTING environmental
> variable; see, e.g.,
> http://www.cpantesters.org/cpan/report/7f3fb1ae-e729-11e7-8087-d3e9cf4e8a6e.
>
>
> However, the Lancaster consensus guidance above indicates that in all
> circumstances other than preparing reports for cpantesters.org,
> AUTOMATED_TESTING should not be turned on. An OpenBSD port of a CPAN
> distribution is, essentially, a set of instructions for fetching a
> tarball from CPAN and installing it on OpenBSD. It is therefore a "CPAN
> client" as defined by the Lancaster consensus and, hence, should *not*
> turn on the AUTOMATED_TESTING environmental variable.
>
> The author of Devel-Platform-Info, Barbie, has participated in many Perl
> QA hackathons and was the administrator of cpantesters.org for many
> years. Hence, we should presume that he is using 'AUTOMATED_TESTING' in
> the way described above. Here's how Barbie structures his tests:
>
> #####
> # From:t/90podtest.t
>
> plan skip_all => "Author tests not required for installation" unless
> ($ENV{AUTOMATED_TESTING});
> eval "use Test::Pod 1.00";
> plan skip_all => "Test::Pod 1.00 required for testing POD" if $@;
> #####
>
> He characterizes the tests in this file as "[a]uthor tests not required
> for installation" which are to be run only if the AUTOMATED_TESTING
> envvar is set. I interpret this to mean that they should only be run in
> the context of CPAN smokers. He then goes on to say that the tests in
> this file should only be run if Test::Pod version 1.00 or greater is
> already present on the user's machine.
>
> If the first 'plan skip_all' statement were not present in this file,
> then the author would be telling the user, "Go ahead and run this file
> if you've already got Test::Pod installed, but don't bother if you don't
> have it installed." If that were the case, then I think it would be
> appropriate for an OpenBSD port's Makefile to include the port
> corresponding to Test::Pod as a value for TEST_DEPENDS. But that's not
> the case here. The author is indicating that the Test::Pod tests should
> only be run by CPAN smokers, not by CPAN clients. If a CPAN client
> turns on AUTOMATED_TESTING, it is, in effect, second-guessing the author.
>
> Now I concede that:
>
> * The CPAN author may be incorrect; hence he or she should be
> second-guessed.
>
> * OpenBSD is free to set policies for ports that adhere to standards
> different from those set by the authors of the code being ported.
>
> * I'm just a beginner with respect to OpenBSD ports, so I may not know
> the historical context.
>
> So, I'm perfectly willing to revise the port's Makefile per your
> suggestion. But it would be good to have an explicit and well-reasoned
> policy to which I and other porters can conform going forward.
>
> Thank you very much.
> Jim Keenan
>
Attached please find a tarball revised per previous suggestions, i.e.,
not taking into account the questions I raised.
Thank you very much.
Jim Keenan
No comments:
Post a Comment