Re: cmake.port.mk switch to ctest

On Wed Aug 21, 2019 at 06:58:02PM +0200, Rafael Sadowski wrote:
> The following diff switch the cmake test task to ctest(1). This enables us to
> easily debug/report the tests.
>
> In the normal state everything is as always. `make test` looks like:
>
> ===> Regression tests for cmocka-1.1.3
> Test project /usr/ports/pobj/cmocka-1.1.3/build-amd64
> Start 1: test_alloc
> 1/44 Test #1: test_alloc ........................ Passed 0.00 sec
> Start 2: test_group_setup_assert
> 2/44 Test #2: test_group_setup_assert ........... Passed 0.00 sec
>
> ...
>
> 42/44 Test #42: assert_module_test ................ Passed 0.00 sec
> Start 43: waiter_test_wrap
> 43/44 Test #43: waiter_test_wrap .................. Passed 0.00 sec
> Start 44: test_uptime
> 44/44 Test #44: test_uptime ....................... Passed 0.00 sec
>
> 100% tests passed, 0 tests failed out of 44
>
> Total Test time (real) = 0.11 sec
>
>
> ... but with the debug options it looks like:
>
> `env MODCMAKE_DEBUG=yes make test`
>
> 41/44 Test #41: assert_macro_test ................. Passed 0.00 sec
> test 42
> Start 42: assert_module_test
>
> 42: Test command: /usr/ports/pobj/cmocka-1.1.3/build-amd64/example/assert_module_test
> 42: Test timeout computed to be: 1500
> 42: [==========] Running 3 test(s).
> 42: [ RUN ] increment_value_fail
> 42: [ ERROR ] --- ASSERT: value
> 42: [ LINE ] --- /usr/ports/pobj/cmocka-1.1.3/cmocka-1.1.3/example/assert_module.c:30: error: Failure!
> 42: [ FAILED ] increment_value_fail
> 42: [ RUN ] increment_value_assert
> 42: Expected assertion value occurred
> 42: [ OK ] increment_value_assert
> 42: [ RUN ] decrement_value_fail
> 42: Expected assert in decrement_value(NULL)
> 42: [ ERROR ] --- [ LINE ] --- /usr/ports/pobj/cmocka-1.1.3/cmocka-1.1.3/example/assert_module_test.c:46: error: Failure!
> 42: [ FAILED ] decrement_value_fail
> 42: [==========] 3 test(s) run.
> 42: [ PASSED ] 1 test(s).
> 42: [ FAILED ] 2 test(s), listed below:
> 42: [ FAILED ] increment_value_fail
> 42: [ FAILED ] decrement_value_fail
> 42:
> 42: 2 FAILED TEST(S)
> 42/44 Test #42: assert_module_test ................ Passed 0.00 sec
> test 43
> Start 43: waiter_test_wrap
>
> 43: Test command: /usr/ports/pobj/cmocka-1.1.3/build-amd64/example/mock/chef_wrap/waiter_test_wrap
> 43: Test timeout computed to be: 1500
> 43: [==========] Running 2 test(s).
> 43: [ RUN ] test_order_hotdog
> 43: [ OK ] test_order_hotdog
> 43: [ RUN ] test_bad_dish
> 43: [ OK ] test_bad_dish
> 43: [==========] 2 test(s) run.
> 43: [ PASSED ] 2 test(s).
> 43/44 Test #43: waiter_test_wrap .................. Passed 0.00 sec
>
> That means that also stderr from the test can be seen. It's very helpful to me.
> You can also override CTEST_FLAGS and use a lot of ctest options to improve the
> output:
>
> https://cmake.org/cmake/help/latest/manual/ctest.1.html
>
> What do you say?
>
> Best Regards,
>
> RS
>
>

Tweaked version blow, MODCMAKE_DEBUG was in the wrong scope.
Objections?

Index: cmake.port.mk
===================================================================
RCS file: /cvs/ports/devel/cmake/cmake.port.mk,v
retrieving revision 1.65
diff -u -p -u -p -r1.65 cmake.port.mk
--- cmake.port.mk 21 Aug 2019 05:47:40 -0000 1.65
+++ cmake.port.mk 2 Sep 2019 06:22:45 -0000
@@ -20,14 +20,21 @@ NINJA_FLAGS ?= -v -j ${MAKE_JOBS}
CONFIGURE_ARGS += -DCMAKE_MAKE_PROGRAM=${NINJA}
.endif

+MODCMAKE_DEBUG ?= "no"
+
.if ${USE_NINJA:L} == "yes" || ${USE_NINJA:L} == "samurai"
_MODCMAKE_GEN = Ninja
MODCMAKE_BUILD_TARGET = cd ${WRKBUILD} && exec ${SETENV} ${MAKE_ENV} \
${NINJA} ${NINJA_FLAGS} ${ALL_TARGET}
MODCMAKE_INSTALL_TARGET = cd ${WRKBUILD} && exec ${SETENV} ${MAKE_ENV} \
${FAKE_SETUP} ${NINJA} ${NINJA_FLAGS} ${FAKE_TARGET}
+
+.if ${MODCMAKE_DEBUG:L} == "yes"
+CTEST_FLAGS ?= -VV
+.endif
+
MODCMAKE_TEST_TARGET = cd ${WRKBUILD} && exec ${SETENV} ${ALL_TEST_ENV} \
- ${NINJA} ${NINJA_FLAGS} ${TEST_FLAGS} ${TEST_TARGET}
+ ${LOCALBASE}/bin/ctest -j ${MAKE_JOBS} ${CTEST_FLAGS}

.if !target(do-build)
do-build:
@@ -68,11 +75,10 @@ CONFIGURE_ENV += MODJAVA_VER=${MODJAVA_V
MODTCL_LIB=${MODTCL_LIB} \
MODTK_LIB=${MODTK_LIB}

-MODCMAKE_DEBUG ?= No
-
.if empty(CONFIGURE_STYLE)
CONFIGURE_STYLE= cmake
.endif
+
MODCMAKE_configure= cd ${WRKBUILD} && ${SETENV} \
CC="${CC}" CFLAGS="${CFLAGS}" \
CXX="${CXX}" CXXFLAGS="${CXXFLAGS}" \
@@ -92,8 +98,6 @@ MODCMAKE_BUILD_SUFFIX = -release.cmake
SUBST_VARS += MODCMAKE_BUILD_SUFFIX

SEPARATE_BUILD ?= Yes
-
-TEST_TARGET ?= test

MODCMAKE_WANTCOLOR ?= No
MODCMAKE_VERBOSE ?= Yes