[rt.cpan.org #85336] Fails often when tested in parallel

[Andreas Koenig via RT]

Wed May 15 23:11:13 2013: Request 85336 was acted upon.
Transaction: Ticket created by ANDK
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org
Status: new
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >


Yesterday my smoker produced two fails while 4 processes were testing Inline 0.53:

http://www.cpantesters.org/cpan/report/62c114d6-bda4-11e2-874b-cb6a10d45c6b
http://www.cpantesters.org/cpan/report/64619edc-bda4-11e2-924f-916b10d45c6b

It's possible that these two were disturbing each other, they were both distrubed by some other, succeeding test run. I smoked with 4 parallel processes. It's not for the first time this happens, I have produced more fail reports in the past but not always they come in pairs. But I have not seen more than two in a row.

There seems to be a race condition somewhere.

HTH,

[Sisyphus via RT]

Sat May 18 02:20:38 2013: Request 85336 was acted upon.
Transaction: Correspondence added by SISYPHUS
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org
Status: new
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

Post by Andreas Koenig via RT
There seems to be a race condition somewhere.

Both of the affected test scripts (t/21read_DATA.t and t/22read_DATA.t) use the same Inline build directory (.Inline) - and they therefore need to access the same configuration file.

If you think that could be the source of the problem, I could work around that by designating 2 different (separate) build directories for those 2 test scripts - thereby assuring that each script has its own copy of the configuration details.
Is that worth trying ? (If so, let me know, and I'll upload a new developer version of Inline that does exactly that and we can see how that goes.)

Cheers,
Rob

[(Andreas J. Koenig) via RT]

Sat May 18 10:40:47 2013: Request 85336 was acted upon.
Transaction: Correspondence added by andreas.koenig.7os6VVqR-tc5b/zKpD7+***@public.gmane.orgd.de
Queue: Inline
Subject: Re: [rt.cpan.org #85336] Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

Post by Sisyphus via RT
Is that worth trying ?

This, or maybe apply some locking? Depends on how much work it is, and
on how relevant for later real world behaviour it is.

Except for this, I'd leave the judgement to the implementor;)

[David Oswald via RT]

Sat May 18 11:23:47 2013: Request 85336 was acted upon.
Transaction: Correspondence added by daoswald
Queue: Inline
Subject: Re: [rt.cpan.org #85336] Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >


I get test failures in Inline::CPP when running tests in parallel.
I've gone through Inline::CPP and implemented file locking to
eliminate race conditions, but the issue persists. That leads me to
believe that there needs to be a similar fix in Inline. Changing
Inline's tests to use separate build directories wouldn't fix the
underlying issue of Inline not supporting concurrency. Proper file
locking probably would resolve the issue, not only for Inline, but
also for plugins such as Inline::CPP.


On Sat, May 18, 2013 at 8:40 AM, (Andreas J. Koenig) via RT

Post by (Andreas J. Koenig) via RT
Sat May 18 10:40:47 2013: Request 85336 was acted upon.
Queue: Inline
Subject: Re: [rt.cpan.org #85336] Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

Post by Sisyphus via RT
Is that worth trying ?

This, or maybe apply some locking? Depends on how much work it is, and
on how relevant for later real world behaviour it is.
Except for this, I'd leave the judgement to the implementor;)
--
andreas

[Sisyphus via RT]

Sun May 19 01:37:37 2013: Request 85336 was acted upon.
Transaction: Correspondence added by SISYPHUS
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

Post by (Andreas J. Koenig) via RT

Post by Sisyphus via RT
Is that worth trying ?

This, or maybe apply some locking? Depends on how much work it is, and
on how relevant for later real world behaviour it is.
Except for this, I'd leave the judgement to the implementor;)

I don't think it's very relevant to real world usage - and hence my (personal) interest in this is nowhere near as keen as it would otherwise be.

However, David Oswald wants to write some patches that will put in place file locking. If there's no problem with those patches, I'll apply them (when I receive them) and release a new devel version of Inline.

Hopefully, that will keep everyone happy :-)

Cheers,
Rob

[Andreas Koenig]

Post by Sisyphus via RT
Is that worth trying ?

This, or maybe apply some locking? Depends on how much work it is, and
on how relevant for later real world behaviour it is.

Except for this, I'd leave the judgement to the implementor;)

[Kent Fredric via RT]

Thu Feb 20 03:51:10 2014: Request 85336 was acted upon.
Transaction: Correspondence added by KENTNL
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org, KENTNL-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >


You'll note quite a few fail reports on cpan testers from me.

I had in fact just assumed Inline was entirely broken somehow due to how long
I've been seeing these errors, and due to the way they failed suggesting
strange things ( For instance, telling me I have a MAD perl when I do not is
very very weird )

Indeed, double checking and running the tests without parallelism makes all the
problems mysteriously vanish!

[Reini Urban via RT]

Fri Apr 04 11:51:31 2014: Request 85336 was acted upon.
Transaction: Correspondence added by rurban-***@public.gmane.org
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org, KENTNL-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

Post by Kent Fredric via RT
You'll note quite a few fail reports on cpan testers from me.

Confirmed and repro even standalone.

The pure Inline/C tests fail on SMP systems with make (tested with gmake 4.0)
in t/25proto.t and t/08taint_1.p
On 5.6 and 5.14 perls (linux)

$@: make[1]: Entering directory `/home/rurban/Perl/Inline/C/_Inline_test/build/PROTO4_7cc8'
make[1]: *** write jobserver: Bad file descriptor. Stop.
make[1]: *** Waiting for unfinished jobs....
make[1]: *** write jobserver: Bad file descriptor. Stop.

I'll try to fix it at https://github.com/rurban/Inline

[Reini Urban via RT]

Fri Apr 04 13:56:12 2014: Request 85336 was acted upon.
Transaction: Correspondence added by rurban-***@public.gmane.org
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org, KENTNL-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

Post by Reini Urban via RT

Post by Kent Fredric via RT
You'll note quite a few fail reports on cpan testers from me.

Confirmed and repro even standalone.
The pure Inline/C tests fail on SMP systems with make (tested with
gmake 4.0)
in t/25proto.t and t/08taint_1.p
On 5.6 and 5.14 perls (linux)
`/home/rurban/Perl/Inline/C/_Inline_test/build/PROTO4_7cc8'
make[1]: *** write jobserver: Bad file descriptor. Stop.
make[1]: *** Waiting for unfinished jobs....
make[1]: *** write jobserver: Bad file descriptor. Stop.
I'll try to fix it at https://github.com/rurban/Inline

The problem is obvious.
make inherits these ENV values from the master make test:
MAKEFLAGS = w --jobserver-fds=3,4 -j -- PREFIX=/usr/local LINKTYPE=dynamic LIBPERL_A=libperl.a
MAKELEVEL = 2
MAKEOVERRIDES = ${-*-command-variables-*-}

The jobserver-fds are wrong, need to be stripped.

All tests with make -j4 pass now. See https://github.com/rurban/Inline

[Reini Urban via RT]

Fri Apr 04 14:02:09 2014: Request 85336 was acted upon.
Transaction: Correspondence added by rurban-***@public.gmane.org
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org, KENTNL-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >


Patches for the next release attached

[Sisyphus via RT]

Sun Apr 06 03:55:40 2014: Request 85336 was acted upon.
Transaction: Correspondence added by SISYPHUS
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org, KENTNL-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

Post by Reini Urban via RT
Patches for the next release attached

Thanks Reini.
These patches have been applied (by hand) .... hope I got 'em right. Inline-0.54_01 (which contains these patches) has now been uploaded to CPAN.
All being well, it will be re-released as Inline-0.55 in a week or so.

For the Changes and C/Changes files I generally structure it so that Changes lists alterations to Inline and C/Changes alterations to Inline::C. (All this really achieves is to make it difficult for myself - especially in those cases where both packages are affected by the one set of changes.)

One will find that Changes and C/Changes have not been altered in strict accordance with the provided patches.

The only other alteration made to the provided patches was to rewrite the 2 occurrences of:

local $ENV{MAKEFLAGS} = $ENV{MAKEFLAGS} =~ s/(--jobserver-fds=[\d,]+)//;

as:

if($ENV{MAKEFLAGS}) {
local $ENV{MAKEFLAGS} = $ENV{MAKEFLAGS} =~ s/(--jobserver-fds=[\d,]+)//;
}

This was done to avoid 'uninitialized' warnings on systems where $ENV{MAKEFLAGS} was unset.

Cheers,
Rob

[Shawn Laffan via RT]

Sat Apr 12 18:46:34 2014: Request 85336 was acted upon.
Transaction: Correspondence added by SLAFFAN
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org, KENTNL-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >


The local assignment to $ENV{MAKEFLAGS} won't apply outside the if block, will it?

This should, though:


local $ENV{MAKEFLAGS} = $ENV{MAKEFLAGS} =~ s/(--jobserver-fds=[\d,]+)//
if $ENV{MAKEFLAGS};


Or it could be localised before that condition if postfix conditions are not preferred (albeit postfix conditions are used elsewhere in the package):


local $ENV{MAKEFLAGS} = $ENV{MAKEFLAGS};
if($ENV{MAKEFLAGS}) {
$ENV{MAKEFLAGS} = $ENV{MAKEFLAGS} =~ s/(--jobserver-fds=[\d,]+)//;
}


Regards,
Shawn.

Post by Sisyphus via RT

Post by Reini Urban via RT
Patches for the next release attached

Thanks Reini.
These patches have been applied (by hand) .... hope I got 'em right.
Inline-0.54_01 (which contains these patches) has now been uploaded to
CPAN.
All being well, it will be re-released as Inline-0.55 in a week or so.
For the Changes and C/Changes files I generally structure it so that
Changes lists alterations to Inline and C/Changes alterations to
Inline::C. (All this really achieves is to make it difficult for
myself - especially in those cases where both packages are affected by
the one set of changes.)
One will find that Changes and C/Changes have not been altered in
strict accordance with the provided patches.
The only other alteration made to the provided patches was to rewrite
local $ENV{MAKEFLAGS} = $ENV{MAKEFLAGS} =~ s/(--jobserver-
fds=[\d,]+)//;
if($ENV{MAKEFLAGS}) {
local $ENV{MAKEFLAGS} = $ENV{MAKEFLAGS} =~ s/(--jobserver-
fds=[\d,]+)//;
}
This was done to avoid 'uninitialized' warnings on systems where
$ENV{MAKEFLAGS} was unset.
Cheers,
Rob

[s***@public.gmane.org]

-----Original Message-----
From: Shawn Laffan via RT

Post by Shawn Laffan via RT
The local assignment to $ENV{MAKEFLAGS} won't apply outside the if block, will it?

Duh !!

Post by Shawn Laffan via RT
local $ENV{MAKEFLAGS} = $ENV{MAKEFLAGS} =~ s/(--jobserver-fds=[\d,]+)//
if $ENV{MAKEFLAGS};

Thanks - fixed in git.

Cheers,
Rob

[sisyphus1-sFbbPxZDHXw0n/[e-mail] via RT]

Sun Apr 13 07:17:59 2014: Request 85336 was acted upon.
Transaction: Correspondence added by sisyphus1-sFbbPxZDHXw0n/***@public.gmane.org
Queue: Inline
Subject: Re: [rt.cpan.org #85336] Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org, KENTNL-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >


-----Original Message-----
From: Shawn Laffan via RT

Post by Shawn Laffan via RT
The local assignment to $ENV{MAKEFLAGS} won't apply outside the if block, will it?

Duh !!

Post by Shawn Laffan via RT
local $ENV{MAKEFLAGS} = $ENV{MAKEFLAGS} =~ s/(--jobserver-fds=[\d,]+)//
if $ENV{MAKEFLAGS};

Thanks - fixed in git.

Cheers,
Rob

[Ed J via RT]

<URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >

I'd be very interested to know whether the change proposed in https://github.com/mohawk2/inline-pm/commit/9fef7cfbd731249579deb3510d96a318115a0928 fixes this issue.

[Ed J via RT]

Thu Jun 12 04:37:14 2014: Request 85336 was acted upon.
Transaction: Correspondence added by ETJ
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org, KENTNL-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >


I'd be very interested to know whether the change proposed in https://github.com/mohawk2/inline-pm/commit/9fef7cfbd731249579deb3510d96a318115a0928 fixes this issue.

[Sisyphus via RT]

Tue Jun 24 05:06:27 2014: Request 85336 was acted upon.
Transaction: Correspondence added by SISYPHUS
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org, KENTNL-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >



Fixed as of 0.55_01

[Sisyphus via RT]

Tue Jul 01 04:27:36 2014: Request 85336 was acted upon.
Transaction: Correspondence added by SISYPHUS
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org, KENTNL-***@public.gmane.org
Status: resolved
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >


Re-opening because of the following patch that was applied to Inline-0.55 as part of this ticket:

####################################
Subject: [PATCH 4/5] MSWin32: disable BUILD_NOISY redirects on MSWin32 with
cmd.exe

also print exitcode with failed commands

diff --git C/C.pm C/C.pm
index f76e34b..21f7dfe 100644
--- C/C.pm
+++ C/C.pm
@@ -804,6 +804,7 @@ sub makefile_pl {
-f ($perl = $Config::Config{perlpath})
or ($perl = $^X)
or croak "Can't locate your perl binary";
+ $perl = qq{"$perl"} if $perl =~ m/\s/;
$o->system_call("$perl Makefile.PL", 'out.Makefile_PL');
$o->fix_make;
}
@@ -841,6 +842,7 @@ sub system_call {
defined $ENV{PERL_INLINE_BUILD_NOISY}
? $ENV{PERL_INLINE_BUILD_NOISY}
: $o->{CONFIG}{BUILD_NOISY};
+ $build_noisy = undef if $build_noisy and $^O eq 'MSWin32' and $Config::Config{sh} =~ /^cmd/;
if (not $build_noisy) {
$cmd = "$cmd > $output_file 2>&1";
}
@@ -861,11 +863,12 @@ sub build_error_message {
close OUTPUT;
}

+ my $errcode = $? >> 8;
return $output . <<END;

A problem was encountered while attempting to compile and install your Inline
$o->{API}{language} code. The command that failed was:
- $cmd
+ \"$cmd\" with error code $errcode

The build directory was:
$build_dir
####################################

I wondered at the time (and still wonder) what that was about - but I applied it anyway, as it didn't break any tests.

However, it does break BUILD_NOISY on Win32 - to the extent that the compiler/linker commands/warnings of a successful build are not seen.

simply removing the line:

$build_noisy = undef if $build_noisy and $^O eq 'MSWin32' and $Config::Config{sh} =~ /^cmd/;

from the patched (0.55) C.pm is sufficient to regain correct functioning of BUILD_NOISY on Windows.
However, doing that probably also destroys whatever it was that the patch was designed to fix.

This episode exposes a need for a test script that examines the output of a BUILD_NOISY build to detect that this output is present.
It would be hard to check that the entire output is as it should be, but we should at least be able to check for the presence of certain key elements like - eg that the output matches the string "perl", that it matches the (interpolated)"$Config{LD}" and that it matches the name of any Inline-C function whose compilation is expected to emit a warning.

I think this breakage of BUILD_NOISY needs to be fixed for the next stable release.

Attached is a try.pl that demonstrates the problem. It shows the expected output of the script, and the actual output I get on one of my Win32 perls using Inline-0.55.

Cheers,
Rob

[Ed J via RT]

Wed Jul 02 00:55:23 2014: Request 85336 was acted upon.
Transaction: Correspondence added by ETJ
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org, KENTNL-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >


FYI, the code to show the correct operation of BUILD_NOISY can be one-linered like so:

perl -MInline=C,Config,BUILD_NOISY,1,FORCE_BUILD,1 -e "use Inline C => q[void inline_warner() { int *x = 2; }]"

My reading of the patch in question is that it turns off BUILD_NOISY when it's Windows and the shell is cmd. If BUILD_NOISY does the right thing with Win32 and CMD, let's undo that change?

[Sisyphus]

----- Original Message -----

Post by Ed J via RT
perl -MInline=C,Config,BUILD_NOISY,1,FORCE_BUILD,1 -e "use Inline C =>
q[void inline_warner() { int *x = 2; }]"

Yes, for a test I'm thinking just have the test script run something like
that as a system command with output redirected to a file - and then check
that file (to an extent that allows us to be confident that BUILD_NOISY is
behaving as expected).

Post by Ed J via RT
My reading of the patch in question is that it turns off BUILD_NOISY when
it's Windows and the shell is cmd.

That's about the extent of it. But I'm damned if I can think of any reason
that ought to be done.

Post by Ed J via RT
If BUILD_NOISY does the right thing with Win32 and CMD, let's undo that
change?

BUILD_NOISY has always done the right thing for me on Win32 in the cmd.exe
shell - that is, until 0.55 ;-)
So yes - we definitely need to revert to pre-0.55 behaviour.

(This Windows laptop I'm using while I'm not at home doesn't have a git
client, and I can't be bothered installing one on it. I'll be back home
tomorrow night and will attend to this BUILD_NOISY issue then, if no-one
else has.)

Cheers,
Rob

[sisyphus1-sFbbPxZDHXw0n/[e-mail] via RT]

Wed Jul 02 05:52:27 2014: Request 85336 was acted upon.
Transaction: Correspondence added by sisyphus1-sFbbPxZDHXw0n/***@public.gmane.org
Queue: Inline
Subject: Re: [rt.cpan.org #85336] Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org, KENTNL-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >


----- Original Message -----

Post by Ed J via RT
perl -MInline=C,Config,BUILD_NOISY,1,FORCE_BUILD,1 -e "use Inline C =>
q[void inline_warner() { int *x = 2; }]"

Yes, for a test I'm thinking just have the test script run something like
that as a system command with output redirected to a file - and then check
that file (to an extent that allows us to be confident that BUILD_NOISY is
behaving as expected).

Post by Ed J via RT
My reading of the patch in question is that it turns off BUILD_NOISY when
it's Windows and the shell is cmd.

That's about the extent of it. But I'm damned if I can think of any reason
that ought to be done.

Post by Ed J via RT
If BUILD_NOISY does the right thing with Win32 and CMD, let's undo that
change?

BUILD_NOISY has always done the right thing for me on Win32 in the cmd.exe
shell - that is, until 0.55 ;-)
So yes - we definitely need to revert to pre-0.55 behaviour.

(This Windows laptop I'm using while I'm not at home doesn't have a git
client, and I can't be bothered installing one on it. I'll be back home
tomorrow night and will attend to this BUILD_NOISY issue then, if no-one
else has.)

Cheers,
Rob

[Ed J via RT]

Wed Jul 02 18:01:36 2014: Request 85336 was acted upon.
Transaction: Correspondence added by ETJ
Queue: Inline
Subject: Fails often when tested in parallel
Broken in: 0.53
Severity: (no value)
Owner: Nobody
Requestors: ANDK-***@public.gmane.org, KENTNL-***@public.gmane.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=85336 >


commit 0bcdb0f7dfa065ff5bf68f2f3033ec7c549e38c3
Author: ...
Date: Wed Jul 2 22:43:44 2014 +0100

Undo change disabling BUILD_NOISY for Win32 when shell eq "cmd".

In new 0.55_03:

diff --git a/C/C.pm b/C/C.pm
index 0b8073e..cc2f4a0 100644
--- a/C/C.pm
+++ b/C/C.pm
@@ -852,7 +852,8 @@ sub system_call {
defined $ENV{PERL_INLINE_BUILD_NOISY}
? $ENV{PERL_INLINE_BUILD_NOISY}
: $o->{CONFIG}{BUILD_NOISY};
- $build_noisy = undef if $build_noisy and $^O eq 'MSWin32' and $Config::Conf
+ # test this functionality with:
+ #perl -MInline=C,Config,BUILD_NOISY,1,FORCE_BUILD,1 -e "use Inline C => q[v
if (not $build_noisy) {
$cmd = "$cmd > $output_file 2>&1";
}