Inline::CPP v0.44: Implement flocking

[David Oswald]

Inline::CPP v0.44 has been uploaded to CPAN.

This version implements flocking on the typemap output file within
CPP.pm, and on two temp files used by Makefile.PL. This should
eliminate race conditions that could possibly be caused within
Inline::CPP, particularly when testing under "HARNESS_OPTIONS=jX", or
by smokers that are testing more than one install at the same time.
The general use case is unlikely to encounter problems, but you never
know, so better safe than sorry.

Rob: I notice there are numerous places in the Inline distribution
where we're opening for output but not flocking. I didn't even look
at possible issues where we might be opening for input, reading, then
opening for output and expecting to not tromp on another process that
may have gotten in the middle of the cycle.

I think that Inline probably should be implementing flocking on all
outputs, and anytime an output follows a read, we should use a flocked
semaphore file to maintain the lock across the read/write cycle.

I can confirm that even after fixing Inline::CPP, we still have race
conditions that must be coming from inside of Inline, that rear their
ugly head while executing tests in parallel.

The fixes within Inline shouldn't be too difficult, but there are so
many places that Inline interacts with files, it will be hard to
assure we've gotten all of them.

I suppose I could implement a semaphore flock at the beginning of
compilation phase for Inline::CPP and release it after compilation
phase finishes. That would make Inline::CPP concurrency-safe even if
Inline were to remain not concurrency-safe. But that's just sweeping
the dust under the carpet. :)

--

David Oswald
daoswald-***@public.gmane.org

[s***@public.gmane.org]

-----Original Message-----
From: David Oswald

Rob: I notice there are numerous places in the Inline distribution where
we're opening for output but not flocking. I didn't even look at possible
issues where we might be opening for input, reading, then opening for
output and expecting to not tromp on another process that may have gotten
in the middle of the cycle.

The key thing for me is that this absence of flocking has never (afaik) been
reported as an issue for any *user*.
Not wishing to denigrate cpan-testers (who do a fantastic job), but I've
always been loathe to address something that exists only in a smoking
set-up, for fear it's going to open up a can of worms for those who actually
want to make use of the module.
I still see the occasional cpan-tester FAIL report where 'use diagnostics;'
is a fatal error ... which only reinforces my resolve to be a little bit
cautious about the significance of the failures that the cpan-testers throw
up :-)

I think that Inline probably should be
implementing flocking on all outputs, and anytime an output follows a
read, we should use a flocked semaphore file to maintain the lock across
the read/write cycle.

I can't see that this should be necessary except perhaps for the
configuration file that gets written in the top level build directory.
I don't insist on not implementing stuff that's pointless (as along as it
comes at little cost), but I'm a bit wary of *flocking* across the board
because I can never get the damn thing to work *portably* && *silently*.
(Maybe that's just because I'm inexperienced in the use of flock.)
If we do flock, I'd rather do it only where it's proven to be necessary.

I can confirm that even after fixing Inline::CPP,
we still have race conditions that must be coming from inside of Inline,
that rear their ugly head while executing tests in parallel.

Can you give me an actual "use" case - one that's outside the
cpan-tester/smoker/Test::Harness environment ?
That would probably encourage me to become a more responsible and caring
Inline maintainer :-)

The fixes
within Inline shouldn't be too difficult, but there are so many places
that Inline interacts with files, it will be hard to assure we've gotten
all of them.

My thinking is that we can't be missing too much or we would've received at
least one bug report about it from a user in the last 15 years.

I suppose I could implement a semaphore flock at the
beginning of compilation phase for Inline::CPP and release it after
compilation phase finishes. That would make Inline::CPP concurrency-safe
even if Inline were to remain not concurrency-safe. But that's just
sweeping the dust under the carpet. :)

Well ... that'd be good enough for me. Does that work ?

Although (as is clearly evident) I'm not all that passionate about this at
the moment, I would probably find it hard to ignore any patch that:
a) broke none of the current tests;
&&
b) created no noise.

(But I'd still prefer that we flocked only where it's demonstrably needed.)

And I still can't help feeling that cpan-testers/Test::Harness/smokers might
be assuming something of Inline that it was never designed (or intended) to
deliver ... dunno ... again, an actual "use" case demonstrating the problem
would be helpful here.

As you can see, David, I don't entirely agree with your POV at the moment.
But I *do* appreciate that you took the time and effort to express it. (And
I thank you for doing that !!)

Cheers,
Rob

[David Oswald]

I appreciate your point of view as well. Disagreeing is healthy. If
we all just fell in line with each other's whims and opinions we would
risk heading off on a path to irrelevancy.

I also agree that I'm hard-pressed to come up with a situation where a
normal use of Inline would result in concurrent compilation. Race
conditions only occur when there's a race. Perhaps if a script
included two or more modules that both relied upon Inline, and both
hadn't previously been compiled, but that's far-fetched.

On the one hand, we can reasonably assert that the CPAN testers are
testing incorrectly if they're testing with concurrency, and that our
failure to pass their tests is due to them not taking into
consideration the fact that some modules will never be used that way.
On the other hand, they have a monumental task in trying to test
everything that shows up on CPAN, and expecting special case treatment
for Inline might be more than they're able to accommodate. Yes,
that's broken. But people look at the test results when weighing
whether or not we're an appropriate solution, and unfortunately, they
don't always dig into the "why" when they see a FAIL.

The easiest solution for Inline would be to simply flock a semaphore
file before it compiles any target code, and unlock it after it's done
with the build directory. This would require only a single flock, and
would prevent concurrent use of the build resources. It would have
zero impact on normal users (it would never block since they would
never be compiling in parallel), but would cause the tests to block
for a moment occasionally. That should avoid all race conditions.

Here's what I'll do: I'm going to release a dev release of Inline::CPP
that does just that. If its smoke testing doesn't get any worse than
it already is, I'll submit a patch for Inline that does the same
thing. If you want to use it you can, and if you don't, no hard
feelings. :)

Dave

-----Original Message----- From: David Oswald

Rob: I notice there are numerous places in the Inline distribution where
we're opening for output but not flocking. I didn't even look at possible
issues where we might be opening for input, reading, then opening for
output and expecting to not tromp on another process that may have gotten
in the middle of the cycle.

The key thing for me is that this absence of flocking has never (afaik) been
reported as an issue for any *user*.
Not wishing to denigrate cpan-testers (who do a fantastic job), but I've
always been loathe to address something that exists only in a smoking
set-up, for fear it's going to open up a can of worms for those who actually
want to make use of the module.
I still see the occasional cpan-tester FAIL report where 'use diagnostics;'
is a fatal error ... which only reinforces my resolve to be a little bit
cautious about the significance of the failures that the cpan-testers throw
up :-)

I think that Inline probably should be
implementing flocking on all outputs, and anytime an output follows a
read, we should use a flocked semaphore file to maintain the lock across
the read/write cycle.

I can't see that this should be necessary except perhaps for the
configuration file that gets written in the top level build directory.
I don't insist on not implementing stuff that's pointless (as along as it
comes at little cost), but I'm a bit wary of *flocking* across the board
because I can never get the damn thing to work *portably* && *silently*.
(Maybe that's just because I'm inexperienced in the use of flock.)
If we do flock, I'd rather do it only where it's proven to be necessary.

I can confirm that even after fixing Inline::CPP,
we still have race conditions that must be coming from inside of Inline,
that rear their ugly head while executing tests in parallel.

Can you give me an actual "use" case - one that's outside the
cpan-tester/smoker/Test::Harness environment ?
That would probably encourage me to become a more responsible and caring
Inline maintainer :-)

The fixes
within Inline shouldn't be too difficult, but there are so many places
that Inline interacts with files, it will be hard to assure we've gotten
all of them.

My thinking is that we can't be missing too much or we would've received at
least one bug report about it from a user in the last 15 years.

I suppose I could implement a semaphore flock at the
beginning of compilation phase for Inline::CPP and release it after
compilation phase finishes. That would make Inline::CPP concurrency-safe
even if Inline were to remain not concurrency-safe. But that's just
sweeping the dust under the carpet. :)

Well ... that'd be good enough for me. Does that work ?
Although (as is clearly evident) I'm not all that passionate about this at
a) broke none of the current tests;
&&
b) created no noise.
(But I'd still prefer that we flocked only where it's demonstrably needed.)
And I still can't help feeling that cpan-testers/Test::Harness/smokers might
be assuming something of Inline that it was never designed (or intended) to
deliver ... dunno ... again, an actual "use" case demonstrating the problem
would be helpful here.
As you can see, David, I don't entirely agree with your POV at the moment.
But I *do* appreciate that you took the time and effort to express it. (And
I thank you for doing that !!)
Cheers,
Rob

[s***@public.gmane.org]

-----Original Message-----
From: David Oswald

Post by David Oswald
Here's what I'll do: I'm going to release a dev release of Inline::CPP
that does just that. If its smoke testing doesn't get any worse than it
already is, I'll submit a patch for Inline that does the same thing. If
you want to use it you can, and if you don't, no hard feelings. :)

If you provide a patch, I'll certainly apply it and release a new Inline
devel version - providing, of course, that it doesn't break anything.

Cheers,
Rob