Discussion:
An Inline question, and a user's experience: Fwd: Using Inline::CPP and Math::Primes::FastSieve
David Oswald
2012-05-17 20:01:44 UTC
Permalink
Today I received the following email message relating to Inline::CPP,
Math::Prime::FastSieve, and Inline. I thought I would forward part of
it to the list to see if we can come up with a solution for some of
Dana's experiences:


---------- Forwarded message ----------
From: Dana Jacobsen <dana.jacobsen-***@public.gmane.org>
Date: Thu, May 17, 2012 at 12:24 PM
Subject: Using Inline::CPP and Math::Primes::FastSieve
To: davido-***@public.gmane.org


David,

 I started looking into prime generators after I added the Goldbach
codes to my variable length coding module.  I thought I'd send you a
message since you've been working on Math::Primes::FastSieve (MPFS)
this month.

 One of my machines is running Ubuntu 11 32-bit.  When I try to
install Inline::CPP I get:

t/01fn_cans.t ....... ok
Had problems bootstrapping Inline module '_02basic_t_bc90'

Can't load '/home/dana/.cpan/build/Inline-CPP-0.39-ftYEEF/_Inline/lib/auto/_02basic_t_bc90/_02basic_t_bc90.so'
for module _02basic_t_bc90:
/home/dana/.cpan/build/Inline-CPP-0.39-ftYEEF/_Inline/lib/auto/_02basic_t_bc90/_02basic_t_bc90.so:
undefined symbol: _ZNSt8ios_base4InitD1Ev at
/home/dana/localperl/lib/5.14.2/i686-linux-64int/DynaLoader.pm line
190.
 at /home/dana/localperl/lib/site_perl/5.14.2/Inline.pm line 536


 at t/02basic.t line 34
BEGIN failed--compilation aborted at t/02basic.t line 92.


I've tried giving it direction to other stdc libraries including the
static linked version, but so far with no success.  Since I can't get
Inline::CPP working, I can't use MPFS on that machine.


Another machine I'm using runs Fedora 16, and installs Inline::CPP and
MPFS just fine.

<snip> --------------------------------------------

Ok, so this is a familiar install issue for Inline::CPP, but I still
haven't gotten my hands on a failure case that I can attempt to debug.
I've asked Dana if there are any suggestions on how I might configure
an Ubuntu install (within a VM) so that I can replicate the failure
and try to debug it.

Dana then went on to show some benchmarks showing M::P::FS to be the
fastest primes generator among the samples chosen (including an XS
alternative on CPAN).

Then Dana's message presents another problem, here:

----------------------------------------------------<message follows>

However .... when I do:

 if (eval { require Math::Prime::FastSieve; 1;}) {
    # ... use MPFS
 } elsif (eval { require Math::Prime::XS;
Math::Prime::XS->import(qw(primes is_prime)); 1; }) {
    # ... use MPXS
 } elsif (....) {
    # ... use DBXS
 } else {
    # pure perl
 }

I get the Inline M51 warning (DATA section not used) if anyone does a
require on my module.  I realize this is a harmless warning, but I
really don't want it happening to my module when I don't even use
Inline.  So for now I've disabled use of MPFS to avoid it.

<snip>------------------------------------------------

That message is coming from Inline, but I can't figure out how one
might squelch it. Are there any suggestions that might help here?


Dave
--
David Oswald
daoswald-***@public.gmane.org


(This message is being sent to inline-***@public.gmane.org, and CC'ed back to Dana
Jacobsen <dana-***@public.gmane.org>.)
Sisyphus
2012-05-18 08:48:39 UTC
Permalink
----- Original Message -----
From: "David Oswald"
Post by David Oswald
----------------------------------------------------<message follows>
if (eval { require Math::Prime::FastSieve; 1;}) {
# ... use MPFS
} elsif (eval { require Math::Prime::XS;
Math::Prime::XS->import(qw(primes is_prime)); 1; }) {
# ... use MPXS
} elsif (....) {
# ... use DBXS
} else {
# pure perl
}
I get the Inline M51 warning (DATA section not used) if anyone does a
require on my module. I realize this is a harmless warning, but I
really don't want it happening to my module when I don't even use
Inline. So for now I've disabled use of MPFS to avoid it.
<snip>------------------------------------------------
That message is coming from Inline, but I can't figure out how one
might squelch it. Are there any suggestions that might help here?
This can perhaps be addressed from within Inline.pm - though I'm not yet
sure what would be entailed in doing that.

David, if you wanted to come up with a workaround for this in
Math::Primes::FastSieve, you could try putting the code inside:

use Inline CPP => <<'EOCPP';
<code>
EOCPP

instead of putting it in a __DATA__ section.
I think that would eliminate the warning, but I was unable to test that
because I couldn't get it to compile. My attempts kept getting hammered
with:
"Can't install an Inline extension module with AUTONAME enabled."

Maybe you know how to get around that - I don't ... and I don't have time to
go digging right now.
Perhaps it's not trivial to do, and is therefore an unsatisfactory approach.

Another workaround that I *have* been able to check, and which appears to
work fine is as follows:

Once the module has been compiled, the following 3 lines in FastSieve.pm are
no longer needed:
use Inline CPP => 'DATA',
VERSION => '0.07',
NAME => 'Math::Prime::FastSieve';

If you get rid of those 3 lines from FastSieve.pm, then you no longer get
the warning of which Dana has complained.
It would therefore be a matter of having 'make install' remove (or comment
out) those 3 lines from the FastSieve.pm that gets installed.

Maybe there's other workarounds that are easier to implement.
That's all I could think of "off the top". I'll look in more depth over the
coming few days.

In the meantime, any other ideas/observations are welcome.

Better, of course, if we can get it fixed satisfactorily in Inline.

Cheers,
Rob
David Oswald
2012-05-18 09:02:17 UTC
Permalink
I wanted to follow up here with some additional information that Dana provided:

First, it's now been identified that the problem in installing
Inline::CPP was due to the fact that the system's g++ compiler had
been upgraded to a newer version than the compiler used to build Perl.
This is really good to know; now I can include a test in the test
suite that gives users a heads up if they're trying to install
Inline::CPP and running into difficulty. We can compare
$Config{gccversion} with 'g++ -v', and if they differ, complain loudly
so that potential users will know what they need to do to get a
successful installation. I have a feeling that when I release a new
developer's release we'll find out by the new test's diagnostics that
those recent Linux failures are due to this issue. So I'm really
grateful to Dana for uncovering it.

Next: Dana identified that Inline will not install on Perl version
5.8.0 or earlier. 5.8.1 is ok. The reason is because Digest::MD5 (a
dependency of Inline, I believe) has an XS component that uses a
feature that wasn't implemented until 5.8.1. There is a bug report in
the RT for Digest::MD5, as well as two recommended patches. But the
reports are fairly old, so I'm not sure if the author intends to fix
them. If you look at the smoke test matrix for Digest::MD5 you will
find that starting with the most recent version, no Perl older than
5.8.1 has met with success.

In the next day or so I'll send the author an email and see if I can
get his attention.

The bad news is that if it doesn't get fixed, it will be necessary to
document that the minimum Perl version supported by Inline, Inline::C,
and Inline::CPP is now 5.8.1. Previously Inline::CPP supported 5.6.0.
Hopefully we can get this fixed, and then put into our Makefile.PL's
a minimum version number for Digest::MD5 that overcomes the issue.

Dave
--
David Oswald
daoswald-***@public.gmane.org
Sisyphus
2012-05-18 13:51:00 UTC
Permalink
----- Original Message -----
From: "David Oswald"
Post by David Oswald
Next: Dana identified that Inline will not install on Perl version
5.8.0 or earlier. 5.8.1 is ok. The reason is because Digest::MD5 (a
dependency of Inline, I believe) has an XS component that uses a
feature that wasn't implemented until 5.8.1. There is a bug report in
the RT for Digest::MD5, as well as two recommended patches. But the
reports are fairly old, so I'm not sure if the author intends to fix
them. If you look at the smoke test matrix for Digest::MD5 you will
find that starting with the most recent version, no Perl older than
5.8.1 has met with success.
There's a related bug report (which I never fully understood) at
https://rt.cpan.org/Public/Bug/Display.html?id=69884

I would've expected that Digest::MD5 was now under the maintainership of
p5p, so approaces to get recent Digest::MD5 versions made backwards
compatible would therefore be better directed there. (I could be wrong about
that, but.)

Cheers,
Rob
Sisyphus
2012-05-19 00:45:10 UTC
Permalink
----- Original Message -----
From: "David Oswald"
Post by David Oswald
Next: Dana identified that Inline will not install on Perl version
5.8.0 or earlier.
This is not quite correct.
Digest::MD5 is core by the time 5.8.0 arrives. Later versions of Digest::MD5
might be unbuildable on 5.8.0 but that doesn't matter, because the version
of Digest::MD5 that ships with the 5.8.0 sources is fine for the purposes of
Inline.
Digest::MD5 was not core for 5.6, however. So there may be a problem there
in that one cannot install the most recent version of Digest::MD5, and has
to settle for an older version. My perl-5.6 has Digest-MD5-2.16, which is
fine for Inline.

Is there a way for a Makefile.PL to specify a *maximum* version for a
dependency ?
Currently the Inline Makefile.PL simply specifies a minimum version of 2.09.

I guess it wouldn't hurt if the Inline Makefile.PL displayed a warning about
this whenever it gets run on perl-5.6.
Maybe this should also be mentioned in the docs.

Cheers,
Rob
David Oswald
2012-05-19 00:56:25 UTC
Permalink
----- Original Message ----- From: "David Oswald"
Post by David Oswald
Next: Dana identified that Inline will not install on Perl version
5.8.0 or earlier.
This is not quite correct.
Digest::MD5 is core by the time 5.8.0 arrives. Later versions of Digest::MD5
might be unbuildable on 5.8.0 but that doesn't matter, because the version
of Digest::MD5 that ships with the 5.8.0 sources is fine for the purposes of
Inline.
Digest::MD5 was not core for 5.6, however. So there may be a problem there
in that one cannot install the most recent version of Digest::MD5, and has
to settle for an older version. My perl-5.6 has Digest-MD5-2.16, which is
fine for Inline.
Is there a way for a Makefile.PL to specify a *maximum* version for a
dependency ?
Currently the Inline Makefile.PL simply specifies a minimum version of 2.09.
I guess it wouldn't hurt if the Inline Makefile.PL displayed a warning about
this whenever it gets run on perl-5.6.
Maybe this should also be mentioned in the docs.
You're right, the main issue is where 'v' is 5.6.0 <= v < 5.8.0. If
there isn't an older version already installed, cpan/cpanm/cpanp will
pull in the latest, which won't install.

Though the Meta Specification (ExtUtils::Meta::Spec, I think) does
permit a maximum version, to my knowledge that feature isn't supported
by the various installers and MakeMakers. I'm not sure for
Module::Install. Anyway, that feature will eventually work, but at
the moment I don't think it does.

Maybe Makefile.PL could detect whether the Perl version number is
within the trouble-range, and if so, test whether or not D::MD5 is
installed. If not, bail out with a diagnostic message stating that
D::MD5 version whatever must be installed manually before proceeding.

The issue with v5.8.0 is simply one of incompatibility with newer
D::MD5's. But that's a non-issue, since it was bundled with the core
distribution. If later versions are incompatible, they won't be
getting installed, and will never conflict.

So... unless D::MD5 gets patched we either need to detect and warn, or
abandon 5.6.x. Detect and warn seems the lesser evil.
--
David Oswald
daoswald-***@public.gmane.org
Sisyphus
2012-05-19 02:18:09 UTC
Permalink
----- Original Message -----
From: "David Oswald"
Post by David Oswald
Maybe Makefile.PL could detect whether the Perl version number is
within the trouble-range, and if so, test whether or not D::MD5 is
installed. If not, bail out with a diagnostic message stating that
D::MD5 version whatever must be installed manually before proceeding.
Yes, I think that's a good compromise.
I'll stick that in and release an Inline-0.50_02 early next week.

Cheers,
Rob
Sisyphus
2012-05-19 18:52:22 UTC
Permalink
----- Original Message -----
From: "David Oswald"
Post by David Oswald
I get the Inline M51 warning (DATA section not used) if anyone does a
require on my module. I realize this is a harmless warning, but I
really don't want it happening to my module when I don't even use
Inline. So for now I've disabled use of MPFS to avoid it.
<snip>------------------------------------------------
That message is coming from Inline, but I can't figure out how one
might squelch it. Are there any suggestions that might help here?
Found the time to have a further look tonight.

It all happens in this section of Inline.pm:

######################################################
#==============================================================================
# Process delayed objects that don't have source code yet.
#==============================================================================
# This code is an ugly hack because of the fact that you can't use an
# INIT block at "run-time proper". So we kill the warning for 5.6+ users
# and tell them to use a Inline->init() call if they run into problems.
(rare)
my $lexwarn = ($] >= 5.006) ? 'no warnings;' : '';

eval <<END;
$lexwarn
\$INIT = \$INIT; # Needed by Sarathy's patch.
sub INIT {
\$INIT++;
&init;
}
END

sub init {
local ($/, $") = ("\n", ' '); local ($\, $,);

while (my $o = shift(@DATA_OBJS)) {
$o->read_DATA;
$o->glue;
}
}

sub END {
warn M51_unused_DATA() if @DATA_OBJS;
print_version() if $version_requested && not $version_printed;
}

#######################################################

If @DATA_OBJS doesn't get emptied out then you get the M51 warning.

If you "require" Math::Prime::FastSieve, sub INIT does not get called
(because INIT blocks can only be executed *before* runtime), therefore sub
init doesn't get called, therefore @DATA_OBJS doesn't get emptied, therefore
you get the M51 warning.

One solution is to take the advice offered in the comments in the above code
and have FastSieve.pm do:

Inline->init();

(immediately prior to the closing '1;' seems to suffice).

I'm a bit nervous about messing too much with this code - I don't know what
"Sarathy's patch" is, or why it needs the line of code that is claimed to be
necessary. I've no idea why it does '\$INIT++;' (or even why $INIT has to
exist) and I don't really know much about how that code fits in to the
general scheme.

It's clear that $lexwarn can now be removed and replaced by an unconditional
'no warnings;' at the begining of the eval() ... and I'll do that.

For the moment, I think I'll just document the 'Inline->init()' solution,
and leave it at that.

Eventually I'd like to think that I'll get around to poking and prodding a
bit, and try to come up with a better way of silencing the M51 warning when
an Inline module is "required". (If someone else gets to it before I do,
then that's all the better :-)

Cheers,
Rob

PS If it's not obvious, the reason that 'use Math::Prime::FastSieve;' does
*not* produce the same
warning is that loading with "use" enables the INIT sub to get run (straight
after compile-time,
and before runtime). Therefore, @DATA_OBJS gets emptied out, and the M51
warning is avoided.
Dana Jacobsen
2012-05-21 19:10:59 UTC
Permalink
The use vs. require issue is a little harder when I have chains of
modules, or things get loaded my Moose/Mouse roles via 'with' since
they load at runtime. In my immediate use, looking for
Math::Prime::FastSieve is all inside a Role, so there's no real way I
can change how it is loaded. Even if I write:

use Math::Prime::FastSieve;
# .... use MPFS

in my role, I get the warning. No direct eval or require on my part.


The chain of modules example, with MyPack.pm:

package MyPack;
use strict;
use warnings;
use Math::Prime::FastSieve;
# ...
1;

Everything is good. No warnings. But along comes someone who wants
to use Mypack and creates his MyScript:

#!/usr/bin/env perl
no strict;
no warnings;
require MyPack;
# ...

Oops, they get the warning. Similarly if they use an eval with use or
require, and 'no warnings' doesn't work for suppression.

As you wrote, putting 'Inline->init()' after the inclusion of M::P::FS
solves the issue in both cases. I can do this successfully:

if (eval {require Math::Prime::FastSieve; Inline->init(); 1;}) {
# We have MPFS, do fast stuff.
} else {
...
}

The goal would be to have Inline handle it, or the module using Inline
(e.g. Math::Prime::FastSieve) do it. But for now this works.

Dana
----- Original Message ----- From: "David Oswald"
Post by David Oswald
I get the Inline M51 warning (DATA section not used) if anyone does a
require on my module. I realize this is a harmless warning, but I
really don't want it happening to my module when I don't even use
Inline. So for now I've disabled use of MPFS to avoid it.
<snip>------------------------------------------------
That message is coming from Inline, but I can't figure out how one
might squelch it.  Are there any suggestions that might help here?
Found the time to have a further look tonight.
######################################################
#==============================================================================
# Process delayed objects that don't have source code yet.
#==============================================================================
# This code is an ugly hack because of the fact that you can't use an
# INIT block at "run-time proper". So we kill the warning for 5.6+ users
# and tell them to use a Inline->init() call if they run into problems.
(rare)
my $lexwarn = ($] >= 5.006) ? 'no warnings;' : '';
eval <<END;
$lexwarn
\$INIT = \$INIT; # Needed by Sarathy's patch.
sub INIT {
  \$INIT++;
  &init;
}
END
sub init {
  local ($/, $") = ("\n", ' '); local ($\, $,);
$o->read_DATA;
$o->glue;
  }
}
sub END {
  print_version() if $version_requested && not $version_printed;
}
#######################################################
If you "require" Math::Prime::FastSieve, sub INIT does not get called
(because INIT blocks can only be executed *before* runtime), therefore sub
you get the M51 warning.
One solution is to take the advice offered in the comments in the above code
Inline->init();
(immediately prior to the closing '1;' seems to suffice).
I'm a bit nervous about messing too much with this code - I don't know what
"Sarathy's patch" is, or why it needs the line of code that is claimed to be
necessary. I've no idea why it does '\$INIT++;' (or even why $INIT has to
exist) and I don't really know much about how that code fits in to the
general scheme.
It's clear that $lexwarn can now be removed and replaced by an unconditional
'no warnings;' at the begining of the eval() ... and I'll do that.
For the moment, I think I'll just document the 'Inline->init()' solution,
and leave it at that.
Eventually I'd like to think that I'll get around to poking and prodding a
bit, and try to come up with a better way of silencing the M51 warning when
an Inline module is "required". (If someone else gets to it before I do,
then that's all the better :-)
Cheers,
Rob
PS If it's not obvious, the reason that 'use Math::Prime::FastSieve;' does
*not* produce the same
warning is that loading with "use" enables the INIT sub to get run (straight
after compile-time,
warning is avoided.
Sisyphus
2012-05-22 05:46:50 UTC
Permalink
----- Original Message -----
From: "Dana Jacobsen"
Post by Dana Jacobsen
As you wrote, putting 'Inline->init()' after the inclusion of M::P::FS
if (eval {require Math::Prime::FastSieve; Inline->init(); 1;}) {
# We have MPFS, do fast stuff.
} else {
...
}
The goal would be to have Inline handle it, or the module using Inline
(e.g. Math::Prime::FastSieve) do it. But for now this works.
By the end of rhis week I'll have released Inline-0.50_02.
The documentation on how to write modules that use Inline will be revised to
include the Inline->init() option for working around this problem.

In case an *early* call to Inline->init() might interfere with the initial
compilation, I'll probably be recommending that it be done in and END{}
block:

END {
Inline->init();
}

That seems fine for me, and I don't think that will ever be too late a time
to make the Inline->init() call. (If I'm ever proven wrong about that, then
I can just re-write the docs again :-)

I don't really want to go altering Inline itself until I understand the
purpose(s) this weird section of code serves. (And that might take a while
;-)

Thanks for raising this Dana, David.

Cheers,
Rob

Loading...