Using Inline Python to import Python vars

[Chris Nighswonger]

I seem to be misunderstanding something about importing Python package vars
with Inline. I'm thinking it may be some syntax problem.

This code returns the value of __version__ as expected:

use Inline Python => "from pykota.version import __version__; print
__version__;";



However, this code borks:

use Inline Python => 'DATA',
DIRECTORY => '/usr/share/webmin/pykota/.Inline/',
NAME => 'PyKota::Test';

print pykota_version();
1;

__DATA__

__Python__

def pykota_version():
from pykota.version import __version__
return __version__


The error is:

File "<string>", line 3
from pykota.version import
^
SyntaxError: invalid syntax
Error -- py_eval raised an exception at
/usr/local/lib/perl/5.14.2/Inline/Python.pm line 177.
INIT failed--call queue aborted.

Kind Regards,
Chris

[Chris Nighswonger]

On Mon, Apr 1, 2013 at 12:57 PM, Chris Nighswonger <

Post by Chris Nighswonger
use Inline Python => 'DATA',
DIRECTORY => '/usr/share/webmin/pykota/.Inline/',
NAME => 'PyKota::Test';
print pykota_version();
1;
__DATA__
__Python__
from pykota.version import __version__
return __version__
File "<string>", line 3
from pykota.version import
^
SyntaxError: invalid syntax
Error -- py_eval raised an exception at
/usr/local/lib/perl/5.14.2/Inline/Python.pm line 177.
INIT failed--call queue aborted.

So I threw together some C to see if the problem was with my Python syntax
in the context of Inline::Python. Below is what I did and it works as
expected.

Any suggestions as to where I'm going wrong here?

Kind Regards,
Chris



#include <Python.h>

main( ) {
PyObject *pstr, *pmod, *pdict;
Py_Initialize( );

pmod = PyImport_ImportModule("__main__");
pdict = PyModule_GetDict(pmod);
pstr = PyRun_String(
"def pykota_version():\n"
" from pykota.version import __version__\n"
" return __version__\n"
"print pykota_version()\n"
, Py_file_input, pdict, pdict);

if (!pstr) {
PyErr_Print();
exit (1);
}

Py_DECREF(pmod);
Py_DECREF(pstr);
}

[Stefan Seifert]

Hi Chris,

Post by Chris Nighswonger
...
__DATA__
__Python__
from pykota.version import __version__
return __version__
File "<string>", line 3
from pykota.version import
^
SyntaxError: invalid syntax

Using perl Makefile.PL -debug when building Inline::Python gives a good hint.
With this flag it will print loads of information including Python source code
it compiles to stderr.

This gives us:

py_eval: code:
def pykota_version():
from pykota.version import
py_eval: main_module=0x7fd023985be8
py_eval: globals=0x229aac0
py_eval: type=1
py_eval: context=257
File "<string>", line 3
from pykota.version import
^
SyntaxError: invalid syntax

I've never used DATA file handles in Perl much less with Inline::Python, so
I'm not exactly sure. But the debug output looks like the Python source code
ends at __version__. So I guess, the code reading the DATA file processes
everything from __Python__ to the next marker and treats every word enclosed
in double underscores as such marker.

Indeed, every other way of giving Python source code to Inline::Python seems
to work ok.

Regards,
Stefan

[Chris Nighswonger]

Post by Stefan Seifert
Hi Chris,
Using perl Makefile.PL -debug when building Inline::Python gives a good hint.
With this flag it will print loads of information including Python source code
it compiles to stderr.

Thanks for that valuable piece of information. I was wondering how to get
the XS foo to spit out the Printf's.

Post by Stefan Seifert
I've never used DATA file handles in Perl much less with Inline::Python, so
I'm not exactly sure. But the debug output looks like the Python source code
ends at __version__. So I guess, the code reading the DATA file processes
everything from __Python__ to the next marker and treats every word enclosed
in double underscores as such marker.

I really should have thought to look at the compiled module. Here is what
it is:

namespace : %
classes : %
functions : @
pykota_version
filtered : <<EOV-

def pykota_version():
import pykota.version
return
EOV


Clearly the problem is exactly as you point out.

I'll see if I can fix it.

Kind Regards,
Chris

[Chris Nighswonger]

On Mon, Apr 8, 2013 at 4:25 PM, Chris Nighswonger <

Post by Stefan Seifert
I've never used DATA file handles in Perl much less with Inline::Python, so
I'm not exactly sure. But the debug output looks like the Python source code
ends at __version__. So I guess, the code reading the DATA file processes
everything from __Python__ to the next marker and treats every word enclosed
in double underscores as such marker.

So it does look as though the DATA splitting algorithm is a bit "loose."

In Inline.pm abt line 354:

@{$DATA{$pkg}} = split /(?m)(__\S+?__\n)/, $data;
shift @{$DATA{$pkg}} unless ($ {$DATA{$pkg}}[0] || '') =~ /__\S+?__\n/;
}

It appears we split on anything prefixed and suffixed with double
underscores.

Assuming that I'm not way off here:

I'm guessing that this is because we really don't know whether this is C,
Java, Python, or FooBar?

Can we have the language extension pass in what it is and actually search
for that marker rather than for __ANYTHING__?

Could we actually check for some specific end-of-code-block maker? Say
'__EOI__' for "end-of-inline" or some such?

Kind Regards,
Chris

[s***@public.gmane.org]

-----Original Message-----
From: Chris Nighswonger
Sent: Tuesday, April 09, 2013 6:39 AM
To: inline-***@public.gmane.org
Subject: Re: Using Inline Python to import Python vars

Post by Chris Nighswonger
So it does look as though the DATA splitting algorithm is a bit "loose."
@{$DATA{$pkg}} = split /(?m)(__\S+?__\n)/, $data;
/__\S+?__\n/;
}
It appears we split on anything prefixed and suffixed with double
underscores.

A newline (immediately after the closing __) is also needed before any
splitting takes place:

######################
use warnings;

$data = "__DATA__ __C__ __other__ __Python__\n__version__";
@stuff = split /(?m)(__\S+?__\n)/, $data;

print ">>$_<<\n" for @stuff;
######################

Post by Chris Nighswonger

__DATA__ __C__ __other__ <<
__Python__

<<

Post by Chris Nighswonger

__version__<<

I can get the same type of failure with this (non-portable) Inline::C
script:

######################
use strict;
use warnings;

use Inline C => 'DATA';

foo();

1;

__DATA__

__C__

void foo() {
printf("%d %s\n",__MINGW32__
,"Hello");
}
######################

However, there's no problem at all if I rewrite foo() as:

void foo() {
printf("%d %s\n",__MINGW32__,
"Hello");
}

Thanks for digging this up, Chris !!

I don't personally use the "__DATA__" approach.
I'll do some digging myself right now, and see if I can come up with a
simple improvement to the regex.
(All the better if someone else beats me to it.)

As a quick guess, perhaps we split only if there's nothing other than
whitespace between the previous newline and the opening double underscore.

In the meantime, if (in Python) it's permissible to add whitespace at the
end of a line, then adding a space after __version__ should do the trick
methinks.

Cheers,
Rob

[s***@public.gmane.org]

-----Original Message-----
From: sisyphus1-sFbbPxZDHXw0n/***@public.gmane.org
Sent: Tuesday, April 09, 2013 12:44 PM
To: Chris Nighswonger ; inline-***@public.gmane.org
Subject: Re: Using Inline Python to import Python vars



-----Original Message-----
From: Chris Nighswonger

Post by Chris Nighswonger
@{$DATA{$pkg}} = split /(?m)(__\S+?__\n)/, $data;

Try replacing that line (in Inline.pm) with:

#warn "\n########\n\$data:\n$data\n########\n";
@{$DATA{$pkg}} = split /(?m)(\n[ \t]{0,}__\S+?__\n)/, $data;
for(@{$DATA{$pkg}}) {$_ = (split /\s/, $_)[-1] . "\n" if $_ =~
/^\s{0,}__\S+?__\n$/} # remove any whitespace that precedes marker
#warn "\n########\n", scalar(@{$DATA{$pkg}}), "\n";l
#for(@{$DATA{$pkg}}) {warn ">>$_<<\n"}

It's only 2 lines of code - the (commented out) warnings might be useful if
things go awry. They won't appear in the final version of Inline.pm.

That works ok for my failing Inline::C script, and without causing any of
the Inline tests in the Inline-0.52 test suite to fail.
I'm hopeful it will do the trick for your Inline::Python script, too.

It's possibly a little convoluted at the moment ... maybe someone can
improve on that.

Cheers,
Rob

[Chris Nighswonger]

Hi Rob,

Post by s***@public.gmane.org
Sent: Tuesday, April 09, 2013 12:44 PM
Subject: Re: Using Inline Python to import Python vars
-----Original Message----- From: Chris Nighswonger
@{$DATA{$pkg}} = split /(?m)(__\S+?__\n)/, $data;
#warn "\n########\n\$data:\n$data\n#**#######\n";
@{$DATA{$pkg}} = split /(?m)(\n[ \t]{0,}__\S+?__\n)/, $data;
/^\s{0,}__\S+?__\n$/} # remove any whitespace that precedes marker
It's only 2 lines of code - the (commented out) warnings might be useful
if things go awry. They won't appear in the final version of Inline.pm.
That works ok for my failing Inline::C script, and without causing any of
the Inline tests in the Inline-0.52 test suite to fail.
I'm hopeful it will do the trick for your Inline::Python script, too.
It's possibly a little convoluted at the moment ... maybe someone can
improve on that.

I'll be out of the office most of the day today, but will try this out
later this afternoon/evening and post back.

Kind Regards,
Chris

[Chris Nighswonger]

Post by s***@public.gmane.org
Sent: Tuesday, April 09, 2013 12:44 PM
Subject: Re: Using Inline Python to import Python vars
-----Original Message----- From: Chris Nighswonger
@{$DATA{$pkg}} = split /(?m)(__\S+?__\n)/, $data;
#warn "\n########\n\$data:\n$data\n#**#######\n";
@{$DATA{$pkg}} = split /(?m)(\n[ \t]{0,}__\S+?__\n)/, $data;
/^\s{0,}__\S+?__\n$/} # remove any whitespace that precedes marker
It's only 2 lines of code - the (commented out) warnings might be useful
if things go awry. They won't appear in the final version of Inline.pm.
That works ok for my failing Inline::C script, and without causing any of
the Inline tests in the Inline-0.52 test suite to fail.
I'm hopeful it will do the trick for your Inline::Python script, too.

This fixes the problem! Fantastic!

Post by s***@public.gmane.org
It's possibly a little convoluted at the moment ... maybe someone can
improve on that.

Is there any case where the marker would not be on a line by itself? If not
perhaps we could just do such as

@{$DATA{$pkg}} = split /(?m)^(__\S+?__\n)/, $data;

and eliminate the need to remove whitespace?

Kind Regards,
Chris

[s***@public.gmane.org]

From: Chris Nighswonger

Post by Chris Nighswonger
This fixes the problem! Fantastic!

Post by s***@public.gmane.org
It's possibly a little convoluted at the moment ... maybe someone can
improve on that.

Is there any case where the marker would not be on a line by itself?

If it's not on a line by itself, it will be deemed to *not* be a marker.
If it matches /__\S+?__\n/ and is on a line by itself, it *will* be regarded
as a marker.

There is therefore still an opening (in Inline::C, at least) to break
Inline.

Even after the alteration to sub read_DATA has been made, this Inline::C
code is still broken:

void foo() {
printf ("%d\n",
__MINGW32__
+ 7);
}

Similarly, if it's valid python to rewrite

return __version__

as

return
__version__

then Inline::Python would still be broken by the latter.

I reckon it's probably a real can of worms to try and fix that - so I'll
just leave it as is (until someone complains, anyway).
I think we would have to start checking for matches against known markers
(__C__, __Python__, etc.) as you originally suggested. Even then there would
be nothing to stop me from breaking Inline::C by defining a symbol named
__Python__ (or __CPP__, etc.) and using that symbol in my Inline::C code.

Hopefully, everyone writes their code *across* the page, not *down* the page
!!

Post by Chris Nighswonger
If not perhaps we could just do such as
@{$DATA{$pkg}} = split /(?m)^(__\S+?__\n)/, $data;
and eliminate the need to remove whitespace?

In the past, Inline has removed the whitespace that precedes the marker -
and it performs a check to verify that the whitespace has been removed.
So ... we need to remove the whitespace to satisfy that check.
(Alternatively, I could rewrite the check, but I think it's safer to stick
to the original formatting rules.)

In fact, in the past, Inline has removed everything (not just whitespace)
that precedes the marker on the line. So, in the past, you should have been
able to write:

__DATA__
some arbitrary stuff __Python__
......

Inline would then remove the " some arbitrary stuff " that precedes the
__Python__ marker.

This will no longer be the case. If the marker is preceded by anything other
than whitespace, it will be regarded as not being a marker.
So .... if anyone has been preceding the marker with anything other than
whitespace, they'll find their code broken.
Surely, no-one has been doing that ?

Anyway, I'll chew things over for a day or two - wait and see if there's any
other thoughts on what ought to be done.
Then I'll upload a 0.52_01 release that contains this amendment to
Inline.pm. (I'll let the list know when that happens.)

Thanks for pursuing this, Chris, Stefan.

Cheers,
Rob

[Chris Nighswonger]

Post by s***@public.gmane.org
Thanks for pursuing this, Chris, Stefan.

Many thanks to Stefan for helping me along and to Rob for fixing things up.

Kind Regards,
Chris