25 replies to this topic

[Tempel]

Dear everyone,

If you're a Python developer, you're probably aware that Distutils is the standard mechanism for distributing Python software. As long as you have a well-made setup script, you can distribute the sources, install to root or home, create an RPM, or, with the right extensions, package it just about any way else.

distPND is a Distutils extension I've created to build PND and PXML files easily. It provides two commands: gen_pxml will slap together a PXML file based on parameters in the setup script and command-line options, while bdist_pnd will assemble your project into a PND file (calling gen_pxml if it needs a PXML file).

To use it, download the sources from its github page and install like any Python package (sudo python setup.py install). After that, you can use the commands with any other Distutils-based setup script by specifying distpnd in command-packages (ie: "python setup.py --command-packages distpnd [command]", or make a setup.cfg file with "command-packages=distpnd" included under the "global" heading). To make iso- or squash-based PNDs, you'll need to have mkisofs or mksquashfs (respectively) available. Even without those, though, you can make a folder that will be read by libpnd.

So far, it works pretty well, but it's kind of hackish and somewhat limited. I'm hoping my fellow Python devs will take distPND for a ride and abuse the crap out of it. Tell me where it breaks or could behave more nicely and I'll fix it up. Or, if you're feeling generous, send along some patches. Some things to note:

• Almost none of this respects the dry-run flag. It shouldn't make any system changes, but watch out for that.
  
• gen_pxml does not nearly expose all the functionality in a PXML. Is there an elegant way to include associations, multiple applications, and multilingual titles and descriptions? One solution might be to have PXML templates; can you concoct a good approach to doing that?
  
• bdist_pnd doesn't yet check that it's not overwriting existing files. Again, watch out.
  
• If you have an icon or info file to include, bdist_pnd expects that file will make its own way into the build_pnd folder (presumably during the "install" portion of the job). Is there a better way to handle this? I think I've solved this pretty well. If you tell bdist_pnd about your icon and info, it ensures they end up in the root of the PND and in your PXML. I've updated the Lonely Tower sources to show two different ways to do it.

As an example of how it works, I've packed up my opus; you can download the sources to see how setup.py and setup.cfg work, or you can download the resulting PND to see the results (with no Pandora, I can't actually confirm that this works, so please don't laugh at me if it fails!).

Have fun everyone! I look forward to hearing back from you.

PS: If your package includes extension modules written in a compiled language (C or Cython or whatever), they'll need to be built for the Pandora. You could just run the whole build process on the Pandora, but if you can make cross-compiling work through the build command, then it should be no additional effort to make it work with bdist_pnd. You might want to take a look at distutilscross to help with this.
PPS: You might also be interested in stdeb, a Distutils extension to build deb packages. Since debs are identical to ipks, this would allow you to build PNDs and Pandora-installable ipks from the same build script! I find that exciting.

Edited by Tempel, 07 November 2010 - 03:18 AM.

[B-ZaR]

I have some simple pygame projects I could try this with. They're not finished/polished, but are playable. I'll be sure to try it out when I get my pandora

[Tempel]

According to people over in the Sparks thread, it looks like distPND isn't producing valid files yet! Without a Pandora of my own to test on, I'm making one last-ditch effort to fix it; if it fails, I'll just have to wait for mine to arrive.

I noticed that the PXML archetype includes an "encoding" attribute that my generated PXMLs did not. But I've fixed that and have updated the PNDs for Sparks and The Lonely Tower. Sparks is made with an isofs, while Lonely Tower is done with a squashfs, just so I can make sure fs-type isn't the culprit.

Pandora owners, please try them out; let me know if they get detected and run. There will be many thanks if you try!

[Tempel]

Thanks to much testing from Gruso, and vminko's VNC-accessible Pandora, I've managed to get enough testing to work out distPND's kinks. If you're curious, the problem stemmed from the fact that Angstrom does not include a python2 symlink; since my Arch box has the python symlink pointed at python3, the Distutils shebang rewriting magic was trying to make the programs start with python2. I kindly asked Distutils to look elsewhere, and now everything works.

Also, I added in specification of subcategories. I discovered that including multiple subcategories seemed to break things (at least on one PND I tried), so watch out. But otherwise, we're pretty much production-ready.

Therefore, this is likely the last build of The Lonely Tower I'll need to put up. And, as always, you can peruse its sources to see how to do the same with your own programs.

Awesome.

[Gruso]

So, join the dots for a newbie here. Would this help me get Jan-Nik's game Tudi happening on the Pandora?

His thread: http://www.gp32x.com/board/index.php?/topic/51274-tudi

It's a simple but extremely addictive game, I love it.

[Tempel]

For a newbie, it probably doesn't mean much This won't help something run on the Pandora, it simply makes it easy to take already-working code and package it into a PND. And it does it in a way that works along with Python's standard distribution utilities, so it makes it easy for Python devs to distribute in many ways including PNDs.

It could be used to package up Tudi, except that its distribution method looks like it's not quite standard. There seems to be a lot of Windows-specific stuff and weird imports, but I'll take a stab at figuring it out.

[Esn]

Hey, is this very useful program mentioned in the wiki somewhere?

I'd do it myself, but I know very little about devving.

[Tempel]

I hadn't thought of putting it there. I'm not sure where in the wiki it would belong. Perhaps a "Tools for packaging" subheading on the Development Tools page?

EDIT: I finally did it.

Edited by Tempel, 09 February 2011 - 04:30 PM.

[valhalla]

Arch linux users can get it from the AUR

[Tempel]

That's cool, valhalla! I was intending to make a PKGBUILD once it got some more testing, but no one's testing it yet. A couple notes:

• Technically it's under the MIT license, not BSD, even though the two are so similar.
  
• The git repo is structured such that official releases are on the master branch, while development happens on the develop branch. Your PKGBUILD seems to be pulling from the master branch, so it doesn't install all the latest code (and *-git packages generally want to have the latest developments, right?)
  
• Since it's on GitHub, you don't need to do a full clone to get the latest sources. It automatically provides tar.gz packages of the latest source for both the master and develop branches. This should make for a simpler PKGBUILD, while still giving the latest source.

Thanks for doing this, though. Maybe a few more people will try it now.

[valhalla]

That's cool, valhalla! I was intending to make a PKGBUILD once it got some more testing, but no one's testing it yet. A couple notes:

• Technically it's under the MIT license, not BSD, even though the two are so similar.
  
• The git repo is structured such that official releases are on the master branch, while development happens on the develop branch. Your PKGBUILD seems to be pulling from the master branch, so it doesn't install all the latest code (and *-git packages generally want to have the latest developments, right?)
  
• Since it's on GitHub, you don't need to do a full clone to get the latest sources. It automatically provides tar.gz packages of the latest source for both the master and develop branches. This should make for a simpler PKGBUILD, while still giving the latest source.

Thanks for doing this, though. Maybe a few more people will try it now.

• fixed, sorry
  
• right, I've changed it to use the develop branch; I' ve also created a new PKGBUILD for the release
  
• the PKGBUILD wouldn't be really simpler, since it has to keep track of changing md5sums and directory names for said tarballs.

I have a couple of python programs, and I plan to try distpnd as soon as I get my pandora, so I needed the PKGBUILD (I don't like to install stuff outside the package manager)

[Tempel]

Nicely done, thank you

Of course, if you need any help or have any trouble using distPND, let me know.

[quartercast]

Hi Tempel,

I'm attempting to port RedNotebook. I'm getting the following error:

petes-computer:rednotebook-1.1 2.3 pete$ python setup.py bdist_pnd
data_base_dir /System/Library/Frameworks/Python.framework/Versions/2.6
running bdist_pnd
Traceback (most recent call last):
  File "setup.py", line 273, in <module>
    setup(**parameters)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/core.py", line 152, in setup
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.py", line 975, in run_commands
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.py", line 995, in run_command
  File "/Library/Python/2.6/site-packages/distpnd/bdist_pnd.py", line 90, in run
    '--executable=/usr/bin/python', '--force'))
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/core.py", line 221, in run_setup
  File "<string>", line 102, in <module>
  File "<string>", line 76, in build_mo_files
NameError: global name 'os' is not defined

I have looked to check 'import os' is included on all files, and it is! Any ideas what might be the cause?

[Tempel]

Woah, Distutils is being weird. It's always been weird, but I thought I had it all figured out until it did this.

If you look inside RedNotebook's setup.py, you'll see that it defines a bunch of functions that get used in the build process. It looks like what's happening is that, when bdist_pnd calls the build command, it runs one (or more) of those functions, but it runs them outside of the context of the rest of the file. I have no idea how you can even do that in Python, but the result is that all names defined outside of the function can't be seen within the function.

Presumably there's something I'm doing in distPND that just isn't how it's supposed to be. But this is looking like black magic to me, so I'm clearly going to have to spend some more time digging through Distutils. As a temporary workaround to get RedNotebook going, try adding the missing names inside the function. So, in function build_mo_files, adding "import os" fixes the immediate problem. Then it complains about "i18n_dir"; copy the definition from a few lines up. Then it complains about msgfmt; copy the "from rednotebook.external import msgfmt" from a few lines above that. Keep going until the errors stop or you get bored. Or just wait (indefinitely) for me to dig deeper into this bizarre world of Distutils.

EDIT: Just noticed it also depends on PyYAML, which is not installed by default on the Pandora. You'll have to make sure it gets into the PND yourself (I'm planning on making distPND do this automatically, but haven't yet). Just download its source, then run on its setup script:

python setup.py install --root=/path/to/rednotebook/build_pnd --install-lib=/

After that, all of PyYAML's files will be in the right place for bdist_pnd to pick them up automatically.

Edited by Tempel, 06 March 2011 - 05:51 PM.

[Tempel]

I found the problem and a workaround, but it's certainly not ideal. It all comes down to Python's "exec" command. It's known to behave weirdly when given both global and local dicts. Apparently that's how it's supposed to behave, it's just not really what you would expect. Then the Distutils developers fell into exec's trap and seem to have misused it within the run_setup function that I call from distPND.

So that's the problem; now for the workaround. The easiest approach is to simply fix Distutils yourself. Open up /usr/lib/python2.?/distutils/core.py, then on line 221, remove the trailing ", l". Getting rid of those three characters allows bdist_pnd to work on RedNotebook (just remember to install PyYAML first, as I mentioned before).

I'm going to try to submit this as a bug report to the Distutils developers, if I can find out how, but who knows how long it'll take for the fix to make it in. I'll also see if I can take a different approach in distPND to avoid the broken run_setup function, but I don't think there's much else I can do.

Thanks for finding interesting problems for me