REVISION: Python25 readline variant and now builds as a framework

[dinosaur@…]

Here's an updated portfile for Python25 that will build it as a framework so it will now work with py-sip and PyQt4, and a variant that will allow readline support. Please integrate it into the tree.

[dinosaur@…]

Updated Portfile

[mww@…]

what do we need the framework for? As far as I remembered, it brought more havoc than benefit with python24; for the readline part: I'd prefer a py25-readline port (or however we're gonna handle the stuff with python24 vs python25)

[guido.soranzio@…]

You have forgotten to remove port:readline from the original depends_lib.

[mww@…]

I'm currently working on a clean python25 port with py25-readline etc. - these ports will get commited when v1.4 gets final (1.4 will have support for python25 ports)

[meissnem@…]

proposed patch

[meissnem@…]

Please consider the patch I just attached. It adds two variants to the python25 port, +framework and +default_python. Both are entirely optional -- and without them, the port remains as it is today.

+default_python simply installs python2.5 as ${prefix}/bin/python -- useful if you don't use many modules (like me).

+framework builds and installs python2.5 as a framework. In order to facilitate frameworks, the py25-* modules need to remove the hard-coded "--prefix=${prefix}" line so modules are installed into the framework. Additionally, the pyexpat module needs to be built to get the toolbox glue working -- so I removed it from the list of modules to be excluded. No explicit dependency on port:expat is necessary since the source code for expat is included in the Python-2.5 tarball and is linked in statically.

[mww@…]

btw. the py25-readline port exists now (as do some other py25-ports); I'm currently a bit short on time & ressources - perhaps pmq@ can have a look at the framework thing.

The thing about the "python" binary: This is a problem as other ports might stop working if they expect python 2.4 for the "python" binary. I'd rather say, create an alias in your .bashrc (alias python=/opt/local/bin/python2.5), if this is a solution for you.

[meissnem@…]

at the very least, the "pyexpat" stuff included in this patch could be used to fix #11442.

[mdickens@…]

There are 2 primary problems with the current python25 scripts and the py25-XXX ports that come from inside the python25 distro.

1: If the +framework variant is enabled (as per meissnem@…'s patch), then "/Applications/MacPython 2.5" is populated with some of the installed apps (instead of all of them in "/Applications/MacPorts/MacPython 2.5"). Reviewing the patches for python24, this can be fixed either with patches or with "post-patch" reinplace's ... for example, the following works well for me:

post-patch {
        reinplace "s|/Applications/MacPython|/Applications/MacPorts/MacPython|g\
" ${worksrcpath}/Mac/Makefile.in
        reinplace "s|/Applications/MacPython|/Applications/MacPorts/MacPython|g\
" ${worksrcpath}/Mac/IDLE/Makefile.in
        reinplace "s|/Applications/MacPython|/Applications/MacPorts/MacPython|g\
" ${worksrcpath}/Mac/PythonLauncher/Makefile.in
        reinplace "s|/Applications/MacPython|/Applications/MacPorts/MacPython|g\
" ${worksrcpath}/Mac/BuildScript/build-installer.py
        reinplace "s|2.4|2.5|g" ${worksrcpath}/Mac/Tools/Doc/setup.py
        reinplace "s|/Applications/MacPython|/Applications/MacPorts/MacPython|g\
" ${worksrcpath}/Mac/Tools/Doc/setup.py
}

2: Some of the py25-XXX modules from inside the python25 distro are disabled via the patch-setup.py file. This only disables the inclusion of the object (compiled .c) files in the python shared library; it does not prevent the inclusion of files that may use those Python modules in .../lib/python2.5/ (originals found in ${python25-distro}/Lib ). For whatever reason, Python2.5 seems to require the .py and .so files for a module to "import" to be in the same directory ... but that will -not- be the case for some of these modules. For example, hashlib is required by some of the files in ${python25-distro}/Lib , and these files cannot be disabled or not installed (beyond removing them all in post-destroot).

That said, none of the critical python25 components seem to rely upon any of the modules disabled by patch-setup.py . It should be possible in the py25-XXX Portfile's to extract the -related- .py file(s) in ${python25-distro}/Lib for the given module, and install them along with the .so created by setup.py .

For my needs, I'm just removing the patch-setup.py , then removing dependence on the py25-XXX modules that come from the python25 distro. This seems to be the "safest" way to get functionality.

[mdickens@…]

This ticket is getting too old. The addition of ticket:12803 to python25 (rev 4), means that creating a framework is no longer as easy as it was with the patch here. This ticket should probably be merged with ticket:12817 since it covers the framework install as well. The framework does work (almost) correctly if the dylib changes are negated. I think the dylib issue needs to be re-addressed more in the form of how 2.4.4 does it, which still provides framework compatibility. I do not yet have a Portfile that does all of this correctly, since it's gotten a bit complicated.

[mdickens@…]

I finally figured out and resolved the issues between framework and dylib in 'configure' and 'Makefile.pre.in'. I'll attach a tarball of the Portfile and related files. These work for me for doing both +framework and without (just standard dylib) on 10.4 (not at the same time). I hope someone else is testing this on other Darwin installs (10.2, .3, .5)!

[mdickens@…]

Could someone add me to the "cc:" list? I'd like to know what's going on with the Python25, +framework, and dylib issues.

[mdickens@…]

I've added another ticket (ticket:13778) with a replacement for patch-setup.py, which supersedes the one included in the attached tarball (of 2007-12-28).

I can confirm that these files (from the tarball, and the new patch-setup.py) provide working python25 on Darwin 8 & 9, using either +Framework or not.

Please also note that the _OSA module will not compile or install on Darwin 9, since Apple removed the "debug" commands from the OSA Framework. This doesn't error out the install process, since it happens when executing setup.py (which doesn't check for those errors).

[pschumm@…]

Thanks very much for your work on this. I need to use a frameworks build because -- among other things -- I need to use mod_wsgi and I can't seem to get it to compile against a non-frameworks Python. I downloaded the attachment you posted on 28-Dec-07 and replaced with the patch-setup.py from 01-Jan-08. The install went fine (OS X 10.4.11), but now I'm having problems because Python seems unable to find other modules which have been installed. For example, if I try to run ez_setup.py, I get the following error:

ImportError: No module named _md5

despite the fact that I have py25-hashlib installed (I get the same problem interactively, BTW). Is this a problem with the python25+frameworks port, or is this something that'll need to be changed in the other ports that depend on python25?

[mdickens@…]

> ImportError: No module named _md5
> 
> despite the fact that I have py25-hashlib installed (I get the same problem interactively, BTW).  Is this a problem with the python25+frameworks port, or is this something that'll need to be changed in the other ports that depend on python25?

This might be related to ticket #13777. Make sure the PYTHONPATH environment variable (in your preferred shell) includes ${prefix}/lib/python25/site-packages, since that's where the py25-hashlib stuff will be placed. e.g. in bash:

export PYTHONPATH=/opt/local/lib/python25/site-packages

You can check this via:

python
>>> import sys
>>> sys.path

and the printout should include both ${prefix}/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5 and ${prefix}/lib/python25/site-packages

[mdickens@…]

I've just attached a new tarball, which contains the changes needed to get the PYTHONPATH correct for the Framework install ; to include both ${Framework}/.../lib/python2.5/ and ${prefix}/lib/python2.5/site-packages/ . Checking the sys.path, these mods work as desired (I borrowed them from the python24 port, so this is no surprise). py25-hashlib and related ports should install and execute without needing the augment the PYTHONPATH in the Port environment (as was the case in python24).

[mdickens@…]

Updated tarball of Python25 port files, which works on 10.4 and 10.5, for no variants, +framework, +universal, and +framework+universal variants. See next entry for more info.

[mdickens@…]

Changes to the tarball of 2008-01-06, from previous:

* Instead of hard-coding "/opt/local" into setup.py, use "__PREFIX__" then replace it with the actual ${prefix} in the post-patch.

* Corrected +universal install on 10.5: '-isysroot' does not work with gcc, but '-isystem' does. This corrects ticket #13701, if that was still an issue.

I've tested all combinations of variants (plain, +framework, +universal, +universal+framework) on both 10.4 and 10.5, and they all work as desired. While the resulting binaries using +universal are for "i386" and "ppc", it would not be difficult to add variants to do "ppc64" and "x86_64" (though, of course, 64-bit apps are not well-tested on OSX just yet). I will look into this variant, and see if the apps work.

[gjznituv@…]

I replaced the tarball contents with the corresponding default macports python25 files.

Is there anything else I should do?

After reinstalling it I still get the same error for py25-wxpython (see ticket #13756 ).

[mdickens@…]

Instructions for using the tarball, because others have asked for it:

0) Deactivate or uninstall any py25-YYY ports, and any other ports that rely upon them. If possible, uninstall all of these first and (re)install them afterwards (in step 6) just to make sure they all work with the "new" Python install.

1) Make sure 'port' is version 1.6.0 or newer; this takes care of some issues.

sudo port selfupdate

Optionally, also upgrade the other installed ports; while this shouldn't make any difference, it might and really can't hurt unless you need a specific version of some particular port.

sudo port upgrade installed

2) Make sure python25 is not installed, e.g.

sudo port -f uninstall python25

3) Download the tarball; let's say this is in ~/Downloads/python25.tar

4) Make the updated Portfile and related files available for 'port' to use:

pushd `port dir python25`
cd ..
sudo rm -rf python25
sudo tar xf ~/Downloads/python25.tar
popd

5) Install the new version of Python, with whatever variant you desire, e.g.:

sudo port install python25 +framework

6) Reactivate or (re)install any py25-YYY ports that were deactivated or uninstalled in step 0.

[mdickens@…]

Replying to gjznituv@gmail.com:

I replaced the tarball contents with the corresponding default macports python25 files. Is there anything else I should do? After reinstalling it I still get the same error for py25-wxpython (see ticket #13756 ).

Please see my answer in ticket #13756, comment 5. A solution is to put a check into each pyXX-wxpython Portfile to make sure that the version of wxWidgets that's currently installed is compatible with the version of wxpython that is about to be installed. If not, tell the user what to do to get the correct version of wxWidgets, or something else useful ... not just a "random" compiler error message.

[jmr@…]

Adding cc as requested in comment:13.

[mdickens@…]

Upgrade to Python 2.5.2, including Portfile and all updated files; provides the same functionality as the previous tarball.

[reiffert@…]

I'm currently working on a python25 framework installation as per default. Your help gets appreciated much.

[reiffert@…]

The following problems got already fixed:

Change /Applications/MacPython 2.5 -> /Applications/MacPorts/MacPython 2.5

Symlinking Library/Frameworks/Python.framework Headers, Python, Resources and Versions/Current by the help of python_select. Thanks to raimue.

Symlinks lib/libpython2.5.dylib lib/python2.5 include/python2.5 into the framework

Currently I'm testing the port on various platform. Please contact me for this purpose.

Whats needs to be done after submitting: Fixing the prefix for all py25-* ports

Whats needs to be done for the next major macports release: Reflecting the new prefix into py25 Portgroup.

Kind regards Thomas

[reiffert@…]

Please test the atteched file python25.tar.bz2 and report back, thank you!

[mdickens@…]

I applaud your efforts towards finally creating the framework Python install. Here are my Initial Note and Comments from a quick review and testing.

• I'm concerned with sym-linking $prefix/lib/python2.5 and $prefix/include/python2.5 into the framework -- doing so is very non-standard. For the former, both paths ($prefix/Frameworks/.../lib/python2.5 and $prefix/lib/python2.5) can easily be made part of the default PYTHONPATH (before user additions via the environment variable) - this was done by my changes to setup.py - and Python is quite good about loading scripts and libraries from anywhere in its path - so there really is no reason -for- doing this. Reasons for avoiding it include:
  • The "upgrade" command will not work properly for already-installed py25-xxx ports that occupy $prefix/lib/python2.5 . Any of these currently installed ports must be first deactivated, then the new python upgraded, then those ports either activated or uninstalled/re-installed. At least in my testing, trying to activate those ports (individually, doing in a group didn't work) results in "Error: port activate failed: Not a directory", even though the port was successfully activated. While this might be a bug on 'port', it is off-putting to the end user.
  • It adds complexity to the install and could be confusing to end-users.
  • If it ain't broke, don't fix it: it's simple to allow for both of these paths to be part of the default PYTHONPATH, so why not leave things alone?

• in "post-patch", there are two "Mac/IDLE/Makefile.in" ... only one is needed.

• the manpages are not installed correctly, and do not work. I think the correct Portfile line is:

  	system "cd ${destroot}${prefix}/share/man/man1 && \
  		ln -sf ${prefix}/Library/Frameworks/Python.framework/Versions/2.5/share/man/man1/python.1.gz python2.5.2.1.gz"
  

• variant +universal does not work for me on Mac OS X 10.5.2, when doing "make libpython2.5.dylib". It looks like someone added $(LDFLAGS) to the dylib Makefile rule. When doing universal, the "-arch" command will be added to LDFLAGS, which isn't compatible with Apple's libtool libtool can handle this without any further flags. Please remove $(LDFLAGS) to fix this problem. You can check the resulting .dylib file for universality by executing "otool -vf XXX.dylib".
  • Even removing the LDFLAGS, there is still an install error for me, during destroot. I don't have time to debug this further right now.

• "destroot.target"
  • should not contain compiling or linking commands, such as "libpython2.5.dylib". This should be part of "build.target". Putting it here saves a line of Portfile code, but that 1 line really doesn't matter.
  • In the top-level Makefile, the target "frameworkinstall" is an alias for the target "install", which in turn does the "maninstall". Thus setting the "destroot.target" to "install" should be sufficient.

• python_select:
  • Since "python_select" isn't part of Python, the port-activate note should probably read something like "To fully complete your installation and make python 2.5 the default, please run 'sudo python_select python25' using the separate port 'python_select'." .
  • Running "python_select --" results in errors.
  • after removing python25, the symlinks created by python_select are still around. This is not a good idea.
  • While I like the intent of python_select, does MacPorts really want to be distributing this level of new, untested, code that leaves issues to be resolved?
  • I think moving this "feature" into the Python25 (or other) Portfile would be the wisest way ... that way the end results are owned by a Port, and can be manipulated by said Port.

[mdickens@…]

YA issue with the symlinks: I just uninstalled this revision of python25, after going through the process of deactivating and reactivating the py25-XXX ports I use. For some reason 'port' let me do this .. I don't think I did "port -f". Anyway, removing this test port removed the symlinks, which then removes my ability to properly uninstall the py25-XXX ports - there were residual files left in $prefix/Library/..../include and .../lib . I should -always- be able to properly uninstall ports, no matter the possibly odd state of whatever is installed. Using symlinks creates this possibility. This will be confusing, at best, to end-users.

[mdickens@…]

I also see that python_select somehow gets ownership (in the sense of "port provides foo") of /opt/local/bin/{pydoc, python, pythonw} and such. While this is a good thing, it is still awkward when doing "sudo port uninstall python25" and the residual symlinks owned by python_select hanging around. There has to be a better way.

[reiffert@…]

A roadmap for the python25 transition can be found here:

http://trac.macosforge.org/projects/macports/wiki/PythonFrameworkTransition

[sanchom@…]

Has there been any progress towards the framework build?

[raimue@…]

See this thread on macports-dev.

Also, I checked in a new version of python25 in the python-frameworks branch.

Reassigning to myself, because reiffert won't have time work on this anymore as reported on the thread linked above.

[mdickens@…]

I applaud the effort to "do the right thing" with all Python installs and "python_select"; I think the end result will be fantastic. That said, it is taking a very long time - now around 6 months since I joined this discussion, and more than 1 year since this ticket was opened - to get what would otherwise be a "simple" framework variant added to this port. I would have greatly preferred to have had a revision early on that allowed for the framework variant, such as that provided by my tarball, and then working towards "doing the right thing" as time allows. This is still my preference because I need the Python 2.5 framework for my work ... hence could whoever is in charge please "First make it work, then make it work well - while being as backwards compatible as possible to the first iteration" (in the words of some other OSS project managers I know). Thanks for listening.

[raimue@…]

Finally merged the new python25 port building as a framework to trunk in r37817.