Get MacroPy working on Python 3.4

[lihaoyi]

I have no idea how hard this will be, but MacroPy does not have many dependencies on the underlying runtime except for PEP302 and the ast library.

Some of the ASTs look slightly different (e.g. functions can't have nested parameter lists, no print statement) and we may have to remove any print statements from the implementation code, but I don't imagine it will be very difficult or require big changes.

[lihaoyi]

It looks like the import hook mechanism works somewhat differently, although it still exists. That will need to be rewritten too.

[macobo]

I'm currently working on this.

An issue I ran into - the AST of With statement has changed in a way that breaks the current block macros. To demonstrate:

Python 2.7:

>>> print(dump(parse("with a as b, c: pass").body[0]))
With(context_expr= Name(id='a', ctx=Load()), 
     optional_vars=Name(id='b', ctx=Store()), 
     body=[With(context_expr=Name(id='c', ctx=Load()), 
                optional_vars=None, 
                body=[Pass()])])

Python 3.4:

>>> print(dump(parse("with a as b, c: pass").body[0]))
With(items=[withitem(context_expr= Name(id='a', ctx=Load()), 
                     optional_vars=Name(id='b', ctx=Store())), 
            withitem(context_expr= Name(id='c', ctx=Load()), 
                     optional_vars=None)], 
     body=[Pass()])

Since I'm not quite sure on how to handle this (what are the semantics of multiple block macros within a single with-statement?), I'd like to hear what would be the nice way of solving this?

[lihaoyi]

It looks to me that the primary difference is that you now have a list of (context -> var)s rather than a single one. The correct thing to do would be to abstract the AST of the with statement away from the macro, providing the macro with the body (and maybe optional_var) directly rather than have it reach into the With() object to dig it out itself.

That will require some pretty fundamental changes to the expand_ast function, as well as updating the various macros to work with the new interface. On the other hand, we'll probably want to do these changes at some point anyway. Give me a few days and I'll get these changes in.

[lihaoyi]

This has been done; now MacroPy takes care of extracting the useful bits of information from the with ... as ... statement, and the macros can just get this information (as well as other things) as arguments. In particular if you need to handle a different-looking With() AST, you should look at funging the code here

[macobo]

python3.4 branch

Finally got around to do this (mainly because I would like to use macropy for another project) and there's been some progress.
Sadly, I've been stumped today by the import hooks not working correctly - adding MacroFinder to meta_path, all imports seem to fail. I didn't find anything to indicate the cause or help debug this in the docs, perhaps you have some ideas?

[lihaoyi]

Not sure; I reached that point too w.r.t 3.4 support before getting stuck. Presumably 3.4 uses a different mechanism for import hooks from 2.7, but I didn't get to the point where I know what the problem is or how to fix it.

I'm gonna be travelling, so probably won't be responsive for the next few days.

[macobo]

Made significant progress on this - the core tests pass (except one which I commented out and exporters), also quick_lambda and trace macros work both under PY2 and PY3.
Since this is all I require for my project, I won't be porting stuff for a while. Hopefully that's enough so porting the rest won't be a pain.

Note that the library six is now required for the PY3 flag: pip install six

[lihaoyi]

Cool =) Here's to hoping you get tempted enough by case classes and other things to port those too. Otherwise, I'll take a look at your branch and see what I can do next time I have time to work on this.

[macobo]

Note that lots of the changes I did are not very pretty (using the PY3 flag is usually a good indication of that). I did clean up the import_hooks however.

Main thing that caused headaches is that the arguments ast has changed (this is also what breaks case classes), but it isn't always obvious where it is used and where the arguments come from. Other changes are more relatively straight-forward, e.g. wrapping map() in list() and so forth.

Also, the literal_eval function changed in py3, breaking current trace macro. The quick-fix was to just copy the old python2 literal_eval to tracing.py. macobo@0075708

Anyways, when you have time and want to review the branch and perhaps merge, let me know. I'd be glad to fix up the style or other issues you might find.

[ChadSki]

I am also interested in using MacroPy in a Python 3 project of mine.

@macobo I downloaded and tested your "python3" branch (tested withPython 3.2 x86 Windows), but I have errors involving undefined symbols.

  File "macropy-python3\macropy\core\__init__.py", line 219, in <module>
    YieldFrom:  lambda tree, i: "(yield from " + rec(tree.value, i) + ")",
NameError: name 'YieldFrom' is not defined

In order to get the tests to run, I had to comment out YieldFrom, Try, and withitem from the trec.update({...}) dictionary. The result of running the tests is FAILED (failures=1, errors=35).

Any insight into why Python is failing to find the YieldFrom, Try, and withitem symbols?

[lihaoyi]

I can't really speak for macobo, since i havent worked with the Py3 branch, but YieldFrom only exists in Python 3.3 and above if i read the docs right. Not sure about the other missing symbols.

[macobo]

@ChadSki Yep, like lihaoyi said the problem is that Python AST changed (again) in Python 3.3. Getting it to work in Python 3.3 was already a hack, so I didn't invest any time into getting it to work under 3.0 <= x <= 3.2.

However, you are welcome to continue my half-made port as I currently don't have the motivation or the time 👍.

[fjarri]

Any updates on this? I would really like to use macropy in Py3 (currently I mostly need basic stuff, like core functions and Walker), but the current PyPi version (1.03) won't even install (because of the relative imports). What about @macobo 's Py3 branch, what's left to be fixed and can its current state be merged?

[lihaoyi]

I personally haven't had the time or motivation to work on it (I'm not using MacroPy on any projects right now at the moment) so it's kind of been on ice =(. The Py3 branch mostly works, but will need some more work just going over the existing test suite and making them pass.

The hard/icky work of getting the import hooks and everything compatible has already been done on @macobo 's branch, which is still up to date since I haven't done much with the main branch since he started on his.

[fjarri]

I wouldn't mind doing that, but is it possible to merge the @macobo 's branch before that so I could base on it?

[lihaoyi]

It's kind of in a incomplete-state; would forking his branch directly work? It's basically up to date, except for a single typo in the readme that was fixed since he branched off. I can answer any questions you have w.r.t. architecture and things, but probably don't have the time/interest to do the merge myself right now unless I find a Python 3.4 project to use macropy in.

[fjarri]

I'm not sure it'll work, it seems that I can only send pull request to @macobo and not to you directly, and he seems not to be around. For me, on Py3.3.2 all the tests pass with his branch after I fix this minor thing.

[macobo]

You could make a patch out of my branch and directly apply it to your freshly-made copy of this repo. This would lose the commit history, so I'm not sure how @lihaoyi would like that.

Another possibility would be that I could give you commit rights to my fork?

EDIT: just remembered, you can also change the upstream of a cloned repository - this would allow you to push changes from a clone of my fork to your own fork of this repo.

[fjarri]

Or maybe I'll just send PR to you, you merge it and send PR to @lihaoyi? :)

[macobo]

Yeah, that would also work for me.

[lihaoyi]

I don't mind losing commit history, so you could just Copy&Paste the tree from @macobo's branch onto your own that you cloned from trunk. Then you could send the PR directly to trunk. =)

Another alternative is you could fork trunk and pull from @macobo's branch, and I believe it'd let you send the PR to me too while keeping history.

I had no idea 3.4 was so close to all tests passing (save for the experimental ones, but I don't care about those much).

What I suggest is you start with @macobo's branch, fix it up to a working state, and run with it for a while (few weeks?) on whatever project you're using it for. I bet you're going to bump into bugs/rough-edges/things-to-fix while actually doing real stuff with it, and it'll be easier for you to fix locally rather than going through the whole pr/review/merge/publish-to-PyPI/update-setup.py thing each time you want a bugfix. That way you can quickly fix up/tweak whatever you want on your side and keep the number of PR-round-trips to a minimum

WDYT?

[fjarri]

Since @macobo is reading this, I don't see why can't we do it the normal way. I've sent him a PR, he'll merge it and send the PR with his python3 branch to you. No need to tinker with commits.

I had no idea 3.4 was so close to all tests passing (save for the experimental ones, but I don't care about those much).

I must note that I've tested it on 3.3, not 3.4. 3.4 is in beta stage now, anyway, and does not seem to have any changes related to the scope of macropy.

What I suggest is you start with @macobo's branch, fix it up to a working state, and run with it for a while (few weeks?) on whatever project you're using it for.

The thing is, I only need very basic functionality, mostly Walker and unparse. I won't notice the majority of macro-related bugs, if there are any. I also think it is better to have some Py3 support than not having any. It can even be a separate branch, if you want to be extra-careful — but this branch should be in the root repo so that I could create many small independent PRs instead of a single giant one.

[lihaoyi]

Sounds good to me =)

On Thu, Jan 30, 2014 at 5:59 PM, Bogdan Opanchuk
[email protected]:

Since @macobo https://github.com/macobo is reading this, I don't see
why can't we do it the normal way. I've sent him a PR, he'll merge it and
send the PR with his python3 branch to you. No need to tinker with
commits.

I had no idea 3.4 was so close to all tests passing (save for the
experimental ones, but I don't care about those much).

I must note that I've tested it on 3.3, not 3.4. 3.4 is in beta stage now,
anyway, and does not seem to have any changes related to the scope of
macropy.

What I suggest is you start with @macobo https://github.com/macobo's
branch, fix it up to a working state, and run with it for a while (few
weeks?) on whatever project you're using it for.

The thing is, I only need very basic functionality, mostly Walker and
unparse. I won't notice the majority of macro-related bugs, if there are
any. I also think it is better to have some Py3 support than not having
any. It can even be a separate branch, if you want to be extra-careful --
but this branch should be in the root repo so that I could create many
small independent PRs instead of a single giant one.

Reply to this email directly or view it on GitHubhttps://github.com//issues/32#issuecomment-33674025
.

[lihaoyi]

@macobo 's branch has been pulled into https://github.com/lihaoyi/macropy/commits/python3, which isn't quite complete, but already supports a lot of things

[ceridwen]

What still needs to be done for this branch? I'd like to explore MacroPy on 3.4.

[lihaoyi]

Take the python3 branch and see whether your can build it and what breaks ^_^

[f1shbot]

From the first example:

Python 3.4.2 (default, Jan 12 2015, 11:38:40)
[GCC 4.9.2 20141224 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import macropy.console
0=[]=====> MacroPy Enabled <=====[]=0
from macropy.case_classes import macros, case
@case
... class Point(x, y): pass
...
Traceback (most recent call last):
File "", line 1, in
File "/home/sfisher/dev/macropy/macropy/core/failure.py", line 17, in raise_error
raise ex
macropy.core.failure.MacroExpansionError: arguments constructor takes either 0 or 6 positional arguments
. . .

I'll look at this a bit more when I get home.

[ceridwen]

I've been working on this. Before I start making pull requests, would you prefer if I lump everything together into one request or break it up into smaller logical units?

[lihaoyi]

Just lump it all together ^_^

[scherrey]

Also interested in the status of python 3 support and the future of this promising library.

[raingloom]

I would also love to see this lib on Python 3

[azazel75]

I'm working on the port on my master branch, where I have merged @lihaoyi 's master and the python3 branch and also merged in contribution by some other developer that started working on this long before me but left the ranch in an incomplete, non functional state.

In my master branch there's no python 2 compatibility code ( there is still some left but i'm going to drop it ) and it should work on python 3.5+. All of the core tests pass and mosto of the other as well, there is just someting wrong with the PEG tests

[lihaoyi]

@azazel75 👍 👍 👍

[azazel75]

all the tests pass, i've also updated pinq to work... Now there is the need to update the code here and there to take into account the "async" variant of some statement and expressions, like async for, async with, async comprehensions

[lihaoyi]

@azazel75 @thedrow if you guys want to do serious work on this, I can give both of you access to this repo. You could also fork it, but working on the main repo would make sure your works gets the attention of anyone who's interested in this project, which would be more difficult on a fork.

I could probably give you publishing access to PyPI too, except that it's been so long ago I don't remember how PyPI works or what (if any) my credentials are...

[azazel75]

Well, it's still unfinished and it's my take on macropy. I mean that as i have no time to debug it on non current pythons, I dropped all the Python 2 and Python 3.4- (maybe also with 3.4 included, i have to check out this to be sure) code. This is mostly beacuse function's arguments have changed very much in their format, and there was code to cast them as they were defined in Py2 and this was causing some problems. While using tools like astroid maybe a solution, it is really poorly documented and at the same time macropy seemed a bit... abandoned. So I draw a line and started fresh with Python 3.5 and 3.6 as targets. This was a decision I made because i needed the tool and it may not be the general consensus on what macropy should target as interpreters. So i'm not so fond that a PR is the right thing to do... in the meantime you can always add a remote to my repo with git remote add azazels [email protected]:azazel75/macropy.git, but I'm sure you know already.

[Daenyth]

I'd also be interested in helping with python3 support and general development. I feel like this could make my work stuff easier, and I'm quite familiar with 2+3 compatibility issues and testing approaches.

[lihaoyi]

Going forward on latest version of Python 3.6 seems totally reasonable to me. Python3 is the future.

[Daenyth]

I'd be happy to help either with 2.7/3.5+ support or only 3.6+ support. There's merits for both. 3.6 brings some real benefits with f-strings, async support, type annotations / mypy, and typing.NamedTuple class syntax

[azazel75]

@lihaoyi as you are pleased, I've started using it in my python to js transpiler and want to use it to have a more layered approach on transpiling, with the help of type inference (and typescript definitions), so i'll give it some love ;-)

[azazel75]

@thedrow I think that we should reach the goal to run with a current python and fully support it. Then we can decide where to go from there. About astroid, it frightens me to se how many outstanding bugs there are and I think that if the goals are to harmonize Py2 and Py3 calling conventions in a way that informations present in Py3 ast aren't lost or degraded and to have a somewhat more friendly ast base that plays well and is more hackable is no such big deal, but it's a "not now" for me, the port has to be completed first, together with doc refreshing to let interested users know what's going on.

About support for Py2.7, well, it's dead for me. I don't use it anymore and I'm not going to maintain support for it. If you really use it and you want to spend hours debugging on it the I'll leave it to you ;-)

[azazel75]

There is good documentation around about the differences on each version's ast

[azazel75]

So if you want to help, start from my branch, try it out, add support for async stuff, and so on, then we will wait for @lihaoyi decision and we'll decide .... where to merge ;-) About the doc (the readme), i'm in the process of converting it to rst, because I'm more familiar with it and also think that there is too much in that readme that can generate some confusion... reading an intro that never ends :-) The format change could come in handy for a later split up in an overview+intro and and a sphinx doc

[azazel75]

ah, there is also JoinedStr from 3.6.... that the PyPy guys backported to their 3.5 version of the interpreter... wtf, anyway... its doc is here http://greentreesnakes.readthedocs.io/en/latest/nodes.html?#JoinedStr

[ceridwen]

I've been really busy in real life, so I haven't been able to take updating Macropy to Python 3 as far as I'd like. I have some work in https://github.com/ceridwen/macropy, but there's still quite a ways to go. I would encourage anyone hoping to port it to look at it, since I fixed some annoying bugs.

I've also done a lot of work at improving astroid's AST for use with Macropy, but there's still more to be done and realistically I don't know when I'm going to get to it. I just started a new job, so my free time for open source is quite limited right now, and the other major astroid maintainer is also busy. I do think that using astroid would have advantages. Python 2.7's EOL isn't until 2020, and there's enormous amounts of legacy code that still exists. As a library author, I still feel compelled to write Python 2/3 compatible code. That said, there are also some difficulties. Here are the biggest ones I've run into:

• astroid's nodes have pointers to their parents as well as their children. This used to be much worse to work with, I've improved it by creating real __init__() and postinit() methods for every node class, but initialization still takes two steps, which will require reworking all the places where nodes are built in Macropy. I have some future plans to turn this into a singly-linked tree, but I haven't finished them because they require extensive refactoring.

• astroid uses the standard library AST parser and then transforms the tree to convert it into an astroid AST. Custom parsers aren't possible at the moment.

• astroid's static analysis is deeply entwined with the rest of the AST code in some ways that are particular problems for Macropy. The worst is that astroid stores information about run-time properties like local variables on the AST nodes themselves, and it adds these attributes to the nodes during tree creation using the static analysis code. If you create an astroid tree without creating these attributes, the static analysis code could misfire badly if it's called on any tree containing these new nodes. However, I don't know what will happen if you tried to call the attribute-assignment on the root of a tree with attributes already assigned to it. astroid actually does some Macropy-style tree mutation to handle some libraries that do things beyond the capability of its static analysis to analyze and has run into some problems with doing it in a way that doesn't break the static analysis. The right solution to this is to not use mutable objects on the nodes themselves for the static analysis, but that will take another large amount of work. Macropy does some limited static analysis itself, which could be replaced with astroid static analysis, but only if these other issues are solved.

• astroid is almost entirely undocumented.

Aside from getting Python 2/3 compatibility, there are also some other benefits:

• Macropy currently has its own methods for converting ASTs into source code, but astroid provides that functionality. I found a number of bugs in Macropy's implementation in my testing, and there are probably others that I haven't found yet. astroid's implementation is better-tested and has fewer bugs.

• The Python AST is a moving target, it's changed in every recent Python3 release. Keeping up with Python 3 support in astroid and Macropy involves duplication of effort.

If anyone wants further advice on this or to ask questions about astroid, I can answer them.

[azazel75]

Thanks for your work @ceridwen, I based my branch on yours so nothing has been lost.

You say that you found bugs in macropy ast conversion code... do you remember them?
Is there any alternative to astroid? (Already done and maintained, i mean)

[ceridwen]

It's been long enough that I don't, unfortunately. I'll look through my notes and see if I wrote anything down about them.

There isn't really an alternative to astroid, unfortunately. There are some other libraries that do static analysis in service of refactoring or code completion with their own tree classes, but those are (AFAIK) concrete syntax trees that are even more inappropriate for Macropy than they are for Pylint. There's been some discussion about collaborating to improve Python's collection of tree libraries that's never gotten off the ground because of the different needs of static analysis, code completion, and refactoring tools and because of lack of time on the part of the authors.

[lihaoyi]

If you really want to make MacroPy work cross-{2.7,3.6}, doesn't MyPy have cross-version parsers and a cross-version AST module that you could C&P into MacroPy? My understanding is that MyPy works for both 2.7 and 3.6, and I imagine their "main" type-checking logic isn't implemented twice.

[azazel75]

I think they use 'typed-ast', or was it named something else?

[ceridwen]

Yeah, mypy now uses two parsers that are part of the typed-ast package. As far as I know, they're modifications of the CPython 2 and 3 parsers. Their Github page says they have experimental support for converting Python 2 ASTs into 3 ASTs, but I don't think that helps for Macropy.

[azazel75]

@thedrow, Travis is enabled now.

What it lacks mostly is a full coverage of the Python 3.5-3.6 syntax, in https://github.com/azazel75/macropy/blob/master/macropy/core/__init__.py we miss handling of async ( for, with, async compehension), await, and the already mentioned ast.JoinedStr node ( the node for f"Hello from {planet}"), there is some marked TODO: that needs to be completed :-/. The fact is that the testsuite tests only a subset of the nodes available in a language release. I was thinking about a way to test this, about testing what's handled by macropy at least in terms of parsing/repr capability and know what's missing automatically, I have similar needs in https://github.com/azazel75/metapensiero.pj ( see https://github.com/azazel75/metapensiero.pj/blob/master/tests/test_ast/test_ast_dump/fstring_format_3.6.0.py and https://github.com/azazel75/metapensiero.pj/blob/master/tests/test_ast/test_ast_dump/fstring_format_3.6.0.ast for example), and I was thinking about sharing some or part of the testuite and the overall value/cost of such a test.
The code also maybe needs to be aware of them. I've added ast.AsyncFunctionDef where ast.Class and ast.FunctionDef where considered, maybe there are other details i'm not considering.

[fferri]

Try https://gitter.im it has nice integration with GitHub and it will send you email notifications of unread messages.

[azazel75]

I'm azazel on freenode's IRC. I'm always connected during work days, i'll join a channel named #macropy laterali today or tomorrow Il 17 giugno 2017 10:57:30 CEST, Omer Katz <[email protected]> ha scritto:

…

I run all the tests and everything passes on 3.6. What is missing? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #32 (comment)

-- Inviato dal mio dispositivo Android con K-9 Mail. Perdonate la brevità.

[azazel75]

If you prefer anything else, just let me know.

[lihaoyi]

I went and opened a gitter chat room for lihaoyi/macropy:

• https://gitter.im/lihaoyi/macropy

You guys can use it, or not, up to you

[azazel75]

I've completed the port to Python 3.4+ and I've revamped the docs... I've already published my fork under the name macropy3 because I needed it in another project and I needed it be published. I can submit a pull request I you guys think it's better (or you can just merge it) let me know... the docs are here http://macropy3.readthedocs.io/en/latest/ and they are still in the works ;-) let me know what you think

[azazel75]

I'm closing this as Python 3.4+ support has landed