New Features

• Support for new compiler: Microsoft Visual C++.

  You can now use Visual Studio 2008 or Visual Studio 2010 for compiling under Windows.

• Support for NetBSD.

  Nuitka works for at least NetBSD 6.0, older versions may or may not work. This required fixing bugs in the generic "fibers" implementation.

• Support for Python3 under Windows too.

  Nuitka uses Scons to build the generated C++ files. Unfortunately it requires Python2 to execute, which is not readily available to call from Python3. It now guesses the default installation paths of CPython 2.7 or CPython 2.6 and it will use it for running Scons instead. You have to install it to" C:Python26" or "C:Python27" for Nuitka to be able to find it.

• Enhanced Python 3.3 compatibility.

  The support the newest version of Python has been extended, improving compatibility for many minor corner cases.

• Added warning when a user compiles a module and executes it immediately when that references __name__.

  Because very likely the intention was to create an executable. And esp. if there is code like this:

  if __name__ == "__main__":
     main()
  

  In module mode, Nuitka will optimize it away, and nothing will happen on execution. This is because the command

  nuitka --execute module
  

  is behavioral more like

  python -c "import module"

  and that was a trap for new users.

• All Linux architectures are now supported. Due to changes in how evaluation order is enforced, we don't have to implement for specific architectures anymore.

Bug Fixes

• Dictionary creation was not fully compatible.

  As revealed by using Nuitka with CPython3.3, the order in which dictionaries are to be populated needs to be reversed, i.e. CPython adds the last item first. We didn't observe this before, and it's likely the new dictionary implementation that finds it.

  Given that hash randomization makes dictionaries item order undetermined anyway, this is more an issue of testing.

• Evaluation order for arguments of calls was not effectively enforced. It is now done in a standards compliant and therefore fully portable way. The compilers and platforms so far supported were not affected, but the newly supported Visual Studio C++ compiler was.

• Using a __future__ import inside a function was giving an assertion, instead of the proper syntax error.

• Python3: Do not set the attributes sys.exc_type, sys.exc_value, sys.exc_traceback.

• Python3: Annotations of function worked only as long as their definition was not referring to local variables.

New Optimization

• Calls with no positional arguments are now using the faster call methods.

  The generated C++ code was using the () constant at call site, when doing calls that use no positional arguments, which is of course useless.

• For Windows now uses OS "Fibers" for Nuitka "Fibers".

  Using threads for fibers was causing only overhead and with this API, MSVC had less issues too.

Organizational

• Accepting Donations via Paypal, please support funding travels, website, etc.
• The User Manual has been updated with new content. We now do support Visual Studio, documented the required LLVM version for clang, Win64 and modules may include modules too, etc. Lots of information was no longer accurate and has been updated.
• The Changelog has been improved for consistency, wordings, and styles.
• Nuitka is now available on the social code platforms as well
  • Bitbucket
  • Github
  • Gitorious
  • Google Code
• Removed "clean-up.sh", which is practically useless, as tests now clean up after themselves reasonably, and with git clean -dfx working better.
• Removed "create-environment.sh" script, which was only setting the PATH variable, which is not necessary.
• Added check-with-pylint --emacs option to make output its work with Emacs compilation mode, to allow easier fixing of warnings from PyLint.
• Documentation is formatted for 80 columns now, source code will gradually aim at it too. So far 90 columns were used, and up to 100 tolerated.

Cleanups

• Removed useless manifest and resource file creation under Windows.

  Turns out this is no longer needed at all. Either CPython, MinGW, or Windows improved to no longer need it.

• PyLint massive cleanups and annotations bringing down the number of warnings by a lot.

• Avoid use of strings and built-ins as run time pre-computed constants that are not needed for specific Python versions, or Nuitka modes.

• Do not track needed tuple, list, and dict creation code variants in context, but e.g. in nuitka.codegen.TupleCodes module instead.

• Introduced an "internal" module to host the complex call helper functions, instead of just adding it to any module that first uses it.

New Tests

• Added basic tests for order evaluation, where there currently were None.
• Added support for "2to3" execution under Windows too, so we can run tests for Python3 installations too.

Summary

The release is clearly major step ahead. The new platform support triggered a whole range of improvements, and means this is truly complete now.

Also there is very much polish in this release, reducing the number of warnings, updated documentation, the only thing really missing is visible progress with optimization.

Things to talk about

Perspective

Last time, I spent a lot of time on justification, "why a new project?", "why not work with the others?", what goals do I have that others do not. Giving examples of how code generation works. Generally to give people an idea of the project.

With this out of the way, I can now focus on inclusion, and success.

Funding

And, well yes, this time I may not have to pay for it all by myself. Last time I spent close to 1000 Euros for the trip (ticket to enter, hotel, flight, food), because I am accepting donations for this specific reason.

For a strange reason, I devote substantial amounts of time to the project, only to put it under the most liberal license. It's probably fair to allow people to make donations if they feel they want to further the project, but don't know how. Or if they just consider it too important for me to loose interest. That kind of feels unlikley though. Too much fun.

Final Picture

And lets have an image I made during Europython 2012 in the city of Florence. It shows what a vibrant place this town is.

Florence is a place full of kind people. The mood not only of the conference, but the whole city is very open minded and helpful. It was very easy to get adopted by strangers to their party.

Final Words

I am looking forward to meeting the friends I made there last time, and new friends. I kind of a great time there last time, one of these "times of my life". Even if the reception was not always as warm as I had deserved. I remember laughing a lot, learning a lot. And making unique experiences.

New Features

• Support for FreeBSD.

  Nuitka works for at least FreeBSD 9.1, older versions may or may not work. This required only fixing some "Linuxisms" in the build process.

• New option for warning about compile time detected exception raises.

  Nuitka can now warn about exceptions that will be raised at run time.

• Basic Python3.3 support.

  The test suite of CPython3.2 passes and fails in a compatible way. New feature "yield from" is not yet supported, and the improved argument parsing error messages are not implemented yet.

Bug Fixes

• Nuitka already supported compilation of "main directories", i.e. directories with a "__main__.py" file inside. The resulting binary name was "__main__.exe" though, but now it is "directory.exe"

  # ls directory
  __main__.py
  
  # nuitka --exe directory
  # ls
  directory directory.exe
  

  This makes this usage more obvious, and fixes the older issue Issue#49 for this feature.

• Evaluation order of binary operators was not enforced.

  Nuitka already enforces evaluation order for just about everything. But not for binary operators it seems. Corrects Issue#61.

• Providing an # coding: no-exist was crashing under Python2, and ignored under Python3, now it does the compatible thing for both.

• Global statements on the compiler level are legal in Python, and were not handled by Nuitka, they now are.

  global a # Not in a function, but on module level. Pointless but legal!
  a = 1
  

  Effectively these statements can be ignored. Corrects part of Issue#65.

• Future imports are only legal when they are at the start of the file. This was not enforced by Nuitka, making it accept code, which CPython would reject. It now properly raises a syntax error. Corrects part of Issue#65.

• Raising exceptions from context was leaking references.

  raise ValueError() from None
  

  Under CPython3.2 the above is not allowed (it is acceptable starting CPython3.3), and was also leaking references to its arguments. Corrects Issue#76.

• Importing the module that became __main__ through the module name, didn't recurse to it.

  This also gives a warning. PyBench does it, and then stumbles over the non-found "pybench" module. Of course, programmers should use sys.modules[ "__main__" ] to access main module code. Not only because the duplicated modules don't share data. Corrects Issue#68.

• Compiled method repr leaked references when printed.

  When printing them, they would not be freed, and subsequently hold references to the object (and class) they belong to. This could trigger bugs for code that expects __del__ to run at some point. Corrects Issue#81.

• The super built-in leaked references to given object.

  This was added, because Python3 needs it. It supplies the arguments to super automatically, whereas for Python2 the programmer had to do it. And now it turns out that the object lost a reference, causing similar issues as above, preventing __del__ to run. Corrects Issue#81.

• The raise statement didn't enforce type of third argument.

  This Python2-only form of exception raising now checks the type of the third argument before using it. Plus, when it's None (which is also legal), no reference to None is leaked.

• Python3 built-in exceptions were strings instead of exceptions.

  A gross mistake that went uncaught by test suites. I wonder how. Them being strings doesn't help their usage of course, fixed. Corrects Issue#82.

• The -nan and nan both exist and make a difference.

  A older story continued. There is a sign to nan, which can be copied away and should be present. This is now also supported by Nuitka. Corrects Issue#75.

• Wrong optimization of a == a, a != a, a < a on C++ level.

  While it's not done during Nuitka optimization, the rich comparison helpers still contained short cuts for "==", "!=", and "<".

• The sys.executable for nuitka-python --python-version 3.2 was still python.

  When determining the value for sys.executable the CPython library code looks at the name exec had received. It was python in all cases, but now it depends on the running version, so it propagates.

• Keyword only functions with default values were loosing references to defaults.

  def f( *, a = X() )
     pass
  
  f()
  f() # Can crash, X() should already be released.
  

  This is now corrected. Of course, a Python3 only issue.

• Pressing CTRL-C didn't generate KeyboardInterrupt in compiled code.

  Nuitka never executes "pending calls". It now does, with the upside, that the solution used, appears to be suitable for threading in Nuitka too. Expect more to come out of this.

• For with statements with return, break, or continue to leave their body, the __exit__ was not called.

  with a:      # This called a.__enter__().
      return 2 # This didn't call a.__exit__( None, None, None ).
  

  This is of course quite huge, and unfortunately wasn't covered by any test suite so far. Turns out, the re-formulation of with statements, was wrongly using try/except/else, but these ignore the problematic statements. Only try/finally does. The enhanced re-formulation now does the correct thing. Corrects Issue#59.

• Starting with Python3, absolute imports are now the default.

  This was already present for Python3.3, and it turns out that all of Python3 does it.

New Optimization

• Constants are now much less often created with pickle module, but created directly.

  This esp. applies for nested constants, now more values become is identical instead of only == identical, which indicates a reduced memory usage.

  a = ( "something_special", )
  b = "something_special"
  
  assert a[0] is b # Now true
  

  This is not only about memory efficiency, but also about performance. Less memory usage is more cache friendly, and the "==" operator will be able to shortcut dramatically in cases of identical objects.

  Constants now created without pickle usage, cover float, list, and dict, which is enough for PyStone to not use it at all, which has been added support for as well.

• Continue statements might be optimized away.

  A terminal continue in a loop, was not optimized away:

  while 1:
      something
      continue   # Now optimized away
  

  The trailing continue has no effect and can therefore be removed.

  while 1:
      something
  

• Loops with only break statements are optimized away.

  while 1:
      break
  

  A loop immediately broken has of course no effect. Loop conditions are re-formulated to immediate "if ... : break" checks. Effectively this means that loops with conditions detected to be always false to see the loop entirely removed.

New Tests

• Added tests for the found issues.
• Running the programs test suite (i.e. recursion) for Python3.2 and Python3.2 as well, after making adaptation so that the absolute import changes are now covered.
• Running the "CPython3.2" test suite with Python3.3 based Nuitka works and found a few minor issues.

Organizational

• The Downloads page now offers RPMs for RHEL6, CentOS6, F17, F18, and openSUSE 12.1, 12.2, 12.3. This large coverage is thanks to openSUSE build service and "ownssh" for contributing an RPM spec file.

  The page got improved with logos for the distributions.

• Added "ownssh" as contributor.

• Revamped the User Manual in terms of layout, structure, and content.

Summary

This release is the result of much validation work. The amount of fixes the largest of any release so far. New platforms, basic Python3.3 support, consolidation all around.

New Features

• Python 3.3 experimental support

  • Now compiles many basic tests. Ported the dictionary quick access and update code to a more generic and useful interface.
  • Added support for __qualname__ to classes and functions.
  • Small compatibility changes. Some exceptions changed, absolute imports are now default, etc.
  • For comparison tests, the hash randomization is disabled.

• Python 3.2 support has been expanded.

  The Python 3.2 on Ubuntu is not providing a helper function that was used by Nuitka, replaced it with out own code.

Bug fixes

• Default values were not "is" identical.

  def defaultKeepsIdentity( arg = "str_value" ):
      print arg is "str_value"
  
  defaultKeepsIdentity()
  

  This now prints "True" as it does with CPython. The solution is actually a general code optimization, see below. Issue#55

• Usage of unicode built-in with more than one argument could corrupt the encoding argument string.

  An implementation error of the unicode was releasing references to arguments converted to default encoding, which could corrupt it.

• Assigning Python3 function annotations could cause a segmentation fault.

New Optimization

• Improved propagation of exception raise statements, eliminating more code. They are now also propagated from all kinds of expressions. Previously this was more limited. An assertion added will make sure that all raises are propagated. Also finally, raise expressions are converted into raise statements, but without any normalization.

  # Now optimizing:
  raise TypeError, 1/0
  # into (minus normalization):
  raise ZeroDivisionError, "integer division or modulo by zero"
  
  # Now optimizing:
  (1/0).something
  # into (minus normalization):
  raise ZeroDivisionError, "integer division or modulo by zero"
  
  # Now optimizing:
  function( a, 1/0 ).something
  # into (minus normalization), notice the side effects of first checking
  # function and a as names to be defined, these may be removed only if
  # they can be demonstrated to have no effect.
  function
  a
  raise ZeroDivisionError, "integer division or modulo by zero"
  

  There is more examples, where the raise propagation is new, but you get the idea.

• Conditional expression nodes are now optimized according to the truth value of the condition, and not only for compile time constants. This covers e.g. container creations, and other things.

  # This was already optimized, as it's a compile time constant.
  a if ( "a", ) else b
  a if True else b
  
  # These are now optimized, as their truth value is known.
  a if ( c, ) else b
  a if not (c, ) else b
  

  This is simply taking advantage of infrastructure that now exists. Each node kind can overload "getTruthValue" and benefit from it. Help would be welcome to review which ones can be added.

• Function creations only have side effects, when their defaults or annotations (Python3) do. This allows to remove them entirely, should they be found to be unused.

• Code generation for constants now shares element values used in tuples.

  The general case is currently too complex to solve, but we now make sure constant tuples (as e.g. used in the default value for the compiled function), and string constants share the value. This should reduce memory usage and speed up program start-up.

Cleanups

• Optimization was initially designed around visitors that each did one thing, and did it well. It turns out though, that this approach is unnecessary, and constraint collection, allows for the most consistent results. All remaining optimization has been merged into constraint collection.
• The names of modules containing node classes were harmonized to always be plural. In the beginning, this was used to convey the information that only a single node kind would be contained, but that has long changed, and is unimportant information.
• The class names of nodes were stripped from the "CPython" prefix. Originally the intent was to express strict correlation to CPython, but with increasing amounts of re-formulations, this was not used at all, and it's also not important enough to dominate the class name.
• The re-formulations performed in tree building have moved out of the "Building" module, into names "ReformulationClasses" e.g., so they are easier to locate and review. Helpers for node building are now in a separate module, and generally it's much easier to find the content of interest now.
• Added new re-formulation of print statements. The conversion to strings is now made explicit in the node tree.

New Tests

• Added test to cover default value identity.

Organizational

• The upload of Nuitka to PyPI has been repaired and now properly displays project information again.

Summary

The quicker release is mostly a consolidation effort, without actual performance progress. The progress towards SSA form matter a lot on the outlook front. Once this is finished, standard compiler algorithms can be added to Nuitka which go beyond the current peephole optimization.