From 4e26cf8cf10c04c94965cbf5507509bc05fd1b9e Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Mon, 26 Aug 2019 14:24:42 -0700 Subject: [PATCH 01/31] Moved documentation to separate files. In order to reduce the size of README.md, I moved some documentation to their own files, and updated README.md with appropriate navigation. --- README.md | 2769 +------------------------------------------ doc/design-notes.md | 100 ++ doc/features.md | 2642 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 2760 insertions(+), 2751 deletions(-) create mode 100644 doc/design-notes.md create mode 100644 doc/features.md diff --git a/README.md b/README.md index 2b7d0002..01c4421a 100644 --- a/README.md +++ b/README.md @@ -1,2781 +1,49 @@ # Unpythonic: Python meets Lisp and Haskell -In the spirit of [toolz](https://github.com/pytoolz/toolz), we provide missing features for Python, mainly from the list processing tradition, but with some haskellisms mixed in. We place a special emphasis on **clear, pythonic syntax**. +In the spirit of [toolz](https://github.com/pytoolz/toolz), we provide missing features for Python, mainly from the list processing tradition, but with some Haskellisms mixed in. We place a special emphasis on **clear, pythonic syntax**. We also provide extensions to the Python language as a set of [syntactic macros](https://en.wikipedia.org/wiki/Macro_(computer_science)#Syntactic_macros) that are designed to work together. The macros provide features such as *automatic* currying, *automatic* tail-call optimization, call-by-need (lazy functions), continuations (``call/cc``), lexically scoped ``let`` and ``do`` with lean syntax, implicit return statements, and easy-to-use multi-expression lambdas with local variables. Each macro adds an orthogonal piece of functionality that can (mostly) be mixed and matched with the others. -Design considerations are simplicity, robustness, and minimal dependencies. Currently none required; [MacroPy](https://github.com/azazel75/macropy) optional, to enable the syntactic macros. +Design considerations are based in simplicity, robustness, and with minimal dependencies. See our [design notes](doc/design-notes.md) for more information. -This README documents the pure-Python part of ``unpythonic``. See also [documentation for the macros](macro_extras/). +### Dependencies -**Contents**: +Currently none required; [MacroPy](https://github.com/azazel75/macropy) optional, to enable the syntactic macros. - - [**Bindings**](#bindings) - - [``let``, ``letrec``: local bindings in an expression](#let-letrec-local-bindings-in-an-expression) - - [Lispylet: alternative syntax](#lispylet-alternative-syntax) - - [``env``: the environment](#env-the-environment) - - [``assignonce``](#assignonce), a relative of ``env``. - - [``dyn``: dynamic assignment](#dyn-dynamic-assignment) a.k.a. parameterize, special variables, "dynamic scoping". - - - [**Containers**](#containers) - - [``frozendict``: an immutable dictionary](#frozendict-an-immutable-dictionary) - - [`cons` and friends: pythonic lispy linked lists](#cons-and-friends-pythonic-lispy-linked-lists) - - [``box``: a mutable single-item container](#box-a-mutable-single-item-container) - - [Container utilities](#container-utilities): ``get_abcs``, ``in_slice``, ``index_in_slice`` - - - [**Sequencing**](#sequencing), run multiple expressions in any expression position (incl. inside a ``lambda``). - - [``begin``: sequence side effects](#begin-sequence-side-effects) - - [``do``: stuff imperative code into an expression](#do-stuff-imperative-code-into-an-expression) - - [``pipe``, ``piped``, ``lazy_piped``: sequence functions](#pipe-piped-lazy_piped-sequence-functions) - - - [**Batteries**](#batteries) missing from the standard library. - - [**Batteries for functools**](#batteries-for-functools): `memoize`, `curry`, `compose`, `withself` and more. - - [``curry`` and reduction rules](#curry-and-reduction-rules): we provide some extra features for bonus haskellness. - - [**Batteries for itertools**](#batteries-for-itertools): multi-input folds, scans (lazy partial folds); unfold; lazy partial unpacking of iterables, etc. - - [``islice``: slice syntax support for ``itertools.islice``](#islice-slice-syntax-support-for-itertoolsislice) - - [`gmemoize`, `imemoize`, `fimemoize`: memoize generators](#gmemoize-imemoize-fimemoize-memoize-generators), iterables and iterator factories. - - [``fup``: functional update; ``ShadowedSequence``](#fup-functional-update-shadowedsequence): like ``collections.ChainMap``, but for sequences. - - [``view``: writable, sliceable view into a sequence](#view-writable-sliceable-view-into-a-sequence) with scalar broadcast on assignment. - - [``mogrify``: update a mutable container in-place](#mogrify-update-a-mutable-container-in-place) - - [``s``, ``m``, ``mg``: lazy mathematical sequences with infix arithmetic](#s-m-mg-lazy-mathematical-sequences-with-infix-arithmetic) - - - [**Control flow tools**](#control-flow-tools) - - [``trampolined``, ``jump``: tail call optimization (TCO) / explicit continuations](#trampolined-jump-tail-call-optimization-tco--explicit-continuations) - - [``looped``, ``looped_over``: loops in FP style (with TCO)](#looped-looped_over-loops-in-fp-style-with-tco) - - [``gtrampolined``: generators with TCO](#gtrampolined-generators-with-tco): tail-chaining; like ``itertools.chain``, but from inside a generator. - - [``setescape``, ``escape``: escape continuations (ec)](#setescape-escape-escape-continuations-ec) - - [``call_ec``: first-class escape continuations](#call_ec-first-class-escape-continuations), like Racket's ``call/ec``. - - [``forall``: nondeterministic evaluation](#forall-nondeterministic-evaluation), a tuple comprehension with multiple body expressions. - - - [**Other**](#other) - - [``def`` as a code block: ``@call``](#def-as-a-code-block-call): run a block of code immediately, in a new lexical scope. - - [``@callwith``: freeze arguments, choose function later](#callwith-freeze-arguments-choose-function-later) - - [``raisef``: ``raise`` as a function](#raisef-raise-as-a-function), useful inside a lambda. - - [``pack``: multi-arg constructor for tuple](#pack-multi-arg-constructor-for-tuple) - - [``namelambda``, rename a function](#namelambda-rename-a-function) - - [``timer``: a context manager for performance testing](#timer-a-context-manager-for-performance-testing) - - [``getattrrec``, ``setattrrec``: access underlying data in an onion of wrappers](#getattrrec-setattrrec-access-underlying-data-in-an-onion-of-wrappers) - - [``arities``, ``kwargs``: Function signature inspection utilities](#arities-kwargs-function-signature-inspection-utilities) - - [``Popper``: a pop-while iterator](#popper-a-pop-while-iterator) - - - [**Advanced: syntactic macros**](macro_extras/): the second half of ``unpythonic``. - - - [**Meta**](#meta) - - [Design notes](#design-notes) - - [Installation](#installation) - - [License](#license) - - [Acknowledgements](#acknowledgements) - - [Python-related FP resources](#python-related-fp-resources) - -For many examples, see the unit tests located in [unpythonic/test/](unpythonic/test/), the docstrings of the individual features, and this README. - -*This README doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out of date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this README) should agree with the unit tests. So if something fails to work as advertised, check what the tests say - and optionally file an issue on GitHub so that the documentation can be fixed.* - - -## Bindings - -Tools to bind identifiers in ways not ordinarily supported by Python. - -### ``let``, ``letrec``: local bindings in an expression - -Introduce bindings local to an expression, like Scheme's ``let`` and ``letrec``. For easy-to-use versions of these constructs that look almost like normal Python, see [macros](macro_extras/). This section documents the underlying pure-Python API, which can also be used directly if you don't want to depend on MacroPy. - -In ``let``, the bindings are independent (do not see each other). A binding is of the form ``name=value``, where ``name`` is a Python identifier, and ``value`` is any expression. - -Use a `lambda e: ...` to supply the environment to the body: - -```python -from unpythonic import let, letrec, dlet, dletrec, blet, bletrec - -u = lambda lst: let(seen=set(), - body=lambda e: - [e.seen.add(x) or x for x in lst if x not in e.seen]) -L = [1, 1, 3, 1, 3, 2, 3, 2, 2, 2, 4, 4, 1, 2, 3] -u(L) # --> [1, 3, 2, 4] -``` - -Generally speaking, `body` is a one-argument function, which takes in the environment instance as the first positional parameter (by convention, named `e` or `env`). In typical inline usage, `body` is `lambda e: expr`. - -*Let over lambda*. Here the inner ``lambda`` is the definition of the function ``counter``: - -```python -counter = let(x=0, - body=lambda e: - lambda: - begin(e.set("x", e.x + 1), # can also use e << ("x", e.x + 1) - e.x)) -counter() # --> 1 -counter() # --> 2 -``` - -Compare the sweet-exp [Racket](http://racket-lang.org/) (see [SRFI-110](https://srfi.schemers.org/srfi-110/srfi-110.html) and [sweet](https://docs.racket-lang.org/sweet/)): - -```racket -define counter - let ([x 0]) ; In Racket, the (λ (e) (...)) in "let" is implicit, and needs no explicit "e". - λ () ; Racket's λ has an implicit (begin ...), so we don't need a begin. - set! x {x + 1} - x -counter() ; --> 1 -counter() ; --> 2 -``` - -*Let over def* decorator ``@dlet``, to *let over lambda* more pythonically: - -```python -@dlet(x=0) -def counter(*, env=None): # named argument "env" filled in by decorator - env.x += 1 - return env.x -counter() # --> 1 -counter() # --> 2 -``` - -In `letrec`, bindings may depend on ones above them in the same `letrec`, by using `lambda e: ...` (**Python 3.6+**): - -```python -x = letrec(a=1, - b=lambda e: - e.a + 1, - body=lambda e: - e.b) # --> 2 -``` - -In `letrec`, the ``value`` of each binding is either a simple value (non-callable, and doesn't use the environment), or an expression of the form ``lambda e: valexpr``, providing access to the environment as ``e``. If ``valexpr`` itself is callable, the binding **must** have the ``lambda e: ...`` wrapper to prevent any misunderstandings in the environment initialization procedure. - -In a non-callable ``valexpr``, trying to depend on a binding below it raises ``AttributeError``. - -A callable ``valexpr`` may depend on any bindings (also later ones) in the same `letrec`. Mutually recursive functions: - -```python -letrec(evenp=lambda e: - lambda x: - (x == 0) or e.oddp(x - 1), - oddp=lambda e: - lambda x: - (x != 0) and e.evenp(x - 1), - body=lambda e: - e.evenp(42)) # --> True -``` - -Order-preserving list uniqifier: - -```python -u = lambda lst: letrec(seen=set(), - see=lambda e: - lambda x: - begin(e.seen.add(x), - x), - body=lambda e: - [e.see(x) for x in lst if x not in e.seen]) -``` - -**CAUTION**: in Pythons older than 3.6, bindings are **initialized in an arbitrary order**, also in `letrec`. This is a limitation of the kwargs abuse. Hence mutually recursive functions are possible, but a non-callable `valexpr` cannot depend on other bindings in the same `letrec`. - -Trying to access `e.foo` from `e.bar` arbitrarily produces either the intended value of `e.foo`, or the uninitialized `lambda e: ...`, depending on whether `e.foo` has been initialized or not at the point of time when `e.bar` is being initialized. - -This has been fixed in Python 3.6, see [PEP 468](https://www.python.org/dev/peps/pep-0468/). - -#### Lispylet: alternative syntax - -If you need **guaranteed left-to-right initialization** of `letrec` bindings in Pythons older than 3.6, there is also an alternative implementation for all the `let` constructs, with positional syntax and more parentheses. The only difference is the syntax; the behavior is identical with the default implementation. - -These constructs are available in the top-level `unpythonic` namespace, with the ``ordered_`` prefix: ``ordered_let``, ``ordered_letrec``, ``ordered_dlet``, ``ordered_dletrec``, ``ordered_blet``, ``ordered_bletrec``. - -It is also possible to override the default `let` constructs by the `ordered_` variants, like this: - -```python -from unpythonic.lispylet import * # override the default "let" implementation - -letrec((('a', 1), - ('b', lambda e: - e.a + 1)), # may refer to any bindings above it in the same letrec, also in Python < 3.6 - lambda e: - e.b) # --> 2 - -letrec((("evenp", lambda e: - lambda x: # callable, needs the lambda e: ... - (x == 0) or e.oddp(x - 1)), - ("oddp", lambda e: - lambda x: - (x != 0) and e.evenp(x - 1))), - lambda e: - e.evenp(42)) # --> True -``` - -The syntax is `let(bindings, body)` (respectively `letrec(bindings, body)`), where `bindings` is `((name, value), ...)`, and `body` is like in the default variants. The same rules concerning `name` and `value` apply. - -The let macros internally use this *lispylet* implementation. - - -### ``env``: the environment - -The environment used by all the ``let`` constructs and ``assignonce`` (but **not** by `dyn`) is essentially a bunch with iteration, subscripting and context manager support. For details, see `unpythonic.env`. - -This allows things like: - -```python -let(x=1, y=2, z=3, - body=lambda e: - [(name, 2*e[name]) for name in e]) # --> [('y', 4), ('z', 6), ('x', 2)] -``` - -It also works as a bare bunch, and supports printing for debugging: - -```python -from unpythonic.env import env - -e = env(s="hello", orange="fruit", answer=42) -print(e) # --> -print(e.s) # --> hello - -d = {'monty': 'python', 'pi': 3.14159} -e = env(**d) -print(e) # --> -print(e.monty) # --> python -``` - -Finally, it supports the context manager: - -```python -with env(x=1, y=2, z=3) as e: - print(e) # --> - print(e.x) # --> 1 -print(e) # empty! -``` - -When the `with` block exits, the environment clears itself. The environment instance itself remains alive due to Python's scoping rules. - -*Changed in v0.13.1.* ``env`` now provides the ``collections.abc.Mapping`` API. - -*Changed in v0.14.0.* ``env`` now provides also the ``collections.abc.MutableMapping`` API. - - -### ``assignonce`` - -In Scheme terms, make `define` and `set!` look different: - -```python -from unpythonic import assignonce - -with assignonce() as e: - e.foo = "bar" # new definition, ok - e.set("foo", "tavern") # explicitly rebind e.foo, ok - e << ("foo", "tavern") # same thing (but return e instead of new value, suitable for chaining) - e.foo = "quux" # AttributeError, e.foo already defined. -``` - -It's a subclass of ``env``, so it shares most of the same [features](#env-the-environment) and allows similar usage. - - -### ``dyn``: dynamic assignment - -([As termed by Felleisen.](https://groups.google.com/forum/#!topic/racket-users/2Baxa2DxDKQ)) - -Like global variables, but better-behaved. Useful for sending some configuration parameters through several layers of function calls without changing their API. Best used sparingly. Like [Racket](http://racket-lang.org/)'s [`parameterize`](https://docs.racket-lang.org/guide/parameterize.html). The *special variables* in Common Lisp work somewhat similarly (but with indefinite scope). - -There's a singleton, `dyn`: - -```python -from unpythonic import dyn - -def f(): # no "a" in lexical scope here - assert dyn.a == 2 - -def g(): - with dyn.let(a=2, b="foo"): - assert dyn.a == 2 - - f() - - with dyn.let(a=3): # dynamic assignments can be nested - assert dyn.a == 3 - - # now "a" has reverted to its previous value - assert dyn.a == 2 - - print(dyn.b) # AttributeError, dyn.b no longer exists -g() -``` - -Dynamic variables are set using `with dyn.let(...)`. There is no `set`, `<<`, unlike in the other `unpythonic` environments. - -The values of dynamic variables remain bound for the dynamic extent of the `with` block. Exiting the `with` block then pops the stack. Inner dynamic scopes shadow outer ones. Dynamic variables are seen also by code that is outside the lexical scope where the `with dyn.let` resides. - -Each thread has its own dynamic scope stack. A newly spawned thread automatically copies the then-current state of the dynamic scope stack **from the main thread** (not the parent thread!). Any copied bindings will remain on the stack for the full dynamic extent of the new thread. Because these bindings are not associated with any `with` block running in that thread, and because aside from the initial copying, the dynamic scope stacks are thread-local, any copied bindings will never be popped, even if the main thread pops its own instances of them. - -The source of the copy is always the main thread mainly because Python's `threading` module gives no tools to detect which thread spawned the current one. (If someone knows a simple solution, PRs welcome!) - -Finally, there is one global dynamic scope shared between all threads, where the default values of dynvars live. The default value is used when ``dyn`` is queried for the value outside the dynamic extent of any ``with dyn.let()`` blocks. Having a default value is convenient for eliminating the need for ``if "x" in dyn`` checks, since the variable will always exist (after the global definition has been executed). - -To create a dynvar and set its default value, use ``make_dynvar``. Each dynamic variable, of the same name, should only have one default set; the (dynamically) latest definition always overwrites. However, we do not prevent overwrites, because in some codebases the same module may run its top-level initialization code multiple times (e.g. if a module has a ``main()`` for tests, and the file gets loaded both as a module and as the main program). - -See also the methods of ``dyn``; particularly noteworthy are ``asdict`` and ``items``, which give access to a live view to dyn's contents in a dictionary format (intended for reading only!). The ``asdict`` method essentially creates a ``collections.ChainMap`` instance, while ``items`` is an abbreviation for ``asdict().items()``. The ``dyn`` object itself can also be iterated over; this creates a ``ChainMap`` instance and redirects to iterate over it. - -To support dictionary-like idioms in iteration, dynvars can alternatively be accessed by subscripting; ``dyn["x"]`` has the same meaning as ``dyn.x``, so you can do things like: - -```python -print(tuple((k, dyn[k]) for k in dyn)) -``` - -Finally, ``dyn`` supports membership testing as ``"x" in dyn``, ``"y" not in dyn``, where the string is the name of the dynvar whose presence is being tested. - -For some more details, see [the unit tests](unpythonic/test/test_dynassign.py). - -*Changed in v0.13.0.* The ``asdict`` and ``items`` methods previously returned a snapshot; now they return a live view. - -*Changed in v0.13.1.* ``dyn`` now provides the ``collections.abc.Mapping`` API. - - -## Containers - -We provide some additional containers. - -The class names are lowercase, because these are intended as low-level utility classes in principle on par with the builtins. The immutable containers are hashable. All containers are pickleable (if their contents are). - -### ``frozendict``: an immutable dictionary - -*Added in v0.13.0.* - -Given the existence of ``dict`` and ``frozenset``, this one is oddly missing from the standard library. - -```python -from unpythonic import frozendict - -d = frozendict({'a': 1, 'b': 2}) -d['a'] # OK -d['c'] = 3 # TypeError, not writable -``` - -Functional updates are supported: - -```python -d2 = frozendict(d, a=42) -assert d2['a'] == 42 and d2['b'] == 2 -assert d['a'] == 1 # original not mutated - -d3 = frozendict({'a': 1, 'b': 2}, {'a': 42}) # rightmost definition of each key wins -assert d3['a'] == 42 and d3['b'] == 2 - -# ...also using unpythonic.fupdate -d4 = fupdate(d3, a=23) -assert d4['a'] == 23 and d4['b'] == 2 -assert d3['a'] == 42 and d3['b'] == 2 # ...of course without touching the original -``` - -Any mappings used when creating an instance are shallow-copied, so that the bindings of the ``frozendict`` do not change even if the original input is later mutated: - -```python -d = {1:2, 3:4} -fd = frozendict(d) -d[5] = 6 -assert d == {1: 2, 3: 4, 5: 6} -assert fd == {1: 2, 3: 4} -``` - -**The usual caution** concerning immutable containers in Python applies: the container protects only the bindings against changes. If the values themselves are mutable, the container cannot protect from mutations inside them. - -All the usual read-access stuff works: - -```python -d7 = frozendict({1:2, 3:4}) -assert 3 in d7 -assert len(d7) == 2 -assert set(d7.keys()) == {1, 3} -assert set(d7.values()) == {2, 4} -assert set(d7.items()) == {(1, 2), (3, 4)} -assert d7 == frozendict({1:2, 3:4}) -assert d7 != frozendict({1:2}) -assert d7 == {1:2, 3:4} # like frozenset, __eq__ doesn't care whether mutable or not -assert d7 != {1:2} -assert {k for k in d7} == {1, 3} -assert d7.get(3) == 4 -assert d7.get(5, 0) == 0 -assert d7.get(5) is None -``` - -In terms of ``collections.abc``, a ``frozendict`` is a hashable immutable mapping: - -```python -assert issubclass(frozendict, Mapping) -assert not issubclass(frozendict, MutableMapping) - -assert issubclass(frozendict, Hashable) -assert hash(d7) == hash(frozendict({1:2, 3:4})) -assert hash(d7) != hash(frozendict({1:2})) -``` - -The abstract superclasses are virtual, just like for ``dict`` (i.e. they do not appear in the MRO). - -Finally, ``frozendict`` obeys the empty-immutable-container singleton invariant: - -```python -assert frozendict() is frozendict() -``` - -...but don't pickle the empty ``frozendict`` and expect this invariant to hold; it's freshly created in each session. - - -### `cons` and friends: pythonic lispy linked lists - -*Laugh, it's funny.* - -```python -from unpythonic import cons, nil, ll, llist, car, cdr, caar, cdar, cadr, cddr, \ - member, lreverse, lappend, lzip, BinaryTreeIterator - -c = cons(1, 2) -assert car(c) == 1 and cdr(c) == 2 - -# ll(...) is like [...] or (...), but for linked lists: -assert ll(1, 2, 3) == cons(1, cons(2, cons(3, nil))) - -t = cons(cons(1, 2), cons(3, 4)) # binary tree -assert [f(t) for f in [caar, cdar, cadr, cddr]] == [1, 2, 3, 4] - -# default iteration scheme is "single cell or linked list": -a, b = cons(1, 2) # unpacking a cons cell -a, b, c = ll(1, 2, 3) # unpacking a linked list -a, b, c, d = BinaryTreeIterator(t) # unpacking a binary tree: use a non-default iteration scheme - -assert list(ll(1, 2, 3)) == [1, 2, 3] -assert tuple(ll(1, 2, 3)) == (1, 2, 3) -assert llist((1, 2, 3)) == ll(1, 2, 3) # llist() is like list() or tuple(), but for linked lists - -l = ll(1, 2, 3) -assert member(2, l) == ll(2, 3) -assert not member(5, l) - -assert lreverse(ll(1, 2, 3)) == ll(3, 2, 1) -assert lappend(ll(1, 2), ll(3, 4), ll(5, 6)) == ll(1, 2, 3, 4, 5, 6) -assert lzip(ll(1, 2, 3), ll(4, 5, 6)) == ll(ll(1, 4), ll(2, 5), ll(3, 6)) -``` - -Cons cells are immutable à la Racket (no `set-car!`/`rplaca`, `set-cdr!`/`rplacd`). Accessors are provided up to `caaaar`, ..., `cddddr`. - -Although linked lists are created with ``ll`` or ``llist``, the data type (for e.g. ``isinstance``) is ``cons``. - -Iterators are supported to walk over linked lists (this also gives sequence unpacking support). When ``next()`` is called, we return the car of the current cell the iterator points to, and the iterator moves to point to the cons cell in the cdr, if any. When the cdr is not a cons cell, it is the next (and last) item returned; except if it `is nil`, then iteration ends without returning the `nil`. - -Python's builtin ``reversed`` can be applied to linked lists; it will internally ``lreverse`` the list (which is O(n)), then return an iterator to that. The ``llist`` constructor is special-cased so that if the input is ``reversed(some_ll)``, it just returns the internal already reversed list. (This is safe because cons cells are immutable.) - -Cons structures can optionally print like in Lisps: - -```python -print(cons(1, 2).lispyrepr()) # --> (1 . 2) -print(ll(1, 2, 3).lispyrepr()) # --> (1 2 3) -print(cons(cons(1, 2), cons(3, 4)).lispyrepr()) # --> ((1 . 2) . (3 . 4)) -``` - -However, by default, they print in a pythonic format suitable for ``eval`` (if all elements are): - -```python -print(cons(1, 2)) # --> cons(1, 2) -print(ll(1, 2, 3)) # --> ll(1, 2, 3) -print(cons(cons(1, 2), cons(3, 4)) # --> cons(cons(1, 2), cons(3, 4)) -``` - -*Changed in v0.11.0.* In previous versions, the Lisp format was always used for printing. - -For more, see the ``llist`` submodule. - -#### Notes - -There is no ``copy`` method or ``lcopy`` function, because cons cells are immutable; which makes cons structures immutable. - -(However, for example, it is possible to `cons` a new item onto an existing linked list; that's fine because it produces a new cons structure - which shares data with the original, just like in Racket.) - -In general, copying cons structures can be error-prone. Given just a starting cell it is impossible to tell if a given instance of a cons structure represents a linked list, or something more general (such as a binary tree) that just happens to locally look like one, along the path that would be traversed if it was indeed a linked list. - -The linked list iteration strategy does not recurse in the ``car`` half, which could lead to incomplete copying. The tree strategy that recurses on both halves, on the other hand, will flatten nested linked lists and produce also the final ``nil``. - -*Added in v0.13.0.* We provide a ``JackOfAllTradesIterator`` as a compromise that understands both trees and linked lists. Nested lists will be flattened, and in a tree any ``nil`` in a ``cdr`` position will be omitted from the output. - -*Changed in v0.13.1.* ``BinaryTreeIterator`` and ``JackOfAllTradesIterator`` now use an explicit data stack instead of implicitly using the call stack for keeping track of the recursion. Hence now all ``cons`` iterators work for arbitrarily deep cons structures without causing Python's call stack to overflow, and without the need for TCO. - -``cons`` has no ``collections.abc`` virtual superclasses (except the implicit ``Hashable`` since ``cons`` provides ``__hash__`` and ``__eq__``), because general cons structures do not fit into the contracts represented by membership in those classes. For example, size cannot be known without iterating, and depends on which iteration scheme is used (e.g. ``nil`` dropping, flattening); which scheme is appropriate depends on the content. - -**Caution**: the ``nil`` singleton is freshly created in each session; newnil is not oldnil, so don't pickle a standalone ``nil``. The unpickler of ``cons`` automatically refreshes any ``nil`` instances inside a pickled cons structure, so that **cons structures** support the illusion that ``nil`` is a special value like ``None`` or ``...``. After unpickling, ``car(c) is nil`` and ``cdr(c) is nil`` still work as expected, even though ``id(nil)`` has changed between sessions. - - -### ``box``: a mutable single-item container - -*Added in v0.12.0.* - -*Changed in v0.13.0.* The class and the data attribute have been renamed to ``box`` and ``x``, respectively. - -No doubt anyone programming in an imperative language has run into the situation caricatured by this highly artificial example: - -```python -a = 23 - -def f(x): - x = 17 # but I want to update the existing a! - -f(a) -assert a == 23 -``` - -Many solutions exist. Common pythonic ones are abusing a ``list`` to represent a box (and then trying to remember it is supposed to hold only a single item), or using the ``global`` or ``nonlocal`` keywords to tell Python, on assignment, to overwrite a name that already exists in a surrounding scope. - -As an alternative to the rampant abuse of lists, we provide a rackety ``box``, which is a minimalistic mutable container that holds exactly one item. The data in the box is accessed via an attribute, so any code that has a reference to the box can update the data in it: - -```python -from unpythonic import box - -a = box(23) - -def f(b): - b.x = 17 - -f(a) -assert a == 17 -``` - -The attribute name is just ``x`` to reduce the number of additional keystrokes required. The ``box`` API is summarized by: - -```python -b1 = box(23) -b2 = box(23) -b3 = box(17) - -assert b1.x == 23 # data lives in the attribute .x -assert 23 in b1 # content is "in" the box, also syntactically -assert 17 not in b1 - -assert [x for x in b1] == [23] # box is iterable -assert len(b1) == 1 # and always has length 1 - -assert b1 == 23 # for equality testing, a box is considered equal to its content - -assert b2 == b1 # contents are equal, but -assert b2 is not b1 # different boxes - -assert b3 != b1 # different contents -``` - -The expression ``item in b`` has the same meaning as ``b.x == item``. Note ``box`` is a mutable container, so it is **not hashable**. - - -### Container utilities - -**Inspect the superclasses** that a particular container type has: - -```python -from unpythonic import get_abcs -print(get_abcs(list)) -``` - -This includes virtual superclasses, i.e. those that are not part of the MRO. This works by ``issubclass(cls, v)`` on all classes defined in ``collections.abc``. - -**Reflection on slices**: - -```python -from unpythonic import in_slice, index_in_slice - -s = slice(1, 11, 2) # 1, 3, 5, 7, 9 -assert in_slice(5, s) -assert not in_slice(6, s) -assert index_in_slice(5, s) == 2 -``` - -An optional length argument can be given to interpret negative indices. See the docstrings for details. - - - -## Sequencing - -Sequencing refers to running multiple expressions, in sequence, in place of one expression. - -Keep in mind the only reason to ever need multiple expressions: *side effects.* (Assignment is a side effect, too; it modifies the environment. In functional style, intermediate named definitions to increase readability are perhaps the most useful kind of side effect.) - -See also ``multilambda`` in [macros](macro_extras/). - - -### ``begin``: sequence side effects - -```python -from unpythonic import begin, begin0 - -f1 = lambda x: begin(print("cheeky side effect"), - 42*x) -f1(2) # --> 84 - -f2 = lambda x: begin0(42*x, - print("cheeky side effect")) -f2(2) # --> 84 -``` - -Actually a tuple in disguise. If worried about memory consumption, use `lazy_begin` and `lazy_begin0` instead, which indeed use loops. The price is the need for a lambda wrapper for each expression to delay evaluation, see [`unpythonic.seq`](unpythonic/seq.py) for details. - - -### ``do``: stuff imperative code into an expression - -No monadic magic. Basically, ``do`` is: - - - An improved ``begin`` that can bind names to intermediate results and then use them in later items. - - - A ``let*`` (technically, ``letrec``) where making a binding is optional, so that some items can have only side effects if so desired. No semantically distinct ``body``; all items play the same role. - -Like in ``letrec`` (see below), use ``lambda e: ...`` to access the environment, and to wrap callable values (to prevent misunderstandings). - -We also provide a ``do[]`` [macro](macro_extras/) that makes the construct easier to use. - -```python -from unpythonic import do, assign - -y = do(assign(x=17), # create and set e.x - lambda e: print(e.x), # 17; uses environment, needs lambda e: ... - assign(x=23), # overwrite e.x - lambda e: print(e.x), # 23 - 42) # return value -assert y == 42 - -y = do(assign(x=17), - assign(z=lambda e: 2*e.x), - lambda e: e.z) -assert y == 34 - -y = do(assign(x=5), - assign(f=lambda e: lambda x: x**2), # callable, needs lambda e: ... - print("hello from 'do'"), # value is None; not callable - lambda e: e.f(e.x)) -assert y == 25 -``` - -If you need to return the first value instead of the last one, use this trick: - -```python -y = do(assign(result=17), - print("assigned 'result' in env"), - lambda e: e.result) # return value -assert y == 17 -``` - -Or use ``do0``, which does it for you: - -```python -from unpythonic import do0, assign - -y = do0(17, - assign(x=42), - lambda e: print(e.x), - print("hello from 'do0'")) -assert y == 17 - -y = do0(assign(x=17), # the first item of do0 can be an assignment, too - lambda e: print(e.x)) -assert y == 17 -``` - -Beware of this pitfall: - -```python -do(lambda e: print("hello 2 from 'do'"), # delayed because lambda e: ... - print("hello 1 from 'do'"), # Python prints immediately before do() - "foo") # gets control, because technically, it is - # **the return value** that is an argument - # for do(). -``` - -Unlike ``begin`` (and ``begin0``), there is no separate ``lazy_do`` (``lazy_do0``), because using a ``lambda e: ...`` wrapper will already delay evaluation of an item. If you want a lazy variant, just wrap each item (also those which don't otherwise need it). - -The above pitfall also applies to using escape continuations inside a ``do``. To do that, wrap the ec call into a ``lambda e: ...`` to delay its evaluation until the ``do`` actually runs: - -```python -call_ec( - lambda ec: - do(assign(x=42), - lambda e: ec(e.x), # IMPORTANT: must delay this! - lambda e: print("never reached"))) # and this (as above) -``` - -This way, any assignments made in the ``do`` (which occur only after ``do`` gets control), performed above the line with the ``ec`` call, will have been performed when the ``ec`` is called. - - -### ``pipe``, ``piped``, ``lazy_piped``: sequence functions - -Similar to Racket's [threading macros](https://docs.racket-lang.org/threading/). A pipe performs a sequence of operations, starting from an initial value, and then returns the final value. It's just function composition, but with an emphasis on data flow, which helps improve readability: - -```python -from unpythonic import pipe - -double = lambda x: 2 * x -inc = lambda x: x + 1 - -x = pipe(42, double, inc) -assert x == 85 -``` - -We also provide ``pipec``, which curries the functions before applying them. Useful with passthrough (see below on ``curry``). - -Optional **shell-like syntax**, with purely functional updates: - -```python -from unpythonic import piped, getvalue - -x = piped(42) | double | inc | getvalue -assert x == 85 - -p = piped(42) | double -assert p | inc | getvalue == 85 -assert p | getvalue == 84 # p itself is never modified by the pipe system -``` - -Set up a pipe by calling ``piped`` for the initial value. Pipe into the sentinel ``getvalue`` to exit the pipe and return the current value. - -**Lazy pipes**, useful for mutable initial values. To perform the planned computation, pipe into the sentinel ``runpipe``: - -```python -from unpythonic import lazy_piped1, runpipe - -lst = [1] -def append_succ(l): - l.append(l[-1] + 1) - return l # this return value is handed to the next function in the pipe -p = lazy_piped1(lst) | append_succ | append_succ # plan a computation -assert lst == [1] # nothing done yet -p | runpipe # run the computation -assert lst == [1, 2, 3] # now the side effect has updated lst. -``` - -Lazy pipe as an unfold: - -```python -from unpythonic import lazy_piped, runpipe - -fibos = [] -def nextfibo(a, b): # multiple arguments allowed - fibos.append(a) # store result by side effect - return (b, a + b) # new state, handed to next function in the pipe -p = lazy_piped(1, 1) # load initial state -for _ in range(10): # set up pipeline - p = p | nextfibo -p | runpipe -assert fibos == [1, 1, 2, 3, 5, 8, 13, 21, 34, 55] -``` - -Both one-in-one-out (*1-to-1*) and n-in-m-out (*n-to-m*) pipes are provided. The 1-to-1 versions have names suffixed with ``1``. The use case is one-argument functions that return one value (which may also be a tuple). - -In the n-to-m versions, when a function returns a tuple, it is unpacked to the argument list of the next function in the pipe. At ``getvalue`` or ``runpipe`` time, the tuple wrapper (if any) around the final result is discarded if it contains only one item. (This allows the n-to-m versions to work also with a single value, as long as it is not a tuple.) The main use case is computations that deal with multiple values, the number of which may also change during the computation (as long as there are as many "slots" on both sides of each individual connection). - - -## Batteries - -Things missing from the standard library. - -### Batteries for functools - - - `memoize`: - - Caches also exceptions à la Racket. If the memoized function is called again with arguments with which it raised an exception the first time, the same exception instance is raised again. - - Works also on instance methods, with results cached separately for each instance. - - This is essentially because ``self`` is an argument, and custom classes have a default ``__hash__``. - - Hence it doesn't matter that the memo lives in the ``memoized`` closure on the class object (type), where the method is, and not directly on the instances. The memo itself is shared between instances, but calls with a different value of ``self`` will create unique entries in it. - - For a solution that performs memoization at the instance level, see [this ActiveState recipe](https://github.com/ActiveState/code/tree/master/recipes/Python/577452_memoize_decorator_instance) (and to demystify the magic contained therein, be sure you understand [descriptors](https://docs.python.org/3/howto/descriptor.html)). - - `curry`, with some extra features: - - Passthrough on the right when too many args (à la Haskell; or [spicy](https://github.com/Technologicat/spicy) for Racket) - - If the intermediate result of a passthrough is callable, it is (curried and) invoked on the remaining positional args. This helps with some instances of [point-free style](https://en.wikipedia.org/wiki/Tacit_programming). - - For simplicity, all remaining keyword args are fed in at the first step that has too many positional args. - - If more positional args are still remaining when the top-level curry context exits, by default ``TypeError`` is raised. - - To override, set the dynvar ``curry_context``. It is a list representing the stack of currently active curry contexts. A context is any object, a human-readable label is fine. See below for an example. - - Can be used both as a decorator and as a regular function. - - As a regular function, `curry` itself is curried à la Racket. If it gets extra arguments (beside the function ``f``), they are the first step. This helps eliminate many parentheses. - - **Caution**: If the positional arities of ``f`` cannot be inspected, currying fails, raising ``UnknownArity``. This may happen with builtins such as ``list.append``. - - `composel`, `composer`: both left-to-right and right-to-left function composition, to help readability. - - Any number of positional arguments is supported, with the same rules as in the pipe system. Multiple return values packed into a tuple are unpacked to the argument list of the next function in the chain. - - `composelc`, `composerc`: curry each function before composing them. Useful with passthrough. - - An implicit top-level curry context is inserted around all the functions except the one that is applied last. - - `composel1`, `composer1`: 1-in-1-out chains (faster; also useful for a single value that is a tuple). - - suffix `i` to use with an iterable (`composeli`, `composeri`, `composelci`, `composerci`, `composel1i`, `composer1i`) - - `withself`: essentially, the Y combinator trick as a decorator. Allows a lambda to refer to itself. - - The ``self`` argument is declared explicitly, but passed implicitly (as the first positional argument), just like the ``self`` argument of a method. - - `apply`: the lispy approach to starargs. Mainly useful with the ``prefix`` [macro](macro_extras/). - - `andf`, `orf`, `notf`: compose predicates (like Racket's `conjoin`, `disjoin`, `negate`). - - `flip`: reverse the order of positional arguments. - - `rotate`: a cousin of `flip`. Permute the order of positional arguments in a cycle. - - `to1st`, `to2nd`, `tokth`, `tolast`, `to` to help inserting 1-in-1-out functions into m-in-n-out compose chains. (Currying can eliminate the need for these.) - - `identity`, `const` which sometimes come in handy when programming with higher-order functions. - -Examples (see also the next section): - -```python -from operator import add, mul -from unpythonic import andf, orf, flatmap, rotate, curry, dyn, zipr, rzip, \ - foldl, foldr, composer, to1st, cons, nil, ll, withself - -isint = lambda x: isinstance(x, int) -iseven = lambda x: x % 2 == 0 -isstr = lambda s: isinstance(s, str) -assert andf(isint, iseven)(42) is True -assert andf(isint, iseven)(43) is False -pred = orf(isstr, andf(isint, iseven)) -assert pred(42) is True -assert pred("foo") is True -assert pred(None) is False - -# lambda that refers to itself -fact = withself(lambda self, n: n * self(n - 1) if n > 1 else 1) -assert fact(5) == 120 - -@rotate(-1) # cycle the argument slots to the left by one place, so "acc" becomes last -def zipper(acc, *rest): # so that we can use the *args syntax to declare this - return acc + (rest,) # even though the input is (e1, ..., en, acc). -myzipl = curry(foldl, zipper, ()) # same as (curry(foldl))(zipper, ()) -myzipr = curry(foldr, zipper, ()) -assert myzipl((1, 2, 3), (4, 5, 6), (7, 8)) == ((1, 4, 7), (2, 5, 8)) -assert myzipr((1, 2, 3), (4, 5, 6), (7, 8)) == ((2, 5, 8), (1, 4, 7)) - -# zip and reverse don't commute for inputs with different lengths -assert tuple(zipr((1, 2, 3), (4, 5, 6), (7, 8))) == ((2, 5, 8), (1, 4, 7)) # zip first -assert tuple(rzip((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) # reverse first - -# curry with passthrough on the right -# final result is a tuple of the result(s) and the leftover args -double = lambda x: 2 * x -with dyn.let(curry_context=["whatever"]): # set a context to allow passthrough to the top level - assert curry(double, 2, "foo") == (4, "foo") # arity of double is 1 - -mysum = curry(foldl, add, 0) -myprod = curry(foldl, mul, 1) -a = ll(1, 2) -b = ll(3, 4) -c = ll(5, 6) -append_two = lambda a, b: foldr(cons, b, a) -append_many = lambda *lsts: foldr(append_two, nil, lsts) # see unpythonic.lappend -assert mysum(append_many(a, b, c)) == 21 -assert myprod(b) == 12 - -map_one = lambda f: curry(foldr, composer(cons, to1st(f)), nil) -doubler = map_one(double) -assert doubler((1, 2, 3)) == ll(2, 4, 6) - -assert curry(map_one, double, ll(1, 2, 3)) == ll(2, 4, 6) -``` - -*Minor detail*: We could also write the last example as: - -```python -double = lambda x: 2 * x -rmap_one = lambda f: curry(foldl, composer(cons, to1st(f)), nil) # essentially reversed(map(...)) -map_one = lambda f: composer(rmap_one(f), lreverse) -assert curry(map_one, double, ll(1, 2, 3)) == ll(2, 4, 6) -``` - -which may be a useful pattern for lengthy iterables that could overflow the call stack (although not in ``foldr``, since our implementation uses a linear process). - -In ``rmap_one``, we can use either ``curry`` or ``functools.partial``. In this case it doesn't matter which, since we want just one partial application anyway. We provide two arguments, and the minimum arity of ``foldl`` is 3, so ``curry`` will trigger the call as soon as (and only as soon as) it gets at least one more argument. - -The final ``curry`` uses both of the extra features. It invokes passthrough, since ``map_one`` has arity 1. It also invokes a call to the callable returned from ``map_one``, with the remaining arguments (in this case just one, the ``ll(1, 2, 3)``). - -Yet another way to write ``map_one`` is: - -```python -mymap = lambda f: curry(foldr, composer(cons, curry(f)), nil) -``` - -The curried ``f`` uses up one argument (provided it is a one-argument function!), and the second argument is passed through on the right; this two-tuple then ends up as the arguments to ``cons``. - -Using a currying compose function (name suffixed with ``c``), the inner curry can be dropped: - -```python -mymap = lambda f: curry(foldr, composerc(cons, f), nil) -myadd = lambda a, b: a + b -assert curry(mymap, myadd, ll(1, 2, 3), ll(2, 4, 6)) == ll(3, 6, 9) -``` - -This is as close to ```(define (map f) (foldr (compose cons f) empty)``` (in ``#lang`` [``spicy``](https://github.com/Technologicat/spicy)) as we're gonna get in Python. - -Notice how the last two versions accept multiple input iterables; this is thanks to currying ``f`` inside the composition. An element from each of the iterables is taken by the processing function ``f``. Being the last argument, ``acc`` is passed through on the right. The output from the processing function - one new item - and ``acc`` then become a two-tuple, passed into cons. - -Finally, keep in mind this exercise is intended as a feature demonstration. In production code, the builtin ``map`` is much better. - - -#### ``curry`` and reduction rules - -The provided variant of ``curry``, beside what it says on the tin, is effectively an explicit local modifier to Python's reduction rules, which allows some Haskell-like idioms. When we say: - -```python -curry(f, a0, a1, ..., a[n-1]) -``` - -it means the following. Let ``m1`` and ``m2`` be the minimum and maximum positional arity of the callable ``f``, respectively. - - - If ``n > m2``, call ``f`` with the first ``m2`` arguments. - - If the result is a callable, curry it, and recurse. - - Else form a tuple, where first item is the result, and the rest are the remaining arguments ``a[m2]``, ``a[m2+1]``, ..., ``a[n-1]``. Return it. - - If more positional args are still remaining when the top-level curry context exits, by default ``TypeError`` is raised. Use the dynvar ``curry_context`` to override; see above for an example. - - If ``m1 <= n <= m2``, call ``f`` and return its result (like a normal function call). - - **Any** positional arity accepted by ``f`` triggers the call; beware when working with [variadic](https://en.wikipedia.org/wiki/Variadic_function) functions. - - If ``n < m1``, partially apply ``f`` to the given arguments, yielding a new function with smaller ``m1``, ``m2``. Then curry the result and return it. - - Internally we stack ``functools.partial`` applications, but there will be only one ``curried`` wrapper no matter how many invocations are used to build up arguments before ``f`` eventually gets called. - -In the above example: - -```python -curry(mapl_one, double, ll(1, 2, 3)) -``` - -the callable ``mapl_one`` takes one argument, which is a function. It yields another function, let us call it ``g``. We are left with: - -```python -curry(g, ll(1, 2, 3)) -``` - -The argument is then passed into ``g``; we obtain a result, and reduction is complete. - -A curried function is also a curry context: - -```python -add2 = lambda x, y: x + y -a2 = curry(add2) -a2(a, b, c) # same as curry(add2, a, b, c); reduces to (a + b, c) -``` - -so on the last line, we don't need to say - -```python -curry(a2, a, b, c) -``` - -because ``a2`` is already curried. Doing so does no harm, though; ``curry`` automatically prevents stacking ``curried`` wrappers: - -```python -curry(a2) is a2 # --> True -``` - -If we wish to modify precedence, parentheses are needed, which takes us out of the curry context, unless we explicitly ``curry`` the subexpression. This works: - -```python -curry(f, a, curry(g, x, y), b, c) -``` - -but this **does not**: - -```python -curry(f, a, (g, x, y), b, c) -``` - -because ``(g, x, y)`` is just a tuple of ``g``, ``x`` and ``y``. This is by design; as with all things Python, *explicit is better than implicit*. - -**Note**: to code in curried style, a [contract system](https://github.com/AndreaCensi/contracts) or a [static type checker](http://mypy-lang.org/) is useful; also, be careful with variadic functions. - - -### Batteries for itertools - - - `foldl`, `foldr` with support for multiple input iterables, like in Racket. - - Like in Racket, `op(elt, acc)`; general case `op(e1, e2, ..., en, acc)`. Note Python's own `functools.reduce` uses the ordering `op(acc, elt)` instead. - - No sane default for multi-input case, so the initial value for `acc` must be given. - - One-input versions with optional init are provided as `reducel`, `reducer`, with semantics similar to Python's `functools.reduce`, but with the rackety ordering `op(elt, acc)`. - - By default, multi-input folds terminate on the shortest input. To instead terminate on the longest input, use the ``longest`` and ``fillvalue`` kwargs. - - For multiple inputs with different lengths, `foldr` syncs the **left** ends. - - `rfoldl`, `rreducel` reverse each input and then left-fold. This syncs the **right** ends. - - `scanl`, `scanr`: scan (a.k.a. accumulate, partial fold); a lazy fold that returns a generator yielding intermediate results. - - `scanl` is suitable for infinite inputs. - - Iteration stops after the final result. - - For `scanl`, this is what `foldl` would have returned (if the fold terminates at all, i.e. if the shortest input is finite). - - *Changed in v0.11.0.* For `scanr`, **ordering of output is different from Haskell**: we yield the results in the order they are computed (via a linear process). - - Multiple input iterables and shortest/longest termination supported; same semantics as in `foldl`, `foldr`. - - One-input versions with optional init are provided as `scanl1`, `scanr1`. Note ordering of arguments to match `functools.reduce`, but op is still the rackety `op(elt, acc)`. - - `rscanl`, `rscanl1` reverse each input and then left-scan. This syncs the **right** ends. - - `unfold1`, `unfold`: generate a sequence [corecursively](https://en.wikipedia.org/wiki/Corecursion). The counterpart of `foldl`. - - `unfold1` is for 1-in-2-out functions. The input is `state`, the return value must be `(value, newstate)` or `None`. - - `unfold` is for n-in-(1+n)-out functions. The input is `*states`, the return value must be `(value, *newstates)` or `None`. - - Unfold returns a generator yielding the collected values. The output can be finite or infinite; to signify that a finite sequence ends, the user function must return `None`. - - `unpack`: lazily unpack an iterable. Suitable for infinite inputs. - - Return the first ``n`` items and the ``k``th tail, in a tuple. Default is ``k = n``. - - Use ``k > n`` to fast-forward, consuming the skipped items. Works by `drop`. - - Use ``k < n`` to peek without permanently extracting an item. Works by [tee](https://docs.python.org/3/library/itertools.html#itertools.tee)ing; plan accordingly. - - `flatmap`: map a function, that returns a list or tuple, over an iterable and then flatten by one level, concatenating the results into a single tuple. - - Essentially, ``composel(map(...), flatten1)``; the same thing the bind operator of the List monad does. - - `map_longest`: the final missing battery for `map`. - - Essentially `starmap(func, zip_longest(*iterables))`, so it's [spanned](https://en.wikipedia.org/wiki/Linear_span) by ``itertools``. - - `rmap`, `rzip`, `rmap_longest`, `rzip_longest`: reverse each input, then map/zip. For multiple inputs, syncs the **right** ends. - - `mapr`, `zipr`, `mapr_longest`, `zipr_longest`: map/zip, then reverse the result. For multiple inputs, syncs the **left** ends. - - `uniqify`, `uniq`: remove duplicates (either all or consecutive only, respectively), preserving the original ordering of the items. - - `flatten1`, `flatten`, `flatten_in`: remove nested list structure. - - `flatten1`: outermost level only. - - `flatten`: recursive, with an optional predicate that controls whether to flatten a given sublist. - - `flatten_in`: recursive, with an optional predicate; but recurse also into items which don't match the predicate. - - `take`, `drop`, `split_at`: based on `itertools` [recipes](https://docs.python.org/3/library/itertools.html#itertools-recipes). - - Especially useful for testing generators. - - `islice` is maybe more pythonic than `take` and `drop`; see below for a utility that supports the slice syntax. - - `tail`: return the tail of an iterable. Same as `drop(1, iterable)`; common use case. - - `butlast`, `butlastn`: return a generator that yields from iterable, dropping the last `n` items if the iterable is finite. Inspired by a similar utility in PG's [On Lisp](http://paulgraham.com/onlisp.html). - - Works by using intermediate storage. **Do not** use the original iterator after a call to `butlast` or `butlastn`. - - `first`, `second`, `nth`, `last`: return the specified item from an iterable. Any preceding items are consumed at C speed. - - `iterate1`, `iterate`: return an infinite generator that yields `x`, `f(x)`, `f(f(x))`, ... - - `iterate1` is for 1-to-1 functions; `iterate` for n-to-n, unpacking the return value to the argument list of the next call. - - `partition` from `itertools` [recipes](https://docs.python.org/3/library/itertools.html#itertools-recipes). - - `rev` is a convenience function that tries `reversed`, and if the input was not a sequence, converts it to a tuple and reverses that. The return value is a `reversed` object. - - `scons`: prepend one element to the start of an iterable, return new iterable. ``scons(x, iterable)`` is lispy shorthand for ``itertools.chain((x,), iterable)``, allowing to omit the one-item tuple wrapper. - - `inn`: contains-check (``x in iterable``) with automatic termination for monotonic divergent infinite iterables. *Added in v0.13.1.* - - Only applicable to monotonic divergent inputs (such as ``primes``). Increasing/decreasing is auto-detected from the first non-zero diff, but the function may fail to terminate if the input is actually not monotonic, or has an upper/lower bound. - - `iindex`: like ``list.index``, but for a general iterable. Consumes the iterable, so only makes sense for memoized inputs. *Added in v0.13.1.* - - `prod`: like the builtin `sum`, but compute the product. Oddly missing from the standard library. *Added in v0.13.1.* - - `window`: sliding length-n window iterator for general iterables. Acts like the well-known [n-gram zip trick](http://www.locallyoptimal.com/blog/2013/01/20/elegant-n-gram-generation-in-python/), but the input can be any iterable. *Added in v0.14.1.* - -Examples: - -```python -from functools import partial -from unpythonic import scanl, scanr, foldl, foldr, flatmap, mapr, zipr, \ - uniqify, uniq, flatten1, flatten, flatten_in, take, drop, \ - unfold, unfold1, cons, nil, ll, curry, s, inn, iindex, window - -assert tuple(scanl(add, 0, range(1, 5))) == (0, 1, 3, 6, 10) -assert tuple(scanr(add, 0, range(1, 5))) == (0, 4, 7, 9, 10) -assert tuple(scanl(mul, 1, range(2, 6))) == (1, 2, 6, 24, 120) -assert tuple(scanr(mul, 1, range(2, 6))) == (1, 5, 20, 60, 120) - -assert tuple(scanl(cons, nil, ll(1, 2, 3))) == (nil, ll(1), ll(2, 1), ll(3, 2, 1)) -assert tuple(scanr(cons, nil, ll(1, 2, 3))) == (nil, ll(3), ll(2, 3), ll(1, 2, 3)) - -def step2(k): # x0, x0 + 2, x0 + 4, ... - return (k, k + 2) # value, newstate -assert tuple(take(10, unfold1(step2, 10))) == (10, 12, 14, 16, 18, 20, 22, 24, 26, 28) - -def nextfibo(a, b): - return (a, b, a + b) # value, *newstates -assert tuple(take(10, unfold(nextfibo, 1, 1))) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) - -def fibos(): - a, b = 1, 1 - while True: - yield a - a, b = b, a + b -a1, a2, a3, tl = unpack(3, fibos()) -a4, a5, tl = unpack(2, tl) -print(a1, a2, a3, a4, a5, tl) # --> 1 1 2 3 5 - -# inn: contains-check with automatic termination for monotonic iterables (infinites ok) -evens = imemoize(s(2, 4, ...)) -assert inn(42, evens()) -assert not inn(41, evens()) - -@gmemoize -def primes(): - yield 2 - for n in count(start=3, step=2): - if not any(n % p == 0 for p in takewhile(lambda x: x*x <= n, primes())): - yield n -assert inn(31337, primes()) -assert not inn(1337, primes()) - -# iindex: find index of item in iterable (mostly only makes sense for memoized input) -assert iindex(2, (1, 2, 3)) == 1 -assert iindex(31337, primes()) == 3378 - -# window: length-n sliding window iterator for general iterables -lst = (x for x in range(5)) -out = [] -for a, b, c in window(lst, n=3): - out.append((a, b, c)) -assert out == [(0, 1, 2), (1, 2, 3), (2, 3, 4)] - -# flatmap -def msqrt(x): # multivalued sqrt - if x == 0.: - return (0.,) - else: - s = x**0.5 - return (s, -s) -assert tuple(flatmap(msqrt, (0, 1, 4, 9))) == (0., 1., -1., 2., -2., 3., -3.) - -# zipr reverses, then iterates. -assert tuple(zipr((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) - -zipr2 = partial(mapr, identity) # mapr works the same way. -assert tuple(zipr2((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) - -# foldr doesn't; it walks from the left, but collects results from the right: -zipr1 = curry(foldr, zipper, ()) -assert zipr1((1, 2, 3), (4, 5, 6), (7, 8)) == ((2, 5, 8), (1, 4, 7)) -# so the result is reversed(zip(...)), whereas zipr gives zip(*(reversed(s) for s in ...)) - -assert tuple(uniqify((1, 1, 2, 2, 2, 1, 2, 2, 4, 3, 4, 3, 3))) == (1, 2, 4, 3) # all -assert tuple(uniq((1, 1, 2, 2, 2, 1, 2, 2, 4, 3, 4, 3, 3))) == (1, 2, 1, 2, 4, 3, 4, 3) # consecutive - -assert tuple(flatten1(((1, 2), (3, (4, 5), 6), (7, 8, 9)))) == (1, 2, 3, (4, 5), 6, 7, 8, 9) -assert tuple(flatten(((1, 2), (3, (4, 5), 6), (7, 8, 9)))) == (1, 2, 3, 4, 5, 6, 7, 8, 9) - -is_nested = lambda sublist: all(isinstance(x, (list, tuple)) for x in sublist) -assert tuple(flatten((((1, 2), (3, 4)), (5, 6)), is_nested)) == ((1, 2), (3, 4), (5, 6)) - -data = (((1, 2), ((3, 4), (5, 6)), 7), ((8, 9), (10, 11))) -assert tuple(flatten(data, is_nested)) == (((1, 2), ((3, 4), (5, 6)), 7), (8, 9), (10, 11)) -assert tuple(flatten_in(data, is_nested)) == (((1, 2), (3, 4), (5, 6), 7), (8, 9), (10, 11)) - -with_n = lambda *args: (partial(f, n) for n, f in args) -clip = lambda n1, n2: composel(*with_n((n1, drop), (n2, take))) -assert tuple(clip(5, 10)(range(20))) == tuple(range(5, 15)) -``` - -In the last example, essentially we just want to `clip 5 10 (range 20)`, the grouping of the parentheses being pretty much an implementation detail. With ``curry``, we can rewrite the last line as: - -```python -assert tuple(curry(clip, 5, 10, range(20)) == tuple(range(5, 15)) -``` - - -### ``islice``: slice syntax support for ``itertools.islice`` - -*Added in v0.13.1.* - -Slice an iterable, using the regular slicing syntax: - -```python -from unpythonic import islice, primes, s - -p = primes() -assert tuple(islice(p)[10:15]) == (31, 37, 41, 43, 47) - -assert tuple(islice(primes())[10:15]) == (31, 37, 41, 43, 47) - -p = primes() -assert islice(p)[10] == 31 - -odds = islice(s(1, 2, ...))[::2] -assert tuple(islice(odds)[:5]) == (1, 3, 5, 7, 9) -assert tuple(islice(odds)[:5]) == (11, 13, 15, 17, 19) # five more -``` - -The slicing variant calls ``itertools.islice`` with the corresponding slicing parameters. - -As a convenience feature: a single index is interpreted as a length-1 islice starting at that index. The slice is then immediately evaluated and the item is returned. - -**CAUTION**: Keep in mind ``itertools.islice`` does not support negative indexing for any of ``start``, ``stop`` or ``step``, and that the slicing process consumes elements from the iterable. - -Like ``fup``, our ``islice`` is essentially a manually curried function with unusual syntax; the initial call to ``islice`` passes in the iterable to be sliced. The object returned by the call accepts a subscript to specify the slice or index. Once the slice or index is provided, the call to ``itertools.islice`` triggers. - -Inspired by Python itself. - - -### `gmemoize`, `imemoize`, `fimemoize`: memoize generators - -Make generator functions (gfunc, i.e. a generator definition) which create memoized generators, similar to how streams behave in Racket. - -Memoize iterables; like `itertools.tee`, but no need to know in advance how many copies of the iterator will be made. Provided for both iterables and for factory functions that make iterables. - - - `gmemoize` is a decorator for a gfunc, which makes it memoize the instantiated generators. - - If the gfunc takes arguments, they must be hashable. A separate memoized sequence is created for each unique set of argument values seen. - - For simplicity, the generator itself may use ``yield`` for output only; ``send`` is not supported. - - Any exceptions raised by the generator (except StopIteration) are also memoized, like in ``memoize``. - - Thread-safe. Calls to ``next`` on the memoized generator from different threads are serialized via a lock. Each memoized sequence has its own lock. This uses ``threading.RLock``, so re-entering from the same thread (e.g. in recursively defined sequences) is fine. - - The whole history is kept indefinitely. For infinite iterables, use this only if you can guarantee that only a reasonable number of terms will ever be evaluated (w.r.t. available RAM). - - Typically, this should be the outermost decorator if several are used on the same gfunc. - - `imemoize`: memoize an iterable. Like `itertools.tee`, but keeps the whole history, so more copies can be teed off later. - - Same limitation: **do not** use the original iterator after it is memoized. The danger is that if anything other than the memoization mechanism advances the original iterator, some values will be lost before they can reach the memo. - - Returns a gfunc with no parameters which, when called, returns a generator that yields items from the memoized iterable. The original iterable is used to retrieve more terms when needed. - - Calling the gfunc essentially tees off a new instance, which begins from the first memoized item. - - `fimemoize`: convert a factory function, that returns an iterable, into the corresponding gfunc, and `gmemoize` that. Return the memoized gfunc. - - Especially convenient with short lambdas, where `(yield from ...)` instead of `...` is just too much text. - -```python -from itertools import count, takewhile -from unpythonic import gmemoize, imemoize, fimemoize, take, nth - -@gmemoize -def primes(): # FP sieve of Eratosthenes - yield 2 - for n in count(start=3, step=2): - if not any(n % p == 0 for p in takewhile(lambda x: x*x <= n, primes())): - yield n -assert tuple(take(10, primes())) == (2, 3, 5, 7, 11, 13, 17, 19, 23, 29) -assert nth(3378, primes()) == 31337 # with memo, linear process; no crash - -# but be careful: -31337 in primes() # --> True -1337 in takewhile(lambda p: p <= 1337, primes()) # not prime, need takewhile() to stop - -# or use unpythonic.inn, which auto-terminates on monotonic iterables: -from unpythonic import inn -inn(31337, primes()) # --> True -inn(1337, primes()) # --> False -``` - -Memoizing only a part of an iterable. This is where `imemoize` and `fimemoize` can be useful. The basic idea is to make a chain of generators, and only memoize the last one: - -```python -from unpythonic import gmemoize, drop, last - -def evens(): # the input iterable - yield from (x for x in range(100) if x % 2 == 0) - -@gmemoize -def some_evens(n): # we want to memoize the result without the n first terms - yield from drop(n, evens()) - -assert last(some_evens(25)) == last(some_evens(25)) # iterating twice! -``` - -Using a lambda, we can also write ``some_evens`` as: - -```python -se = gmemoize(lambda n: (yield from drop(n, evens()))) -assert last(se(25)) == last(se(25)) -``` - -Using `fimemoize`, we can omit the ``yield from``, shortening this to: - -```python -se = fimemoize(lambda n: drop(n, evens())) -assert last(se(25)) == last(se(25)) -``` - -If we don't need to take an argument, we can memoize the iterable directly, using ``imemoize``: - -```python -se = imemoize(drop(25, evens())) -assert last(se()) == last(se()) # se is a gfunc, so call it to get a generator instance -``` - -Finally, compare the `fimemoize` example, rewritten using `def`, to the original `gmemoize` example: - -```python -@fimemoize -def some_evens(n): - return drop(n, evens()) - -@gmemoize -def some_evens(n): - yield from drop(n, evens()) -``` - -The only differences are the name of the decorator and ``return`` vs. ``yield from``. The point of `fimemoize` is that in simple cases like this, it allows us to use a regular factory function that makes an iterable, instead of a gfunc. Of course, the gfunc could have several `yield` expressions before it finishes, whereas the factory function terminates at the `return`. - - -### ``fup``: Functional update; ``ShadowedSequence`` - -We provide ``ShadowedSequence``, which is a bit like ``collections.ChainMap``, but for sequences, and only two levels (but it's a sequence; instances can be chained). See its docstring for details. - -The function ``fupdate`` functionally updates sequences and mappings. Whereas ``ShadowedSequence`` reads directly from the original sequences at access time, ``fupdate`` makes a shallow copy (of the same type as the given input sequence) when it finalizes its output. The utility function ``fup`` is a specialization of ``fupdate`` to sequences, and adds support for the standard slicing syntax. - -First, let's look at ``fupdate``: - -```python -from unpythonic import fupdate - -lst = [1, 2, 3] -out = fupdate(lst, 1, 42) -assert lst == [1, 2, 3] # the original remains untouched -assert out == [1, 42, 3] - -lst = [1, 2, 3] -out = fupdate(lst, -1, 42) # negative indices also supported -assert lst == [1, 2, 3] -assert out == [1, 2, 42] -``` - -Immutable input sequences are allowed. Replacing a slice of a tuple by a sequence: - -```python -from itertools import repeat -lst = (1, 2, 3, 4, 5) -assert fupdate(lst, slice(0, None, 2), tuple(repeat(10, 3))) == (10, 2, 10, 4, 10) -assert fupdate(lst, slice(1, None, 2), tuple(repeat(10, 2))) == (1, 10, 3, 10, 5) -assert fupdate(lst, slice(None, None, 2), tuple(repeat(10, 3))) == (10, 2, 10, 4, 10) -assert fupdate(lst, slice(None, None, -1), tuple(range(5))) == (4, 3, 2, 1, 0) -``` - -Slicing supports negative indices and steps, and default starts, stops and steps, as usual in Python. Just remember ``a[start:stop:step]`` actually means ``a[slice(start, stop, step)]`` (with ``None`` replacing omitted ``start``, ``stop`` and ``step``), and everything should follow. Multidimensional arrays are **not** supported. - -When ``fupdate`` constructs its output, the replacement occurs by walking *the input sequence* left-to-right, and pulling an item from the replacement sequence when the given replacement specification so requires. Hence the replacement sequence is not necessarily accessed left-to-right. (In the last example above, ``tuple(range(5))`` was read in the order ``(4, 3, 2, 1, 0)``.) - -The replacement sequence must have at least as many items as the slice requires (when applied to the original input). Any extra items in the replacement sequence are simply ignored (so e.g. an infinite ``repeat`` is fine), but if the replacement is too short, ``IndexError`` is raised. (*Changed in v0.13.1.* This was previously ``ValueError``.) - -It is also possible to replace multiple individual items. These are treated as separate specifications, applied left to right (so later updates shadow earlier ones, if updating at the same index): - -```python -lst = (1, 2, 3, 4, 5) -out = fupdate(lst, (1, 2, 3), (17, 23, 42)) -assert lst == (1, 2, 3, 4, 5) -assert out == (1, 17, 23, 42, 5) -``` - -Multiple specifications can be used with slices and sequences as well: - -```python -lst = tuple(range(10)) -out = fupdate(lst, (slice(0, 10, 2), slice(1, 10, 2)), - (tuple(repeat(2, 5)), tuple(repeat(3, 5)))) -assert lst == tuple(range(10)) -assert out == (2, 3, 2, 3, 2, 3, 2, 3, 2, 3) -``` - -Strictly speaking, each specification can be either a slice/sequence pair or an index/item pair: - -```python -lst = tuple(range(10)) -out = fupdate(lst, (slice(0, 10, 2), slice(1, 10, 2), 6), - (tuple(repeat(2, 5)), tuple(repeat(3, 5)), 42)) -assert lst == tuple(range(10)) -assert out == (2, 3, 2, 3, 2, 3, 42, 3, 2, 3) -``` - -Also mappings can be functionally updated: - -```python -d1 = {'foo': 'bar', 'fruit': 'apple'} -d2 = fupdate(d1, foo='tavern') -assert sorted(d1.items()) == [('foo', 'bar'), ('fruit', 'apple')] -assert sorted(d2.items()) == [('foo', 'tavern'), ('fruit', 'apple')] -``` - -For immutable mappings, ``fupdate`` supports ``frozendict`` (see below). Any other mapping is assumed mutable, and ``fupdate`` essentially just performs ``copy.copy()`` and then ``.update()``. - -We can also functionally update a namedtuple: - -```python -from collections import namedtuple -A = namedtuple("A", "p q") -a = A(17, 23) -out = fupdate(a, 0, 42) -assert a == A(17, 23) -assert out == A(42, 23) -``` - -Namedtuples export only a sequence interface, so they cannot be treated as mappings. - -Support for ``namedtuple`` requires an extra feature, which is available for custom classes, too. When constructing the output sequence, ``fupdate`` first checks whether the input type has a ``._make()`` method, and if so, hands the iterable containing the final data to that to construct the output. Otherwise the regular constructor is called (and it must accept a single iterable). - -**The preferred way** to use ``fupdate`` on sequences is through the ``fup`` utility function, which adds support for Python's standard slicing syntax: - -```python -from unpythonic import fup -from itertools import repeat - -lst = (1, 2, 3, 4, 5) -assert fup(lst)[3] << 42 == (1, 2, 3, 42, 5) -assert fup(lst)[0::2] << tuple(repeat(10, 3)) == (10, 2, 10, 4, 10) -``` - -Currently only one update specification is supported in a single ``fup()``. - -The notation follows the ``unpythonic`` convention that ``<<`` denotes an assignment of some sort. Here it denotes a functional update, which returns a modified copy, leaving the original untouched. - -The ``fup`` call is essentially curried. It takes in the sequence to be functionally updated. The object returned by the call accepts a subscript to specify the index or indices. This then returns another object that accepts a left-shift to specify the values. Once the values are provided, the underlying call to ``fupdate`` triggers, and the result is returned. - -*Changed in v0.13.1.* Added support to ``ShadowedSequence`` for slicing (read-only), equality comparison, ``str`` and ``repr``. Out-of-range read access to a single item emits a meaningful error, like in ``list``. The utility ``fup`` was previously a macro; now it is a regular function, with slightly changed syntax to accommodate. - - -### ``view``: writable, sliceable view into a sequence - -*Added in v0.14.0.* Added the read-only cousin ``roview``, which behaves the same except it has no ``__setitem__`` or ``reverse``. This can be useful for giving read-only access to an internal sequence. The constructor of the writable ``view`` now checks that the input is not read-only (``roview``, or a ``Sequence`` that is not also a ``MutableSequence``) before allowing creation of the writable view. - -*Added in v0.13.1.* - -A writable view into a sequence, with slicing, so you can take a slice of a slice (of a slice ...), and it reflects the original both ways: - -```python -from unpythonic import view - -lst = list(range(10)) -v = view(lst)[::2] -assert v == [0, 2, 4, 6, 8] -v2 = v[1:-1] -assert v2 == [2, 4, 6] -v2[1:] = (10, 20) -assert lst == [0, 1, 2, 3, 10, 5, 20, 7, 8, 9] - -lst[2] = 42 -assert v == [0, 42, 10, 20, 8] -assert v2 == [42, 10, 20] - -lst = list(range(5)) -v = view(lst)[2:4] -v[:] = 42 # scalar broadcast -assert lst == [0, 1, 42, 42, 4] -``` - -While ``fupdate`` lets you be more functional than Python otherwise allows, ``view`` lets you be more imperative than Python otherwise allows. - -We store slice specs, not actual indices, so this works also if the underlying sequence undergoes length changes. - -Slicing a view returns a new view. Slicing anything else will usually copy, because the object being sliced does, before we get control. To slice lazily, first view the sequence itself and then slice that. The initial no-op view is optimized away, so it won't slow down accesses. Alternatively, pass a ``slice`` object into the ``view`` constructor. - -The view can be efficiently iterated over. As usual, iteration assumes that no inserts/deletes in the underlying sequence occur during the iteration. - -Getting/setting an item (subscripting) checks whether the index cache needs updating during each access, so it can be a bit slow. Setting a slice checks just once, and then updates the underlying iterable directly. Setting a slice to a scalar value broadcasts the scalar à la NumPy. - -The ``unpythonic.collections`` module also provides the ``SequenceView`` and ``MutableSequenceView`` abstract base classes; ``view`` is a ``MutableSequenceView``. - - -### ``mogrify``: update a mutable container in-place - -*Added in v0.13.0.* - -Recurse on given container, apply a function to each atom. If the container is mutable, then update in-place; if not, then construct a new copy like ``map`` does. - -If the container is a mapping, the function is applied to the values; keys are left untouched. - -Unlike ``map`` and its cousins, only a single input container is supported. (Supporting multiple containers as input would require enforcing some compatibility constraints on their type and shape, since ``mogrify`` is not limited to sequences.) - -```python -from unpythonic import mogrify - -lst1 = [1, 2, 3] -lst2 = mogrify(lst1, lambda x: x**2) -assert lst2 == [2, 4, 6] -assert lst2 is lst1 -``` - -Containers are detected by checking for instances of ``collections.abc`` superclasses (also virtuals are ok). Supported abcs are ``MutableMapping``, ``MutableSequence``, ``MutableSet``, ``Mapping``, ``Sequence`` and ``Set``. Any value that does not match any of these is treated as an atom. Containers can be nested, with an arbitrary combination of the types supported. - -For convenience, we introduce some special cases: - - - Any classes created by ``collections.namedtuple``, because they do not conform to the standard constructor API for a ``Sequence``. - - Thus, for (an immutable) ``Sequence``, we first check for the presence of a ``._make()`` method, and if found, use it as the constructor. Otherwise we use the regular constructor. - - - ``str`` is treated as an atom, although technically a ``Sequence``. - - It doesn't conform to the exact same API (its constructor does not take an iterable), and often we don't want to treat strings as containers anyway. - - If you want to process strings, implement it in your function that is called by ``mogrify``. - - - The ``box`` container from ``unpythonic.collections``; although mutable, its update is not conveniently expressible by the ``collections.abc`` APIs. - - - The ``cons`` container from ``unpythonic.llist`` (including the ``ll``, ``llist`` linked lists). This is treated with the general tree strategy, so nested linked lists will be flattened, and the final ``nil`` is also processed. - - Note that since ``cons`` is immutable, anyway, if you know you have a long linked list where you need to update the values, just iterate over it and produce a new copy - that will work as intended. - - -### ``s``, ``m``, ``mg``: lazy mathematical sequences with infix arithmetic - -*Added in v0.13.0.* - -*Added in v0.13.1:* ``primes`` and ``fibonacci``. - -*Added in v0.14.0:* ``mg``, a decorator to mathify a gfunc, so that it will ``m()`` the generator instances it makes. Combo with ``imemoize`` for great justice, e.g. ``a = mg(imemoize(s(1, 2, ...)))``. - -We provide a compact syntax to create lazy constant, arithmetic, geometric and power sequences: ``s(...)``. Numeric (``int``, ``float``, ``mpmath``) and symbolic (SymPy) formats are supported. We avoid accumulating roundoff error when used with floating-point formats. - -We also provide arithmetic operation support for iterables (termwise). To make any iterable infix math aware, use ``m(iterable)``. The arithmetic is lazy; it just plans computations, returning a new lazy mathematical sequence. To extract values, iterate over the result. (Note this implies that expressions consisting of thousands of operations will overflow Python's call stack. In practice this shouldn't be a problem.) - -The function versions of the arithmetic operations (also provided, à la the ``operator`` module) have an **s** prefix (short for mathematical **sequence**), because in Python the **i** prefix (which could stand for *iterable*) is already used to denote the in-place operators. - -We provide the [Cauchy product](https://en.wikipedia.org/wiki/Cauchy_product), and its generalization, the diagonal combination-reduction, for two (possibly infinite) iterables. Note ``cauchyprod`` **does not sum the series**; given the input sequences ``a`` and ``b``, the call ``cauchyprod(a, b)`` computes the elements of the output sequence ``c``. - -Finally, we provide ready-made generators that yield some common sequences (currently, the Fibonacci numbers and the prime numbers). The prime generator is an FP-ized sieve of Eratosthenes. - -```python -from unpythonic import s, m, cauchyprod, take, last, fibonacci, primes - -assert tuple(take(10, s(1, ...))) == (1,)*10 -assert tuple(take(10, s(1, 2, ...))) == tuple(range(1, 11)) -assert tuple(take(10, s(1, 2, 4, ...))) == (1, 2, 4, 8, 16, 32, 64, 128, 256, 512) -assert tuple(take(5, s(2, 4, 16, ...))) == (2, 4, 16, 256, 65536) # 2, 2**2, (2**2)**2, ... - -assert tuple(s(1, 2, ..., 10)) == tuple(range(1, 11)) -assert tuple(s(1, 2, 4, ..., 512)) == (1, 2, 4, 8, 16, 32, 64, 128, 256, 512) - -assert tuple(take(5, s(1, -1, 1, ...))) == (1, -1, 1, -1, 1) - -assert tuple(take(5, s(1, 3, 5, ...) + s(2, 4, 6, ...))) == (3, 7, 11, 15, 19) -assert tuple(take(5, s(1, 3, ...) * s(2, 4, ...))) == (2, 12, 30, 56, 90) - -assert tuple(take(5, s(1, 3, ...)**s(2, 4, ...))) == (1, 3**4, 5**6, 7**8, 9**10) -assert tuple(take(5, s(1, 3, ...)**2)) == (1, 3**2, 5**2, 7**2, 9**2) -assert tuple(take(5, 2**s(1, 3, ...))) == (2**1, 2**3, 2**5, 2**7, 2**9) - -assert tuple(take(3, cauchyprod(s(1, 3, 5, ...), s(2, 4, 6, ...)))) == (2, 10, 28) - -assert tuple(take(10, primes())) == (2, 3, 5, 7, 11, 13, 17, 19, 23, 29) -assert tuple(take(10, fibonacci())) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) -``` - -A math iterable (i.e. one that has infix math support) is an instance of the class ``m``: - -```python -a = s(1, 3, ...) -b = s(2, 4, ...) -c = a + b -assert isinstance(a, m) -assert isinstance(b, m) -assert isinstance(c, m) -assert tuple(take(5, c)) == (3, 7, 11, 15, 19) - -d = 1 / (a + b) -assert isinstance(d, m) -``` - -Applying an operation meant for regular (non-math) iterables will drop the arithmetic support, but it can be restored by m'ing manually: - -```python -e = take(5, c) -assert not isinstance(e, m) - -f = m(take(5, c)) -assert isinstance(f, m) -``` - -Symbolic expression support with SymPy: - -```python -from unpythonic import s -from sympy import symbols - -x0 = symbols("x0", real=True) -k = symbols("k", positive=True) - -assert tuple(take(4, s(x0, ...))) == (x0, x0, x0, x0) -assert tuple(take(4, s(x0, x0 + k, ...))) == (x0, x0 + k, x0 + 2*k, x0 + 3*k) -assert tuple(take(4, s(x0, x0*k, x0*k**2, ...))) == (x0, x0*k, x0*k**2, x0*k**3) - -assert tuple(s(x0, x0 + k, ..., x0 + 3*k)) == (x0, x0 + k, x0 + 2*k, x0 + 3*k) -assert tuple(s(x0, x0*k, x0*k**2, ..., x0*k**5)) == (x0, x0*k, x0*k**2, x0*k**3, x0*k**4, x0*k**5) - -x0, k = symbols("x0, k", positive=True) -assert tuple(s(x0, x0**k, x0**(k**2), ..., x0**(k**4))) == (x0, x0**k, x0**(k**2), x0**(k**3), x0**(k**4)) - -x = symbols("x", real=True) -px = lambda stream: stream * s(1, x, x**2, ...) # powers of x -s1 = px(s(1, 3, 5, ...)) # 1, 3*x, 5*x**2, ... -s2 = px(s(2, 4, 6, ...)) # 2, 4*x, 6*x**2, ... -assert tuple(take(3, cauchyprod(s1, s2))) == (2, 10*x, 28*x**2) -``` - -**CAUTION**: Symbolic sequence detection is sensitive to the assumptions on the symbols, because very pythonically, ``SymPy`` only simplifies when the result is guaranteed to hold in the most general case under the given assumptions. - -Inspired by Haskell. - - -## Control flow tools - -Tools related to control flow. - -### ``trampolined``, ``jump``: tail call optimization (TCO) / explicit continuations - -**v0.10.0**: ``fasttco`` has been renamed ``tco``, and the exception-based old default implementation has been removed. See also [macros](macro_extras/) for an easy-to-use solution. - -**v0.11.1**: The special jump target ``SELF`` (keep current target) has been removed. If you need tail recursion in a lambda, use ``unpythonic.fun.withself`` to get a reference to the lambda itself. See example below. - -Express algorithms elegantly without blowing the call stack - with explicit, clear syntax. - -*Tail recursion*: - -```python -from unpythonic import trampolined, jump - -@trampolined -def fact(n, acc=1): - if n == 0: - return acc - else: - return jump(fact, n - 1, n * acc) -print(fact(4)) # 24 -``` - -Functions that use TCO **must** be `@trampolined`. Calling a trampolined function normally starts the trampoline. - -Inside a trampolined function, a normal call `f(a, ..., kw=v, ...)` remains a normal call. - -A tail call with target `f` is denoted `return jump(f, a, ..., kw=v, ...)`. This explicitly marks that it is indeed a tail call (due to the explicit ``return``). Note that `jump` is **a noun, not a verb**. The `jump(f, ...)` part just evaluates to a `jump` instance, which on its own does nothing. Returning it to the trampoline actually performs the tail call. - -If the jump target has a trampoline, don't worry; the trampoline implementation will automatically strip it and jump into the actual entrypoint. - -Trying to ``jump(...)`` without the ``return`` does nothing useful, and will **usually** print an *unclaimed jump* warning. It does this by checking a flag in the ``__del__`` method of ``jump``; any correctly used jump instance should have been claimed by a trampoline before it gets garbage-collected. - -(Some *unclaimed jump* warnings may appear also if the process is terminated by Ctrl+C (``KeyboardInterrupt``). This is normal; it just means that the termination occurred after a jump object was instantiated but before it was claimed by the trampoline.) - -The final result is just returned normally. This shuts down the trampoline, and returns the given value from the initial call (to a ``@trampolined`` function) that originally started that trampoline. - - -*Tail recursion in a lambda*: - -```python -t = trampolined(withself(lambda self, n, acc=1: - acc if n == 0 else jump(self, n - 1, n * acc))) -print(t(4)) # 24 -``` - -Here the jump is just `jump` instead of `return jump`, since lambda does not use the `return` syntax. - -To denote tail recursion in an anonymous function, use ``unpythonic.fun.withself``. The ``self`` argument is declared explicitly, but passed implicitly, just like the ``self`` argument of a method. - - -*Mutual recursion with TCO*: - -```python -@trampolined -def even(n): - if n == 0: - return True - else: - return jump(odd, n - 1) -@trampolined -def odd(n): - if n == 0: - return False - else: - return jump(even, n - 1) -assert even(42) is True -assert odd(4) is False -assert even(10000) is True # no crash -``` - -*Mutual recursion in `letrec` with TCO*: - -```python -letrec(evenp=lambda e: - trampolined(lambda x: - (x == 0) or jump(e.oddp, x - 1)), - oddp=lambda e: - trampolined(lambda x: - (x != 0) and jump(e.evenp, x - 1)), - body=lambda e: - e.evenp(10000)) -``` - - -#### Reinterpreting TCO as explicit continuations - -TCO from another viewpoint: - -```python -@trampolined -def foo(): - return jump(bar) -@trampolined -def bar(): - return jump(baz) -@trampolined -def baz(): - print("How's the spaghetti today?") -foo() -``` - -Each function in the TCO call chain tells the trampoline where to go next (and with what arguments). All hail [lambda, the ultimate GOTO](http://hdl.handle.net/1721.1/5753)! - -Each TCO call chain brings its own trampoline, so they nest as expected: - -```python -@trampolined -def foo(): - return jump(bar) -@trampolined -def bar(): - t = even(42) # start another trampoline for even/odd - return jump(baz, t) -@trampolined -def baz(result): - print(result) -foo() # start trampoline -``` - - -### ``looped``, ``looped_over``: loops in FP style (with TCO) - -*Functional loop with automatic tail call optimization* (for calls re-invoking the loop body): - -```python -from unpythonic import looped, looped_over - -@looped -def s(loop, acc=0, i=0): - if i == 10: - return acc - else: - return loop(acc + i, i + 1) -print(s) # 45 -``` - -Compare the sweet-exp Racket: - -```racket -define s - let loop ([acc 0] [i 0]) - cond - {i = 10} - acc - else - loop {acc + i} {i + 1} -displayln s ; 45 -``` - -The `@looped` decorator is essentially sugar. Behaviorally equivalent code: - -```python -@trampolined -def s(acc=0, i=0): - if i == 10: - return acc - else: - return jump(s, acc + i, i + 1) -s = s() -print(s) # 45 -``` - -In `@looped`, the function name of the loop body is the name of the final result, like in `@call`. The final result of the loop is just returned normally. - -The first parameter of the loop body is the magic parameter ``loop``. It is *self-ish*, representing a jump back to the loop body itself, starting a new iteration. Just like Python's ``self``, ``loop`` can have any name; it is passed positionally. - -Note that ``loop`` is **a noun, not a verb.** This is because the expression ``loop(...)`` is essentially the same as ``jump(...)`` to the loop body itself. However, it also inserts the magic parameter ``loop``, which can only be set up via this mechanism. - -Additional arguments can be given to ``loop(...)``. When the loop body is called, any additional positional arguments are appended to the implicit ones, and can be anything. Additional arguments can also be passed by name. The initial values of any additional arguments **must** be declared as defaults in the formal parameter list of the loop body. The loop is automatically started by `@looped`, by calling the body with the magic ``loop`` as the only argument. - -Any loop variables such as ``i`` in the above example are **in scope only in the loop body**; there is no ``i`` in the surrounding scope. Moreover, it's a fresh ``i`` at each iteration; nothing is mutated by the looping mechanism. (But be careful if you use a mutable object instance as a loop variable. The loop body is just a function call like any other, so the usual rules apply.) - -FP loops don't have to be pure: - -```python -out = [] -@looped -def _(loop, i=0): - if i <= 3: - out.append(i) # cheeky side effect - return loop(i + 1) - # the implicit "return None" terminates the loop. -assert out == [0, 1, 2, 3] -``` - -Keep in mind, though, that this pure-Python FP looping mechanism is slow, so it may make sense to use it only when "the FP-ness" (no mutation, scoping) is important. - -Also be aware that `@looped` is specifically neither a ``for`` loop nor a ``while`` loop; instead, it is a general looping mechanism that can express both kinds of loops. - -*Typical `while True` loop in FP style*: - -```python -@looped -def _(loop): - print("Enter your name (or 'q' to quit): ", end='') - s = input() - if s.lower() == 'q': - return # ...the implicit None. In a "while True:", "break" here. - else: - print("Hello, {}!".format(s)) - return loop() -``` - -#### FP loop over an iterable - -In Python, loops often run directly over the elements of an iterable, which markedly improves readability compared to dealing with indices. Enter ``@looped_over``: - -```python -@looped_over(range(10), acc=0) -def s(loop, x, acc): - return loop(acc + x) -assert s == 45 -``` - -The ``@looped_over`` decorator is essentially sugar. Behaviorally equivalent code: - -```python -@call -def s(iterable=range(10)): - it = iter(iterable) - @looped - def _tmp(loop, acc=0): - try: - x = next(it) - return loop(acc + x) - except StopIteration: - return acc - return _tmp -assert s == 45 -``` - -In ``@looped_over``, the loop body takes three magic positional parameters. The first parameter ``loop`` works like in ``@looped``. The second parameter ``x`` is the current element. The third parameter ``acc`` is initialized to the ``acc`` value given to ``@looped_over``, and then (functionally) updated at each iteration, taking as the new value the first positional argument given to ``loop(...)``, if any positional arguments were given. Otherwise ``acc`` retains its last value. - -Additional arguments can be given to ``loop(...)``. The same notes as above apply. For example, here we have the additional parameters ``fruit`` and ``number``. The first one is passed positionally, and the second one by name: - -```python -@looped_over(range(10), acc=0) -def s(loop, x, acc, fruit="pear", number=23): - print(fruit, number) - newfruit = "orange" if fruit == "apple" else "apple" - newnumber = number + 1 - return loop(acc + x, newfruit, number=newnumber) -assert s == 45 -``` - -The loop body is called once for each element in the iterable. When the iterable runs out of elements, the last ``acc`` value that was given to ``loop(...)`` becomes the return value of the loop. If the iterable is empty, the body never runs; then the return value of the loop is the initial value of ``acc``. - -To terminate the loop early, just ``return`` your final result normally, like in ``@looped``. (It can be anything, does not need to be ``acc``.) - -Multiple input iterables work somewhat like in Python's ``for``, except any sequence unpacking must be performed inside the body: - -```python -@looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=()) -def p(loop, item, acc): - numb, lett = item - return loop(acc + ("{:d}{:s}".format(numb, lett),)) -assert p == ('1a', '2b', '3c') - -@looped_over(enumerate(zip((1, 2, 3), ('a', 'b', 'c'))), acc=()) -def q(loop, item, acc): - idx, (numb, lett) = item - return loop(acc + ("Item {:d}: {:d}{:s}".format(idx, numb, lett),)) -assert q == ('Item 0: 1a', 'Item 1: 2b', 'Item 2: 3c') -``` - -FP loops can be nested (also those over iterables): - -```python -@looped_over(range(1, 4), acc=()) -def outer_result(outer_loop, y, outer_acc): - @looped_over(range(1, 3), acc=()) - def inner_result(inner_loop, x, inner_acc): - return inner_loop(inner_acc + (y*x,)) - return outer_loop(outer_acc + (inner_result,)) -assert outer_result == ((1, 2), (2, 4), (3, 6)) -``` - -If you feel the trailing commas ruin the aesthetics, see ``unpythonic.misc.pack``. - -#### Accumulator type and runtime cost - -As [the reference warns (note 6)](https://docs.python.org/3/library/stdtypes.html#common-sequence-operations), repeated concatenation of tuples has an O(n²) runtime cost, because each concatenation creates a new tuple, which needs to copy all of the already existing elements. To keep the runtime O(n), there are two options: - - - *Pythonic solution*: Destructively modify a mutable sequence. Particularly, ``list`` is a dynamic array that has a low amortized cost for concatenation (most often O(1), with the occasional O(n) when the allocated storage grows). - - *Unpythonic solution*: ``cons`` a linked list, and reverse it at the end. Cons cells are immutable; consing a new element to the front costs O(1). Reversing the list costs O(n). - -Mutable sequence (Python ``list``): - -```python -@looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=[]) -def p(loop, item, acc): - numb, lett = item - acc.append("{:d}{:s}".format(numb, lett)) - return loop(acc) -assert p == ['1a', '2b', '3c'] -``` - -Linked list: - -```python -from unpythonic import cons, nil, ll - -@lreverse -@looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=nil) -def p(loop, item, acc): - numb, lett = item - return loop(cons("{:d}{:s}".format(numb, lett), acc)) -assert p == ll('1a', '2b', '3c') -``` - -Note the unpythonic use of the ``lreverse`` function as a decorator. ``@looped_over`` overwrites the def'd name by the return value of the loop; then ``lreverse`` takes that as input, and overwrites once more. Thus ``p`` becomes the final list. - -To get the output as a tuple, we can add ``tuple`` to the decorator chain: - -```python -@tuple -@lreverse -@looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=nil) -def p(loop, item, acc): - numb, lett = item - return loop(cons("{:d}{:s}".format(numb, lett), acc)) -assert p == ('1a', '2b', '3c') -``` - -This works in both solutions. The cost is an additional O(n) step. - -#### ``break`` - -The main way to exit an FP loop (also early) is, at any time, to just ``return`` the final result normally. - -If you want to exit the function *containing* the loop from inside the loop, see **escape continuations** below. - -#### ``continue`` - -The main way to *continue* an FP loop is, at any time, to ``loop(...)`` with the appropriate arguments that will make it proceed to the next iteration. Or package the appropriate `loop(...)` expression into your own function ``cont``, and then use ``cont(...)``: - -```python -@looped -def s(loop, acc=0, i=0): - cont = lambda newacc=acc: loop(newacc, i + 1) # always increase i; by default keep current value of acc - if i <= 4: - return cont() # essentially "continue" for this FP loop - elif i == 10: - return acc - else: - return cont(acc + i) -print(s) # 35 -``` - -This approach separates the computations of the new values for the iteration counter and the accumulator. - -#### Prepackaged ``break`` and ``continue`` - -See ``@breakably_looped`` (offering `brk`) and ``@breakably_looped_over`` (offering `brk` and `cnt`). - -The point of `brk(value)` over just `return value` is that `brk` is first-class, so it can be passed on to functions called by the loop body (so that those functions then have the power to directly terminate the loop). - -In ``@looped``, a library-provided ``cnt`` wouldn't make sense, since all parameters except ``loop`` are user-defined. *The client code itself defines what it means to proceed to the "next" iteration*. Really the only way in a construct with this degree of flexibility is for the client code to fill in all the arguments itself. - -Because ``@looped_over`` is a more specific abstraction, there the concept of *continue* is much more clear-cut. We define `cnt` to mean *proceed to take the next element from the iterable, keeping the current value of `acc`*. Essentially `cnt` is a partially applied `loop(...)` with the first positional argument set to the current value of `acc`. - -#### FP loops using a lambda as body - -Just call the `looped()` decorator manually: - -```python -s = looped(lambda loop, acc=0, i=0: - loop(acc + i, i + 1) if i < 10 else acc) -print(s) -``` - -It's not just a decorator; in Lisps, a construct like this would likely be named ``call/looped``. - -We can also use ``let`` to make local definitions: - -```python -s = looped(lambda loop, acc=0, i=0: - let(cont=lambda newacc=acc: - loop(newacc, i + 1), - body=lambda e: - e.cont(acc + i) if i < 10 else acc) -print(s) -``` - -The `looped_over()` decorator also works, if we just keep in mind that parameterized decorators in Python are actually decorator factories: - -```python -r10 = looped_over(range(10), acc=0) -s = r10(lambda loop, x, acc: - loop(acc + x)) -assert s == 45 -``` - -If you **really** need to make that into an expression, bind ``r10`` using ``let`` (if you use ``letrec``, keeping in mind it is a callable), or to make your code unreadable, just inline it. - -With ``curry``, this is also a possible solution: - -```python -s = curry(looped_over, range(10), 0, - lambda loop, x, acc: - loop(acc + x)) -assert s == 45 -``` - -### ``gtrampolined``: generators with TCO - -In ``unpythonic``, a generator can tail-chain into another generator. This is like invoking ``itertools.chain``, but as a tail call from inside the generator - so the generator itself can choose the next iterable in the chain. If the next iterable is a generator, it can again tail-chain into something else. If it is not a generator, it becomes the last iterable in the TCO chain. - -Python provides a convenient hook to build things like this, in the guise of ``return``: - -```python -from unpythonic import gtco, take, last - -def march(): - yield 1 - yield 2 - return march() # tail-chain to a new instance of itself -assert tuple(take(6, gtco(march()))) == (1, 2, 1, 2, 1, 2) -last(take(10000, gtco(march()))) # no crash -``` - -Note the calls to ``gtco`` at the use sites. For convenience, we provide ``@gtrampolined``, which automates that: - -```python -from unpythonic import gtrampolined, take, last - -@gtrampolined -def ones(): - yield 1 - return ones() -assert tuple(take(10, ones())) == (1,) * 10 -last(take(10000, ones())) # no crash -``` - -It is safe to tail-chain into a ``@gtrampolined`` generator; the system strips the TCO target's trampoline if it has one. - -Like all tail calls, this works for any *iterative* process. In contrast, this **does not work**: - -```python -from operator import add -from unpythonic import gtrampolined, scanl, take - -@gtrampolined -def fibos(): # see numerics.py - yield 1 - return scanl(add, 1, fibos()) -print(tuple(take(10, fibos()))) # --> (1, 1, 2), only 3 terms?! -``` - -This sequence (technically iterable, but in the mathematical sense) is recursively defined, and the ``return`` shuts down the generator before it can yield more terms into ``scanl``. With ``yield from`` instead of ``return`` the second example works (but since it is recursive, it eventually blows the call stack). - -This particular example can be converted into a linear process with a different higher-order function, no TCO needed: - -```python -from unpythonic import unfold, take, last -def fibos(): - def nextfibo(a, b): - return a, b, a + b # value, *newstates - return unfold(nextfibo, 1, 1) -assert tuple(take(10, fibos())) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) -last(take(10000, fibos())) # no crash -``` - - -### ``setescape``, ``escape``: escape continuations (ec) - -Escape continuations can be used as a *multi-return*: - -```python -from unpythonic import setescape, escape - -@setescape() # note the parentheses -def f(): - def g(): - escape("hello from g") # the argument becomes the return value of f() - print("not reached") - g() - print("not reached either") -assert f() == "hello from g" -``` - -**CAUTION**: The implementation is based on exceptions, so catch-all ``except:`` statements will intercept also escapes, breaking the escape mechanism. As you already know, be specific in what you catch! - -In Lisp terms, `@setescape` essentially captures the escape continuation (ec) of the function decorated with it. The nearest (dynamically) surrounding ec can then be invoked by `escape(value)`. The function decorated with `@setescape` immediately terminates, returning ``value``. - -In Python terms, an escape means just raising a specific type of exception; the usual rules concerning ``try/except/else/finally`` and ``with`` blocks apply. It is a function call, so it works also in lambdas. - -Escaping the function surrounding an FP loop, from inside the loop: - -```python -@setescape() -def f(): - @looped - def s(loop, acc=0, i=0): - if i > 5: - escape(acc) - return loop(acc + i, i + 1) - print("never reached") -f() # --> 15 -``` - -For more control, both ``@setescape`` points and escape instances can be tagged: - -```python -@setescape(tags="foo") # setescape point tags can be single value or tuple (tuples OR'd, like isinstance()) -def foo(): - @call - @setescape(tags="bar") - def bar(): - @looped - def s(loop, acc=0, i=0): - if i > 5: - escape(acc, tag="foo") # escape instance tag must be a single value - return loop(acc + i, i + 1) - print("never reached") - return False - print("never reached either") - return False -assert foo() == 15 -``` - -For details on tagging, especially how untagged and tagged escapes and points interact, and how to make one-to-one connections, see the docstring for ``@setescape``. - - -#### ``call_ec``: first-class escape continuations - -We provide ``call/ec`` (a.k.a. ``call-with-escape-continuation``), in Python spelled as ``call_ec``. It's a decorator that, like ``@call``, immediately runs the function and replaces the def'd name with the return value. The twist is that it internally sets up an escape point, and hands a **first-class escape continuation** to the callee. - -The function to be decorated **must** take one positional argument, the ec instance. - -The ec instance itself is another function, which takes one positional argument: the value to send to the escape point. The ec instance and the escape point are connected one-to-one. No other ``@setescape`` point will catch the ec instance, and the escape point catches only this particular ec instance and nothing else. - -Any particular ec instance is only valid inside the dynamic extent of the ``call_ec`` invocation that created it. Attempting to call the ec later raises ``RuntimeError``. - -This builds on ``@setescape`` and ``escape``, so the caution about catch-all ``except:`` statements applies here, too. - -```python -from unpythonic import call_ec - -@call_ec -def result(ec): # effectively, just a code block, capturing the ec as an argument - answer = 42 - ec(answer) # here this has the same effect as "return answer"... - print("never reached") - answer = 23 - return answer -assert result == 42 - -@call_ec -def result(ec): - answer = 42 - def inner(): - ec(answer) # ...but here this directly escapes from the outer def - print("never reached") - return 23 - answer = inner() - print("never reached either") - return answer -assert result == 42 -``` - -The ec doesn't have to be called from the lexical scope of the call_ec'd function, as long as the call occurs within the dynamic extent of the ``call_ec``. It's essentially a *return from me* for the original function: - -```python -def f(ec): - print("hi from f") - ec(42) - -@call_ec -def result(ec): - f(ec) # pass on the ec - it's a first-class value - print("never reached") -assert result == 42 -``` - -This also works with lambdas, by using ``call_ec()`` directly. No need for a trampoline: - -```python -result = call_ec(lambda ec: - begin(print("hi from lambda"), - ec(42), - print("never reached"))) -assert result == 42 -``` - -Normally ``begin()`` would return the last value, but the ec overrides that; it is effectively a ``return`` for multi-expression lambdas! - -But wait, doesn't Python evaluate all the arguments of `begin(...)` before the `begin` itself has a chance to run? Why doesn't the example print also *never reached*? This is because escapes are implemented using exceptions. Evaluating the ec call raises an exception, preventing any further elements from being evaluated. - -This usage is valid with named functions, too - ``call_ec`` is not only a decorator: - -```python -def f(ec): - print("hi from f") - ec(42) - print("never reached") - -# ...possibly somewhere else, possibly much later... - -result = call_ec(f) -assert result == 42 -``` - - -### ``forall``: nondeterministic evaluation - -We provide a simple variant of nondeterministic evaluation. This is essentially a toy that has no more power than list comprehensions or nested for loops. See also the easy-to-use [macro](macro_extras/) version with natural syntax and a clean implementation. - -An important feature of McCarthy's [`amb` operator](https://rosettacode.org/wiki/Amb) is its nonlocality - being able to jump back to a choice point, even after the dynamic extent of the function where that choice point resides. If that sounds a lot like ``call/cc``, that's because that's how ``amb`` is usually implemented. See examples [in Ruby](http://www.randomhacks.net/2005/10/11/amb-operator/) and [in Racket](http://www.cs.toronto.edu/~david/courses/csc324_w15/extra/choice.html). - -Python can't do that, short of transforming the whole program into [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style), while applying TCO everywhere to prevent stack overflow. **If that's what you want**, see ``continuations`` in [the macros](macro_extras/). - -This ``forall`` is essentially a tuple comprehension that: - - - Can have multiple body expressions (side effects also welcome!), by simply listing them in sequence. - - Allows filters to be placed at any level of the nested looping. - - Presents the source code in the same order as it actually runs. - -The ``unpythonic.amb`` module defines four operators: - - - ``forall`` is the control structure, which marks a section with nondeterministic evaluation. - - ``choice`` binds a name: ``choice(x=range(3))`` essentially means ``for e.x in range(3):``. - - ``insist`` is a filter, which allows the remaining lines to run if the condition evaluates to truthy. - - ``deny`` is ``insist not``; it allows the remaining lines to run if the condition evaluates to falsey. - -Choice variables live in the environment, which is accessed via a ``lambda e: ...``, just like in ``letrec``. Lexical scoping is emulated. In the environment, each line only sees variables defined above it; trying to access a variable defined later raises ``AttributeError``. - -The last line in a ``forall`` describes one item of the output. The output items are collected into a tuple, which becomes the return value of the ``forall`` expression. - -```python -out = forall(choice(y=range(3)), - choice(x=range(3)), - lambda e: insist(e.x % 2 == 0), - lambda e: (e.x, e.y)) -assert out == ((0, 0), (2, 0), (0, 1), (2, 1), (0, 2), (2, 2)) - -out = forall(choice(y=range(3)), - choice(x=range(3)), - lambda e: deny(e.x % 2 == 0), - lambda e: (e.x, e.y)) -assert out == ((1, 0), (1, 1), (1, 2)) -``` - -Pythagorean triples: - -```python -pt = forall(choice(z=range(1, 21)), # hypotenuse - choice(x=lambda e: range(1, e.z+1)), # shorter leg - choice(y=lambda e: range(e.x, e.z+1)), # longer leg - lambda e: insist(e.x*e.x + e.y*e.y == e.z*e.z), - lambda e: (e.x, e.y, e.z)) -assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), - (8, 15, 17), (9, 12, 15), (12, 16, 20)) - -``` - -Beware: - -```python -out = forall(range(2), # do the rest twice! - choice(x=range(1, 4)), - lambda e: e.x) -assert out == (1, 2, 3, 1, 2, 3) -``` - -The initial ``range(2)`` causes the remaining lines to run twice - because it yields two output values - regardless of whether we bind the result to a variable or not. In effect, each line, if it returns more than one output, introduces a new nested loop at that point. - -For more, see the docstring of ``forall``. - -#### For haskellers - -The implementation is based on the List monad, and a bastardized variant of do-notation. Quick vocabulary: - - - ``forall(...)`` = ``do ...`` (for a List monad) - - ``choice(x=foo)`` = ``x <- foo``, where ``foo`` is an iterable - - ``insist x`` = ``guard x`` - - ``deny x`` = ``guard (not x)`` - - Last line = implicit ``return ...`` - - -## Other - -Stuff that didn't fit elsewhere. - -### ``def`` as a code block: ``@call`` - -Fuel for different thinking. Compare `call-with-something` in Lisps - but without parameters, so just `call`. A `def` is really just a new lexical scope to hold code to run later... or right now! - -At the top level of a module, this is seldom useful, but keep in mind that Python allows nested function definitions. Used with an inner ``def``, this becomes a versatile tool. - -*Make temporaries fall out of scope as soon as no longer needed*: - -```python -from unpythonic import call - -@call -def x(): - a = 2 # many temporaries that help readability... - b = 3 # ...of this calculation, but would just pollute locals... - c = 5 # ...after the block exits - return a * b * c -print(x) # 30 -``` - -*Multi-break out of nested loops* - `continue`, `break` and `return` are really just second-class [ec](https://docs.racket-lang.org/reference/cont.html#%28def._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._call%2Fec%29%29)s. So `def` to make `return` escape to exactly where you want: - -```python -@call -def result(): - for x in range(10): - for y in range(10): - if x * y == 42: - return (x, y) -print(result) # (6, 7) -``` - -(But see ``@setescape``, ``escape``, and ``call_ec``.) - -Compare the sweet-exp Racket: - -```racket -define result - let/ec return ; name the (first-class) ec to break out of this let/ec block - for ([x in-range(10)]) - for ([y in-range(10)]) - cond - {{x * y} = 42} - return (list x y) -displayln result ; (6 7) -``` - -Noting [what ``let/ec`` does](https://docs.racket-lang.org/reference/cont.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%2Fec%29%29), using ``call_ec`` we can make the Python even closer to the Racket: - -```python -@call_ec -def result(rtn): - for x in range(10): - for y in range(10): - if x * y == 42: - rtn((x, y)) -print(result) # (6, 7) -``` - -*Twist the meaning of `def` into a "let statement"*: - -```python -@call -def result(x=1, y=2, z=3): - return x * y * z -print(result) # 6 -``` - -(But see `blet`, `bletrec` if you want an `env` instance.) - -*Letrec without `letrec`*, when it doesn't have to be an expression: - -```python -@call -def t(): - def evenp(x): return x == 0 or oddp(x - 1) - def oddp(x): return x != 0 and evenp(x - 1) - return evenp(42) -print(t) # True -``` - -Essentially the implementation is just `def call(thunk): return thunk()`. The point is to: - - - Make it explicit right at the definition site that this block is *going to be called now* (in contrast to an explicit call and assignment *after* the definition). Centralize the related information. Align the presentation order with the thought process. - - - Help eliminate errors, in the same way as the habit of typing parentheses only in pairs. No risk of forgetting to call the block after writing the definition. - - - Document that the block is going to be used only once. Tell the reader there's no need to remember this definition. - -Note [the grammar](https://docs.python.org/3/reference/grammar.html) requires a newline after a decorator. - -**NOTE**: ``call`` can also be used as a normal function: ``call(f, *a, **kw)`` is the same as ``f(*a, **kw)``. This is occasionally useful. - - -### ``@callwith``: freeze arguments, choose function later - -*Added in v0.11.0.* If you need to pass arguments when using ``@call`` as a decorator, use its cousin ``@callwith``: - -```python -from unpythonic import callwith - -@callwith(3) -def result(x): - return x**2 -assert result == 9 -``` - -Like ``call``, it can also be called normally. It's essentially an argument freezer: - -```python -def myadd(a, b): - return a + b -def mymul(a, b): - return a * b -apply23 = callwith(2, 3) -assert apply23(myadd) == 5 -assert apply23(mymul) == 6 -``` - -When called normally, the two-step application is mandatory. The first step stores the given arguments. It returns a function ``f(callable)``. When ``f`` is called, it calls its ``callable`` argument, passing in the arguments stored in the first step. - -In other words, ``callwith`` is similar to ``functools.partial``, but without specializing to any particular function. The function to be called is given later, in the second step. - -Hence, ``callwith(2, 3)(myadd)`` means "make a function that passes in two positional arguments, with values ``2`` and ``3``. Then call this function for the callable ``myadd``". But if we instead write``callwith(2, 3, myadd)``, it means "make a function that passes in three positional arguments, with values ``2``, ``3`` and ``myadd`` - not what we want in the above example. - -If you want to specialize some arguments now and some later, combine with ``partial``: - -```python -from functools import partial - -p1 = partial(callwith, 2) -p2 = partial(p1, 3) -p3 = partial(p2, 4) -apply234 = p3() # actually call callwith, get the function -def add3(a, b, c): - return a + b + c -def mul3(a, b, c): - return a * b * c -assert apply234(add3) == 9 -assert apply234(mul3) == 24 -``` - -If the code above feels weird, it should. Arguments are gathered first, and the function to which they will be passed is chosen in the last step. - -Another use case of ``callwith`` is ``map``, if we want to vary the function instead of the data: - -```python -m = map(callwith(3), [lambda x: 2*x, lambda x: x**2, lambda x: x**(1/2)]) -assert tuple(m) == (6, 9, 3**(1/2)) -``` - -If you have MacroPy, this combines nicely with ``quick_lambda``: - -```python -from macropy.quick_lambda import macros, f, _ -from unpythonic import callwith - -m = map(callwith(3), [f[2 * _], f[_**2], f[_**(1/2)]]) -assert tuple(m) == (6, 9, 3**(1/2)) -``` - -A pythonic alternative to the above examples is: - -```python -a = [2, 3] -def myadd(a, b): - return a + b -def mymul(a, b): - return a * b -assert myadd(*a) == 5 -assert mymul(*a) == 6 - -a = [2] -a += [3] -a += [4] -def add3(a, b, c): - return a + b + c -def mul3(a, b, c): - return a * b * c -assert add3(*a) == 9 -assert mul3(*a) == 24 - -m = (f(3) for f in [lambda x: 2*x, lambda x: x**2, lambda x: x**(1/2)]) -assert tuple(m) == (6, 9, 3**(1/2)) -``` - -Inspired by *Function application with $* in [LYAH: Higher Order Functions](http://learnyouahaskell.com/higher-order-functions). - - -### ``raisef``: ``raise`` as a function - -Raise an exception from an expression position: - -```python -from unpythonic import raisef - -f = lambda x: raisef(RuntimeError, "I'm in ur lambda raising exceptions") -``` - - -### ``pack``: multi-arg constructor for tuple - -The default ``tuple`` constructor accepts a single iterable. But sometimes one needs to pass in the elements separately. Most often a literal tuple such as ``(1, 2, 3)`` is then the right solution, but there are situations that do not admit a literal tuple. Enter ``pack``: - -```python -from unpythonic import pack - -myzip = lambda lol: map(pack, *lol) -lol = ((1, 2), (3, 4), (5, 6)) -assert tuple(myzip(lol)) == ((1, 3, 5), (2, 4, 6)) -``` - - -### ``namelambda``, rename a function - -*Changed in v0.13.0.* Now supports renaming any function object (``isinstance(f, (types.LambdaType, types.FunctionType))``), and will rename a lambda even if it has already been named. - -*Changed in v0.13.1.* Now the return value is a modified copy; the original function object is not mutated. - -For those situations where you return a lambda as a closure, call it much later, and it happens to crash - so you can tell from the stack trace *which* of the *N* lambdas in the codebase it is. - -For technical reasons, ``namelambda`` conforms to the parametric decorator API. Usage: - -```python -from unpythonic import namelambda - -square = namelambda("square")(lambda x: x**2) -assert square.__name__ == "square" - -kaboom = namelambda("kaboom")(lambda: some_typoed_name) -kaboom() # --> stack trace, showing the function name "kaboom" -``` - -The first call returns a *foo-renamer*, which takes a function object and returns a copy that has its name changed to *foo*. - -Technically, this updates ``__name__`` (the obvious place), ``__qualname__`` (used by ``repr()``), and ``__code__.co_name`` (used by stack traces). - -**CAUTION**: There is one pitfall: - -```python -from unpythonic import namelambda, withself - -nested = namelambda("outer")(lambda: namelambda("inner")(withself(lambda self: self))) -print(nested.__qualname__) # "outer" -print(nested().__qualname__) # "..inner" -``` - -The inner lambda does not see the outer's new name; the parent scope names are baked into a function's ``__qualname__`` too early for the outer rename to be in effect at that time. - - -### ``timer``: a context manager for performance testing - -*Added in v0.13.0.* - -```python -from unpythonic import timer - -with timer() as tim: - for _ in range(int(1e6)): - pass -print(tim.dt) # elapsed time in seconds (float) - -with timer(p=True): # if p, auto-print result - for _ in range(int(1e6)): - pass -``` - -The auto-print mode is a convenience feature to minimize bureaucracy if you just want to see the *Δt*. To instead access the *Δt* programmatically, name the timer instance using the ``with ... as ...`` syntax. After the context exits, the *Δt* is available in its ``dt`` attribute. - - -### ``getattrrec``, ``setattrrec``: access underlying data in an onion of wrappers - -*Added in v0.13.1.* - -```python -from unpythonic import getattrrec, setattrrec - -class Wrapper: - def __init__(self, x): - self.x = x - -w = Wrapper(Wrapper(42)) -assert type(getattr(w, "x")) == Wrapper -assert type(getattrrec(w, "x")) == int -assert getattrrec(w, "x") == 42 - -setattrrec(w, "x", 23) -assert type(getattr(w, "x")) == Wrapper -assert type(getattrrec(w, "x")) == int -assert getattrrec(w, "x") == 23 -``` - - -### ``arities``, ``kwargs``: Function signature inspection utilities - -Convenience functions providing an easy-to-use API for inspecting a function's signature. The heavy lifting is done by ``inspect``. - -Methods on objects and classes are treated specially, so that the reported arity matches what the programmer actually needs to supply when calling the method (i.e., implicit ``self`` and ``cls`` are ignored). - -```python -from unpythonic import arities, arity_includes, UnknownArity, \ - kwargs, required_kwargs, optional_kwargs, - -f = lambda a, b: None -assert arities(f) == (2, 2) # min, max positional arity - -f = lambda a, b=23: None -assert arities(f) == (1, 2) -assert arity_includes(f, 2) is True -assert arity_includes(f, 3) is False - -f = lambda a, *args: None -assert arities(f) == (1, float("+inf")) - -f = lambda *, a, b, c=42: None -assert arities(f) == (0, 0) -assert required_kwargs(f) == set(('a', 'b')) -assert optional_kwargs(f) == set(('c')) -assert kwargs(f) == (set(('a', 'b')), set(('c'))) - -class A: - def __init__(self): - pass - def meth(self, x): - pass - @classmethod - def classmeth(cls, x): - pass - @staticmethod - def staticmeth(x): - pass -assert arities(A) == (0, 0) # constructor of "A" takes no args beside the implicit self -# methods on the class -assert arities(A.meth) == (2, 2) -assert arities(A.classmeth) == (1, 1) -assert arities(A.staticmeth) == (1, 1) -# methods on an instance -a = A() -assert arities(a.meth) == (1, 1) # self is implicit, so just one -assert arities(a.classmeth) == (1, 1) # cls is implicit -assert arities(a.staticmeth) == (1, 1) -``` - -We special-case the builtin functions that either fail to return any arity (are uninspectable) or report incorrect arity information, so that also their arities are reported correctly. Note we **do not** special-case the *methods* of any builtin classes, so e.g. ``list.append`` remains uninspectable. This limitation might or might not be lifted in a future version. - -If the arity cannot be inspected, and the function is not one of the special-cased builtins, the ``UnknownArity`` exception is raised. - -These functions are internally used in various places in unpythonic, particularly ``curry``. The ``let`` and FP looping constructs also use these to emit a meaningful error message if the signature of user-provided function does not match what is expected. - -Inspired by various Racket functions such as ``(arity-includes?)`` and ``(procedure-keywords)``. - - -### ``Popper``: a pop-while iterator - -*Added in v0.14.1.* - -Consider this highly artificial example: - -```python -from collections import deque - -inp = deque(range(5)) -out = [] -while inp: - x = inp.pop(0) - out.append(x) -assert inp == deque([]) -assert out == list(range(5)) -``` - -``Popper`` condenses the ``while`` and ``pop`` into a ``for``, while allowing the loop body to mutate the input iterable in arbitrary ways (we never actually ``iter()`` it): - -```python -from collections import deque -from unpythonic import Popper - -inp = deque(range(5)) -out = [] -for x in Popper(inp): - out.append(x) -assert inp == deque([]) -assert out == list(range(5)) - -inp = deque(range(3)) -out = [] -for x in Popper(inp): - out.append(x) - if x < 10: - inp.appendleft(x + 10) -assert inp == deque([]) -assert out == [0, 10, 1, 11, 2, 12] -``` - -``Popper`` comboes with other iterable utilities, such as ``window``: - -```python -from collections import deque -from unpythonic import Popper, window - -inp = deque(range(3)) -out = [] -for a, b in window(Popper(inp)): - out.append((a, b)) - if a < 10: - inp.append(a + 10) -assert inp == deque([]) -assert out == [(0, 1), (1, 2), (2, 10), (10, 11), (11, 12)] -``` - -(Although ``window`` invokes ``iter()`` on the ``Popper``, this works because the ``Popper`` never invokes ``iter()`` on the underlying container. Any mutations to the input container performed by the loop body will be understood by ``Popper`` and thus also seen by the ``window``. The first ``n`` elements, though, are read before the loop body gets control, because the window needs them to initialize itself.) - -One possible real use case for ``Popper`` is to split sequences of items, stored as lists in a deque, into shorter sequences where some condition is contiguously ``True`` or ``False``. When the condition changes state, just commit the current subsequence, and push the rest of that input sequence (still requiring analysis) back to the input deque, to be dealt with later. - -The argument to ``Popper`` (here ``lst``) contains the **remaining** items. Each iteration pops an element **from the left**. The loop terminates when ``lst`` is empty. - -The input container must support either ``popleft()`` or ``pop(0)``. This is fully duck-typed. At least ``collections.deque`` and any ``collections.abc.MutableSequence`` (including ``list``) are fine. - -Per-iteration efficiency is O(1) for ``collections.deque``, and O(n) for a ``list``. - -Named after [Karl Popper](https://en.wikipedia.org/wiki/Karl_Popper). - - - -## Meta - -### Design notes - -#### On ``let`` and Python - -Why no `let*`, as a function? In Python, name lookup always occurs at runtime. Python gives us no compile-time guarantees that no binding refers to a later one - in [Racket](http://racket-lang.org/), this guarantee is the main difference between `let*` and `letrec`. - -Even Racket's `letrec` processes the bindings sequentially, left-to-right, but *the scoping of the names is mutually recursive*. Hence a binding may contain a lambda that, when eventually called, uses a binding defined further down in the `letrec` form. - -In contrast, in a `let*` form, attempting such a definition is *a compile-time error*, because at any point in the sequence of bindings, only names found earlier in the sequence have been bound. See [TRG on `let`](https://docs.racket-lang.org/guide/let.html). - -Our `letrec` behaves like `let*` in that if `valexpr` is not a function, it may only refer to bindings above it. But this is only enforced at run time, and we allow mutually recursive function definitions, hence `letrec`. - -Note the function versions of our `let` constructs, presented here, are **not** properly lexically scoped; in case of nested ``let`` expressions, one must be explicit about which environment the names come from. - -The [macro versions](macro_extras/) of the `let` constructs **are** lexically scoped. The macros also provide a ``letseq[]`` that, similarly to Racket's ``let*``, gives a compile-time guarantee that no binding refers to a later one. - -Inspiration: [[1]](https://nvbn.github.io/2014/09/25/let-statement-in-python/) [[2]](https://stackoverflow.com/questions/12219465/is-there-a-python-equivalent-of-the-haskell-let) [[3]](http://sigusr2.net/more-about-let-in-python.html). - -#### Python is not a Lisp - -The point behind providing `let` and `begin` (and the ``let[]`` and ``do[]`` [macros](macro_extras/)) is to make Python lambdas slightly more useful - which was really the starting point for this whole experiment. - -The oft-quoted single-expression limitation of the Python ``lambda`` is ultimately a herring, as this library demonstrates. The real problem is the statement/expression dichotomy. In Python, the looping constructs (`for`, `while`), the full power of `if`, and `return` are statements, so they cannot be used in lambdas. We can work around some of this: - - - The expression form of `if` can be used, but readability suffers if nested. Actually, [`and` and `or` are sufficient for full generality](https://www.ibm.com/developerworks/library/l-prog/), but readability suffers there too. Another possibility is to use MacroPy to define a ``cond`` expression, but it's essentially duplicating a feature the language already almost has. (Our [macros](macro_extras/) do exactly that, providing a ``cond`` expression as a macro.) - - Functional looping (with TCO, to boot) is possible. - - ``unpythonic.ec.call_ec`` gives us ``return`` (the ec), and ``unpythonic.misc.raisef`` gives us ``raise``. - - Exception handling (``try``/``except``/``else``/``finally``) and context management (``with``) are currently **not** available for lambdas, even in ``unpythonic``. - -Still, ultimately one must keep in mind that Python is not a Lisp. Not all of Python's standard library is expression-friendly; some standard functions and methods lack return values - even though a call is an expression! For example, `set.add(x)` returns `None`, whereas in an expression context, returning `x` would be much more useful, even though it does have a side effect. - -#### Assignment syntax - -Why the clunky `e.set("foo", newval)` or `e << ("foo", newval)`, which do not directly mention `e.foo`? This is mainly because in Python, the language itself is not customizable. If we could define a new operator `e.foo newval` to transform to `e.set("foo", newval)`, this would be easily solved. - -Our [macros](macro_extras/) essentially do exactly this, but by borrowing the ``<<`` operator to provide the syntax ``foo << newval``, because even with MacroPy, it is not possible to define new [BinOp](https://greentreesnakes.readthedocs.io/en/latest/nodes.html#BinOp)s in Python. That would be possible essentially as a *reader macro* (as it's known in the Lisp world), to transform custom BinOps into some syntactically valid Python code before proceeding with the rest of the import machinery, but it seems as of this writing, no one has done this. - -Without macros, in raw Python, we could abuse `e.foo << newval`, which transforms to `e.foo.__lshift__(newval)`, to essentially perform `e.set("foo", newval)`, but this requires some magic, because we then need to monkey-patch each incoming value (including the first one when the name "foo" is defined) to set up the redirect and keep it working. - - - Methods of builtin types such as `int` are read-only, so we can't just override `__lshift__` in any given `newval`. - - For many types of objects, at the price of some copy-constructing, we can provide a wrapper object that inherits from the original's type, and just adds an `__lshift__` method to catch and redirect the appropriate call. See commented-out proof-of-concept in [`unpythonic/env.py`](unpythonic/env.py). - - But that approach doesn't work for function values, because `function` is not an acceptable base type to inherit from. In this case we could set up a proxy object, whose `__call__` method calls the original function (but what about the docstring and such? Is `@functools.wraps` enough?). But then there are two kinds of wrappers, and the re-wrapping logic (which is needed to avoid stacking wrappers when someone does `e.a << e.b`) needs to know about that. - - It's still difficult to be sure these two approaches cover all cases; a read of `e.foo` gets a wrapped value, not the original; and this already violates [The Zen of Python](https://www.python.org/dev/peps/pep-0020/) #1, #2 and #3. - -If we later choose go this route nevertheless, `<<` is a better choice for the syntax than `<<=`, because `let` needs `e.set(...)` to be valid in an expression context. - -The current solution for the assignment syntax issue is to use macros, to have both clean syntax at the use site and a relatively hackfree implementation. - -#### TCO syntax and speed - -Benefits and costs of ``return jump(...)``: - - - Explicitly a tail call due to ``return``. - - The trampoline can be very simple and (relatively speaking) fast. Just a dumb ``jump`` record, a ``while`` loop, and regular function calls and returns. - - The cost is that ``jump`` cannot detect whether the user forgot the ``return``, leaving a possibility for bugs in the client code (causing an FP loop to immediately exit, returning ``None``). Unit tests of client code become very important. - - This is somewhat mitigated by the check in `__del__`, but it can only print a warning, not stop the incorrect program from proceeding. - - We could mandate that trampolined functions must not return ``None``, but: - - Uniformity is lost between regular and trampolined functions, if only one kind may return ``None``. - - This breaks the *don't care about return value* use case, which is rather common when using side effects. - - Failing to terminate at the intended point may well fall through into what was intended as another branch of the client code, which may correctly have a ``return``. So this would not even solve the problem. - -The other simple-ish solution is to use exceptions, making the jump wrest control from the caller. Then ``jump(...)`` becomes a verb, but this approach is 2-5x slower, when measured with a do-nothing loop. (See the old default TCO implementation in v0.9.2.) - -Our [macros](macro_extras/) provide an easy-to use solution. Just wrap the relevant section of code in a ``with tco:``, to automatically apply TCO to code that looks exactly like standard Python. With the macro, function definitions (also lambdas) and returns are automatically converted. It also knows enough not to add a ``@trampolined`` if you have already declared a ``def`` as ``@looped`` (or any of the other TCO-enabling decorators in ``unpythonic.fploop``). - -For other libraries bringing TCO to Python, see: - - - [tco](https://github.com/baruchel/tco) by Thomas Baruchel, based on exceptions. - - [ActiveState recipe 474088](https://github.com/ActiveState/code/tree/master/recipes/Python/474088_Tail_Call_Optimization_Decorator), based on ``inspect``. - - ``recur.tco`` in [fn.py](https://github.com/fnpy/fn.py), the original source of the approach used here. - - [MacroPy](https://github.com/azazel75/macropy) uses an approach similar to ``fn.py``. - -#### Wait, no monads? - -(Beside List inside ``forall``.) - -Admittedly unpythonic, but Haskell feature, not Lisp. Besides, already done elsewhere, see [OSlash](https://github.com/dbrattli/OSlash) if you need them. - -If you want to roll your own monads for whatever reason, there's [this silly hack](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/monads.py) that wasn't packaged into this; or just read Stephan Boyer's quick introduction [[part 1]](https://www.stephanboyer.com/post/9/monads-part-1-a-design-pattern) [[part 2]](https://www.stephanboyer.com/post/10/monads-part-2-impure-computations) [[super quick intro]](https://www.stephanboyer.com/post/83/super-quick-intro-to-monads) and figure it out, it's easy. (Until you get to `State` and `Reader`, where [this](http://brandon.si/code/the-state-monad-a-tutorial-for-the-confused/) and maybe [this](https://gaiustech.wordpress.com/2010/09/06/on-monads/) can be helpful.) - -#### Your hovercraft is full of eels! - -[Naturally](http://stupidpythonideas.blogspot.com/2015/05/spam-spam-spam-gouda-spam-and-tulips.html), they come with the territory. - -Some have expressed the opinion [the statement-vs-expression dichotomy is a feature](http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html). The BDFL himself has famously stated that TCO has no place in Python [[1]](http://neopythonic.blogspot.com/2009/04/tail-recursion-elimination.html) [[2]](http://neopythonic.blogspot.fi/2009/04/final-words-on-tail-calls.html), and less famously that multi-expression lambdas or continuations have no place in Python [[3]](https://www.artima.com/weblogs/viewpost.jsp?thread=147358). Several potentially interesting PEPs have been deferred [[1]](https://www.python.org/dev/peps/pep-3150/) [[2]](https://www.python.org/dev/peps/pep-0403/) or rejected [[3]](https://www.python.org/dev/peps/pep-0511/) [[4]](https://www.python.org/dev/peps/pep-0463/) [[5]](https://www.python.org/dev/peps/pep-0472/). - -Of course, if I agreed, I wouldn't be doing this (or [this](https://github.com/Technologicat/pydialect)). - -On a point raised [here](https://www.artima.com/weblogs/viewpost.jsp?thread=147358) with respect to indentation-sensitive vs. indentation-insensitive parser modes, having seen [sweet expressions](https://srfi.schemers.org/srfi-110/srfi-110.html) I think Python is confusing matters by linking the mode to statements vs. expressions. A workable solution is to make *everything* support both modes (or even preprocess the source code text to use only one of the modes), which *uniformly* makes parentheses an alternative syntax for grouping. - -It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose ``lambda x: [expr0, expr1, ...]`` will have to do for a multiple-expression lambda in MacroPy. Unless I decide at some point to make a source filter for [Pydialect](https://github.com/Technologicat/pydialect) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. +### Documentation +- [Basic: pure-Python](doc/features.md) +- [Advanced: syntactic macros](macro_extras/): the second half of ``unpythonic``. ### Installation #### PyPI -Usually one of: +`pip3 install unpythonic --user` -```pip3 install unpythonic --user``` +or -```sudo pip3 install unpythonic``` - -depending on what you want. +`sudo pip3 install unpythonic` #### GitHub -Clone (or pull) from GitHub. Then, usually one of: +Clone (or pull) from GitHub. Then, -```python3 setup.py install --user``` +`python3 setup.py install --user` -```sudo python3 setup.py install``` +or -depending on what you want. +`sudo python3 setup.py install` #### Uninstall -```pip3 uninstall unpythonic``` +Uninstallation must be invoked in a folder which has no subfolder called `unpythonic`, so that `pip` recognizes it as a package name (instead of a filename). Then, -with ``sudo`` if needed. +`pip3 uninstall unpythonic` -Must be invoked in a folder which has no subfolder called `unpythonic`, so that `pip` recognizes it as a package name (instead of a filename). +or +`sudo pip3 uninstall unpythonic` ### License @@ -2822,4 +90,3 @@ Old, but interesting: - [Peter Norvig (2000): Python for Lisp Programmers](http://www.norvig.com/python-lisp.html) - [David Mertz (2001): Charming Python - Functional programming in Python, part 2](https://www.ibm.com/developerworks/library/l-prog2/index.html) - diff --git a/doc/design-notes.md b/doc/design-notes.md new file mode 100644 index 00000000..0608ffd6 --- /dev/null +++ b/doc/design-notes.md @@ -0,0 +1,100 @@ +# Design Notes + +## Contents + +- [On ``let`` and Python](#on-let-and-python) +- [Python is Not a Lisp](#python-is-not-a-lisp) +- [Assignment Syntax](#assignment-syntax) +- [TCO Syntax and Speed](#tco-syntax-and-speed) +- [No Monads?](#wait-no-monads) +- [Further Explanation](#your-hovercraft-is-full-of-eels) + +## On ``let`` and Python + +Why no `let*`, as a function? In Python, name lookup always occurs at runtime. Python gives us no compile-time guarantees that no binding refers to a later one - in [Racket](http://racket-lang.org/), this guarantee is the main difference between `let*` and `letrec`. + +Even Racket's `letrec` processes the bindings sequentially, left-to-right, but *the scoping of the names is mutually recursive*. Hence a binding may contain a lambda that, when eventually called, uses a binding defined further down in the `letrec` form. + +In contrast, in a `let*` form, attempting such a definition is *a compile-time error*, because at any point in the sequence of bindings, only names found earlier in the sequence have been bound. See [TRG on `let`](https://docs.racket-lang.org/guide/let.html). + +Our `letrec` behaves like `let*` in that if `valexpr` is not a function, it may only refer to bindings above it. But this is only enforced at run time, and we allow mutually recursive function definitions, hence `letrec`. + +Note the function versions of our `let` constructs, presented here, are **not** properly lexically scoped; in case of nested ``let`` expressions, one must be explicit about which environment the names come from. + +The [macro versions](macro_extras/) of the `let` constructs **are** lexically scoped. The macros also provide a ``letseq[]`` that, similarly to Racket's ``let*``, gives a compile-time guarantee that no binding refers to a later one. + +Inspiration: [[1]](https://nvbn.github.io/2014/09/25/let-statement-in-python/) [[2]](https://stackoverflow.com/questions/12219465/is-there-a-python-equivalent-of-the-haskell-let) [[3]](http://sigusr2.net/more-about-let-in-python.html). + +## Python is not a Lisp + +The point behind providing `let` and `begin` (and the ``let[]`` and ``do[]`` [macros](macro_extras/)) is to make Python lambdas slightly more useful - which was really the starting point for this whole experiment. + +The oft-quoted single-expression limitation of the Python ``lambda`` is ultimately a herring, as this library demonstrates. The real problem is the statement/expression dichotomy. In Python, the looping constructs (`for`, `while`), the full power of `if`, and `return` are statements, so they cannot be used in lambdas. We can work around some of this: + + - The expression form of `if` can be used, but readability suffers if nested. Actually, [`and` and `or` are sufficient for full generality](https://www.ibm.com/developerworks/library/l-prog/), but readability suffers there too. Another possibility is to use MacroPy to define a ``cond`` expression, but it's essentially duplicating a feature the language already almost has. (Our [macros](macro_extras/) do exactly that, providing a ``cond`` expression as a macro.) + - Functional looping (with TCO, to boot) is possible. + - ``unpythonic.ec.call_ec`` gives us ``return`` (the ec), and ``unpythonic.misc.raisef`` gives us ``raise``. + - Exception handling (``try``/``except``/``else``/``finally``) and context management (``with``) are currently **not** available for lambdas, even in ``unpythonic``. + +Still, ultimately one must keep in mind that Python is not a Lisp. Not all of Python's standard library is expression-friendly; some standard functions and methods lack return values - even though a call is an expression! For example, `set.add(x)` returns `None`, whereas in an expression context, returning `x` would be much more useful, even though it does have a side effect. + +## Assignment syntax + +Why the clunky `e.set("foo", newval)` or `e << ("foo", newval)`, which do not directly mention `e.foo`? This is mainly because in Python, the language itself is not customizable. If we could define a new operator `e.foo newval` to transform to `e.set("foo", newval)`, this would be easily solved. + +Our [macros](macro_extras/) essentially do exactly this, but by borrowing the ``<<`` operator to provide the syntax ``foo << newval``, because even with MacroPy, it is not possible to define new [BinOp](https://greentreesnakes.readthedocs.io/en/latest/nodes.html#BinOp)s in Python. That would be possible essentially as a *reader macro* (as it's known in the Lisp world), to transform custom BinOps into some syntactically valid Python code before proceeding with the rest of the import machinery, but it seems as of this writing, no one has done this. + +Without macros, in raw Python, we could abuse `e.foo << newval`, which transforms to `e.foo.__lshift__(newval)`, to essentially perform `e.set("foo", newval)`, but this requires some magic, because we then need to monkey-patch each incoming value (including the first one when the name "foo" is defined) to set up the redirect and keep it working. + + - Methods of builtin types such as `int` are read-only, so we can't just override `__lshift__` in any given `newval`. + - For many types of objects, at the price of some copy-constructing, we can provide a wrapper object that inherits from the original's type, and just adds an `__lshift__` method to catch and redirect the appropriate call. See commented-out proof-of-concept in [`unpythonic/env.py`](unpythonic/env.py). + - But that approach doesn't work for function values, because `function` is not an acceptable base type to inherit from. In this case we could set up a proxy object, whose `__call__` method calls the original function (but what about the docstring and such? Is `@functools.wraps` enough?). But then there are two kinds of wrappers, and the re-wrapping logic (which is needed to avoid stacking wrappers when someone does `e.a << e.b`) needs to know about that. + - It's still difficult to be sure these two approaches cover all cases; a read of `e.foo` gets a wrapped value, not the original; and this already violates [The Zen of Python](https://www.python.org/dev/peps/pep-0020/) #1, #2 and #3. + +If we later choose go this route nevertheless, `<<` is a better choice for the syntax than `<<=`, because `let` needs `e.set(...)` to be valid in an expression context. + +The current solution for the assignment syntax issue is to use macros, to have both clean syntax at the use site and a relatively hackfree implementation. + +## TCO syntax and speed + +Benefits and costs of ``return jump(...)``: + + - Explicitly a tail call due to ``return``. + - The trampoline can be very simple and (relatively speaking) fast. Just a dumb ``jump`` record, a ``while`` loop, and regular function calls and returns. + - The cost is that ``jump`` cannot detect whether the user forgot the ``return``, leaving a possibility for bugs in the client code (causing an FP loop to immediately exit, returning ``None``). Unit tests of client code become very important. + - This is somewhat mitigated by the check in `__del__`, but it can only print a warning, not stop the incorrect program from proceeding. + - We could mandate that trampolined functions must not return ``None``, but: + - Uniformity is lost between regular and trampolined functions, if only one kind may return ``None``. + - This breaks the *don't care about return value* use case, which is rather common when using side effects. + - Failing to terminate at the intended point may well fall through into what was intended as another branch of the client code, which may correctly have a ``return``. So this would not even solve the problem. + +The other simple-ish solution is to use exceptions, making the jump wrest control from the caller. Then ``jump(...)`` becomes a verb, but this approach is 2-5x slower, when measured with a do-nothing loop. (See the old default TCO implementation in v0.9.2.) + +Our [macros](macro_extras/) provide an easy-to use solution. Just wrap the relevant section of code in a ``with tco:``, to automatically apply TCO to code that looks exactly like standard Python. With the macro, function definitions (also lambdas) and returns are automatically converted. It also knows enough not to add a ``@trampolined`` if you have already declared a ``def`` as ``@looped`` (or any of the other TCO-enabling decorators in ``unpythonic.fploop``). + +For other libraries bringing TCO to Python, see: + + - [tco](https://github.com/baruchel/tco) by Thomas Baruchel, based on exceptions. + - [ActiveState recipe 474088](https://github.com/ActiveState/code/tree/master/recipes/Python/474088_Tail_Call_Optimization_Decorator), based on ``inspect``. + - ``recur.tco`` in [fn.py](https://github.com/fnpy/fn.py), the original source of the approach used here. + - [MacroPy](https://github.com/azazel75/macropy) uses an approach similar to ``fn.py``. + +## Wait, no monads? + +(Beside List inside ``forall``.) + +Admittedly unpythonic, but Haskell feature, not Lisp. Besides, already done elsewhere, see [OSlash](https://github.com/dbrattli/OSlash) if you need them. + +If you want to roll your own monads for whatever reason, there's [this silly hack](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/monads.py) that wasn't packaged into this; or just read Stephan Boyer's quick introduction [[part 1]](https://www.stephanboyer.com/post/9/monads-part-1-a-design-pattern) [[part 2]](https://www.stephanboyer.com/post/10/monads-part-2-impure-computations) [[super quick intro]](https://www.stephanboyer.com/post/83/super-quick-intro-to-monads) and figure it out, it's easy. (Until you get to `State` and `Reader`, where [this](http://brandon.si/code/the-state-monad-a-tutorial-for-the-confused/) and maybe [this](https://gaiustech.wordpress.com/2010/09/06/on-monads/) can be helpful.) + +## Your hovercraft is full of eels! + +[Naturally](http://stupidpythonideas.blogspot.com/2015/05/spam-spam-spam-gouda-spam-and-tulips.html), they come with the territory. + +Some have expressed the opinion [the statement-vs-expression dichotomy is a feature](http://stupidpythonideas.blogspot.com/2015/01/statements-and-expressions.html). The BDFL himself has famously stated that TCO has no place in Python [[1]](http://neopythonic.blogspot.com/2009/04/tail-recursion-elimination.html) [[2]](http://neopythonic.blogspot.fi/2009/04/final-words-on-tail-calls.html), and less famously that multi-expression lambdas or continuations have no place in Python [[3]](https://www.artima.com/weblogs/viewpost.jsp?thread=147358). Several potentially interesting PEPs have been deferred [[1]](https://www.python.org/dev/peps/pep-3150/) [[2]](https://www.python.org/dev/peps/pep-0403/) or rejected [[3]](https://www.python.org/dev/peps/pep-0511/) [[4]](https://www.python.org/dev/peps/pep-0463/) [[5]](https://www.python.org/dev/peps/pep-0472/). + +Of course, if I agreed, I wouldn't be doing this (or [this](https://github.com/Technologicat/pydialect)). + +On a point raised [here](https://www.artima.com/weblogs/viewpost.jsp?thread=147358) with respect to indentation-sensitive vs. indentation-insensitive parser modes, having seen [sweet expressions](https://srfi.schemers.org/srfi-110/srfi-110.html) I think Python is confusing matters by linking the mode to statements vs. expressions. A workable solution is to make *everything* support both modes (or even preprocess the source code text to use only one of the modes), which *uniformly* makes parentheses an alternative syntax for grouping. + +It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose ``lambda x: [expr0, expr1, ...]`` will have to do for a multiple-expression lambda in MacroPy. Unless I decide at some point to make a source filter for [Pydialect](https://github.com/Technologicat/pydialect) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. diff --git a/doc/features.md b/doc/features.md new file mode 100644 index 00000000..97cf870c --- /dev/null +++ b/doc/features.md @@ -0,0 +1,2642 @@ +This README documents the pure-Python part of ``unpythonic``. See also [documentation for the macros](macro_extras/). + +**Contents**: + + - [**Bindings**](#bindings) + - [``let``, ``letrec``: local bindings in an expression](#let-letrec-local-bindings-in-an-expression) + - [Lispylet: alternative syntax](#lispylet-alternative-syntax) + - [``env``: the environment](#env-the-environment) + - [``assignonce``](#assignonce), a relative of ``env``. + - [``dyn``: dynamic assignment](#dyn-dynamic-assignment) a.k.a. parameterize, special variables, "dynamic scoping". + + - [**Containers**](#containers) + - [``frozendict``: an immutable dictionary](#frozendict-an-immutable-dictionary) + - [`cons` and friends: pythonic lispy linked lists](#cons-and-friends-pythonic-lispy-linked-lists) + - [``box``: a mutable single-item container](#box-a-mutable-single-item-container) + - [Container utilities](#container-utilities): ``get_abcs``, ``in_slice``, ``index_in_slice`` + + - [**Sequencing**](#sequencing), run multiple expressions in any expression position (incl. inside a ``lambda``). + - [``begin``: sequence side effects](#begin-sequence-side-effects) + - [``do``: stuff imperative code into an expression](#do-stuff-imperative-code-into-an-expression) + - [``pipe``, ``piped``, ``lazy_piped``: sequence functions](#pipe-piped-lazy_piped-sequence-functions) + + - [**Batteries**](#batteries) missing from the standard library. + - [**Batteries for functools**](#batteries-for-functools): `memoize`, `curry`, `compose`, `withself` and more. + - [``curry`` and reduction rules](#curry-and-reduction-rules): we provide some extra features for bonus haskellness. + - [**Batteries for itertools**](#batteries-for-itertools): multi-input folds, scans (lazy partial folds); unfold; lazy partial unpacking of iterables, etc. + - [``islice``: slice syntax support for ``itertools.islice``](#islice-slice-syntax-support-for-itertoolsislice) + - [`gmemoize`, `imemoize`, `fimemoize`: memoize generators](#gmemoize-imemoize-fimemoize-memoize-generators), iterables and iterator factories. + - [``fup``: functional update; ``ShadowedSequence``](#fup-functional-update-shadowedsequence): like ``collections.ChainMap``, but for sequences. + - [``view``: writable, sliceable view into a sequence](#view-writable-sliceable-view-into-a-sequence) with scalar broadcast on assignment. + - [``mogrify``: update a mutable container in-place](#mogrify-update-a-mutable-container-in-place) + - [``s``, ``m``, ``mg``: lazy mathematical sequences with infix arithmetic](#s-m-mg-lazy-mathematical-sequences-with-infix-arithmetic) + + - [**Control flow tools**](#control-flow-tools) + - [``trampolined``, ``jump``: tail call optimization (TCO) / explicit continuations](#trampolined-jump-tail-call-optimization-tco--explicit-continuations) + - [``looped``, ``looped_over``: loops in FP style (with TCO)](#looped-looped_over-loops-in-fp-style-with-tco) + - [``gtrampolined``: generators with TCO](#gtrampolined-generators-with-tco): tail-chaining; like ``itertools.chain``, but from inside a generator. + - [``setescape``, ``escape``: escape continuations (ec)](#setescape-escape-escape-continuations-ec) + - [``call_ec``: first-class escape continuations](#call_ec-first-class-escape-continuations), like Racket's ``call/ec``. + - [``forall``: nondeterministic evaluation](#forall-nondeterministic-evaluation), a tuple comprehension with multiple body expressions. + + - [**Other**](#other) + - [``def`` as a code block: ``@call``](#def-as-a-code-block-call): run a block of code immediately, in a new lexical scope. + - [``@callwith``: freeze arguments, choose function later](#callwith-freeze-arguments-choose-function-later) + - [``raisef``: ``raise`` as a function](#raisef-raise-as-a-function), useful inside a lambda. + - [``pack``: multi-arg constructor for tuple](#pack-multi-arg-constructor-for-tuple) + - [``namelambda``, rename a function](#namelambda-rename-a-function) + - [``timer``: a context manager for performance testing](#timer-a-context-manager-for-performance-testing) + - [``getattrrec``, ``setattrrec``: access underlying data in an onion of wrappers](#getattrrec-setattrrec-access-underlying-data-in-an-onion-of-wrappers) + - [``arities``, ``kwargs``: Function signature inspection utilities](#arities-kwargs-function-signature-inspection-utilities) + - [``Popper``: a pop-while iterator](#popper-a-pop-while-iterator) + + - [**Advanced: syntactic macros**](macro_extras/): the second half of ``unpythonic``. + + - [**Meta**](#meta) + - [Design notes](#design-notes) + - [Installation](#installation) + - [License](#license) + - [Acknowledgements](#acknowledgements) + - [Python-related FP resources](#python-related-fp-resources) + +For many examples, see the unit tests located in [unpythonic/test/](unpythonic/test/), the docstrings of the individual features, and this README. + +*This README doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out of date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this README) should agree with the unit tests. So if something fails to work as advertised, check what the tests say - and optionally file an issue on GitHub so that the documentation can be fixed.* + + +## Bindings + +Tools to bind identifiers in ways not ordinarily supported by Python. + +### ``let``, ``letrec``: local bindings in an expression + +Introduce bindings local to an expression, like Scheme's ``let`` and ``letrec``. For easy-to-use versions of these constructs that look almost like normal Python, see [macros](macro_extras/). This section documents the underlying pure-Python API, which can also be used directly if you don't want to depend on MacroPy. + +In ``let``, the bindings are independent (do not see each other). A binding is of the form ``name=value``, where ``name`` is a Python identifier, and ``value`` is any expression. + +Use a `lambda e: ...` to supply the environment to the body: + +```python +from unpythonic import let, letrec, dlet, dletrec, blet, bletrec + +u = lambda lst: let(seen=set(), + body=lambda e: + [e.seen.add(x) or x for x in lst if x not in e.seen]) +L = [1, 1, 3, 1, 3, 2, 3, 2, 2, 2, 4, 4, 1, 2, 3] +u(L) # --> [1, 3, 2, 4] +``` + +Generally speaking, `body` is a one-argument function, which takes in the environment instance as the first positional parameter (by convention, named `e` or `env`). In typical inline usage, `body` is `lambda e: expr`. + +*Let over lambda*. Here the inner ``lambda`` is the definition of the function ``counter``: + +```python +counter = let(x=0, + body=lambda e: + lambda: + begin(e.set("x", e.x + 1), # can also use e << ("x", e.x + 1) + e.x)) +counter() # --> 1 +counter() # --> 2 +``` + +Compare the sweet-exp [Racket](http://racket-lang.org/) (see [SRFI-110](https://srfi.schemers.org/srfi-110/srfi-110.html) and [sweet](https://docs.racket-lang.org/sweet/)): + +```racket +define counter + let ([x 0]) ; In Racket, the (λ (e) (...)) in "let" is implicit, and needs no explicit "e". + λ () ; Racket's λ has an implicit (begin ...), so we don't need a begin. + set! x {x + 1} + x +counter() ; --> 1 +counter() ; --> 2 +``` + +*Let over def* decorator ``@dlet``, to *let over lambda* more pythonically: + +```python +@dlet(x=0) +def counter(*, env=None): # named argument "env" filled in by decorator + env.x += 1 + return env.x +counter() # --> 1 +counter() # --> 2 +``` + +In `letrec`, bindings may depend on ones above them in the same `letrec`, by using `lambda e: ...` (**Python 3.6+**): + +```python +x = letrec(a=1, + b=lambda e: + e.a + 1, + body=lambda e: + e.b) # --> 2 +``` + +In `letrec`, the ``value`` of each binding is either a simple value (non-callable, and doesn't use the environment), or an expression of the form ``lambda e: valexpr``, providing access to the environment as ``e``. If ``valexpr`` itself is callable, the binding **must** have the ``lambda e: ...`` wrapper to prevent any misunderstandings in the environment initialization procedure. + +In a non-callable ``valexpr``, trying to depend on a binding below it raises ``AttributeError``. + +A callable ``valexpr`` may depend on any bindings (also later ones) in the same `letrec`. Mutually recursive functions: + +```python +letrec(evenp=lambda e: + lambda x: + (x == 0) or e.oddp(x - 1), + oddp=lambda e: + lambda x: + (x != 0) and e.evenp(x - 1), + body=lambda e: + e.evenp(42)) # --> True +``` + +Order-preserving list uniqifier: + +```python +u = lambda lst: letrec(seen=set(), + see=lambda e: + lambda x: + begin(e.seen.add(x), + x), + body=lambda e: + [e.see(x) for x in lst if x not in e.seen]) +``` + +**CAUTION**: in Pythons older than 3.6, bindings are **initialized in an arbitrary order**, also in `letrec`. This is a limitation of the kwargs abuse. Hence mutually recursive functions are possible, but a non-callable `valexpr` cannot depend on other bindings in the same `letrec`. + +Trying to access `e.foo` from `e.bar` arbitrarily produces either the intended value of `e.foo`, or the uninitialized `lambda e: ...`, depending on whether `e.foo` has been initialized or not at the point of time when `e.bar` is being initialized. + +This has been fixed in Python 3.6, see [PEP 468](https://www.python.org/dev/peps/pep-0468/). + +#### Lispylet: alternative syntax + +If you need **guaranteed left-to-right initialization** of `letrec` bindings in Pythons older than 3.6, there is also an alternative implementation for all the `let` constructs, with positional syntax and more parentheses. The only difference is the syntax; the behavior is identical with the default implementation. + +These constructs are available in the top-level `unpythonic` namespace, with the ``ordered_`` prefix: ``ordered_let``, ``ordered_letrec``, ``ordered_dlet``, ``ordered_dletrec``, ``ordered_blet``, ``ordered_bletrec``. + +It is also possible to override the default `let` constructs by the `ordered_` variants, like this: + +```python +from unpythonic.lispylet import * # override the default "let" implementation + +letrec((('a', 1), + ('b', lambda e: + e.a + 1)), # may refer to any bindings above it in the same letrec, also in Python < 3.6 + lambda e: + e.b) # --> 2 + +letrec((("evenp", lambda e: + lambda x: # callable, needs the lambda e: ... + (x == 0) or e.oddp(x - 1)), + ("oddp", lambda e: + lambda x: + (x != 0) and e.evenp(x - 1))), + lambda e: + e.evenp(42)) # --> True +``` + +The syntax is `let(bindings, body)` (respectively `letrec(bindings, body)`), where `bindings` is `((name, value), ...)`, and `body` is like in the default variants. The same rules concerning `name` and `value` apply. + +The let macros internally use this *lispylet* implementation. + + +### ``env``: the environment + +The environment used by all the ``let`` constructs and ``assignonce`` (but **not** by `dyn`) is essentially a bunch with iteration, subscripting and context manager support. For details, see `unpythonic.env`. + +This allows things like: + +```python +let(x=1, y=2, z=3, + body=lambda e: + [(name, 2*e[name]) for name in e]) # --> [('y', 4), ('z', 6), ('x', 2)] +``` + +It also works as a bare bunch, and supports printing for debugging: + +```python +from unpythonic.env import env + +e = env(s="hello", orange="fruit", answer=42) +print(e) # --> +print(e.s) # --> hello + +d = {'monty': 'python', 'pi': 3.14159} +e = env(**d) +print(e) # --> +print(e.monty) # --> python +``` + +Finally, it supports the context manager: + +```python +with env(x=1, y=2, z=3) as e: + print(e) # --> + print(e.x) # --> 1 +print(e) # empty! +``` + +When the `with` block exits, the environment clears itself. The environment instance itself remains alive due to Python's scoping rules. + +*Changed in v0.13.1.* ``env`` now provides the ``collections.abc.Mapping`` API. + +*Changed in v0.14.0.* ``env`` now provides also the ``collections.abc.MutableMapping`` API. + + +### ``assignonce`` + +In Scheme terms, make `define` and `set!` look different: + +```python +from unpythonic import assignonce + +with assignonce() as e: + e.foo = "bar" # new definition, ok + e.set("foo", "tavern") # explicitly rebind e.foo, ok + e << ("foo", "tavern") # same thing (but return e instead of new value, suitable for chaining) + e.foo = "quux" # AttributeError, e.foo already defined. +``` + +It's a subclass of ``env``, so it shares most of the same [features](#env-the-environment) and allows similar usage. + + +### ``dyn``: dynamic assignment + +([As termed by Felleisen.](https://groups.google.com/forum/#!topic/racket-users/2Baxa2DxDKQ)) + +Like global variables, but better-behaved. Useful for sending some configuration parameters through several layers of function calls without changing their API. Best used sparingly. Like [Racket](http://racket-lang.org/)'s [`parameterize`](https://docs.racket-lang.org/guide/parameterize.html). The *special variables* in Common Lisp work somewhat similarly (but with indefinite scope). + +There's a singleton, `dyn`: + +```python +from unpythonic import dyn + +def f(): # no "a" in lexical scope here + assert dyn.a == 2 + +def g(): + with dyn.let(a=2, b="foo"): + assert dyn.a == 2 + + f() + + with dyn.let(a=3): # dynamic assignments can be nested + assert dyn.a == 3 + + # now "a" has reverted to its previous value + assert dyn.a == 2 + + print(dyn.b) # AttributeError, dyn.b no longer exists +g() +``` + +Dynamic variables are set using `with dyn.let(...)`. There is no `set`, `<<`, unlike in the other `unpythonic` environments. + +The values of dynamic variables remain bound for the dynamic extent of the `with` block. Exiting the `with` block then pops the stack. Inner dynamic scopes shadow outer ones. Dynamic variables are seen also by code that is outside the lexical scope where the `with dyn.let` resides. + +Each thread has its own dynamic scope stack. A newly spawned thread automatically copies the then-current state of the dynamic scope stack **from the main thread** (not the parent thread!). Any copied bindings will remain on the stack for the full dynamic extent of the new thread. Because these bindings are not associated with any `with` block running in that thread, and because aside from the initial copying, the dynamic scope stacks are thread-local, any copied bindings will never be popped, even if the main thread pops its own instances of them. + +The source of the copy is always the main thread mainly because Python's `threading` module gives no tools to detect which thread spawned the current one. (If someone knows a simple solution, PRs welcome!) + +Finally, there is one global dynamic scope shared between all threads, where the default values of dynvars live. The default value is used when ``dyn`` is queried for the value outside the dynamic extent of any ``with dyn.let()`` blocks. Having a default value is convenient for eliminating the need for ``if "x" in dyn`` checks, since the variable will always exist (after the global definition has been executed). + +To create a dynvar and set its default value, use ``make_dynvar``. Each dynamic variable, of the same name, should only have one default set; the (dynamically) latest definition always overwrites. However, we do not prevent overwrites, because in some codebases the same module may run its top-level initialization code multiple times (e.g. if a module has a ``main()`` for tests, and the file gets loaded both as a module and as the main program). + +See also the methods of ``dyn``; particularly noteworthy are ``asdict`` and ``items``, which give access to a live view to dyn's contents in a dictionary format (intended for reading only!). The ``asdict`` method essentially creates a ``collections.ChainMap`` instance, while ``items`` is an abbreviation for ``asdict().items()``. The ``dyn`` object itself can also be iterated over; this creates a ``ChainMap`` instance and redirects to iterate over it. + +To support dictionary-like idioms in iteration, dynvars can alternatively be accessed by subscripting; ``dyn["x"]`` has the same meaning as ``dyn.x``, so you can do things like: + +```python +print(tuple((k, dyn[k]) for k in dyn)) +``` + +Finally, ``dyn`` supports membership testing as ``"x" in dyn``, ``"y" not in dyn``, where the string is the name of the dynvar whose presence is being tested. + +For some more details, see [the unit tests](unpythonic/test/test_dynassign.py). + +*Changed in v0.13.0.* The ``asdict`` and ``items`` methods previously returned a snapshot; now they return a live view. + +*Changed in v0.13.1.* ``dyn`` now provides the ``collections.abc.Mapping`` API. + + +## Containers + +We provide some additional containers. + +The class names are lowercase, because these are intended as low-level utility classes in principle on par with the builtins. The immutable containers are hashable. All containers are pickleable (if their contents are). + +### ``frozendict``: an immutable dictionary + +*Added in v0.13.0.* + +Given the existence of ``dict`` and ``frozenset``, this one is oddly missing from the standard library. + +```python +from unpythonic import frozendict + +d = frozendict({'a': 1, 'b': 2}) +d['a'] # OK +d['c'] = 3 # TypeError, not writable +``` + +Functional updates are supported: + +```python +d2 = frozendict(d, a=42) +assert d2['a'] == 42 and d2['b'] == 2 +assert d['a'] == 1 # original not mutated + +d3 = frozendict({'a': 1, 'b': 2}, {'a': 42}) # rightmost definition of each key wins +assert d3['a'] == 42 and d3['b'] == 2 + +# ...also using unpythonic.fupdate +d4 = fupdate(d3, a=23) +assert d4['a'] == 23 and d4['b'] == 2 +assert d3['a'] == 42 and d3['b'] == 2 # ...of course without touching the original +``` + +Any mappings used when creating an instance are shallow-copied, so that the bindings of the ``frozendict`` do not change even if the original input is later mutated: + +```python +d = {1:2, 3:4} +fd = frozendict(d) +d[5] = 6 +assert d == {1: 2, 3: 4, 5: 6} +assert fd == {1: 2, 3: 4} +``` + +**The usual caution** concerning immutable containers in Python applies: the container protects only the bindings against changes. If the values themselves are mutable, the container cannot protect from mutations inside them. + +All the usual read-access stuff works: + +```python +d7 = frozendict({1:2, 3:4}) +assert 3 in d7 +assert len(d7) == 2 +assert set(d7.keys()) == {1, 3} +assert set(d7.values()) == {2, 4} +assert set(d7.items()) == {(1, 2), (3, 4)} +assert d7 == frozendict({1:2, 3:4}) +assert d7 != frozendict({1:2}) +assert d7 == {1:2, 3:4} # like frozenset, __eq__ doesn't care whether mutable or not +assert d7 != {1:2} +assert {k for k in d7} == {1, 3} +assert d7.get(3) == 4 +assert d7.get(5, 0) == 0 +assert d7.get(5) is None +``` + +In terms of ``collections.abc``, a ``frozendict`` is a hashable immutable mapping: + +```python +assert issubclass(frozendict, Mapping) +assert not issubclass(frozendict, MutableMapping) + +assert issubclass(frozendict, Hashable) +assert hash(d7) == hash(frozendict({1:2, 3:4})) +assert hash(d7) != hash(frozendict({1:2})) +``` + +The abstract superclasses are virtual, just like for ``dict`` (i.e. they do not appear in the MRO). + +Finally, ``frozendict`` obeys the empty-immutable-container singleton invariant: + +```python +assert frozendict() is frozendict() +``` + +...but don't pickle the empty ``frozendict`` and expect this invariant to hold; it's freshly created in each session. + + +### `cons` and friends: pythonic lispy linked lists + +*Laugh, it's funny.* + +```python +from unpythonic import cons, nil, ll, llist, car, cdr, caar, cdar, cadr, cddr, \ + member, lreverse, lappend, lzip, BinaryTreeIterator + +c = cons(1, 2) +assert car(c) == 1 and cdr(c) == 2 + +# ll(...) is like [...] or (...), but for linked lists: +assert ll(1, 2, 3) == cons(1, cons(2, cons(3, nil))) + +t = cons(cons(1, 2), cons(3, 4)) # binary tree +assert [f(t) for f in [caar, cdar, cadr, cddr]] == [1, 2, 3, 4] + +# default iteration scheme is "single cell or linked list": +a, b = cons(1, 2) # unpacking a cons cell +a, b, c = ll(1, 2, 3) # unpacking a linked list +a, b, c, d = BinaryTreeIterator(t) # unpacking a binary tree: use a non-default iteration scheme + +assert list(ll(1, 2, 3)) == [1, 2, 3] +assert tuple(ll(1, 2, 3)) == (1, 2, 3) +assert llist((1, 2, 3)) == ll(1, 2, 3) # llist() is like list() or tuple(), but for linked lists + +l = ll(1, 2, 3) +assert member(2, l) == ll(2, 3) +assert not member(5, l) + +assert lreverse(ll(1, 2, 3)) == ll(3, 2, 1) +assert lappend(ll(1, 2), ll(3, 4), ll(5, 6)) == ll(1, 2, 3, 4, 5, 6) +assert lzip(ll(1, 2, 3), ll(4, 5, 6)) == ll(ll(1, 4), ll(2, 5), ll(3, 6)) +``` + +Cons cells are immutable à la Racket (no `set-car!`/`rplaca`, `set-cdr!`/`rplacd`). Accessors are provided up to `caaaar`, ..., `cddddr`. + +Although linked lists are created with ``ll`` or ``llist``, the data type (for e.g. ``isinstance``) is ``cons``. + +Iterators are supported to walk over linked lists (this also gives sequence unpacking support). When ``next()`` is called, we return the car of the current cell the iterator points to, and the iterator moves to point to the cons cell in the cdr, if any. When the cdr is not a cons cell, it is the next (and last) item returned; except if it `is nil`, then iteration ends without returning the `nil`. + +Python's builtin ``reversed`` can be applied to linked lists; it will internally ``lreverse`` the list (which is O(n)), then return an iterator to that. The ``llist`` constructor is special-cased so that if the input is ``reversed(some_ll)``, it just returns the internal already reversed list. (This is safe because cons cells are immutable.) + +Cons structures can optionally print like in Lisps: + +```python +print(cons(1, 2).lispyrepr()) # --> (1 . 2) +print(ll(1, 2, 3).lispyrepr()) # --> (1 2 3) +print(cons(cons(1, 2), cons(3, 4)).lispyrepr()) # --> ((1 . 2) . (3 . 4)) +``` + +However, by default, they print in a pythonic format suitable for ``eval`` (if all elements are): + +```python +print(cons(1, 2)) # --> cons(1, 2) +print(ll(1, 2, 3)) # --> ll(1, 2, 3) +print(cons(cons(1, 2), cons(3, 4)) # --> cons(cons(1, 2), cons(3, 4)) +``` + +*Changed in v0.11.0.* In previous versions, the Lisp format was always used for printing. + +For more, see the ``llist`` submodule. + +#### Notes + +There is no ``copy`` method or ``lcopy`` function, because cons cells are immutable; which makes cons structures immutable. + +(However, for example, it is possible to `cons` a new item onto an existing linked list; that's fine because it produces a new cons structure - which shares data with the original, just like in Racket.) + +In general, copying cons structures can be error-prone. Given just a starting cell it is impossible to tell if a given instance of a cons structure represents a linked list, or something more general (such as a binary tree) that just happens to locally look like one, along the path that would be traversed if it was indeed a linked list. + +The linked list iteration strategy does not recurse in the ``car`` half, which could lead to incomplete copying. The tree strategy that recurses on both halves, on the other hand, will flatten nested linked lists and produce also the final ``nil``. + +*Added in v0.13.0.* We provide a ``JackOfAllTradesIterator`` as a compromise that understands both trees and linked lists. Nested lists will be flattened, and in a tree any ``nil`` in a ``cdr`` position will be omitted from the output. + +*Changed in v0.13.1.* ``BinaryTreeIterator`` and ``JackOfAllTradesIterator`` now use an explicit data stack instead of implicitly using the call stack for keeping track of the recursion. Hence now all ``cons`` iterators work for arbitrarily deep cons structures without causing Python's call stack to overflow, and without the need for TCO. + +``cons`` has no ``collections.abc`` virtual superclasses (except the implicit ``Hashable`` since ``cons`` provides ``__hash__`` and ``__eq__``), because general cons structures do not fit into the contracts represented by membership in those classes. For example, size cannot be known without iterating, and depends on which iteration scheme is used (e.g. ``nil`` dropping, flattening); which scheme is appropriate depends on the content. + +**Caution**: the ``nil`` singleton is freshly created in each session; newnil is not oldnil, so don't pickle a standalone ``nil``. The unpickler of ``cons`` automatically refreshes any ``nil`` instances inside a pickled cons structure, so that **cons structures** support the illusion that ``nil`` is a special value like ``None`` or ``...``. After unpickling, ``car(c) is nil`` and ``cdr(c) is nil`` still work as expected, even though ``id(nil)`` has changed between sessions. + + +### ``box``: a mutable single-item container + +*Added in v0.12.0.* + +*Changed in v0.13.0.* The class and the data attribute have been renamed to ``box`` and ``x``, respectively. + +No doubt anyone programming in an imperative language has run into the situation caricatured by this highly artificial example: + +```python +a = 23 + +def f(x): + x = 17 # but I want to update the existing a! + +f(a) +assert a == 23 +``` + +Many solutions exist. Common pythonic ones are abusing a ``list`` to represent a box (and then trying to remember it is supposed to hold only a single item), or using the ``global`` or ``nonlocal`` keywords to tell Python, on assignment, to overwrite a name that already exists in a surrounding scope. + +As an alternative to the rampant abuse of lists, we provide a rackety ``box``, which is a minimalistic mutable container that holds exactly one item. The data in the box is accessed via an attribute, so any code that has a reference to the box can update the data in it: + +```python +from unpythonic import box + +a = box(23) + +def f(b): + b.x = 17 + +f(a) +assert a == 17 +``` + +The attribute name is just ``x`` to reduce the number of additional keystrokes required. The ``box`` API is summarized by: + +```python +b1 = box(23) +b2 = box(23) +b3 = box(17) + +assert b1.x == 23 # data lives in the attribute .x +assert 23 in b1 # content is "in" the box, also syntactically +assert 17 not in b1 + +assert [x for x in b1] == [23] # box is iterable +assert len(b1) == 1 # and always has length 1 + +assert b1 == 23 # for equality testing, a box is considered equal to its content + +assert b2 == b1 # contents are equal, but +assert b2 is not b1 # different boxes + +assert b3 != b1 # different contents +``` + +The expression ``item in b`` has the same meaning as ``b.x == item``. Note ``box`` is a mutable container, so it is **not hashable**. + + +### Container utilities + +**Inspect the superclasses** that a particular container type has: + +```python +from unpythonic import get_abcs +print(get_abcs(list)) +``` + +This includes virtual superclasses, i.e. those that are not part of the MRO. This works by ``issubclass(cls, v)`` on all classes defined in ``collections.abc``. + +**Reflection on slices**: + +```python +from unpythonic import in_slice, index_in_slice + +s = slice(1, 11, 2) # 1, 3, 5, 7, 9 +assert in_slice(5, s) +assert not in_slice(6, s) +assert index_in_slice(5, s) == 2 +``` + +An optional length argument can be given to interpret negative indices. See the docstrings for details. + + + +## Sequencing + +Sequencing refers to running multiple expressions, in sequence, in place of one expression. + +Keep in mind the only reason to ever need multiple expressions: *side effects.* (Assignment is a side effect, too; it modifies the environment. In functional style, intermediate named definitions to increase readability are perhaps the most useful kind of side effect.) + +See also ``multilambda`` in [macros](macro_extras/). + + +### ``begin``: sequence side effects + +```python +from unpythonic import begin, begin0 + +f1 = lambda x: begin(print("cheeky side effect"), + 42*x) +f1(2) # --> 84 + +f2 = lambda x: begin0(42*x, + print("cheeky side effect")) +f2(2) # --> 84 +``` + +Actually a tuple in disguise. If worried about memory consumption, use `lazy_begin` and `lazy_begin0` instead, which indeed use loops. The price is the need for a lambda wrapper for each expression to delay evaluation, see [`unpythonic.seq`](unpythonic/seq.py) for details. + + +### ``do``: stuff imperative code into an expression + +No monadic magic. Basically, ``do`` is: + + - An improved ``begin`` that can bind names to intermediate results and then use them in later items. + + - A ``let*`` (technically, ``letrec``) where making a binding is optional, so that some items can have only side effects if so desired. No semantically distinct ``body``; all items play the same role. + +Like in ``letrec`` (see below), use ``lambda e: ...`` to access the environment, and to wrap callable values (to prevent misunderstandings). + +We also provide a ``do[]`` [macro](macro_extras/) that makes the construct easier to use. + +```python +from unpythonic import do, assign + +y = do(assign(x=17), # create and set e.x + lambda e: print(e.x), # 17; uses environment, needs lambda e: ... + assign(x=23), # overwrite e.x + lambda e: print(e.x), # 23 + 42) # return value +assert y == 42 + +y = do(assign(x=17), + assign(z=lambda e: 2*e.x), + lambda e: e.z) +assert y == 34 + +y = do(assign(x=5), + assign(f=lambda e: lambda x: x**2), # callable, needs lambda e: ... + print("hello from 'do'"), # value is None; not callable + lambda e: e.f(e.x)) +assert y == 25 +``` + +If you need to return the first value instead of the last one, use this trick: + +```python +y = do(assign(result=17), + print("assigned 'result' in env"), + lambda e: e.result) # return value +assert y == 17 +``` + +Or use ``do0``, which does it for you: + +```python +from unpythonic import do0, assign + +y = do0(17, + assign(x=42), + lambda e: print(e.x), + print("hello from 'do0'")) +assert y == 17 + +y = do0(assign(x=17), # the first item of do0 can be an assignment, too + lambda e: print(e.x)) +assert y == 17 +``` + +Beware of this pitfall: + +```python +do(lambda e: print("hello 2 from 'do'"), # delayed because lambda e: ... + print("hello 1 from 'do'"), # Python prints immediately before do() + "foo") # gets control, because technically, it is + # **the return value** that is an argument + # for do(). +``` + +Unlike ``begin`` (and ``begin0``), there is no separate ``lazy_do`` (``lazy_do0``), because using a ``lambda e: ...`` wrapper will already delay evaluation of an item. If you want a lazy variant, just wrap each item (also those which don't otherwise need it). + +The above pitfall also applies to using escape continuations inside a ``do``. To do that, wrap the ec call into a ``lambda e: ...`` to delay its evaluation until the ``do`` actually runs: + +```python +call_ec( + lambda ec: + do(assign(x=42), + lambda e: ec(e.x), # IMPORTANT: must delay this! + lambda e: print("never reached"))) # and this (as above) +``` + +This way, any assignments made in the ``do`` (which occur only after ``do`` gets control), performed above the line with the ``ec`` call, will have been performed when the ``ec`` is called. + + +### ``pipe``, ``piped``, ``lazy_piped``: sequence functions + +Similar to Racket's [threading macros](https://docs.racket-lang.org/threading/). A pipe performs a sequence of operations, starting from an initial value, and then returns the final value. It's just function composition, but with an emphasis on data flow, which helps improve readability: + +```python +from unpythonic import pipe + +double = lambda x: 2 * x +inc = lambda x: x + 1 + +x = pipe(42, double, inc) +assert x == 85 +``` + +We also provide ``pipec``, which curries the functions before applying them. Useful with passthrough (see below on ``curry``). + +Optional **shell-like syntax**, with purely functional updates: + +```python +from unpythonic import piped, getvalue + +x = piped(42) | double | inc | getvalue +assert x == 85 + +p = piped(42) | double +assert p | inc | getvalue == 85 +assert p | getvalue == 84 # p itself is never modified by the pipe system +``` + +Set up a pipe by calling ``piped`` for the initial value. Pipe into the sentinel ``getvalue`` to exit the pipe and return the current value. + +**Lazy pipes**, useful for mutable initial values. To perform the planned computation, pipe into the sentinel ``runpipe``: + +```python +from unpythonic import lazy_piped1, runpipe + +lst = [1] +def append_succ(l): + l.append(l[-1] + 1) + return l # this return value is handed to the next function in the pipe +p = lazy_piped1(lst) | append_succ | append_succ # plan a computation +assert lst == [1] # nothing done yet +p | runpipe # run the computation +assert lst == [1, 2, 3] # now the side effect has updated lst. +``` + +Lazy pipe as an unfold: + +```python +from unpythonic import lazy_piped, runpipe + +fibos = [] +def nextfibo(a, b): # multiple arguments allowed + fibos.append(a) # store result by side effect + return (b, a + b) # new state, handed to next function in the pipe +p = lazy_piped(1, 1) # load initial state +for _ in range(10): # set up pipeline + p = p | nextfibo +p | runpipe +assert fibos == [1, 1, 2, 3, 5, 8, 13, 21, 34, 55] +``` + +Both one-in-one-out (*1-to-1*) and n-in-m-out (*n-to-m*) pipes are provided. The 1-to-1 versions have names suffixed with ``1``. The use case is one-argument functions that return one value (which may also be a tuple). + +In the n-to-m versions, when a function returns a tuple, it is unpacked to the argument list of the next function in the pipe. At ``getvalue`` or ``runpipe`` time, the tuple wrapper (if any) around the final result is discarded if it contains only one item. (This allows the n-to-m versions to work also with a single value, as long as it is not a tuple.) The main use case is computations that deal with multiple values, the number of which may also change during the computation (as long as there are as many "slots" on both sides of each individual connection). + + +## Batteries + +Things missing from the standard library. + +### Batteries for functools + + - `memoize`: + - Caches also exceptions à la Racket. If the memoized function is called again with arguments with which it raised an exception the first time, the same exception instance is raised again. + - Works also on instance methods, with results cached separately for each instance. + - This is essentially because ``self`` is an argument, and custom classes have a default ``__hash__``. + - Hence it doesn't matter that the memo lives in the ``memoized`` closure on the class object (type), where the method is, and not directly on the instances. The memo itself is shared between instances, but calls with a different value of ``self`` will create unique entries in it. + - For a solution that performs memoization at the instance level, see [this ActiveState recipe](https://github.com/ActiveState/code/tree/master/recipes/Python/577452_memoize_decorator_instance) (and to demystify the magic contained therein, be sure you understand [descriptors](https://docs.python.org/3/howto/descriptor.html)). + - `curry`, with some extra features: + - Passthrough on the right when too many args (à la Haskell; or [spicy](https://github.com/Technologicat/spicy) for Racket) + - If the intermediate result of a passthrough is callable, it is (curried and) invoked on the remaining positional args. This helps with some instances of [point-free style](https://en.wikipedia.org/wiki/Tacit_programming). + - For simplicity, all remaining keyword args are fed in at the first step that has too many positional args. + - If more positional args are still remaining when the top-level curry context exits, by default ``TypeError`` is raised. + - To override, set the dynvar ``curry_context``. It is a list representing the stack of currently active curry contexts. A context is any object, a human-readable label is fine. See below for an example. + - Can be used both as a decorator and as a regular function. + - As a regular function, `curry` itself is curried à la Racket. If it gets extra arguments (beside the function ``f``), they are the first step. This helps eliminate many parentheses. + - **Caution**: If the positional arities of ``f`` cannot be inspected, currying fails, raising ``UnknownArity``. This may happen with builtins such as ``list.append``. + - `composel`, `composer`: both left-to-right and right-to-left function composition, to help readability. + - Any number of positional arguments is supported, with the same rules as in the pipe system. Multiple return values packed into a tuple are unpacked to the argument list of the next function in the chain. + - `composelc`, `composerc`: curry each function before composing them. Useful with passthrough. + - An implicit top-level curry context is inserted around all the functions except the one that is applied last. + - `composel1`, `composer1`: 1-in-1-out chains (faster; also useful for a single value that is a tuple). + - suffix `i` to use with an iterable (`composeli`, `composeri`, `composelci`, `composerci`, `composel1i`, `composer1i`) + - `withself`: essentially, the Y combinator trick as a decorator. Allows a lambda to refer to itself. + - The ``self`` argument is declared explicitly, but passed implicitly (as the first positional argument), just like the ``self`` argument of a method. + - `apply`: the lispy approach to starargs. Mainly useful with the ``prefix`` [macro](macro_extras/). + - `andf`, `orf`, `notf`: compose predicates (like Racket's `conjoin`, `disjoin`, `negate`). + - `flip`: reverse the order of positional arguments. + - `rotate`: a cousin of `flip`. Permute the order of positional arguments in a cycle. + - `to1st`, `to2nd`, `tokth`, `tolast`, `to` to help inserting 1-in-1-out functions into m-in-n-out compose chains. (Currying can eliminate the need for these.) + - `identity`, `const` which sometimes come in handy when programming with higher-order functions. + +Examples (see also the next section): + +```python +from operator import add, mul +from unpythonic import andf, orf, flatmap, rotate, curry, dyn, zipr, rzip, \ + foldl, foldr, composer, to1st, cons, nil, ll, withself + +isint = lambda x: isinstance(x, int) +iseven = lambda x: x % 2 == 0 +isstr = lambda s: isinstance(s, str) +assert andf(isint, iseven)(42) is True +assert andf(isint, iseven)(43) is False +pred = orf(isstr, andf(isint, iseven)) +assert pred(42) is True +assert pred("foo") is True +assert pred(None) is False + +# lambda that refers to itself +fact = withself(lambda self, n: n * self(n - 1) if n > 1 else 1) +assert fact(5) == 120 + +@rotate(-1) # cycle the argument slots to the left by one place, so "acc" becomes last +def zipper(acc, *rest): # so that we can use the *args syntax to declare this + return acc + (rest,) # even though the input is (e1, ..., en, acc). +myzipl = curry(foldl, zipper, ()) # same as (curry(foldl))(zipper, ()) +myzipr = curry(foldr, zipper, ()) +assert myzipl((1, 2, 3), (4, 5, 6), (7, 8)) == ((1, 4, 7), (2, 5, 8)) +assert myzipr((1, 2, 3), (4, 5, 6), (7, 8)) == ((2, 5, 8), (1, 4, 7)) + +# zip and reverse don't commute for inputs with different lengths +assert tuple(zipr((1, 2, 3), (4, 5, 6), (7, 8))) == ((2, 5, 8), (1, 4, 7)) # zip first +assert tuple(rzip((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) # reverse first + +# curry with passthrough on the right +# final result is a tuple of the result(s) and the leftover args +double = lambda x: 2 * x +with dyn.let(curry_context=["whatever"]): # set a context to allow passthrough to the top level + assert curry(double, 2, "foo") == (4, "foo") # arity of double is 1 + +mysum = curry(foldl, add, 0) +myprod = curry(foldl, mul, 1) +a = ll(1, 2) +b = ll(3, 4) +c = ll(5, 6) +append_two = lambda a, b: foldr(cons, b, a) +append_many = lambda *lsts: foldr(append_two, nil, lsts) # see unpythonic.lappend +assert mysum(append_many(a, b, c)) == 21 +assert myprod(b) == 12 + +map_one = lambda f: curry(foldr, composer(cons, to1st(f)), nil) +doubler = map_one(double) +assert doubler((1, 2, 3)) == ll(2, 4, 6) + +assert curry(map_one, double, ll(1, 2, 3)) == ll(2, 4, 6) +``` + +*Minor detail*: We could also write the last example as: + +```python +double = lambda x: 2 * x +rmap_one = lambda f: curry(foldl, composer(cons, to1st(f)), nil) # essentially reversed(map(...)) +map_one = lambda f: composer(rmap_one(f), lreverse) +assert curry(map_one, double, ll(1, 2, 3)) == ll(2, 4, 6) +``` + +which may be a useful pattern for lengthy iterables that could overflow the call stack (although not in ``foldr``, since our implementation uses a linear process). + +In ``rmap_one``, we can use either ``curry`` or ``functools.partial``. In this case it doesn't matter which, since we want just one partial application anyway. We provide two arguments, and the minimum arity of ``foldl`` is 3, so ``curry`` will trigger the call as soon as (and only as soon as) it gets at least one more argument. + +The final ``curry`` uses both of the extra features. It invokes passthrough, since ``map_one`` has arity 1. It also invokes a call to the callable returned from ``map_one``, with the remaining arguments (in this case just one, the ``ll(1, 2, 3)``). + +Yet another way to write ``map_one`` is: + +```python +mymap = lambda f: curry(foldr, composer(cons, curry(f)), nil) +``` + +The curried ``f`` uses up one argument (provided it is a one-argument function!), and the second argument is passed through on the right; this two-tuple then ends up as the arguments to ``cons``. + +Using a currying compose function (name suffixed with ``c``), the inner curry can be dropped: + +```python +mymap = lambda f: curry(foldr, composerc(cons, f), nil) +myadd = lambda a, b: a + b +assert curry(mymap, myadd, ll(1, 2, 3), ll(2, 4, 6)) == ll(3, 6, 9) +``` + +This is as close to ```(define (map f) (foldr (compose cons f) empty)``` (in ``#lang`` [``spicy``](https://github.com/Technologicat/spicy)) as we're gonna get in Python. + +Notice how the last two versions accept multiple input iterables; this is thanks to currying ``f`` inside the composition. An element from each of the iterables is taken by the processing function ``f``. Being the last argument, ``acc`` is passed through on the right. The output from the processing function - one new item - and ``acc`` then become a two-tuple, passed into cons. + +Finally, keep in mind this exercise is intended as a feature demonstration. In production code, the builtin ``map`` is much better. + + +#### ``curry`` and reduction rules + +The provided variant of ``curry``, beside what it says on the tin, is effectively an explicit local modifier to Python's reduction rules, which allows some Haskell-like idioms. When we say: + +```python +curry(f, a0, a1, ..., a[n-1]) +``` + +it means the following. Let ``m1`` and ``m2`` be the minimum and maximum positional arity of the callable ``f``, respectively. + + - If ``n > m2``, call ``f`` with the first ``m2`` arguments. + - If the result is a callable, curry it, and recurse. + - Else form a tuple, where first item is the result, and the rest are the remaining arguments ``a[m2]``, ``a[m2+1]``, ..., ``a[n-1]``. Return it. + - If more positional args are still remaining when the top-level curry context exits, by default ``TypeError`` is raised. Use the dynvar ``curry_context`` to override; see above for an example. + - If ``m1 <= n <= m2``, call ``f`` and return its result (like a normal function call). + - **Any** positional arity accepted by ``f`` triggers the call; beware when working with [variadic](https://en.wikipedia.org/wiki/Variadic_function) functions. + - If ``n < m1``, partially apply ``f`` to the given arguments, yielding a new function with smaller ``m1``, ``m2``. Then curry the result and return it. + - Internally we stack ``functools.partial`` applications, but there will be only one ``curried`` wrapper no matter how many invocations are used to build up arguments before ``f`` eventually gets called. + +In the above example: + +```python +curry(mapl_one, double, ll(1, 2, 3)) +``` + +the callable ``mapl_one`` takes one argument, which is a function. It yields another function, let us call it ``g``. We are left with: + +```python +curry(g, ll(1, 2, 3)) +``` + +The argument is then passed into ``g``; we obtain a result, and reduction is complete. + +A curried function is also a curry context: + +```python +add2 = lambda x, y: x + y +a2 = curry(add2) +a2(a, b, c) # same as curry(add2, a, b, c); reduces to (a + b, c) +``` + +so on the last line, we don't need to say + +```python +curry(a2, a, b, c) +``` + +because ``a2`` is already curried. Doing so does no harm, though; ``curry`` automatically prevents stacking ``curried`` wrappers: + +```python +curry(a2) is a2 # --> True +``` + +If we wish to modify precedence, parentheses are needed, which takes us out of the curry context, unless we explicitly ``curry`` the subexpression. This works: + +```python +curry(f, a, curry(g, x, y), b, c) +``` + +but this **does not**: + +```python +curry(f, a, (g, x, y), b, c) +``` + +because ``(g, x, y)`` is just a tuple of ``g``, ``x`` and ``y``. This is by design; as with all things Python, *explicit is better than implicit*. + +**Note**: to code in curried style, a [contract system](https://github.com/AndreaCensi/contracts) or a [static type checker](http://mypy-lang.org/) is useful; also, be careful with variadic functions. + + +### Batteries for itertools + + - `foldl`, `foldr` with support for multiple input iterables, like in Racket. + - Like in Racket, `op(elt, acc)`; general case `op(e1, e2, ..., en, acc)`. Note Python's own `functools.reduce` uses the ordering `op(acc, elt)` instead. + - No sane default for multi-input case, so the initial value for `acc` must be given. + - One-input versions with optional init are provided as `reducel`, `reducer`, with semantics similar to Python's `functools.reduce`, but with the rackety ordering `op(elt, acc)`. + - By default, multi-input folds terminate on the shortest input. To instead terminate on the longest input, use the ``longest`` and ``fillvalue`` kwargs. + - For multiple inputs with different lengths, `foldr` syncs the **left** ends. + - `rfoldl`, `rreducel` reverse each input and then left-fold. This syncs the **right** ends. + - `scanl`, `scanr`: scan (a.k.a. accumulate, partial fold); a lazy fold that returns a generator yielding intermediate results. + - `scanl` is suitable for infinite inputs. + - Iteration stops after the final result. + - For `scanl`, this is what `foldl` would have returned (if the fold terminates at all, i.e. if the shortest input is finite). + - *Changed in v0.11.0.* For `scanr`, **ordering of output is different from Haskell**: we yield the results in the order they are computed (via a linear process). + - Multiple input iterables and shortest/longest termination supported; same semantics as in `foldl`, `foldr`. + - One-input versions with optional init are provided as `scanl1`, `scanr1`. Note ordering of arguments to match `functools.reduce`, but op is still the rackety `op(elt, acc)`. + - `rscanl`, `rscanl1` reverse each input and then left-scan. This syncs the **right** ends. + - `unfold1`, `unfold`: generate a sequence [corecursively](https://en.wikipedia.org/wiki/Corecursion). The counterpart of `foldl`. + - `unfold1` is for 1-in-2-out functions. The input is `state`, the return value must be `(value, newstate)` or `None`. + - `unfold` is for n-in-(1+n)-out functions. The input is `*states`, the return value must be `(value, *newstates)` or `None`. + - Unfold returns a generator yielding the collected values. The output can be finite or infinite; to signify that a finite sequence ends, the user function must return `None`. + - `unpack`: lazily unpack an iterable. Suitable for infinite inputs. + - Return the first ``n`` items and the ``k``th tail, in a tuple. Default is ``k = n``. + - Use ``k > n`` to fast-forward, consuming the skipped items. Works by `drop`. + - Use ``k < n`` to peek without permanently extracting an item. Works by [tee](https://docs.python.org/3/library/itertools.html#itertools.tee)ing; plan accordingly. + - `flatmap`: map a function, that returns a list or tuple, over an iterable and then flatten by one level, concatenating the results into a single tuple. + - Essentially, ``composel(map(...), flatten1)``; the same thing the bind operator of the List monad does. + - `map_longest`: the final missing battery for `map`. + - Essentially `starmap(func, zip_longest(*iterables))`, so it's [spanned](https://en.wikipedia.org/wiki/Linear_span) by ``itertools``. + - `rmap`, `rzip`, `rmap_longest`, `rzip_longest`: reverse each input, then map/zip. For multiple inputs, syncs the **right** ends. + - `mapr`, `zipr`, `mapr_longest`, `zipr_longest`: map/zip, then reverse the result. For multiple inputs, syncs the **left** ends. + - `uniqify`, `uniq`: remove duplicates (either all or consecutive only, respectively), preserving the original ordering of the items. + - `flatten1`, `flatten`, `flatten_in`: remove nested list structure. + - `flatten1`: outermost level only. + - `flatten`: recursive, with an optional predicate that controls whether to flatten a given sublist. + - `flatten_in`: recursive, with an optional predicate; but recurse also into items which don't match the predicate. + - `take`, `drop`, `split_at`: based on `itertools` [recipes](https://docs.python.org/3/library/itertools.html#itertools-recipes). + - Especially useful for testing generators. + - `islice` is maybe more pythonic than `take` and `drop`; see below for a utility that supports the slice syntax. + - `tail`: return the tail of an iterable. Same as `drop(1, iterable)`; common use case. + - `butlast`, `butlastn`: return a generator that yields from iterable, dropping the last `n` items if the iterable is finite. Inspired by a similar utility in PG's [On Lisp](http://paulgraham.com/onlisp.html). + - Works by using intermediate storage. **Do not** use the original iterator after a call to `butlast` or `butlastn`. + - `first`, `second`, `nth`, `last`: return the specified item from an iterable. Any preceding items are consumed at C speed. + - `iterate1`, `iterate`: return an infinite generator that yields `x`, `f(x)`, `f(f(x))`, ... + - `iterate1` is for 1-to-1 functions; `iterate` for n-to-n, unpacking the return value to the argument list of the next call. + - `partition` from `itertools` [recipes](https://docs.python.org/3/library/itertools.html#itertools-recipes). + - `rev` is a convenience function that tries `reversed`, and if the input was not a sequence, converts it to a tuple and reverses that. The return value is a `reversed` object. + - `scons`: prepend one element to the start of an iterable, return new iterable. ``scons(x, iterable)`` is lispy shorthand for ``itertools.chain((x,), iterable)``, allowing to omit the one-item tuple wrapper. + - `inn`: contains-check (``x in iterable``) with automatic termination for monotonic divergent infinite iterables. *Added in v0.13.1.* + - Only applicable to monotonic divergent inputs (such as ``primes``). Increasing/decreasing is auto-detected from the first non-zero diff, but the function may fail to terminate if the input is actually not monotonic, or has an upper/lower bound. + - `iindex`: like ``list.index``, but for a general iterable. Consumes the iterable, so only makes sense for memoized inputs. *Added in v0.13.1.* + - `prod`: like the builtin `sum`, but compute the product. Oddly missing from the standard library. *Added in v0.13.1.* + - `window`: sliding length-n window iterator for general iterables. Acts like the well-known [n-gram zip trick](http://www.locallyoptimal.com/blog/2013/01/20/elegant-n-gram-generation-in-python/), but the input can be any iterable. *Added in v0.14.1.* + +Examples: + +```python +from functools import partial +from unpythonic import scanl, scanr, foldl, foldr, flatmap, mapr, zipr, \ + uniqify, uniq, flatten1, flatten, flatten_in, take, drop, \ + unfold, unfold1, cons, nil, ll, curry, s, inn, iindex, window + +assert tuple(scanl(add, 0, range(1, 5))) == (0, 1, 3, 6, 10) +assert tuple(scanr(add, 0, range(1, 5))) == (0, 4, 7, 9, 10) +assert tuple(scanl(mul, 1, range(2, 6))) == (1, 2, 6, 24, 120) +assert tuple(scanr(mul, 1, range(2, 6))) == (1, 5, 20, 60, 120) + +assert tuple(scanl(cons, nil, ll(1, 2, 3))) == (nil, ll(1), ll(2, 1), ll(3, 2, 1)) +assert tuple(scanr(cons, nil, ll(1, 2, 3))) == (nil, ll(3), ll(2, 3), ll(1, 2, 3)) + +def step2(k): # x0, x0 + 2, x0 + 4, ... + return (k, k + 2) # value, newstate +assert tuple(take(10, unfold1(step2, 10))) == (10, 12, 14, 16, 18, 20, 22, 24, 26, 28) + +def nextfibo(a, b): + return (a, b, a + b) # value, *newstates +assert tuple(take(10, unfold(nextfibo, 1, 1))) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) + +def fibos(): + a, b = 1, 1 + while True: + yield a + a, b = b, a + b +a1, a2, a3, tl = unpack(3, fibos()) +a4, a5, tl = unpack(2, tl) +print(a1, a2, a3, a4, a5, tl) # --> 1 1 2 3 5 + +# inn: contains-check with automatic termination for monotonic iterables (infinites ok) +evens = imemoize(s(2, 4, ...)) +assert inn(42, evens()) +assert not inn(41, evens()) + +@gmemoize +def primes(): + yield 2 + for n in count(start=3, step=2): + if not any(n % p == 0 for p in takewhile(lambda x: x*x <= n, primes())): + yield n +assert inn(31337, primes()) +assert not inn(1337, primes()) + +# iindex: find index of item in iterable (mostly only makes sense for memoized input) +assert iindex(2, (1, 2, 3)) == 1 +assert iindex(31337, primes()) == 3378 + +# window: length-n sliding window iterator for general iterables +lst = (x for x in range(5)) +out = [] +for a, b, c in window(lst, n=3): + out.append((a, b, c)) +assert out == [(0, 1, 2), (1, 2, 3), (2, 3, 4)] + +# flatmap +def msqrt(x): # multivalued sqrt + if x == 0.: + return (0.,) + else: + s = x**0.5 + return (s, -s) +assert tuple(flatmap(msqrt, (0, 1, 4, 9))) == (0., 1., -1., 2., -2., 3., -3.) + +# zipr reverses, then iterates. +assert tuple(zipr((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) + +zipr2 = partial(mapr, identity) # mapr works the same way. +assert tuple(zipr2((1, 2, 3), (4, 5, 6), (7, 8))) == ((3, 6, 8), (2, 5, 7)) + +# foldr doesn't; it walks from the left, but collects results from the right: +zipr1 = curry(foldr, zipper, ()) +assert zipr1((1, 2, 3), (4, 5, 6), (7, 8)) == ((2, 5, 8), (1, 4, 7)) +# so the result is reversed(zip(...)), whereas zipr gives zip(*(reversed(s) for s in ...)) + +assert tuple(uniqify((1, 1, 2, 2, 2, 1, 2, 2, 4, 3, 4, 3, 3))) == (1, 2, 4, 3) # all +assert tuple(uniq((1, 1, 2, 2, 2, 1, 2, 2, 4, 3, 4, 3, 3))) == (1, 2, 1, 2, 4, 3, 4, 3) # consecutive + +assert tuple(flatten1(((1, 2), (3, (4, 5), 6), (7, 8, 9)))) == (1, 2, 3, (4, 5), 6, 7, 8, 9) +assert tuple(flatten(((1, 2), (3, (4, 5), 6), (7, 8, 9)))) == (1, 2, 3, 4, 5, 6, 7, 8, 9) + +is_nested = lambda sublist: all(isinstance(x, (list, tuple)) for x in sublist) +assert tuple(flatten((((1, 2), (3, 4)), (5, 6)), is_nested)) == ((1, 2), (3, 4), (5, 6)) + +data = (((1, 2), ((3, 4), (5, 6)), 7), ((8, 9), (10, 11))) +assert tuple(flatten(data, is_nested)) == (((1, 2), ((3, 4), (5, 6)), 7), (8, 9), (10, 11)) +assert tuple(flatten_in(data, is_nested)) == (((1, 2), (3, 4), (5, 6), 7), (8, 9), (10, 11)) + +with_n = lambda *args: (partial(f, n) for n, f in args) +clip = lambda n1, n2: composel(*with_n((n1, drop), (n2, take))) +assert tuple(clip(5, 10)(range(20))) == tuple(range(5, 15)) +``` + +In the last example, essentially we just want to `clip 5 10 (range 20)`, the grouping of the parentheses being pretty much an implementation detail. With ``curry``, we can rewrite the last line as: + +```python +assert tuple(curry(clip, 5, 10, range(20)) == tuple(range(5, 15)) +``` + + +### ``islice``: slice syntax support for ``itertools.islice`` + +*Added in v0.13.1.* + +Slice an iterable, using the regular slicing syntax: + +```python +from unpythonic import islice, primes, s + +p = primes() +assert tuple(islice(p)[10:15]) == (31, 37, 41, 43, 47) + +assert tuple(islice(primes())[10:15]) == (31, 37, 41, 43, 47) + +p = primes() +assert islice(p)[10] == 31 + +odds = islice(s(1, 2, ...))[::2] +assert tuple(islice(odds)[:5]) == (1, 3, 5, 7, 9) +assert tuple(islice(odds)[:5]) == (11, 13, 15, 17, 19) # five more +``` + +The slicing variant calls ``itertools.islice`` with the corresponding slicing parameters. + +As a convenience feature: a single index is interpreted as a length-1 islice starting at that index. The slice is then immediately evaluated and the item is returned. + +**CAUTION**: Keep in mind ``itertools.islice`` does not support negative indexing for any of ``start``, ``stop`` or ``step``, and that the slicing process consumes elements from the iterable. + +Like ``fup``, our ``islice`` is essentially a manually curried function with unusual syntax; the initial call to ``islice`` passes in the iterable to be sliced. The object returned by the call accepts a subscript to specify the slice or index. Once the slice or index is provided, the call to ``itertools.islice`` triggers. + +Inspired by Python itself. + + +### `gmemoize`, `imemoize`, `fimemoize`: memoize generators + +Make generator functions (gfunc, i.e. a generator definition) which create memoized generators, similar to how streams behave in Racket. + +Memoize iterables; like `itertools.tee`, but no need to know in advance how many copies of the iterator will be made. Provided for both iterables and for factory functions that make iterables. + + - `gmemoize` is a decorator for a gfunc, which makes it memoize the instantiated generators. + - If the gfunc takes arguments, they must be hashable. A separate memoized sequence is created for each unique set of argument values seen. + - For simplicity, the generator itself may use ``yield`` for output only; ``send`` is not supported. + - Any exceptions raised by the generator (except StopIteration) are also memoized, like in ``memoize``. + - Thread-safe. Calls to ``next`` on the memoized generator from different threads are serialized via a lock. Each memoized sequence has its own lock. This uses ``threading.RLock``, so re-entering from the same thread (e.g. in recursively defined sequences) is fine. + - The whole history is kept indefinitely. For infinite iterables, use this only if you can guarantee that only a reasonable number of terms will ever be evaluated (w.r.t. available RAM). + - Typically, this should be the outermost decorator if several are used on the same gfunc. + - `imemoize`: memoize an iterable. Like `itertools.tee`, but keeps the whole history, so more copies can be teed off later. + - Same limitation: **do not** use the original iterator after it is memoized. The danger is that if anything other than the memoization mechanism advances the original iterator, some values will be lost before they can reach the memo. + - Returns a gfunc with no parameters which, when called, returns a generator that yields items from the memoized iterable. The original iterable is used to retrieve more terms when needed. + - Calling the gfunc essentially tees off a new instance, which begins from the first memoized item. + - `fimemoize`: convert a factory function, that returns an iterable, into the corresponding gfunc, and `gmemoize` that. Return the memoized gfunc. + - Especially convenient with short lambdas, where `(yield from ...)` instead of `...` is just too much text. + +```python +from itertools import count, takewhile +from unpythonic import gmemoize, imemoize, fimemoize, take, nth + +@gmemoize +def primes(): # FP sieve of Eratosthenes + yield 2 + for n in count(start=3, step=2): + if not any(n % p == 0 for p in takewhile(lambda x: x*x <= n, primes())): + yield n +assert tuple(take(10, primes())) == (2, 3, 5, 7, 11, 13, 17, 19, 23, 29) +assert nth(3378, primes()) == 31337 # with memo, linear process; no crash + +# but be careful: +31337 in primes() # --> True +1337 in takewhile(lambda p: p <= 1337, primes()) # not prime, need takewhile() to stop + +# or use unpythonic.inn, which auto-terminates on monotonic iterables: +from unpythonic import inn +inn(31337, primes()) # --> True +inn(1337, primes()) # --> False +``` + +Memoizing only a part of an iterable. This is where `imemoize` and `fimemoize` can be useful. The basic idea is to make a chain of generators, and only memoize the last one: + +```python +from unpythonic import gmemoize, drop, last + +def evens(): # the input iterable + yield from (x for x in range(100) if x % 2 == 0) + +@gmemoize +def some_evens(n): # we want to memoize the result without the n first terms + yield from drop(n, evens()) + +assert last(some_evens(25)) == last(some_evens(25)) # iterating twice! +``` + +Using a lambda, we can also write ``some_evens`` as: + +```python +se = gmemoize(lambda n: (yield from drop(n, evens()))) +assert last(se(25)) == last(se(25)) +``` + +Using `fimemoize`, we can omit the ``yield from``, shortening this to: + +```python +se = fimemoize(lambda n: drop(n, evens())) +assert last(se(25)) == last(se(25)) +``` + +If we don't need to take an argument, we can memoize the iterable directly, using ``imemoize``: + +```python +se = imemoize(drop(25, evens())) +assert last(se()) == last(se()) # se is a gfunc, so call it to get a generator instance +``` + +Finally, compare the `fimemoize` example, rewritten using `def`, to the original `gmemoize` example: + +```python +@fimemoize +def some_evens(n): + return drop(n, evens()) + +@gmemoize +def some_evens(n): + yield from drop(n, evens()) +``` + +The only differences are the name of the decorator and ``return`` vs. ``yield from``. The point of `fimemoize` is that in simple cases like this, it allows us to use a regular factory function that makes an iterable, instead of a gfunc. Of course, the gfunc could have several `yield` expressions before it finishes, whereas the factory function terminates at the `return`. + + +### ``fup``: Functional update; ``ShadowedSequence`` + +We provide ``ShadowedSequence``, which is a bit like ``collections.ChainMap``, but for sequences, and only two levels (but it's a sequence; instances can be chained). See its docstring for details. + +The function ``fupdate`` functionally updates sequences and mappings. Whereas ``ShadowedSequence`` reads directly from the original sequences at access time, ``fupdate`` makes a shallow copy (of the same type as the given input sequence) when it finalizes its output. The utility function ``fup`` is a specialization of ``fupdate`` to sequences, and adds support for the standard slicing syntax. + +First, let's look at ``fupdate``: + +```python +from unpythonic import fupdate + +lst = [1, 2, 3] +out = fupdate(lst, 1, 42) +assert lst == [1, 2, 3] # the original remains untouched +assert out == [1, 42, 3] + +lst = [1, 2, 3] +out = fupdate(lst, -1, 42) # negative indices also supported +assert lst == [1, 2, 3] +assert out == [1, 2, 42] +``` + +Immutable input sequences are allowed. Replacing a slice of a tuple by a sequence: + +```python +from itertools import repeat +lst = (1, 2, 3, 4, 5) +assert fupdate(lst, slice(0, None, 2), tuple(repeat(10, 3))) == (10, 2, 10, 4, 10) +assert fupdate(lst, slice(1, None, 2), tuple(repeat(10, 2))) == (1, 10, 3, 10, 5) +assert fupdate(lst, slice(None, None, 2), tuple(repeat(10, 3))) == (10, 2, 10, 4, 10) +assert fupdate(lst, slice(None, None, -1), tuple(range(5))) == (4, 3, 2, 1, 0) +``` + +Slicing supports negative indices and steps, and default starts, stops and steps, as usual in Python. Just remember ``a[start:stop:step]`` actually means ``a[slice(start, stop, step)]`` (with ``None`` replacing omitted ``start``, ``stop`` and ``step``), and everything should follow. Multidimensional arrays are **not** supported. + +When ``fupdate`` constructs its output, the replacement occurs by walking *the input sequence* left-to-right, and pulling an item from the replacement sequence when the given replacement specification so requires. Hence the replacement sequence is not necessarily accessed left-to-right. (In the last example above, ``tuple(range(5))`` was read in the order ``(4, 3, 2, 1, 0)``.) + +The replacement sequence must have at least as many items as the slice requires (when applied to the original input). Any extra items in the replacement sequence are simply ignored (so e.g. an infinite ``repeat`` is fine), but if the replacement is too short, ``IndexError`` is raised. (*Changed in v0.13.1.* This was previously ``ValueError``.) + +It is also possible to replace multiple individual items. These are treated as separate specifications, applied left to right (so later updates shadow earlier ones, if updating at the same index): + +```python +lst = (1, 2, 3, 4, 5) +out = fupdate(lst, (1, 2, 3), (17, 23, 42)) +assert lst == (1, 2, 3, 4, 5) +assert out == (1, 17, 23, 42, 5) +``` + +Multiple specifications can be used with slices and sequences as well: + +```python +lst = tuple(range(10)) +out = fupdate(lst, (slice(0, 10, 2), slice(1, 10, 2)), + (tuple(repeat(2, 5)), tuple(repeat(3, 5)))) +assert lst == tuple(range(10)) +assert out == (2, 3, 2, 3, 2, 3, 2, 3, 2, 3) +``` + +Strictly speaking, each specification can be either a slice/sequence pair or an index/item pair: + +```python +lst = tuple(range(10)) +out = fupdate(lst, (slice(0, 10, 2), slice(1, 10, 2), 6), + (tuple(repeat(2, 5)), tuple(repeat(3, 5)), 42)) +assert lst == tuple(range(10)) +assert out == (2, 3, 2, 3, 2, 3, 42, 3, 2, 3) +``` + +Also mappings can be functionally updated: + +```python +d1 = {'foo': 'bar', 'fruit': 'apple'} +d2 = fupdate(d1, foo='tavern') +assert sorted(d1.items()) == [('foo', 'bar'), ('fruit', 'apple')] +assert sorted(d2.items()) == [('foo', 'tavern'), ('fruit', 'apple')] +``` + +For immutable mappings, ``fupdate`` supports ``frozendict`` (see below). Any other mapping is assumed mutable, and ``fupdate`` essentially just performs ``copy.copy()`` and then ``.update()``. + +We can also functionally update a namedtuple: + +```python +from collections import namedtuple +A = namedtuple("A", "p q") +a = A(17, 23) +out = fupdate(a, 0, 42) +assert a == A(17, 23) +assert out == A(42, 23) +``` + +Namedtuples export only a sequence interface, so they cannot be treated as mappings. + +Support for ``namedtuple`` requires an extra feature, which is available for custom classes, too. When constructing the output sequence, ``fupdate`` first checks whether the input type has a ``._make()`` method, and if so, hands the iterable containing the final data to that to construct the output. Otherwise the regular constructor is called (and it must accept a single iterable). + +**The preferred way** to use ``fupdate`` on sequences is through the ``fup`` utility function, which adds support for Python's standard slicing syntax: + +```python +from unpythonic import fup +from itertools import repeat + +lst = (1, 2, 3, 4, 5) +assert fup(lst)[3] << 42 == (1, 2, 3, 42, 5) +assert fup(lst)[0::2] << tuple(repeat(10, 3)) == (10, 2, 10, 4, 10) +``` + +Currently only one update specification is supported in a single ``fup()``. + +The notation follows the ``unpythonic`` convention that ``<<`` denotes an assignment of some sort. Here it denotes a functional update, which returns a modified copy, leaving the original untouched. + +The ``fup`` call is essentially curried. It takes in the sequence to be functionally updated. The object returned by the call accepts a subscript to specify the index or indices. This then returns another object that accepts a left-shift to specify the values. Once the values are provided, the underlying call to ``fupdate`` triggers, and the result is returned. + +*Changed in v0.13.1.* Added support to ``ShadowedSequence`` for slicing (read-only), equality comparison, ``str`` and ``repr``. Out-of-range read access to a single item emits a meaningful error, like in ``list``. The utility ``fup`` was previously a macro; now it is a regular function, with slightly changed syntax to accommodate. + + +### ``view``: writable, sliceable view into a sequence + +*Added in v0.14.0.* Added the read-only cousin ``roview``, which behaves the same except it has no ``__setitem__`` or ``reverse``. This can be useful for giving read-only access to an internal sequence. The constructor of the writable ``view`` now checks that the input is not read-only (``roview``, or a ``Sequence`` that is not also a ``MutableSequence``) before allowing creation of the writable view. + +*Added in v0.13.1.* + +A writable view into a sequence, with slicing, so you can take a slice of a slice (of a slice ...), and it reflects the original both ways: + +```python +from unpythonic import view + +lst = list(range(10)) +v = view(lst)[::2] +assert v == [0, 2, 4, 6, 8] +v2 = v[1:-1] +assert v2 == [2, 4, 6] +v2[1:] = (10, 20) +assert lst == [0, 1, 2, 3, 10, 5, 20, 7, 8, 9] + +lst[2] = 42 +assert v == [0, 42, 10, 20, 8] +assert v2 == [42, 10, 20] + +lst = list(range(5)) +v = view(lst)[2:4] +v[:] = 42 # scalar broadcast +assert lst == [0, 1, 42, 42, 4] +``` + +While ``fupdate`` lets you be more functional than Python otherwise allows, ``view`` lets you be more imperative than Python otherwise allows. + +We store slice specs, not actual indices, so this works also if the underlying sequence undergoes length changes. + +Slicing a view returns a new view. Slicing anything else will usually copy, because the object being sliced does, before we get control. To slice lazily, first view the sequence itself and then slice that. The initial no-op view is optimized away, so it won't slow down accesses. Alternatively, pass a ``slice`` object into the ``view`` constructor. + +The view can be efficiently iterated over. As usual, iteration assumes that no inserts/deletes in the underlying sequence occur during the iteration. + +Getting/setting an item (subscripting) checks whether the index cache needs updating during each access, so it can be a bit slow. Setting a slice checks just once, and then updates the underlying iterable directly. Setting a slice to a scalar value broadcasts the scalar à la NumPy. + +The ``unpythonic.collections`` module also provides the ``SequenceView`` and ``MutableSequenceView`` abstract base classes; ``view`` is a ``MutableSequenceView``. + + +### ``mogrify``: update a mutable container in-place + +*Added in v0.13.0.* + +Recurse on given container, apply a function to each atom. If the container is mutable, then update in-place; if not, then construct a new copy like ``map`` does. + +If the container is a mapping, the function is applied to the values; keys are left untouched. + +Unlike ``map`` and its cousins, only a single input container is supported. (Supporting multiple containers as input would require enforcing some compatibility constraints on their type and shape, since ``mogrify`` is not limited to sequences.) + +```python +from unpythonic import mogrify + +lst1 = [1, 2, 3] +lst2 = mogrify(lst1, lambda x: x**2) +assert lst2 == [2, 4, 6] +assert lst2 is lst1 +``` + +Containers are detected by checking for instances of ``collections.abc`` superclasses (also virtuals are ok). Supported abcs are ``MutableMapping``, ``MutableSequence``, ``MutableSet``, ``Mapping``, ``Sequence`` and ``Set``. Any value that does not match any of these is treated as an atom. Containers can be nested, with an arbitrary combination of the types supported. + +For convenience, we introduce some special cases: + + - Any classes created by ``collections.namedtuple``, because they do not conform to the standard constructor API for a ``Sequence``. + + Thus, for (an immutable) ``Sequence``, we first check for the presence of a ``._make()`` method, and if found, use it as the constructor. Otherwise we use the regular constructor. + + - ``str`` is treated as an atom, although technically a ``Sequence``. + + It doesn't conform to the exact same API (its constructor does not take an iterable), and often we don't want to treat strings as containers anyway. + + If you want to process strings, implement it in your function that is called by ``mogrify``. + + - The ``box`` container from ``unpythonic.collections``; although mutable, its update is not conveniently expressible by the ``collections.abc`` APIs. + + - The ``cons`` container from ``unpythonic.llist`` (including the ``ll``, ``llist`` linked lists). This is treated with the general tree strategy, so nested linked lists will be flattened, and the final ``nil`` is also processed. + + Note that since ``cons`` is immutable, anyway, if you know you have a long linked list where you need to update the values, just iterate over it and produce a new copy - that will work as intended. + + +### ``s``, ``m``, ``mg``: lazy mathematical sequences with infix arithmetic + +*Added in v0.13.0.* + +*Added in v0.13.1:* ``primes`` and ``fibonacci``. + +*Added in v0.14.0:* ``mg``, a decorator to mathify a gfunc, so that it will ``m()`` the generator instances it makes. Combo with ``imemoize`` for great justice, e.g. ``a = mg(imemoize(s(1, 2, ...)))``. + +We provide a compact syntax to create lazy constant, arithmetic, geometric and power sequences: ``s(...)``. Numeric (``int``, ``float``, ``mpmath``) and symbolic (SymPy) formats are supported. We avoid accumulating roundoff error when used with floating-point formats. + +We also provide arithmetic operation support for iterables (termwise). To make any iterable infix math aware, use ``m(iterable)``. The arithmetic is lazy; it just plans computations, returning a new lazy mathematical sequence. To extract values, iterate over the result. (Note this implies that expressions consisting of thousands of operations will overflow Python's call stack. In practice this shouldn't be a problem.) + +The function versions of the arithmetic operations (also provided, à la the ``operator`` module) have an **s** prefix (short for mathematical **sequence**), because in Python the **i** prefix (which could stand for *iterable*) is already used to denote the in-place operators. + +We provide the [Cauchy product](https://en.wikipedia.org/wiki/Cauchy_product), and its generalization, the diagonal combination-reduction, for two (possibly infinite) iterables. Note ``cauchyprod`` **does not sum the series**; given the input sequences ``a`` and ``b``, the call ``cauchyprod(a, b)`` computes the elements of the output sequence ``c``. + +Finally, we provide ready-made generators that yield some common sequences (currently, the Fibonacci numbers and the prime numbers). The prime generator is an FP-ized sieve of Eratosthenes. + +```python +from unpythonic import s, m, cauchyprod, take, last, fibonacci, primes + +assert tuple(take(10, s(1, ...))) == (1,)*10 +assert tuple(take(10, s(1, 2, ...))) == tuple(range(1, 11)) +assert tuple(take(10, s(1, 2, 4, ...))) == (1, 2, 4, 8, 16, 32, 64, 128, 256, 512) +assert tuple(take(5, s(2, 4, 16, ...))) == (2, 4, 16, 256, 65536) # 2, 2**2, (2**2)**2, ... + +assert tuple(s(1, 2, ..., 10)) == tuple(range(1, 11)) +assert tuple(s(1, 2, 4, ..., 512)) == (1, 2, 4, 8, 16, 32, 64, 128, 256, 512) + +assert tuple(take(5, s(1, -1, 1, ...))) == (1, -1, 1, -1, 1) + +assert tuple(take(5, s(1, 3, 5, ...) + s(2, 4, 6, ...))) == (3, 7, 11, 15, 19) +assert tuple(take(5, s(1, 3, ...) * s(2, 4, ...))) == (2, 12, 30, 56, 90) + +assert tuple(take(5, s(1, 3, ...)**s(2, 4, ...))) == (1, 3**4, 5**6, 7**8, 9**10) +assert tuple(take(5, s(1, 3, ...)**2)) == (1, 3**2, 5**2, 7**2, 9**2) +assert tuple(take(5, 2**s(1, 3, ...))) == (2**1, 2**3, 2**5, 2**7, 2**9) + +assert tuple(take(3, cauchyprod(s(1, 3, 5, ...), s(2, 4, 6, ...)))) == (2, 10, 28) + +assert tuple(take(10, primes())) == (2, 3, 5, 7, 11, 13, 17, 19, 23, 29) +assert tuple(take(10, fibonacci())) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) +``` + +A math iterable (i.e. one that has infix math support) is an instance of the class ``m``: + +```python +a = s(1, 3, ...) +b = s(2, 4, ...) +c = a + b +assert isinstance(a, m) +assert isinstance(b, m) +assert isinstance(c, m) +assert tuple(take(5, c)) == (3, 7, 11, 15, 19) + +d = 1 / (a + b) +assert isinstance(d, m) +``` + +Applying an operation meant for regular (non-math) iterables will drop the arithmetic support, but it can be restored by m'ing manually: + +```python +e = take(5, c) +assert not isinstance(e, m) + +f = m(take(5, c)) +assert isinstance(f, m) +``` + +Symbolic expression support with SymPy: + +```python +from unpythonic import s +from sympy import symbols + +x0 = symbols("x0", real=True) +k = symbols("k", positive=True) + +assert tuple(take(4, s(x0, ...))) == (x0, x0, x0, x0) +assert tuple(take(4, s(x0, x0 + k, ...))) == (x0, x0 + k, x0 + 2*k, x0 + 3*k) +assert tuple(take(4, s(x0, x0*k, x0*k**2, ...))) == (x0, x0*k, x0*k**2, x0*k**3) + +assert tuple(s(x0, x0 + k, ..., x0 + 3*k)) == (x0, x0 + k, x0 + 2*k, x0 + 3*k) +assert tuple(s(x0, x0*k, x0*k**2, ..., x0*k**5)) == (x0, x0*k, x0*k**2, x0*k**3, x0*k**4, x0*k**5) + +x0, k = symbols("x0, k", positive=True) +assert tuple(s(x0, x0**k, x0**(k**2), ..., x0**(k**4))) == (x0, x0**k, x0**(k**2), x0**(k**3), x0**(k**4)) + +x = symbols("x", real=True) +px = lambda stream: stream * s(1, x, x**2, ...) # powers of x +s1 = px(s(1, 3, 5, ...)) # 1, 3*x, 5*x**2, ... +s2 = px(s(2, 4, 6, ...)) # 2, 4*x, 6*x**2, ... +assert tuple(take(3, cauchyprod(s1, s2))) == (2, 10*x, 28*x**2) +``` + +**CAUTION**: Symbolic sequence detection is sensitive to the assumptions on the symbols, because very pythonically, ``SymPy`` only simplifies when the result is guaranteed to hold in the most general case under the given assumptions. + +Inspired by Haskell. + + +## Control flow tools + +Tools related to control flow. + +### ``trampolined``, ``jump``: tail call optimization (TCO) / explicit continuations + +**v0.10.0**: ``fasttco`` has been renamed ``tco``, and the exception-based old default implementation has been removed. See also [macros](macro_extras/) for an easy-to-use solution. + +**v0.11.1**: The special jump target ``SELF`` (keep current target) has been removed. If you need tail recursion in a lambda, use ``unpythonic.fun.withself`` to get a reference to the lambda itself. See example below. + +Express algorithms elegantly without blowing the call stack - with explicit, clear syntax. + +*Tail recursion*: + +```python +from unpythonic import trampolined, jump + +@trampolined +def fact(n, acc=1): + if n == 0: + return acc + else: + return jump(fact, n - 1, n * acc) +print(fact(4)) # 24 +``` + +Functions that use TCO **must** be `@trampolined`. Calling a trampolined function normally starts the trampoline. + +Inside a trampolined function, a normal call `f(a, ..., kw=v, ...)` remains a normal call. + +A tail call with target `f` is denoted `return jump(f, a, ..., kw=v, ...)`. This explicitly marks that it is indeed a tail call (due to the explicit ``return``). Note that `jump` is **a noun, not a verb**. The `jump(f, ...)` part just evaluates to a `jump` instance, which on its own does nothing. Returning it to the trampoline actually performs the tail call. + +If the jump target has a trampoline, don't worry; the trampoline implementation will automatically strip it and jump into the actual entrypoint. + +Trying to ``jump(...)`` without the ``return`` does nothing useful, and will **usually** print an *unclaimed jump* warning. It does this by checking a flag in the ``__del__`` method of ``jump``; any correctly used jump instance should have been claimed by a trampoline before it gets garbage-collected. + +(Some *unclaimed jump* warnings may appear also if the process is terminated by Ctrl+C (``KeyboardInterrupt``). This is normal; it just means that the termination occurred after a jump object was instantiated but before it was claimed by the trampoline.) + +The final result is just returned normally. This shuts down the trampoline, and returns the given value from the initial call (to a ``@trampolined`` function) that originally started that trampoline. + + +*Tail recursion in a lambda*: + +```python +t = trampolined(withself(lambda self, n, acc=1: + acc if n == 0 else jump(self, n - 1, n * acc))) +print(t(4)) # 24 +``` + +Here the jump is just `jump` instead of `return jump`, since lambda does not use the `return` syntax. + +To denote tail recursion in an anonymous function, use ``unpythonic.fun.withself``. The ``self`` argument is declared explicitly, but passed implicitly, just like the ``self`` argument of a method. + + +*Mutual recursion with TCO*: + +```python +@trampolined +def even(n): + if n == 0: + return True + else: + return jump(odd, n - 1) +@trampolined +def odd(n): + if n == 0: + return False + else: + return jump(even, n - 1) +assert even(42) is True +assert odd(4) is False +assert even(10000) is True # no crash +``` + +*Mutual recursion in `letrec` with TCO*: + +```python +letrec(evenp=lambda e: + trampolined(lambda x: + (x == 0) or jump(e.oddp, x - 1)), + oddp=lambda e: + trampolined(lambda x: + (x != 0) and jump(e.evenp, x - 1)), + body=lambda e: + e.evenp(10000)) +``` + + +#### Reinterpreting TCO as explicit continuations + +TCO from another viewpoint: + +```python +@trampolined +def foo(): + return jump(bar) +@trampolined +def bar(): + return jump(baz) +@trampolined +def baz(): + print("How's the spaghetti today?") +foo() +``` + +Each function in the TCO call chain tells the trampoline where to go next (and with what arguments). All hail [lambda, the ultimate GOTO](http://hdl.handle.net/1721.1/5753)! + +Each TCO call chain brings its own trampoline, so they nest as expected: + +```python +@trampolined +def foo(): + return jump(bar) +@trampolined +def bar(): + t = even(42) # start another trampoline for even/odd + return jump(baz, t) +@trampolined +def baz(result): + print(result) +foo() # start trampoline +``` + + +### ``looped``, ``looped_over``: loops in FP style (with TCO) + +*Functional loop with automatic tail call optimization* (for calls re-invoking the loop body): + +```python +from unpythonic import looped, looped_over + +@looped +def s(loop, acc=0, i=0): + if i == 10: + return acc + else: + return loop(acc + i, i + 1) +print(s) # 45 +``` + +Compare the sweet-exp Racket: + +```racket +define s + let loop ([acc 0] [i 0]) + cond + {i = 10} + acc + else + loop {acc + i} {i + 1} +displayln s ; 45 +``` + +The `@looped` decorator is essentially sugar. Behaviorally equivalent code: + +```python +@trampolined +def s(acc=0, i=0): + if i == 10: + return acc + else: + return jump(s, acc + i, i + 1) +s = s() +print(s) # 45 +``` + +In `@looped`, the function name of the loop body is the name of the final result, like in `@call`. The final result of the loop is just returned normally. + +The first parameter of the loop body is the magic parameter ``loop``. It is *self-ish*, representing a jump back to the loop body itself, starting a new iteration. Just like Python's ``self``, ``loop`` can have any name; it is passed positionally. + +Note that ``loop`` is **a noun, not a verb.** This is because the expression ``loop(...)`` is essentially the same as ``jump(...)`` to the loop body itself. However, it also inserts the magic parameter ``loop``, which can only be set up via this mechanism. + +Additional arguments can be given to ``loop(...)``. When the loop body is called, any additional positional arguments are appended to the implicit ones, and can be anything. Additional arguments can also be passed by name. The initial values of any additional arguments **must** be declared as defaults in the formal parameter list of the loop body. The loop is automatically started by `@looped`, by calling the body with the magic ``loop`` as the only argument. + +Any loop variables such as ``i`` in the above example are **in scope only in the loop body**; there is no ``i`` in the surrounding scope. Moreover, it's a fresh ``i`` at each iteration; nothing is mutated by the looping mechanism. (But be careful if you use a mutable object instance as a loop variable. The loop body is just a function call like any other, so the usual rules apply.) + +FP loops don't have to be pure: + +```python +out = [] +@looped +def _(loop, i=0): + if i <= 3: + out.append(i) # cheeky side effect + return loop(i + 1) + # the implicit "return None" terminates the loop. +assert out == [0, 1, 2, 3] +``` + +Keep in mind, though, that this pure-Python FP looping mechanism is slow, so it may make sense to use it only when "the FP-ness" (no mutation, scoping) is important. + +Also be aware that `@looped` is specifically neither a ``for`` loop nor a ``while`` loop; instead, it is a general looping mechanism that can express both kinds of loops. + +*Typical `while True` loop in FP style*: + +```python +@looped +def _(loop): + print("Enter your name (or 'q' to quit): ", end='') + s = input() + if s.lower() == 'q': + return # ...the implicit None. In a "while True:", "break" here. + else: + print("Hello, {}!".format(s)) + return loop() +``` + +#### FP loop over an iterable + +In Python, loops often run directly over the elements of an iterable, which markedly improves readability compared to dealing with indices. Enter ``@looped_over``: + +```python +@looped_over(range(10), acc=0) +def s(loop, x, acc): + return loop(acc + x) +assert s == 45 +``` + +The ``@looped_over`` decorator is essentially sugar. Behaviorally equivalent code: + +```python +@call +def s(iterable=range(10)): + it = iter(iterable) + @looped + def _tmp(loop, acc=0): + try: + x = next(it) + return loop(acc + x) + except StopIteration: + return acc + return _tmp +assert s == 45 +``` + +In ``@looped_over``, the loop body takes three magic positional parameters. The first parameter ``loop`` works like in ``@looped``. The second parameter ``x`` is the current element. The third parameter ``acc`` is initialized to the ``acc`` value given to ``@looped_over``, and then (functionally) updated at each iteration, taking as the new value the first positional argument given to ``loop(...)``, if any positional arguments were given. Otherwise ``acc`` retains its last value. + +Additional arguments can be given to ``loop(...)``. The same notes as above apply. For example, here we have the additional parameters ``fruit`` and ``number``. The first one is passed positionally, and the second one by name: + +```python +@looped_over(range(10), acc=0) +def s(loop, x, acc, fruit="pear", number=23): + print(fruit, number) + newfruit = "orange" if fruit == "apple" else "apple" + newnumber = number + 1 + return loop(acc + x, newfruit, number=newnumber) +assert s == 45 +``` + +The loop body is called once for each element in the iterable. When the iterable runs out of elements, the last ``acc`` value that was given to ``loop(...)`` becomes the return value of the loop. If the iterable is empty, the body never runs; then the return value of the loop is the initial value of ``acc``. + +To terminate the loop early, just ``return`` your final result normally, like in ``@looped``. (It can be anything, does not need to be ``acc``.) + +Multiple input iterables work somewhat like in Python's ``for``, except any sequence unpacking must be performed inside the body: + +```python +@looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=()) +def p(loop, item, acc): + numb, lett = item + return loop(acc + ("{:d}{:s}".format(numb, lett),)) +assert p == ('1a', '2b', '3c') + +@looped_over(enumerate(zip((1, 2, 3), ('a', 'b', 'c'))), acc=()) +def q(loop, item, acc): + idx, (numb, lett) = item + return loop(acc + ("Item {:d}: {:d}{:s}".format(idx, numb, lett),)) +assert q == ('Item 0: 1a', 'Item 1: 2b', 'Item 2: 3c') +``` + +FP loops can be nested (also those over iterables): + +```python +@looped_over(range(1, 4), acc=()) +def outer_result(outer_loop, y, outer_acc): + @looped_over(range(1, 3), acc=()) + def inner_result(inner_loop, x, inner_acc): + return inner_loop(inner_acc + (y*x,)) + return outer_loop(outer_acc + (inner_result,)) +assert outer_result == ((1, 2), (2, 4), (3, 6)) +``` + +If you feel the trailing commas ruin the aesthetics, see ``unpythonic.misc.pack``. + +#### Accumulator type and runtime cost + +As [the reference warns (note 6)](https://docs.python.org/3/library/stdtypes.html#common-sequence-operations), repeated concatenation of tuples has an O(n²) runtime cost, because each concatenation creates a new tuple, which needs to copy all of the already existing elements. To keep the runtime O(n), there are two options: + + - *Pythonic solution*: Destructively modify a mutable sequence. Particularly, ``list`` is a dynamic array that has a low amortized cost for concatenation (most often O(1), with the occasional O(n) when the allocated storage grows). + - *Unpythonic solution*: ``cons`` a linked list, and reverse it at the end. Cons cells are immutable; consing a new element to the front costs O(1). Reversing the list costs O(n). + +Mutable sequence (Python ``list``): + +```python +@looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=[]) +def p(loop, item, acc): + numb, lett = item + acc.append("{:d}{:s}".format(numb, lett)) + return loop(acc) +assert p == ['1a', '2b', '3c'] +``` + +Linked list: + +```python +from unpythonic import cons, nil, ll + +@lreverse +@looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=nil) +def p(loop, item, acc): + numb, lett = item + return loop(cons("{:d}{:s}".format(numb, lett), acc)) +assert p == ll('1a', '2b', '3c') +``` + +Note the unpythonic use of the ``lreverse`` function as a decorator. ``@looped_over`` overwrites the def'd name by the return value of the loop; then ``lreverse`` takes that as input, and overwrites once more. Thus ``p`` becomes the final list. + +To get the output as a tuple, we can add ``tuple`` to the decorator chain: + +```python +@tuple +@lreverse +@looped_over(zip((1, 2, 3), ('a', 'b', 'c')), acc=nil) +def p(loop, item, acc): + numb, lett = item + return loop(cons("{:d}{:s}".format(numb, lett), acc)) +assert p == ('1a', '2b', '3c') +``` + +This works in both solutions. The cost is an additional O(n) step. + +#### ``break`` + +The main way to exit an FP loop (also early) is, at any time, to just ``return`` the final result normally. + +If you want to exit the function *containing* the loop from inside the loop, see **escape continuations** below. + +#### ``continue`` + +The main way to *continue* an FP loop is, at any time, to ``loop(...)`` with the appropriate arguments that will make it proceed to the next iteration. Or package the appropriate `loop(...)` expression into your own function ``cont``, and then use ``cont(...)``: + +```python +@looped +def s(loop, acc=0, i=0): + cont = lambda newacc=acc: loop(newacc, i + 1) # always increase i; by default keep current value of acc + if i <= 4: + return cont() # essentially "continue" for this FP loop + elif i == 10: + return acc + else: + return cont(acc + i) +print(s) # 35 +``` + +This approach separates the computations of the new values for the iteration counter and the accumulator. + +#### Prepackaged ``break`` and ``continue`` + +See ``@breakably_looped`` (offering `brk`) and ``@breakably_looped_over`` (offering `brk` and `cnt`). + +The point of `brk(value)` over just `return value` is that `brk` is first-class, so it can be passed on to functions called by the loop body (so that those functions then have the power to directly terminate the loop). + +In ``@looped``, a library-provided ``cnt`` wouldn't make sense, since all parameters except ``loop`` are user-defined. *The client code itself defines what it means to proceed to the "next" iteration*. Really the only way in a construct with this degree of flexibility is for the client code to fill in all the arguments itself. + +Because ``@looped_over`` is a more specific abstraction, there the concept of *continue* is much more clear-cut. We define `cnt` to mean *proceed to take the next element from the iterable, keeping the current value of `acc`*. Essentially `cnt` is a partially applied `loop(...)` with the first positional argument set to the current value of `acc`. + +#### FP loops using a lambda as body + +Just call the `looped()` decorator manually: + +```python +s = looped(lambda loop, acc=0, i=0: + loop(acc + i, i + 1) if i < 10 else acc) +print(s) +``` + +It's not just a decorator; in Lisps, a construct like this would likely be named ``call/looped``. + +We can also use ``let`` to make local definitions: + +```python +s = looped(lambda loop, acc=0, i=0: + let(cont=lambda newacc=acc: + loop(newacc, i + 1), + body=lambda e: + e.cont(acc + i) if i < 10 else acc) +print(s) +``` + +The `looped_over()` decorator also works, if we just keep in mind that parameterized decorators in Python are actually decorator factories: + +```python +r10 = looped_over(range(10), acc=0) +s = r10(lambda loop, x, acc: + loop(acc + x)) +assert s == 45 +``` + +If you **really** need to make that into an expression, bind ``r10`` using ``let`` (if you use ``letrec``, keeping in mind it is a callable), or to make your code unreadable, just inline it. + +With ``curry``, this is also a possible solution: + +```python +s = curry(looped_over, range(10), 0, + lambda loop, x, acc: + loop(acc + x)) +assert s == 45 +``` + +### ``gtrampolined``: generators with TCO + +In ``unpythonic``, a generator can tail-chain into another generator. This is like invoking ``itertools.chain``, but as a tail call from inside the generator - so the generator itself can choose the next iterable in the chain. If the next iterable is a generator, it can again tail-chain into something else. If it is not a generator, it becomes the last iterable in the TCO chain. + +Python provides a convenient hook to build things like this, in the guise of ``return``: + +```python +from unpythonic import gtco, take, last + +def march(): + yield 1 + yield 2 + return march() # tail-chain to a new instance of itself +assert tuple(take(6, gtco(march()))) == (1, 2, 1, 2, 1, 2) +last(take(10000, gtco(march()))) # no crash +``` + +Note the calls to ``gtco`` at the use sites. For convenience, we provide ``@gtrampolined``, which automates that: + +```python +from unpythonic import gtrampolined, take, last + +@gtrampolined +def ones(): + yield 1 + return ones() +assert tuple(take(10, ones())) == (1,) * 10 +last(take(10000, ones())) # no crash +``` + +It is safe to tail-chain into a ``@gtrampolined`` generator; the system strips the TCO target's trampoline if it has one. + +Like all tail calls, this works for any *iterative* process. In contrast, this **does not work**: + +```python +from operator import add +from unpythonic import gtrampolined, scanl, take + +@gtrampolined +def fibos(): # see numerics.py + yield 1 + return scanl(add, 1, fibos()) +print(tuple(take(10, fibos()))) # --> (1, 1, 2), only 3 terms?! +``` + +This sequence (technically iterable, but in the mathematical sense) is recursively defined, and the ``return`` shuts down the generator before it can yield more terms into ``scanl``. With ``yield from`` instead of ``return`` the second example works (but since it is recursive, it eventually blows the call stack). + +This particular example can be converted into a linear process with a different higher-order function, no TCO needed: + +```python +from unpythonic import unfold, take, last +def fibos(): + def nextfibo(a, b): + return a, b, a + b # value, *newstates + return unfold(nextfibo, 1, 1) +assert tuple(take(10, fibos())) == (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) +last(take(10000, fibos())) # no crash +``` + + +### ``setescape``, ``escape``: escape continuations (ec) + +Escape continuations can be used as a *multi-return*: + +```python +from unpythonic import setescape, escape + +@setescape() # note the parentheses +def f(): + def g(): + escape("hello from g") # the argument becomes the return value of f() + print("not reached") + g() + print("not reached either") +assert f() == "hello from g" +``` + +**CAUTION**: The implementation is based on exceptions, so catch-all ``except:`` statements will intercept also escapes, breaking the escape mechanism. As you already know, be specific in what you catch! + +In Lisp terms, `@setescape` essentially captures the escape continuation (ec) of the function decorated with it. The nearest (dynamically) surrounding ec can then be invoked by `escape(value)`. The function decorated with `@setescape` immediately terminates, returning ``value``. + +In Python terms, an escape means just raising a specific type of exception; the usual rules concerning ``try/except/else/finally`` and ``with`` blocks apply. It is a function call, so it works also in lambdas. + +Escaping the function surrounding an FP loop, from inside the loop: + +```python +@setescape() +def f(): + @looped + def s(loop, acc=0, i=0): + if i > 5: + escape(acc) + return loop(acc + i, i + 1) + print("never reached") +f() # --> 15 +``` + +For more control, both ``@setescape`` points and escape instances can be tagged: + +```python +@setescape(tags="foo") # setescape point tags can be single value or tuple (tuples OR'd, like isinstance()) +def foo(): + @call + @setescape(tags="bar") + def bar(): + @looped + def s(loop, acc=0, i=0): + if i > 5: + escape(acc, tag="foo") # escape instance tag must be a single value + return loop(acc + i, i + 1) + print("never reached") + return False + print("never reached either") + return False +assert foo() == 15 +``` + +For details on tagging, especially how untagged and tagged escapes and points interact, and how to make one-to-one connections, see the docstring for ``@setescape``. + + +#### ``call_ec``: first-class escape continuations + +We provide ``call/ec`` (a.k.a. ``call-with-escape-continuation``), in Python spelled as ``call_ec``. It's a decorator that, like ``@call``, immediately runs the function and replaces the def'd name with the return value. The twist is that it internally sets up an escape point, and hands a **first-class escape continuation** to the callee. + +The function to be decorated **must** take one positional argument, the ec instance. + +The ec instance itself is another function, which takes one positional argument: the value to send to the escape point. The ec instance and the escape point are connected one-to-one. No other ``@setescape`` point will catch the ec instance, and the escape point catches only this particular ec instance and nothing else. + +Any particular ec instance is only valid inside the dynamic extent of the ``call_ec`` invocation that created it. Attempting to call the ec later raises ``RuntimeError``. + +This builds on ``@setescape`` and ``escape``, so the caution about catch-all ``except:`` statements applies here, too. + +```python +from unpythonic import call_ec + +@call_ec +def result(ec): # effectively, just a code block, capturing the ec as an argument + answer = 42 + ec(answer) # here this has the same effect as "return answer"... + print("never reached") + answer = 23 + return answer +assert result == 42 + +@call_ec +def result(ec): + answer = 42 + def inner(): + ec(answer) # ...but here this directly escapes from the outer def + print("never reached") + return 23 + answer = inner() + print("never reached either") + return answer +assert result == 42 +``` + +The ec doesn't have to be called from the lexical scope of the call_ec'd function, as long as the call occurs within the dynamic extent of the ``call_ec``. It's essentially a *return from me* for the original function: + +```python +def f(ec): + print("hi from f") + ec(42) + +@call_ec +def result(ec): + f(ec) # pass on the ec - it's a first-class value + print("never reached") +assert result == 42 +``` + +This also works with lambdas, by using ``call_ec()`` directly. No need for a trampoline: + +```python +result = call_ec(lambda ec: + begin(print("hi from lambda"), + ec(42), + print("never reached"))) +assert result == 42 +``` + +Normally ``begin()`` would return the last value, but the ec overrides that; it is effectively a ``return`` for multi-expression lambdas! + +But wait, doesn't Python evaluate all the arguments of `begin(...)` before the `begin` itself has a chance to run? Why doesn't the example print also *never reached*? This is because escapes are implemented using exceptions. Evaluating the ec call raises an exception, preventing any further elements from being evaluated. + +This usage is valid with named functions, too - ``call_ec`` is not only a decorator: + +```python +def f(ec): + print("hi from f") + ec(42) + print("never reached") + +# ...possibly somewhere else, possibly much later... + +result = call_ec(f) +assert result == 42 +``` + + +### ``forall``: nondeterministic evaluation + +We provide a simple variant of nondeterministic evaluation. This is essentially a toy that has no more power than list comprehensions or nested for loops. See also the easy-to-use [macro](macro_extras/) version with natural syntax and a clean implementation. + +An important feature of McCarthy's [`amb` operator](https://rosettacode.org/wiki/Amb) is its nonlocality - being able to jump back to a choice point, even after the dynamic extent of the function where that choice point resides. If that sounds a lot like ``call/cc``, that's because that's how ``amb`` is usually implemented. See examples [in Ruby](http://www.randomhacks.net/2005/10/11/amb-operator/) and [in Racket](http://www.cs.toronto.edu/~david/courses/csc324_w15/extra/choice.html). + +Python can't do that, short of transforming the whole program into [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style), while applying TCO everywhere to prevent stack overflow. **If that's what you want**, see ``continuations`` in [the macros](macro_extras/). + +This ``forall`` is essentially a tuple comprehension that: + + - Can have multiple body expressions (side effects also welcome!), by simply listing them in sequence. + - Allows filters to be placed at any level of the nested looping. + - Presents the source code in the same order as it actually runs. + +The ``unpythonic.amb`` module defines four operators: + + - ``forall`` is the control structure, which marks a section with nondeterministic evaluation. + - ``choice`` binds a name: ``choice(x=range(3))`` essentially means ``for e.x in range(3):``. + - ``insist`` is a filter, which allows the remaining lines to run if the condition evaluates to truthy. + - ``deny`` is ``insist not``; it allows the remaining lines to run if the condition evaluates to falsey. + +Choice variables live in the environment, which is accessed via a ``lambda e: ...``, just like in ``letrec``. Lexical scoping is emulated. In the environment, each line only sees variables defined above it; trying to access a variable defined later raises ``AttributeError``. + +The last line in a ``forall`` describes one item of the output. The output items are collected into a tuple, which becomes the return value of the ``forall`` expression. + +```python +out = forall(choice(y=range(3)), + choice(x=range(3)), + lambda e: insist(e.x % 2 == 0), + lambda e: (e.x, e.y)) +assert out == ((0, 0), (2, 0), (0, 1), (2, 1), (0, 2), (2, 2)) + +out = forall(choice(y=range(3)), + choice(x=range(3)), + lambda e: deny(e.x % 2 == 0), + lambda e: (e.x, e.y)) +assert out == ((1, 0), (1, 1), (1, 2)) +``` + +Pythagorean triples: + +```python +pt = forall(choice(z=range(1, 21)), # hypotenuse + choice(x=lambda e: range(1, e.z+1)), # shorter leg + choice(y=lambda e: range(e.x, e.z+1)), # longer leg + lambda e: insist(e.x*e.x + e.y*e.y == e.z*e.z), + lambda e: (e.x, e.y, e.z)) +assert tuple(sorted(pt)) == ((3, 4, 5), (5, 12, 13), (6, 8, 10), + (8, 15, 17), (9, 12, 15), (12, 16, 20)) + +``` + +Beware: + +```python +out = forall(range(2), # do the rest twice! + choice(x=range(1, 4)), + lambda e: e.x) +assert out == (1, 2, 3, 1, 2, 3) +``` + +The initial ``range(2)`` causes the remaining lines to run twice - because it yields two output values - regardless of whether we bind the result to a variable or not. In effect, each line, if it returns more than one output, introduces a new nested loop at that point. + +For more, see the docstring of ``forall``. + +#### For haskellers + +The implementation is based on the List monad, and a bastardized variant of do-notation. Quick vocabulary: + + - ``forall(...)`` = ``do ...`` (for a List monad) + - ``choice(x=foo)`` = ``x <- foo``, where ``foo`` is an iterable + - ``insist x`` = ``guard x`` + - ``deny x`` = ``guard (not x)`` + - Last line = implicit ``return ...`` + + +## Other + +Stuff that didn't fit elsewhere. + +### ``def`` as a code block: ``@call`` + +Fuel for different thinking. Compare `call-with-something` in Lisps - but without parameters, so just `call`. A `def` is really just a new lexical scope to hold code to run later... or right now! + +At the top level of a module, this is seldom useful, but keep in mind that Python allows nested function definitions. Used with an inner ``def``, this becomes a versatile tool. + +*Make temporaries fall out of scope as soon as no longer needed*: + +```python +from unpythonic import call + +@call +def x(): + a = 2 # many temporaries that help readability... + b = 3 # ...of this calculation, but would just pollute locals... + c = 5 # ...after the block exits + return a * b * c +print(x) # 30 +``` + +*Multi-break out of nested loops* - `continue`, `break` and `return` are really just second-class [ec](https://docs.racket-lang.org/reference/cont.html#%28def._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._call%2Fec%29%29)s. So `def` to make `return` escape to exactly where you want: + +```python +@call +def result(): + for x in range(10): + for y in range(10): + if x * y == 42: + return (x, y) +print(result) # (6, 7) +``` + +(But see ``@setescape``, ``escape``, and ``call_ec``.) + +Compare the sweet-exp Racket: + +```racket +define result + let/ec return ; name the (first-class) ec to break out of this let/ec block + for ([x in-range(10)]) + for ([y in-range(10)]) + cond + {{x * y} = 42} + return (list x y) +displayln result ; (6 7) +``` + +Noting [what ``let/ec`` does](https://docs.racket-lang.org/reference/cont.html#%28form._%28%28lib._racket%2Fprivate%2Fletstx-scheme..rkt%29._let%2Fec%29%29), using ``call_ec`` we can make the Python even closer to the Racket: + +```python +@call_ec +def result(rtn): + for x in range(10): + for y in range(10): + if x * y == 42: + rtn((x, y)) +print(result) # (6, 7) +``` + +*Twist the meaning of `def` into a "let statement"*: + +```python +@call +def result(x=1, y=2, z=3): + return x * y * z +print(result) # 6 +``` + +(But see `blet`, `bletrec` if you want an `env` instance.) + +*Letrec without `letrec`*, when it doesn't have to be an expression: + +```python +@call +def t(): + def evenp(x): return x == 0 or oddp(x - 1) + def oddp(x): return x != 0 and evenp(x - 1) + return evenp(42) +print(t) # True +``` + +Essentially the implementation is just `def call(thunk): return thunk()`. The point is to: + + - Make it explicit right at the definition site that this block is *going to be called now* (in contrast to an explicit call and assignment *after* the definition). Centralize the related information. Align the presentation order with the thought process. + + - Help eliminate errors, in the same way as the habit of typing parentheses only in pairs. No risk of forgetting to call the block after writing the definition. + + - Document that the block is going to be used only once. Tell the reader there's no need to remember this definition. + +Note [the grammar](https://docs.python.org/3/reference/grammar.html) requires a newline after a decorator. + +**NOTE**: ``call`` can also be used as a normal function: ``call(f, *a, **kw)`` is the same as ``f(*a, **kw)``. This is occasionally useful. + + +### ``@callwith``: freeze arguments, choose function later + +*Added in v0.11.0.* If you need to pass arguments when using ``@call`` as a decorator, use its cousin ``@callwith``: + +```python +from unpythonic import callwith + +@callwith(3) +def result(x): + return x**2 +assert result == 9 +``` + +Like ``call``, it can also be called normally. It's essentially an argument freezer: + +```python +def myadd(a, b): + return a + b +def mymul(a, b): + return a * b +apply23 = callwith(2, 3) +assert apply23(myadd) == 5 +assert apply23(mymul) == 6 +``` + +When called normally, the two-step application is mandatory. The first step stores the given arguments. It returns a function ``f(callable)``. When ``f`` is called, it calls its ``callable`` argument, passing in the arguments stored in the first step. + +In other words, ``callwith`` is similar to ``functools.partial``, but without specializing to any particular function. The function to be called is given later, in the second step. + +Hence, ``callwith(2, 3)(myadd)`` means "make a function that passes in two positional arguments, with values ``2`` and ``3``. Then call this function for the callable ``myadd``". But if we instead write``callwith(2, 3, myadd)``, it means "make a function that passes in three positional arguments, with values ``2``, ``3`` and ``myadd`` - not what we want in the above example. + +If you want to specialize some arguments now and some later, combine with ``partial``: + +```python +from functools import partial + +p1 = partial(callwith, 2) +p2 = partial(p1, 3) +p3 = partial(p2, 4) +apply234 = p3() # actually call callwith, get the function +def add3(a, b, c): + return a + b + c +def mul3(a, b, c): + return a * b * c +assert apply234(add3) == 9 +assert apply234(mul3) == 24 +``` + +If the code above feels weird, it should. Arguments are gathered first, and the function to which they will be passed is chosen in the last step. + +Another use case of ``callwith`` is ``map``, if we want to vary the function instead of the data: + +```python +m = map(callwith(3), [lambda x: 2*x, lambda x: x**2, lambda x: x**(1/2)]) +assert tuple(m) == (6, 9, 3**(1/2)) +``` + +If you have MacroPy, this combines nicely with ``quick_lambda``: + +```python +from macropy.quick_lambda import macros, f, _ +from unpythonic import callwith + +m = map(callwith(3), [f[2 * _], f[_**2], f[_**(1/2)]]) +assert tuple(m) == (6, 9, 3**(1/2)) +``` + +A pythonic alternative to the above examples is: + +```python +a = [2, 3] +def myadd(a, b): + return a + b +def mymul(a, b): + return a * b +assert myadd(*a) == 5 +assert mymul(*a) == 6 + +a = [2] +a += [3] +a += [4] +def add3(a, b, c): + return a + b + c +def mul3(a, b, c): + return a * b * c +assert add3(*a) == 9 +assert mul3(*a) == 24 + +m = (f(3) for f in [lambda x: 2*x, lambda x: x**2, lambda x: x**(1/2)]) +assert tuple(m) == (6, 9, 3**(1/2)) +``` + +Inspired by *Function application with $* in [LYAH: Higher Order Functions](http://learnyouahaskell.com/higher-order-functions). + + +### ``raisef``: ``raise`` as a function + +Raise an exception from an expression position: + +```python +from unpythonic import raisef + +f = lambda x: raisef(RuntimeError, "I'm in ur lambda raising exceptions") +``` + + +### ``pack``: multi-arg constructor for tuple + +The default ``tuple`` constructor accepts a single iterable. But sometimes one needs to pass in the elements separately. Most often a literal tuple such as ``(1, 2, 3)`` is then the right solution, but there are situations that do not admit a literal tuple. Enter ``pack``: + +```python +from unpythonic import pack + +myzip = lambda lol: map(pack, *lol) +lol = ((1, 2), (3, 4), (5, 6)) +assert tuple(myzip(lol)) == ((1, 3, 5), (2, 4, 6)) +``` + + +### ``namelambda``, rename a function + +*Changed in v0.13.0.* Now supports renaming any function object (``isinstance(f, (types.LambdaType, types.FunctionType))``), and will rename a lambda even if it has already been named. + +*Changed in v0.13.1.* Now the return value is a modified copy; the original function object is not mutated. + +For those situations where you return a lambda as a closure, call it much later, and it happens to crash - so you can tell from the stack trace *which* of the *N* lambdas in the codebase it is. + +For technical reasons, ``namelambda`` conforms to the parametric decorator API. Usage: + +```python +from unpythonic import namelambda + +square = namelambda("square")(lambda x: x**2) +assert square.__name__ == "square" + +kaboom = namelambda("kaboom")(lambda: some_typoed_name) +kaboom() # --> stack trace, showing the function name "kaboom" +``` + +The first call returns a *foo-renamer*, which takes a function object and returns a copy that has its name changed to *foo*. + +Technically, this updates ``__name__`` (the obvious place), ``__qualname__`` (used by ``repr()``), and ``__code__.co_name`` (used by stack traces). + +**CAUTION**: There is one pitfall: + +```python +from unpythonic import namelambda, withself + +nested = namelambda("outer")(lambda: namelambda("inner")(withself(lambda self: self))) +print(nested.__qualname__) # "outer" +print(nested().__qualname__) # "..inner" +``` + +The inner lambda does not see the outer's new name; the parent scope names are baked into a function's ``__qualname__`` too early for the outer rename to be in effect at that time. + + +### ``timer``: a context manager for performance testing + +*Added in v0.13.0.* + +```python +from unpythonic import timer + +with timer() as tim: + for _ in range(int(1e6)): + pass +print(tim.dt) # elapsed time in seconds (float) + +with timer(p=True): # if p, auto-print result + for _ in range(int(1e6)): + pass +``` + +The auto-print mode is a convenience feature to minimize bureaucracy if you just want to see the *Δt*. To instead access the *Δt* programmatically, name the timer instance using the ``with ... as ...`` syntax. After the context exits, the *Δt* is available in its ``dt`` attribute. + + +### ``getattrrec``, ``setattrrec``: access underlying data in an onion of wrappers + +*Added in v0.13.1.* + +```python +from unpythonic import getattrrec, setattrrec + +class Wrapper: + def __init__(self, x): + self.x = x + +w = Wrapper(Wrapper(42)) +assert type(getattr(w, "x")) == Wrapper +assert type(getattrrec(w, "x")) == int +assert getattrrec(w, "x") == 42 + +setattrrec(w, "x", 23) +assert type(getattr(w, "x")) == Wrapper +assert type(getattrrec(w, "x")) == int +assert getattrrec(w, "x") == 23 +``` + + +### ``arities``, ``kwargs``: Function signature inspection utilities + +Convenience functions providing an easy-to-use API for inspecting a function's signature. The heavy lifting is done by ``inspect``. + +Methods on objects and classes are treated specially, so that the reported arity matches what the programmer actually needs to supply when calling the method (i.e., implicit ``self`` and ``cls`` are ignored). + +```python +from unpythonic import arities, arity_includes, UnknownArity, \ + kwargs, required_kwargs, optional_kwargs, + +f = lambda a, b: None +assert arities(f) == (2, 2) # min, max positional arity + +f = lambda a, b=23: None +assert arities(f) == (1, 2) +assert arity_includes(f, 2) is True +assert arity_includes(f, 3) is False + +f = lambda a, *args: None +assert arities(f) == (1, float("+inf")) + +f = lambda *, a, b, c=42: None +assert arities(f) == (0, 0) +assert required_kwargs(f) == set(('a', 'b')) +assert optional_kwargs(f) == set(('c')) +assert kwargs(f) == (set(('a', 'b')), set(('c'))) + +class A: + def __init__(self): + pass + def meth(self, x): + pass + @classmethod + def classmeth(cls, x): + pass + @staticmethod + def staticmeth(x): + pass +assert arities(A) == (0, 0) # constructor of "A" takes no args beside the implicit self +# methods on the class +assert arities(A.meth) == (2, 2) +assert arities(A.classmeth) == (1, 1) +assert arities(A.staticmeth) == (1, 1) +# methods on an instance +a = A() +assert arities(a.meth) == (1, 1) # self is implicit, so just one +assert arities(a.classmeth) == (1, 1) # cls is implicit +assert arities(a.staticmeth) == (1, 1) +``` + +We special-case the builtin functions that either fail to return any arity (are uninspectable) or report incorrect arity information, so that also their arities are reported correctly. Note we **do not** special-case the *methods* of any builtin classes, so e.g. ``list.append`` remains uninspectable. This limitation might or might not be lifted in a future version. + +If the arity cannot be inspected, and the function is not one of the special-cased builtins, the ``UnknownArity`` exception is raised. + +These functions are internally used in various places in unpythonic, particularly ``curry``. The ``let`` and FP looping constructs also use these to emit a meaningful error message if the signature of user-provided function does not match what is expected. + +Inspired by various Racket functions such as ``(arity-includes?)`` and ``(procedure-keywords)``. + + +### ``Popper``: a pop-while iterator + +*Added in v0.14.1.* + +Consider this highly artificial example: + +```python +from collections import deque + +inp = deque(range(5)) +out = [] +while inp: + x = inp.pop(0) + out.append(x) +assert inp == deque([]) +assert out == list(range(5)) +``` + +``Popper`` condenses the ``while`` and ``pop`` into a ``for``, while allowing the loop body to mutate the input iterable in arbitrary ways (we never actually ``iter()`` it): + +```python +from collections import deque +from unpythonic import Popper + +inp = deque(range(5)) +out = [] +for x in Popper(inp): + out.append(x) +assert inp == deque([]) +assert out == list(range(5)) + +inp = deque(range(3)) +out = [] +for x in Popper(inp): + out.append(x) + if x < 10: + inp.appendleft(x + 10) +assert inp == deque([]) +assert out == [0, 10, 1, 11, 2, 12] +``` + +``Popper`` comboes with other iterable utilities, such as ``window``: + +```python +from collections import deque +from unpythonic import Popper, window + +inp = deque(range(3)) +out = [] +for a, b in window(Popper(inp)): + out.append((a, b)) + if a < 10: + inp.append(a + 10) +assert inp == deque([]) +assert out == [(0, 1), (1, 2), (2, 10), (10, 11), (11, 12)] +``` + +(Although ``window`` invokes ``iter()`` on the ``Popper``, this works because the ``Popper`` never invokes ``iter()`` on the underlying container. Any mutations to the input container performed by the loop body will be understood by ``Popper`` and thus also seen by the ``window``. The first ``n`` elements, though, are read before the loop body gets control, because the window needs them to initialize itself.) + +One possible real use case for ``Popper`` is to split sequences of items, stored as lists in a deque, into shorter sequences where some condition is contiguously ``True`` or ``False``. When the condition changes state, just commit the current subsequence, and push the rest of that input sequence (still requiring analysis) back to the input deque, to be dealt with later. + +The argument to ``Popper`` (here ``lst``) contains the **remaining** items. Each iteration pops an element **from the left**. The loop terminates when ``lst`` is empty. + +The input container must support either ``popleft()`` or ``pop(0)``. This is fully duck-typed. At least ``collections.deque`` and any ``collections.abc.MutableSequence`` (including ``list``) are fine. + +Per-iteration efficiency is O(1) for ``collections.deque``, and O(n) for a ``list``. + +Named after [Karl Popper](https://en.wikipedia.org/wiki/Karl_Popper). From 9538ddd0bacb42d87b4fea378922034304cc6953 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Mon, 26 Aug 2019 14:31:16 -0700 Subject: [PATCH 02/31] Update README.md Fixed formatting. --- README.md | 26 +++++++++++++------------- doc/design-notes.md | 14 ++++++-------- 2 files changed, 19 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index 01c4421a..7192b4fa 100644 --- a/README.md +++ b/README.md @@ -15,48 +15,48 @@ Currently none required; [MacroPy](https://github.com/azazel75/macropy) optional - [Basic: pure-Python](doc/features.md) - [Advanced: syntactic macros](macro_extras/): the second half of ``unpythonic``. -### Installation +## Installation #### PyPI -`pip3 install unpythonic --user` +``pip3 install unpythonic --user`` or -`sudo pip3 install unpythonic` +``sudo pip3 install unpythonic`` #### GitHub Clone (or pull) from GitHub. Then, -`python3 setup.py install --user` +``python3 setup.py install --user`` or -`sudo python3 setup.py install` +``sudo python3 setup.py install`` #### Uninstall -Uninstallation must be invoked in a folder which has no subfolder called `unpythonic`, so that `pip` recognizes it as a package name (instead of a filename). Then, +Uninstallation must be invoked in a folder which has no subfolder called ``unpythonic``, so that ``pip`` recognizes it as a package name (instead of a filename). Then, -`pip3 uninstall unpythonic` +``pip3 uninstall unpythonic`` or -`sudo pip3 uninstall unpythonic` +``sudo pip3 uninstall unpythonic`` -### License +## License 2-clause [BSD](LICENSE.md). Dynamic assignment based on [StackOverflow answer by Jason Orendorff (2010)](https://stackoverflow.com/questions/2001138/how-to-create-dynamical-scoped-variables-in-python), used under CC-BY-SA. The threading support is original to our version. -Core idea of `lispylet` based on [StackOverflow answer by divs1210 (2017)](https://stackoverflow.com/a/44737147), used under the MIT license. +Core idea of ``lispylet`` based on [StackOverflow answer by divs1210 (2017)](https://stackoverflow.com/a/44737147), used under the MIT license. -Core idea of `view` based on [StackOverflow answer by Mathieu Caroff (2018)](https://stackoverflow.com/a/53253136), used under the MIT license. Our additions include support for sequences with changing length, write support, iteration based on `__iter__`, in-place reverse, and the abstract base classes. +Core idea of ``view`` based on [StackOverflow answer by Mathieu Caroff (2018)](https://stackoverflow.com/a/53253136), used under the MIT license. Our additions include support for sequences with changing length, write support, iteration based on ``__iter__``, in-place reverse, and the abstract base classes. -### Acknowledgements +## Acknowledgements Thanks to [TUT](http://www.tut.fi/en/home) for letting me teach [RAK-19006 in spring term 2018](https://github.com/Technologicat/python-3-scicomp-intro); early versions of parts of this library were originally developed as teaching examples for that course. Thanks to @AgenttiX for feedback. @@ -65,7 +65,7 @@ The trampoline implementation of ``unpythonic.tco`` takes its remarkably clean a Another important source of inspiration was [tco](https://github.com/baruchel/tco) by Thomas Baruchel, for thinking about the possibilities of TCO in Python. -### Python-related FP resources +## Python-related FP resources Python clearly wants to be an impure-FP language. A decorator with arguments *is a curried closure* - how much more FP can you get? diff --git a/doc/design-notes.md b/doc/design-notes.md index 0608ffd6..fc17f22e 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -1,7 +1,5 @@ # Design Notes -## Contents - - [On ``let`` and Python](#on-let-and-python) - [Python is Not a Lisp](#python-is-not-a-lisp) - [Assignment Syntax](#assignment-syntax) @@ -9,7 +7,7 @@ - [No Monads?](#wait-no-monads) - [Further Explanation](#your-hovercraft-is-full-of-eels) -## On ``let`` and Python +### On ``let`` and Python Why no `let*`, as a function? In Python, name lookup always occurs at runtime. Python gives us no compile-time guarantees that no binding refers to a later one - in [Racket](http://racket-lang.org/), this guarantee is the main difference between `let*` and `letrec`. @@ -25,7 +23,7 @@ The [macro versions](macro_extras/) of the `let` constructs **are** lexically sc Inspiration: [[1]](https://nvbn.github.io/2014/09/25/let-statement-in-python/) [[2]](https://stackoverflow.com/questions/12219465/is-there-a-python-equivalent-of-the-haskell-let) [[3]](http://sigusr2.net/more-about-let-in-python.html). -## Python is not a Lisp +### Python is not a Lisp The point behind providing `let` and `begin` (and the ``let[]`` and ``do[]`` [macros](macro_extras/)) is to make Python lambdas slightly more useful - which was really the starting point for this whole experiment. @@ -38,7 +36,7 @@ The oft-quoted single-expression limitation of the Python ``lambda`` is ultimate Still, ultimately one must keep in mind that Python is not a Lisp. Not all of Python's standard library is expression-friendly; some standard functions and methods lack return values - even though a call is an expression! For example, `set.add(x)` returns `None`, whereas in an expression context, returning `x` would be much more useful, even though it does have a side effect. -## Assignment syntax +### Assignment syntax Why the clunky `e.set("foo", newval)` or `e << ("foo", newval)`, which do not directly mention `e.foo`? This is mainly because in Python, the language itself is not customizable. If we could define a new operator `e.foo newval` to transform to `e.set("foo", newval)`, this would be easily solved. @@ -55,7 +53,7 @@ If we later choose go this route nevertheless, `<<` is a better choice for the s The current solution for the assignment syntax issue is to use macros, to have both clean syntax at the use site and a relatively hackfree implementation. -## TCO syntax and speed +### TCO syntax and speed Benefits and costs of ``return jump(...)``: @@ -79,7 +77,7 @@ For other libraries bringing TCO to Python, see: - ``recur.tco`` in [fn.py](https://github.com/fnpy/fn.py), the original source of the approach used here. - [MacroPy](https://github.com/azazel75/macropy) uses an approach similar to ``fn.py``. -## Wait, no monads? +### Wait, no monads? (Beside List inside ``forall``.) @@ -87,7 +85,7 @@ Admittedly unpythonic, but Haskell feature, not Lisp. Besides, already done else If you want to roll your own monads for whatever reason, there's [this silly hack](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/monads.py) that wasn't packaged into this; or just read Stephan Boyer's quick introduction [[part 1]](https://www.stephanboyer.com/post/9/monads-part-1-a-design-pattern) [[part 2]](https://www.stephanboyer.com/post/10/monads-part-2-impure-computations) [[super quick intro]](https://www.stephanboyer.com/post/83/super-quick-intro-to-monads) and figure it out, it's easy. (Until you get to `State` and `Reader`, where [this](http://brandon.si/code/the-state-monad-a-tutorial-for-the-confused/) and maybe [this](https://gaiustech.wordpress.com/2010/09/06/on-monads/) can be helpful.) -## Your hovercraft is full of eels! +### Your hovercraft is full of eels! [Naturally](http://stupidpythonideas.blogspot.com/2015/05/spam-spam-spam-gouda-spam-and-tulips.html), they come with the territory. From d1d9dda8bdf2a85230c9bc2aca44f5f7a0277dc1 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 11:06:37 -0700 Subject: [PATCH 03/31] Update design-notes.md --- doc/design-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index fc17f22e..89887541 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -19,7 +19,7 @@ Our `letrec` behaves like `let*` in that if `valexpr` is not a function, it may Note the function versions of our `let` constructs, presented here, are **not** properly lexically scoped; in case of nested ``let`` expressions, one must be explicit about which environment the names come from. -The [macro versions](macro_extras/) of the `let` constructs **are** lexically scoped. The macros also provide a ``letseq[]`` that, similarly to Racket's ``let*``, gives a compile-time guarantee that no binding refers to a later one. +The [macro versions](../macro_extras/) of the `let` constructs **are** lexically scoped. The macros also provide a ``letseq[]`` that, similarly to Racket's ``let*``, gives a compile-time guarantee that no binding refers to a later one. Inspiration: [[1]](https://nvbn.github.io/2014/09/25/let-statement-in-python/) [[2]](https://stackoverflow.com/questions/12219465/is-there-a-python-equivalent-of-the-haskell-let) [[3]](http://sigusr2.net/more-about-let-in-python.html). From 0e607291c67ff63b78dcb5593eec7974ab067d5d Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 11:29:31 -0700 Subject: [PATCH 04/31] Update README.md --- macro_extras/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macro_extras/README.md b/macro_extras/README.md index 8004c4a6..74fe19ca 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -450,7 +450,7 @@ Macros that run multiple expressions, in sequence, in place of one expression. We provide an ``expr`` macro wrapper for ``unpythonic.seq.do``, with some extra features. -This essentially allows writing imperative code in any expression position. For an `if-elif-else` conditional, see `cond`; for loops, see the functions in `unpythonic.fploop` (esp. `looped`). +This essentially allows writing imperative code in any expression position. For an `if-elif-else` conditional, see `cond`; for loops, see the functions in [`unpythonic.fploop`](../unpythonic/fploop.py) (esp. `looped`). ```python from unpythonic.syntax import macros, do, local, delete From 3721914a2a5c1afb11182985713cfaa8b4abc1b0 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 11:40:04 -0700 Subject: [PATCH 05/31] Update design-notes.md --- doc/design-notes.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/doc/design-notes.md b/doc/design-notes.md index 89887541..834de794 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -1,5 +1,16 @@ # Design Notes +# A collapsible section with markdown +
+ Click to expand! + + ## Heading + 1. A numbered + 2. list + * With some + * Sub bullets +
+ - [On ``let`` and Python](#on-let-and-python) - [Python is Not a Lisp](#python-is-not-a-lisp) - [Assignment Syntax](#assignment-syntax) From f0b3a9002f2be24e2978873b9203843d62539c3f Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 11:40:37 -0700 Subject: [PATCH 06/31] Update design-notes.md --- doc/design-notes.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 834de794..89887541 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -1,16 +1,5 @@ # Design Notes -# A collapsible section with markdown -
- Click to expand! - - ## Heading - 1. A numbered - 2. list - * With some - * Sub bullets -
- - [On ``let`` and Python](#on-let-and-python) - [Python is Not a Lisp](#python-is-not-a-lisp) - [Assignment Syntax](#assignment-syntax) From 02e69c17685d5b0ddf19ff56261377174ea89a80 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 11:41:34 -0700 Subject: [PATCH 07/31] Update README.md --- macro_extras/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/macro_extras/README.md b/macro_extras/README.md index 74fe19ca..18acb19b 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -74,7 +74,8 @@ Macros that introduce new ways to bind identifiers. ### ``let``, ``letseq``, ``letrec`` as macros -Properly lexically scoped ``let`` constructs, no boilerplate: +
+ Properly lexically scoped ``let`` constructs, no boilerplate: ```python from unpythonic.syntax import macros, let, letseq, letrec @@ -91,6 +92,7 @@ letrec((evenp, lambda x: (x == 0) or oddp(x - 1)), # mutually recursive binding (oddp, lambda x: (x != 0) and evenp(x - 1)))[ print(evenp(42))] ``` +
As seen in the examples, the syntax is similar to ``unpythonic.lispylet``. Assignment to variables in the environment is supported via the left-shift syntax ``x << 42``. From 14270536ed20d9c5b5ec8f5e5aad408638ecf23f Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 11:51:45 -0700 Subject: [PATCH 08/31] Update README.md --- macro_extras/README.md | 131 +++++++++++++++++++++-------------------- 1 file changed, 68 insertions(+), 63 deletions(-) diff --git a/macro_extras/README.md b/macro_extras/README.md index 18acb19b..9e7303b4 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -1,72 +1,73 @@ -# ``unpythonic.syntax``: Language extensions +# Language extensions using ``unpythonic.syntax`` -These optional features, providing extensions to the Python language as syntactic macros, are built on [MacroPy](https://github.com/azazel75/macropy), from PyPI package ``macropy3``. -If you want to take language extension a step further, see the sister project [Pydialect](https://github.com/Technologicat/pydialect). +These optional features -- providing extensions to the Python language as syntactic macros -- are built on [MacroPy](https://github.com/azazel75/macropy), from the PyPI package ``macropy3``. If you want to take language extension a step further, see our sister project [Pydialect](https://github.com/Technologicat/pydialect). -Because macro expansion occurs at import time, the unit tests that contain usage examples (located in [unpythonic/syntax/test/](../unpythonic/syntax/test/)) cannot be run directly. Instead, run them via the included [generic MacroPy3 bootstrapper](macropy3). Usage of the bootstrapper is `./macropy3 -m some.module` (like `python3 -m some.module`); see `-h` for options. +Macro expansion occurs at import time, so the unit tests that contain usage examples (located in [unpythonic/syntax/test/](../unpythonic/syntax/test/)) cannot be run directly. Instead, run them via the included [generic MacroPy3 bootstrapper](macropy3). For convenience, ``setup.py`` installs the `macropy3` bootstrapper. -The tests use relative imports; invoke them from the top-level directory of ``unpythonic`` as e.g. ``macro_extras/macropy3 -m unpythonic.syntax.test.test_curry``. This is to make the tests run against the source tree without installing it first; in your own code, once you have installed ``unpythonic``, feel free to use absolute imports, like those shown in this README. +To use the bootstrapper, run: + + `./macropy3 -m some.module` (like `python3 -m some.module`); see `-h` for options. + +The tests use relative imports; invoke them from the top-level directory of ``unpythonic`` as e.g.: + + ``macro_extras/macropy3 -m unpythonic.syntax.test.test_curry`` + + This is to make the tests run against the source tree without installing it first; in your own code, once you have installed ``unpythonic``, feel free to use absolute imports, like those shown in this README. There is no abbreviation for ``memoize(lambda: ...)``, because ``MacroPy`` itself already provides ``lazy`` and ``interned``. !! **Currently** (12/2018) this requires the latest MacroPy from git HEAD. !! -Of the `python3` command-line options, the `macropy3` bootstrapper supports only `-m`. If you need to give other `python3` command-line options when running a MacroPy-enabled source file, run the bootstrapper manually via `python3 ...` and place the options there. - -*Changed in v0.14.0.* The `macropy3` bootstrapper now takes the `-m` option, like `python3 -m mod`. The alternative is to specify a filename positionally, like ``python3 mod.py``. In either case, the bootstrapper will import the module in a special mode that pretends its `__name__ == '__main__'`, to allow using the pythonic conditional main idiom also in macro-enabled code. For convenience, ``setup.py`` now installs the `macropy3` bootstrapper. - -**Contents**: - - - [**Bindings**](#bindings) - - [``let``, ``letseq``, ``letrec`` as macros](#let-letseq-letrec-as-macros); proper lexical scoping, no boilerplate. - - [``dlet``, ``dletseq``, ``dletrec``, ``blet``, ``bletseq``, ``bletrec``: decorator versions](#dlet-dletseq-dletrec-blet-bletseq-bletrec-decorator-versions) - - [``let_syntax``, ``abbrev``: syntactic local bindings](#let_syntax-abbrev-syntactic-local-bindings); splice code at macro expansion time. - - [Bonus: barebones ``let``](#bonus-barebones-let): pure AST transformation of ``let`` into a ``lambda``. - - - [**Sequencing**](#sequencing) - - [``do`` as a macro: stuff imperative code into an expression, *with style*](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style) - - - [**Tools for lambdas**](#tools-for-lambdas) - - [``multilambda``: supercharge your lambdas](#multilambda-supercharge-your-lambdas); multiple expressions, local variables. - - [``namedlambda``: auto-name your lambdas](#namedlambda-auto-name-your-lambdas) by assignment. - - [``quicklambda``: combo with ``macropy.quick_lambda``](#quicklambda-combo-with-macropyquick_lambda) - - [``envify``: make formal parameters live in an unpythonic ``env``](#envify-make-formal-parameters-live-in-an-unpythonic-env) - - - [**Language features**](#language-features) - - [``curry``: automatic currying for Python](#curry-automatic-currying-for-python) - - [``lazify``: call-by-need for Python](#lazify-call-by-need-for-python) - - [Forcing promises manually](#forcing-promises-manually) - - [Binding constructs and auto-lazification](#binding-constructs-and-auto-lazification) - - [Note about TCO](#note-about-tco) - - [``tco``: automatic tail call optimization for Python](#tco-automatic-tail-call-optimization-for-python) - - [TCO and continuations](#tco-and-continuations) - - [``continuations``: call/cc for Python](#continuations-callcc-for-python) - - [Differences between ``call/cc`` and certain other language features](#differences-between-callcc-and-certain-other-language-features) (generators, exceptions) - - [``call_cc`` API reference](#call_cc-api-reference) - - [Combo notes](#combo-notes) - - [Continuations as an escape mechanism](#continuations-as-an-escape-mechanism) - - [What can be used as a continuation?](#what-can-be-used-as-a-continuation) - - [This isn't ``call/cc``!](#this-isnt-callcc) - - [Why this syntax?](#why-this-syntax) - - [``prefix``: prefix function call syntax for Python](#prefix-prefix-function-call-syntax-for-python) - - [``autoreturn``: implicit ``return`` in tail position](#autoreturn-implicit-return-in-tail-position), like in Lisps. - - [``forall``: nondeterministic evaluation](#forall-nondeterministic-evaluation) with monadic do-notation for Python. - - - [**Convenience features**](#convenience-features) - - [``cond``: the missing ``elif`` for ``a if p else b``](#cond-the-missing-elif-for-a-if-p-else-b) - - [``aif``: anaphoric if](#aif-anaphoric-if), the test result is ``it``. - - [``autoref``: implicitly reference attributes of an object](#autoref-implicitly-reference-attributes-of-an-object) - - [``dbg``: debug-print expressions with source code](#dbg-debug-print-expressions-with-source-code) - - *Changed in v0.13.1.* The ``fup[]`` macro is gone, and has been replaced with the ``fup`` function, with slightly changed syntax to accommodate. - - - [**Other**](#other) - - [``nb``: silly ultralight math notebook](#nb-silly-ultralight-math-notebook) - - - [**Meta**](#meta) - - [Comboability](#comboability) - - [The xmas tree combo](#the-xmas-tree-combo): notes on the macros working together. - - [This is semantics, not syntax!](#this-is-semantics-not-syntax) - +The `macropy3` bootstrapper takes the `-m` option, like `python3 -m mod`. The alternative is to specify a filename positionally, like ``python3 mod.py``. In either case, the bootstrapper will import the module in a special mode that pretends its `__name__ == '__main__'`, to allow using the pythonic conditional main idiom also in macro-enabled code. + +**Note: This document is up-to-date for version 0.14.1.** + +[**Bindings**](#bindings) + - [``let``, ``letseq``, ``letrec`` as macros](#let-letseq-letrec-as-macros); proper lexical scoping, no boilerplate. + - [``dlet``, ``dletseq``, ``dletrec``, ``blet``, ``bletseq``, ``bletrec``: decorator versions](#dlet-dletseq-dletrec-blet-bletseq-bletrec-decorator-versions) + - [``let_syntax``, ``abbrev``: syntactic local bindings](#let_syntax-abbrev-syntactic-local-bindings); splice code at macro expansion time. + - [Bonus: barebones ``let``](#bonus-barebones-let): pure AST transformation of ``let`` into a ``lambda``. + +[**Tools for lambdas**](#tools-for-lambdas) + - [``multilambda``: supercharge your lambdas](#multilambda-supercharge-your-lambdas); multiple expressions, local variables. + - [``namedlambda``: auto-name your lambdas](#namedlambda-auto-name-your-lambdas) by assignment. + - [``quicklambda``: combo with ``macropy.quick_lambda``](#quicklambda-combo-with-macropyquick_lambda) + - [``envify``: make formal parameters live in an unpythonic ``env``](#envify-make-formal-parameters-live-in-an-unpythonic-env) + +[**Language features**](#language-features) + - [``curry``: automatic currying for Python](#curry-automatic-currying-for-python) + - [``lazify``: call-by-need for Python](#lazify-call-by-need-for-python) + - [Forcing promises manually](#forcing-promises-manually) + - [Binding constructs and auto-lazification](#binding-constructs-and-auto-lazification) + - [Note about TCO](#note-about-tco) + - [``tco``: automatic tail call optimization for Python](#tco-automatic-tail-call-optimization-for-python) + - [TCO and continuations](#tco-and-continuations) + - [``continuations``: call/cc for Python](#continuations-callcc-for-python) + - [Differences between ``call/cc`` and certain other language features](#differences-between-callcc-and-certain-other-language-features) (generators, exceptions) + - [``call_cc`` API reference](#call_cc-api-reference) + - [Combo notes](#combo-notes) + - [Continuations as an escape mechanism](#continuations-as-an-escape-mechanism) + - [What can be used as a continuation?](#what-can-be-used-as-a-continuation) + - [This isn't ``call/cc``!](#this-isnt-callcc) + - [Why this syntax?](#why-this-syntax) + - [``prefix``: prefix function call syntax for Python](#prefix-prefix-function-call-syntax-for-python) + - [``autoreturn``: implicit ``return`` in tail position](#autoreturn-implicit-return-in-tail-position), like in Lisps. + - [``forall``: nondeterministic evaluation](#forall-nondeterministic-evaluation) with monadic do-notation for Python. + + [**Convenience features**](#convenience-features) + - [``cond``: the missing ``elif`` for ``a if p else b``](#cond-the-missing-elif-for-a-if-p-else-b) + - [``aif``: anaphoric if](#aif-anaphoric-if), the test result is ``it``. + - [``autoref``: implicitly reference attributes of an object](#autoref-implicitly-reference-attributes-of-an-object) + - [``dbg``: debug-print expressions with source code](#dbg-debug-print-expressions-with-source-code) + - *Changed in v0.13.1.* The ``fup[]`` macro is gone, and has been replaced with the ``fup`` function, with slightly changed syntax to accommodate. + +[**Other**](#other) + - [``nb``: silly ultralight math notebook](#nb-silly-ultralight-math-notebook) + + [**Meta**](#meta) + - [Comboability](#comboability) + - [The xmas tree combo](#the-xmas-tree-combo): notes on the macros working together. + - [This is semantics, not syntax!](#this-is-semantics-not-syntax) ## Bindings @@ -94,13 +95,13 @@ letrec((evenp, lambda x: (x == 0) or oddp(x - 1)), # mutually recursive binding ``` -As seen in the examples, the syntax is similar to ``unpythonic.lispylet``. Assignment to variables in the environment is supported via the left-shift syntax ``x << 42``. +As seen in the examples, the syntax is similar to [``unpythonic.lispylet``](../doc/features.md#lispylet-alternative-syntax). Assignment to variables in the environment is supported via the left-shift syntax ``x << 42``. The bindings are given as macro arguments as ``((name, value), ...)``, the body goes into the ``[...]``. #### Alternate syntaxes -*Added in v0.12.0.* The following Haskell-inspired, perhaps more pythonic alternate syntaxes are now available: +The following Haskell-inspired, perhaps more pythonic alternate syntaxes are now available: ```python let[((x, 21), @@ -116,6 +117,9 @@ let[x + y + z, These syntaxes take no macro arguments; both the let-body and the bindings are placed inside the same ``[...]``. +
+ Further explanation for these alternatives: + Semantically, these do the exact same thing as the original lispy syntax: the bindings are evaluated first, and then the body is evaluated with the bindings in place. The purpose of the second variant (the *let-where*) is just readability; sometimes it looks clearer to place the body expression first, and only then explain what the symbols in it mean. These syntaxes are valid for all **expression forms** of ``let``, namely: ``let[]``, ``letseq[]``, ``letrec[]``, ``let_syntax[]`` and ``abbrev[]``. The decorator variants (``dlet`` et al., ``blet`` et al.) and the block variants (``with let_syntax``, ``with abbrev``) support only the original lispy syntax, because there the body is in any case placed differently. @@ -123,6 +127,7 @@ These syntaxes are valid for all **expression forms** of ``let``, namely: ``let[ In the first variant above (the *let-in*), note the bindings block still needs the outer parentheses. This is due to Python's precedence rules; ``in`` binds more strongly than the comma (which makes sense almost everywhere else), so to make it refer to all of the bindings, the bindings block must be parenthesized. If the ``let`` expander complains your code does not look like a ``let`` form and you have used *let-in*, check your parentheses. In the second variant (the *let-where*), note the comma between the body and ``where``; it is compulsory to make the expression into syntactically valid Python. (It's however semi-easyish to remember, since also English requires the comma for a where-expression.) +
#### Special syntax for one binding From 5717aab81e8986ff3dae56222858306efc38ad44 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:14:13 -0700 Subject: [PATCH 09/31] Update design-notes.md --- doc/design-notes.md | 78 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 74 insertions(+), 4 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 89887541..6e70a2ac 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -4,8 +4,11 @@ - [Python is Not a Lisp](#python-is-not-a-lisp) - [Assignment Syntax](#assignment-syntax) - [TCO Syntax and Speed](#tco-syntax-and-speed) -- [No Monads?](#wait-no-monads) -- [Further Explanation](#your-hovercraft-is-full-of-eels) +- [Comboability](#comboability) +- [No Monads?](#no-monads) +- [Further Explanation](#further-explanation) +- [Notes on Macros](#notes-on-macros) +- [This is Just Semantics, Not Syntax!](#this-is-just-semantics-not-syntax) ### On ``let`` and Python @@ -76,8 +79,16 @@ For other libraries bringing TCO to Python, see: - [ActiveState recipe 474088](https://github.com/ActiveState/code/tree/master/recipes/Python/474088_Tail_Call_Optimization_Decorator), based on ``inspect``. - ``recur.tco`` in [fn.py](https://github.com/fnpy/fn.py), the original source of the approach used here. - [MacroPy](https://github.com/azazel75/macropy) uses an approach similar to ``fn.py``. + +### Comboability -### Wait, no monads? +Making macros work together is nontrivial, essentially because *macros don't compose*. [As pointed out by John Shutt](https://fexpr.blogspot.com/2013/12/abstractive-power.html), in a multilayered language extension implemented with macros, the second layer of macros needs to understand all of the first layer. The issue is that the macro abstraction leaks the details of its expansion. Contrast with functions, which operate on values: the process that was used to arrive at a value doesn't matter. It's always possible for a function to take this value and transform it into another value, which can then be used as input for the next layer of functions. That's composability at its finest. + +The need for interaction between macros may arise already in what *feels* like a single layer of abstraction; for example, it's not only that the block macros must understand ``let[]``, but some of them must understand other block macros. This is because what feels like one layer of abstraction is actually implemented as a number of separate macros, which run in a specific order. Thus, from the viewpoint of actually applying the macros, if the resulting software is to work correctly, the mere act of allowing combos between the block macros already makes them into a multilayer system. The compartmentalization of conceptually separate features into separate macros facilitates understanding and maintainability, but fails to reach the ideal of modularity. + +Therefore, any particular combination of macros that has not been specifically tested might not work. That said, if some particular combo doesn't work and *is not at least documented as such*, that's an error; please raise an issue. The unit tests should cover the combos that on the surface seem the most useful, but there's no guarantee that they cover everything that actually is useful somewhere. + +### No Monads (Beside List inside ``forall``.) @@ -85,7 +96,9 @@ Admittedly unpythonic, but Haskell feature, not Lisp. Besides, already done else If you want to roll your own monads for whatever reason, there's [this silly hack](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/monads.py) that wasn't packaged into this; or just read Stephan Boyer's quick introduction [[part 1]](https://www.stephanboyer.com/post/9/monads-part-1-a-design-pattern) [[part 2]](https://www.stephanboyer.com/post/10/monads-part-2-impure-computations) [[super quick intro]](https://www.stephanboyer.com/post/83/super-quick-intro-to-monads) and figure it out, it's easy. (Until you get to `State` and `Reader`, where [this](http://brandon.si/code/the-state-monad-a-tutorial-for-the-confused/) and maybe [this](https://gaiustech.wordpress.com/2010/09/06/on-monads/) can be helpful.) -### Your hovercraft is full of eels! +### Further Explanation + +Your hovercraft is full of eels! [Naturally](http://stupidpythonideas.blogspot.com/2015/05/spam-spam-spam-gouda-spam-and-tulips.html), they come with the territory. @@ -96,3 +109,60 @@ Of course, if I agreed, I wouldn't be doing this (or [this](https://github.com/T On a point raised [here](https://www.artima.com/weblogs/viewpost.jsp?thread=147358) with respect to indentation-sensitive vs. indentation-insensitive parser modes, having seen [sweet expressions](https://srfi.schemers.org/srfi-110/srfi-110.html) I think Python is confusing matters by linking the mode to statements vs. expressions. A workable solution is to make *everything* support both modes (or even preprocess the source code text to use only one of the modes), which *uniformly* makes parentheses an alternative syntax for grouping. It would be nice to be able to use indentation to structure expressions to improve their readability, like one can do in Racket with [sweet](https://docs.racket-lang.org/sweet/), but I suppose ``lambda x: [expr0, expr1, ...]`` will have to do for a multiple-expression lambda in MacroPy. Unless I decide at some point to make a source filter for [Pydialect](https://github.com/Technologicat/pydialect) to auto-convert between indentation and parentheses; but for Python this is somewhat difficult to do, because statements **must** use indentation whereas expressions **must** use parentheses, and this must be done before we can invoke the standard parser to produce an AST. + +### Notes on Macros + + - ``continuations`` and ``tco`` are mutually exclusive, since ``continuations`` already implies TCO. + - However, the ``tco`` macro skips any ``with continuations`` blocks inside it, **for the specific reason** of allowing modules written in the [Lispython dialect](https://github.com/Technologicat/pydialect) (which implies TCO for the whole module) to use ``with continuations``. + + - ``prefix``, ``autoreturn``, ``quicklambda`` and ``multilambda`` are first-pass macros (expand from outside in), because they change the semantics: + - ``prefix`` transforms things-that-look-like-tuples into function calls, + - ``autoreturn`` adds ``return`` statements where there weren't any, + - ``quicklambda`` transforms things-that-look-like-list-lookups into ``lambda`` function definitions, + - ``multilambda`` transforms things-that-look-like-lists (in the body of a ``lambda``) into sequences of multiple expressions, using ``do[]``. + - Hence, a lexically outer block of one of these types *will expand first*, before any macros inside it are expanded, in contrast to the default *from inside out* expansion order. + - This yields clean, standard-ish Python for the rest of the macros, which then don't need to worry about their input meaning something completely different from what it looks like. + + - An already expanded ``do[]`` (including that inserted by `multilambda`) is accounted for by all ``unpythonic.syntax`` macros when handling expressions. + - For simplicity, this is **the only** type of sequencing understood by the macros. + - E.g. the more rudimentary ``unpythonic.seq.begin`` is not treated as a sequencing operation. This matters especially in ``tco``, where it is critically important to correctly detect a tail position in a return-value expression or (multi-)lambda body. + - *Sequencing* is here meant in the Racket/Haskell sense of *running sub-operations in a specified order*, unrelated to Python's *sequences*. + + - The TCO transformation knows about TCO-enabling decorators provided by ``unpythonic``, and adds the ``@trampolined`` decorator to a function definition only when it is not already TCO'd. + - This applies also to lambdas; they are decorated by directly wrapping them with a call: ``trampolined(lambda ...: ...)``. + - This allows ``with tco`` to work together with the functions in ``unpythonic.fploop``, which imply TCO. + + - Macros that transform lambdas (notably ``continuations`` and ``tco``): + - Perform a first pass to take note of all lambdas that appear in the code *before the expansion of any inner macros*. Then in the second pass, *after the expansion of all inner macros*, only the recorded lambdas are transformed. + - This mechanism distinguishes between explicit lambdas in the client code, and internal implicit lambdas automatically inserted by a macro. The latter are a technical detail that should not undergo the same transformations as user-written explicit lambdas. + - The identification is based on the ``id`` of the AST node instance. Hence, if you plan to write your own macros that work together with those in ``unpythonic.syntax``, avoid going overboard with FP. Modifying the tree in-place, preserving the original AST node instances as far as sensible, is just fine. + - For the interested reader, grep the source code for ``userlambdas``. + - Support a limited form of *decorated lambdas*, i.e. trees of the form ``f(g(h(lambda ...: ...)))``. + - The macros will reorder a chain of lambda decorators (i.e. nested calls) to use the correct ordering, when only known decorators are used on a literal lambda. + - This allows some combos such as ``tco``, ``unpythonic.fploop.looped``, ``curry``. + - Only decorators provided by ``unpythonic`` are recognized, and only some of them are supported. For details, see ``unpythonic.regutil``. + - If you need to combo ``unpythonic.fploop.looped`` and ``unpythonic.ec.call_ec``, use ``unpythonic.fploop.breakably_looped``, which does exactly that. + - The problem with a direct combo is that the required ordering is the trampoline (inside ``looped``) outermost, then ``call_ec``, and then the actual loop, but because an escape continuation is only valid for the dynamic extent of the ``call_ec``, the whole loop must be run inside the dynamic extent of the ``call_ec``. + - ``unpythonic.fploop.breakably_looped`` internally inserts the ``call_ec`` at the right step, and gives you the ec as ``brk``. + - For the interested reader, look at ``unpythonic.syntax.util``. + + - ``namedlambda`` is a two-pass macro. In the first pass (outside-in), it names lambdas inside ``let[]`` expressions before they are expanded away. The second pass (inside-out) of ``namedlambda`` must run after ``curry`` to analyze and transform the auto-curried code produced by ``with curry``. In most cases, placing ``namedlambda`` in a separate outer ``with`` block runs both operations in the correct order. + + - ``autoref`` does not need in its output to be curried (hence after ``curry`` to gain some performance), but needs to run before ``lazify``, so that both branches of each transformed reference get the implicit forcing. Its transformation is orthogonal to what ``namedlambda`` does, so it does not matter in which exact order these two run. + + - ``lazify`` is a rather invasive rewrite that needs to see the output from most of the other macros. + + - ``envify`` needs to see the output of ``lazify`` in order to shunt function args into an unpythonic ``env`` without triggering the implicit forcing. + + - Some of the block macros can be comboed as multiple context managers in the same ``with`` statement (expansion order is then *left-to-right*), whereas some (notably ``curry`` and ``namedlambda``) require their own ``with`` statement. + - This is a [known issue in MacroPy](https://github.com/azazel75/macropy/issues/21). I have made a [fix](https://github.com/azazel75/macropy/pull/22), but still need to make proper test cases to get it merged. + - If something goes wrong in the expansion of one block macro in a ``with`` statement that specifies several block macros, surprises may occur. + - When in doubt, use a separate ``with`` statement for each block macro that applies to the same section of code, and nest the blocks. + - Test one step at a time with the ``macropy.tracing.show_expanded`` block macro to make sure the expansion looks like what you intended. + + +### This is semantics, not syntax! + +[Strictly speaking](https://stackoverflow.com/questions/17930267/what-is-the-difference-between-syntax-and-semantics-of-programming-languages), ``True``: we just repurpose Python's existing syntax to give it new meanings. However, in the Racket reference, **a** *syntax* designates a macro, in contrast to a *procedure* (regular function). We provide syntaxes in this particular sense. The name ``unpythonic.syntax`` is also shorter to type than ``unpythonic.semantics``, less abstruse, and close enough to convey the intended meaning. + +If you want custom *syntax* proper, then you may be interested in [Pydialect](https://github.com/Technologicat/pydialect). From f328e5ff957059c919833aa9dd1a96d2751698c3 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:15:38 -0700 Subject: [PATCH 10/31] Update design-notes.md --- doc/design-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index 6e70a2ac..b1f4ad9b 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -8,7 +8,7 @@ - [No Monads?](#no-monads) - [Further Explanation](#further-explanation) - [Notes on Macros](#notes-on-macros) -- [This is Just Semantics, Not Syntax!](#this-is-just-semantics-not-syntax) +- [This is Semantics, Not Syntax!](#this-is-semantics-not-syntax) ### On ``let`` and Python From 805265d96fda4a967b75c230523da2ceb6231528 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:17:51 -0700 Subject: [PATCH 11/31] Update README.md --- macro_extras/README.md | 74 +++--------------------------------------- 1 file changed, 5 insertions(+), 69 deletions(-) diff --git a/macro_extras/README.md b/macro_extras/README.md index 9e7303b4..589f332b 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -1,8 +1,8 @@ # Language extensions using ``unpythonic.syntax`` -These optional features -- providing extensions to the Python language as syntactic macros -- are built on [MacroPy](https://github.com/azazel75/macropy), from the PyPI package ``macropy3``. If you want to take language extension a step further, see our sister project [Pydialect](https://github.com/Technologicat/pydialect). +Our Python language extensions, as syntactic macros, are built on [MacroPy](https://github.com/azazel75/macropy), from the PyPI package ``macropy3``. If you want to take language extension a step further, see our sister project [Pydialect](https://github.com/Technologicat/pydialect). -Macro expansion occurs at import time, so the unit tests that contain usage examples (located in [unpythonic/syntax/test/](../unpythonic/syntax/test/)) cannot be run directly. Instead, run them via the included [generic MacroPy3 bootstrapper](macropy3). For convenience, ``setup.py`` installs the `macropy3` bootstrapper. +The unit tests that contain usage examples (located in [unpythonic/syntax/test/](../unpythonic/syntax/test/)) cannot be run directly because macro expansion occurs at import time. Instead, run them via the included [generic MacroPy3 bootstrapper](macropy3). For convenience, ``setup.py`` installs this bootstrapper. To use the bootstrapper, run: @@ -12,7 +12,7 @@ The tests use relative imports; invoke them from the top-level directory of ``un ``macro_extras/macropy3 -m unpythonic.syntax.test.test_curry`` - This is to make the tests run against the source tree without installing it first; in your own code, once you have installed ``unpythonic``, feel free to use absolute imports, like those shown in this README. + This is to make the tests run against the source tree without installing it first. Once you have installed ``unpythonic``, feel free to use absolute imports in your own code, like those shown in this README. There is no abbreviation for ``memoize(lambda: ...)``, because ``MacroPy`` itself already provides ``lazy`` and ``interned``. @@ -1621,15 +1621,7 @@ Obviously not intended for production use, although is very likely to work anywh Is this just a set of macros, a language extension, or a compiler for a new language that just happens to be implemented in MacroPy, à la *On Lisp*? All of the above, really. - -### Comboability - -Making macros work together is nontrivial, essentially because *macros don't compose*. [As pointed out by John Shutt](https://fexpr.blogspot.com/2013/12/abstractive-power.html), in a multilayered language extension implemented with macros, the second layer of macros needs to understand all of the first layer. The issue is that the macro abstraction leaks the details of its expansion. Contrast with functions, which operate on values: the process that was used to arrive at a value doesn't matter. It's always possible for a function to take this value and transform it into another value, which can then be used as input for the next layer of functions. That's composability at its finest. - -The need for interaction between macros may arise already in what *feels* like a single layer of abstraction; for example, it's not only that the block macros must understand ``let[]``, but some of them must understand other block macros. This is because what feels like one layer of abstraction is actually implemented as a number of separate macros, which run in a specific order. Thus, from the viewpoint of actually applying the macros, if the resulting software is to work correctly, the mere act of allowing combos between the block macros already makes them into a multilayer system. The compartmentalization of conceptually separate features into separate macros facilitates understanding and maintainability, but fails to reach the ideal of modularity. - -Therefore, any particular combination of macros that has not been specifically tested might not work. That said, if some particular combo doesn't work and *is not at least documented as such*, that's an error; please raise an issue. The unit tests should cover the combos that on the surface seem the most useful, but there's no guarantee that they cover everything that actually is useful somewhere. - +See our [notes on comboability](#../doc/design-notes.md#comboability). ### The xmas tree combo @@ -1664,60 +1656,4 @@ with lazify: ... ``` -Other things to note: - - - ``continuations`` and ``tco`` are mutually exclusive, since ``continuations`` already implies TCO. - - However, the ``tco`` macro skips any ``with continuations`` blocks inside it, **for the specific reason** of allowing modules written in the [Lispython dialect](https://github.com/Technologicat/pydialect) (which implies TCO for the whole module) to use ``with continuations``. - - - ``prefix``, ``autoreturn``, ``quicklambda`` and ``multilambda`` are first-pass macros (expand from outside in), because they change the semantics: - - ``prefix`` transforms things-that-look-like-tuples into function calls, - - ``autoreturn`` adds ``return`` statements where there weren't any, - - ``quicklambda`` transforms things-that-look-like-list-lookups into ``lambda`` function definitions, - - ``multilambda`` transforms things-that-look-like-lists (in the body of a ``lambda``) into sequences of multiple expressions, using ``do[]``. - - Hence, a lexically outer block of one of these types *will expand first*, before any macros inside it are expanded, in contrast to the default *from inside out* expansion order. - - This yields clean, standard-ish Python for the rest of the macros, which then don't need to worry about their input meaning something completely different from what it looks like. - - - An already expanded ``do[]`` (including that inserted by `multilambda`) is accounted for by all ``unpythonic.syntax`` macros when handling expressions. - - For simplicity, this is **the only** type of sequencing understood by the macros. - - E.g. the more rudimentary ``unpythonic.seq.begin`` is not treated as a sequencing operation. This matters especially in ``tco``, where it is critically important to correctly detect a tail position in a return-value expression or (multi-)lambda body. - - *Sequencing* is here meant in the Racket/Haskell sense of *running sub-operations in a specified order*, unrelated to Python's *sequences*. - - - The TCO transformation knows about TCO-enabling decorators provided by ``unpythonic``, and adds the ``@trampolined`` decorator to a function definition only when it is not already TCO'd. - - This applies also to lambdas; they are decorated by directly wrapping them with a call: ``trampolined(lambda ...: ...)``. - - This allows ``with tco`` to work together with the functions in ``unpythonic.fploop``, which imply TCO. - - - Macros that transform lambdas (notably ``continuations`` and ``tco``): - - Perform a first pass to take note of all lambdas that appear in the code *before the expansion of any inner macros*. Then in the second pass, *after the expansion of all inner macros*, only the recorded lambdas are transformed. - - This mechanism distinguishes between explicit lambdas in the client code, and internal implicit lambdas automatically inserted by a macro. The latter are a technical detail that should not undergo the same transformations as user-written explicit lambdas. - - The identification is based on the ``id`` of the AST node instance. Hence, if you plan to write your own macros that work together with those in ``unpythonic.syntax``, avoid going overboard with FP. Modifying the tree in-place, preserving the original AST node instances as far as sensible, is just fine. - - For the interested reader, grep the source code for ``userlambdas``. - - Support a limited form of *decorated lambdas*, i.e. trees of the form ``f(g(h(lambda ...: ...)))``. - - The macros will reorder a chain of lambda decorators (i.e. nested calls) to use the correct ordering, when only known decorators are used on a literal lambda. - - This allows some combos such as ``tco``, ``unpythonic.fploop.looped``, ``curry``. - - Only decorators provided by ``unpythonic`` are recognized, and only some of them are supported. For details, see ``unpythonic.regutil``. - - If you need to combo ``unpythonic.fploop.looped`` and ``unpythonic.ec.call_ec``, use ``unpythonic.fploop.breakably_looped``, which does exactly that. - - The problem with a direct combo is that the required ordering is the trampoline (inside ``looped``) outermost, then ``call_ec``, and then the actual loop, but because an escape continuation is only valid for the dynamic extent of the ``call_ec``, the whole loop must be run inside the dynamic extent of the ``call_ec``. - - ``unpythonic.fploop.breakably_looped`` internally inserts the ``call_ec`` at the right step, and gives you the ec as ``brk``. - - For the interested reader, look at ``unpythonic.syntax.util``. - - - ``namedlambda`` is a two-pass macro. In the first pass (outside-in), it names lambdas inside ``let[]`` expressions before they are expanded away. The second pass (inside-out) of ``namedlambda`` must run after ``curry`` to analyze and transform the auto-curried code produced by ``with curry``. In most cases, placing ``namedlambda`` in a separate outer ``with`` block runs both operations in the correct order. - - - ``autoref`` does not need in its output to be curried (hence after ``curry`` to gain some performance), but needs to run before ``lazify``, so that both branches of each transformed reference get the implicit forcing. Its transformation is orthogonal to what ``namedlambda`` does, so it does not matter in which exact order these two run. - - - ``lazify`` is a rather invasive rewrite that needs to see the output from most of the other macros. - - - ``envify`` needs to see the output of ``lazify`` in order to shunt function args into an unpythonic ``env`` without triggering the implicit forcing. - - - Some of the block macros can be comboed as multiple context managers in the same ``with`` statement (expansion order is then *left-to-right*), whereas some (notably ``curry`` and ``namedlambda``) require their own ``with`` statement. - - This is a [known issue in MacroPy](https://github.com/azazel75/macropy/issues/21). I have made a [fix](https://github.com/azazel75/macropy/pull/22), but still need to make proper test cases to get it merged. - - If something goes wrong in the expansion of one block macro in a ``with`` statement that specifies several block macros, surprises may occur. - - When in doubt, use a separate ``with`` statement for each block macro that applies to the same section of code, and nest the blocks. - - Test one step at a time with the ``macropy.tracing.show_expanded`` block macro to make sure the expansion looks like what you intended. - - -### This is semantics, not syntax! - -[Strictly speaking](https://stackoverflow.com/questions/17930267/what-is-the-difference-between-syntax-and-semantics-of-programming-languages), ``True``: we just repurpose Python's existing syntax to give it new meanings. However, in the Racket reference, **a** *syntax* designates a macro, in contrast to a *procedure* (regular function). We provide syntaxes in this particular sense. The name ``unpythonic.syntax`` is also shorter to type than ``unpythonic.semantics``, less abstruse, and close enough to convey the intended meaning. - -If you want custom *syntax* proper, then you may be interested in [Pydialect](https://github.com/Technologicat/pydialect). - +See our [notes on macros](#../doc/design-notes.md#notes-on-macros) for more information. From 80197ad84152c4b8c2fb99bea153e99310a6b263 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:21:50 -0700 Subject: [PATCH 12/31] Update README.md --- macro_extras/README.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/macro_extras/README.md b/macro_extras/README.md index 589f332b..63467af1 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -65,9 +65,7 @@ The `macropy3` bootstrapper takes the `-m` option, like `python3 -m mod`. The al - [``nb``: silly ultralight math notebook](#nb-silly-ultralight-math-notebook) [**Meta**](#meta) - - [Comboability](#comboability) - [The xmas tree combo](#the-xmas-tree-combo): notes on the macros working together. - - [This is semantics, not syntax!](#this-is-semantics-not-syntax) ## Bindings @@ -1621,7 +1619,7 @@ Obviously not intended for production use, although is very likely to work anywh Is this just a set of macros, a language extension, or a compiler for a new language that just happens to be implemented in MacroPy, à la *On Lisp*? All of the above, really. -See our [notes on comboability](#../doc/design-notes.md#comboability). +See our [notes on comboability](../doc/design-notes.md#comboability). ### The xmas tree combo @@ -1656,4 +1654,4 @@ with lazify: ... ``` -See our [notes on macros](#../doc/design-notes.md#notes-on-macros) for more information. +See our [notes on macros](../doc/design-notes.md#notes-on-macros) for more information. From 8890268394cc69e6e1de10268476beed890595fc Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:24:31 -0700 Subject: [PATCH 13/31] Update design-notes.md --- doc/design-notes.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/doc/design-notes.md b/doc/design-notes.md index b1f4ad9b..188cae39 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -8,7 +8,6 @@ - [No Monads?](#no-monads) - [Further Explanation](#further-explanation) - [Notes on Macros](#notes-on-macros) -- [This is Semantics, Not Syntax!](#this-is-semantics-not-syntax) ### On ``let`` and Python @@ -159,10 +158,3 @@ It would be nice to be able to use indentation to structure expressions to impro - If something goes wrong in the expansion of one block macro in a ``with`` statement that specifies several block macros, surprises may occur. - When in doubt, use a separate ``with`` statement for each block macro that applies to the same section of code, and nest the blocks. - Test one step at a time with the ``macropy.tracing.show_expanded`` block macro to make sure the expansion looks like what you intended. - - -### This is semantics, not syntax! - -[Strictly speaking](https://stackoverflow.com/questions/17930267/what-is-the-difference-between-syntax-and-semantics-of-programming-languages), ``True``: we just repurpose Python's existing syntax to give it new meanings. However, in the Racket reference, **a** *syntax* designates a macro, in contrast to a *procedure* (regular function). We provide syntaxes in this particular sense. The name ``unpythonic.syntax`` is also shorter to type than ``unpythonic.semantics``, less abstruse, and close enough to convey the intended meaning. - -If you want custom *syntax* proper, then you may be interested in [Pydialect](https://github.com/Technologicat/pydialect). From e4c213e30e84891d78530b58a7f6a273b70e24fb Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:27:58 -0700 Subject: [PATCH 14/31] Update README.md --- README.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7192b4fa..ccbb4884 100644 --- a/README.md +++ b/README.md @@ -12,8 +12,8 @@ Currently none required; [MacroPy](https://github.com/azazel75/macropy) optional ### Documentation -- [Basic: pure-Python](doc/features.md) -- [Advanced: syntactic macros](macro_extras/): the second half of ``unpythonic``. +- [pure-Python](doc/features.md) +- [syntactic macros](macro_extras/README.md): the second half of ``unpythonic``. ## Installation @@ -64,6 +64,11 @@ The trampoline implementation of ``unpythonic.tco`` takes its remarkably clean a Another important source of inspiration was [tco](https://github.com/baruchel/tco) by Thomas Baruchel, for thinking about the possibilities of TCO in Python. +### This is semantics, not syntax! + +[Strictly speaking](https://stackoverflow.com/questions/17930267/what-is-the-difference-between-syntax-and-semantics-of-programming-languages), ``True``: we just repurpose Python's existing syntax to give it new meanings. However, in the Racket reference, **a** *syntax* designates a macro, in contrast to a *procedure* (regular function). We provide syntaxes in this particular sense. The name ``unpythonic.syntax`` is also shorter to type than ``unpythonic.semantics``, less abstruse, and close enough to convey the intended meaning. + +If you want custom *syntax* proper, then you may be interested in [Pydialect](https://github.com/Technologicat/pydialect). ## Python-related FP resources From 1161706c16f273886c982671f3331982d93a5364 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:30:33 -0700 Subject: [PATCH 15/31] Update README.md --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index ccbb4884..1451c618 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,12 @@ We also provide extensions to the Python language as a set of [syntactic macros] Design considerations are based in simplicity, robustness, and with minimal dependencies. See our [design notes](doc/design-notes.md) for more information. +#### This is semantics, not syntax! + +[Strictly speaking](https://stackoverflow.com/questions/17930267/what-is-the-difference-between-syntax-and-semantics-of-programming-languages), ``True``: we just repurpose Python's existing syntax to give it new meanings. However, in the Racket reference, **a** *syntax* designates a macro, in contrast to a *procedure* (regular function). We provide syntaxes in this particular sense. The name ``unpythonic.syntax`` is also shorter to type than ``unpythonic.semantics``, less abstruse, and close enough to convey the intended meaning. + +If you want custom *syntax* proper, then you may be interested in [Pydialect](https://github.com/Technologicat/pydialect). + ### Dependencies Currently none required; [MacroPy](https://github.com/azazel75/macropy) optional, to enable the syntactic macros. @@ -64,12 +70,6 @@ The trampoline implementation of ``unpythonic.tco`` takes its remarkably clean a Another important source of inspiration was [tco](https://github.com/baruchel/tco) by Thomas Baruchel, for thinking about the possibilities of TCO in Python. -### This is semantics, not syntax! - -[Strictly speaking](https://stackoverflow.com/questions/17930267/what-is-the-difference-between-syntax-and-semantics-of-programming-languages), ``True``: we just repurpose Python's existing syntax to give it new meanings. However, in the Racket reference, **a** *syntax* designates a macro, in contrast to a *procedure* (regular function). We provide syntaxes in this particular sense. The name ``unpythonic.syntax`` is also shorter to type than ``unpythonic.semantics``, less abstruse, and close enough to convey the intended meaning. - -If you want custom *syntax* proper, then you may be interested in [Pydialect](https://github.com/Technologicat/pydialect). - ## Python-related FP resources Python clearly wants to be an impure-FP language. A decorator with arguments *is a curried closure* - how much more FP can you get? From 2411e72e76ac8048479da98ae245a04e3bfd6f27 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:31:02 -0700 Subject: [PATCH 16/31] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1451c618..83e0e870 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ We also provide extensions to the Python language as a set of [syntactic macros] Design considerations are based in simplicity, robustness, and with minimal dependencies. See our [design notes](doc/design-notes.md) for more information. -#### This is semantics, not syntax! +**This is semantics, not syntax!** [Strictly speaking](https://stackoverflow.com/questions/17930267/what-is-the-difference-between-syntax-and-semantics-of-programming-languages), ``True``: we just repurpose Python's existing syntax to give it new meanings. However, in the Racket reference, **a** *syntax* designates a macro, in contrast to a *procedure* (regular function). We provide syntaxes in this particular sense. The name ``unpythonic.syntax`` is also shorter to type than ``unpythonic.semantics``, less abstruse, and close enough to convey the intended meaning. From c91ad0ef957742ec39cfd62fa2b59df557da3bee Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:33:36 -0700 Subject: [PATCH 17/31] Update README.md --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 83e0e870..46c7c49a 100644 --- a/README.md +++ b/README.md @@ -18,12 +18,12 @@ Currently none required; [MacroPy](https://github.com/azazel75/macropy) optional ### Documentation -- [pure-Python](doc/features.md) -- [syntactic macros](macro_extras/README.md): the second half of ``unpythonic``. +- [pure-Python (basic implementation)](doc/features.md) +- [syntactic macros (advanced implementation)](macro_extras/README.md): the second half of ``unpythonic``. ## Installation -#### PyPI +**PyPI** ``pip3 install unpythonic --user`` @@ -31,7 +31,7 @@ or ``sudo pip3 install unpythonic`` -#### GitHub +**GitHub** Clone (or pull) from GitHub. Then, @@ -41,7 +41,7 @@ or ``sudo python3 setup.py install`` -#### Uninstall +**Uninstall** Uninstallation must be invoked in a folder which has no subfolder called ``unpythonic``, so that ``pip`` recognizes it as a package name (instead of a filename). Then, From e97787b4361d38b6d2a2ca5fab031d98d81bb380 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:43:11 -0700 Subject: [PATCH 18/31] Update README.md --- macro_extras/README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/macro_extras/README.md b/macro_extras/README.md index 63467af1..14426eef 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -73,8 +73,7 @@ Macros that introduce new ways to bind identifiers. ### ``let``, ``letseq``, ``letrec`` as macros -
- Properly lexically scoped ``let`` constructs, no boilerplate: +Properly lexically scoped ``let`` constructs, no boilerplate: ```python from unpythonic.syntax import macros, let, letseq, letrec @@ -91,7 +90,6 @@ letrec((evenp, lambda x: (x == 0) or oddp(x - 1)), # mutually recursive binding (oddp, lambda x: (x != 0) and evenp(x - 1)))[ print(evenp(42))] ``` -
As seen in the examples, the syntax is similar to [``unpythonic.lispylet``](../doc/features.md#lispylet-alternative-syntax). Assignment to variables in the environment is supported via the left-shift syntax ``x << 42``. @@ -116,7 +114,7 @@ let[x + y + z, These syntaxes take no macro arguments; both the let-body and the bindings are placed inside the same ``[...]``.
- Further explanation for these alternatives: + Expand for further explanation: Semantically, these do the exact same thing as the original lispy syntax: the bindings are evaluated first, and then the body is evaluated with the bindings in place. The purpose of the second variant (the *let-where*) is just readability; sometimes it looks clearer to place the body expression first, and only then explain what the symbols in it mean. @@ -129,9 +127,11 @@ In the second variant (the *let-where*), note the comma between the body and ``w #### Special syntax for one binding -*Added in v0.12.0.* If there is only one binding, to make the syntax more pythonic, the outer parentheses may be omitted in the bindings block of the **expr forms** of ``let``, ``letseq``, ``letrec``, ``let_syntax`` and ``abbrev``. +If there is only one binding, to make the syntax more pythonic, the outer parentheses may be omitted in the bindings block of the **expr forms** of: -*Changed in v0.13.0.* Now supported also by the bindings block of ``dlet``, ``dletseq``, ``dletrec``, ``blet``, ``bletseq`` and ``bletrec``. +- ``let``, ``letseq``, ``letrec`` +- ``dlet``, ``dletseq``, ``dletrec``, ``blet``, ``bletseq``, ``bletrec`` +- ``let_syntax``, ``abbrev`` ```python let(x, 21)[2*x] @@ -141,11 +141,11 @@ let[2*x, where(x, 21)] This is valid also in the *let-in* variant, because there is still one set of parentheses enclosing the bindings block. -This is essentially special-cased in the ``let`` expander. (If interested in the technical details, look at ``unpythonic.syntax.letdoutil.UnexpandedLetView``, which performs the destructuring. See also ``unpythonic.syntax.__init__.let``; MacroPy itself already destructures the original lispy syntax when the macro is invoked.) +This is essentially special-cased in the ``let`` expander. (If interested in the technical details, look at [``unpythonic.syntax.letdoutil.UnexpandedLetView``](../unpythonic/syntax/letdoutil.py), which performs the destructuring. See also [``unpythonic.syntax.__init__.let``](../unpythonic/syntax/__init__.py); MacroPy itself already destructures the original lispy syntax when the macro is invoked.) #### Multiple expressions in body -*Added in v0.9.2.* The `let` constructs can now use a multiple-expression body. The syntax to activate multiple expression mode is an extra set of brackets around the body (like in `multilambda`; see below): +The `let` constructs can now use a multiple-expression body. The syntax to activate multiple expression mode is an extra set of brackets around the body (like in `multilambda`; see below): ```python let((x, 1), From 935350f873f88a553d2a5fd6d92764f703f479ad Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:46:21 -0700 Subject: [PATCH 19/31] Update README.md --- macro_extras/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macro_extras/README.md b/macro_extras/README.md index 14426eef..6d3bb2f7 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -114,7 +114,7 @@ let[x + y + z, These syntaxes take no macro arguments; both the let-body and the bindings are placed inside the same ``[...]``.
- Expand for further explanation: + **Expand for further explanation:** Semantically, these do the exact same thing as the original lispy syntax: the bindings are evaluated first, and then the body is evaluated with the bindings in place. The purpose of the second variant (the *let-where*) is just readability; sometimes it looks clearer to place the body expression first, and only then explain what the symbols in it mean. From e67e876466943c90e99967832a6787a956a23a8f Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:47:10 -0700 Subject: [PATCH 20/31] Update README.md --- macro_extras/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/macro_extras/README.md b/macro_extras/README.md index 6d3bb2f7..1a575b6e 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -114,7 +114,7 @@ let[x + y + z, These syntaxes take no macro arguments; both the let-body and the bindings are placed inside the same ``[...]``.
- **Expand for further explanation:** +**Expand for further explanation: ** Semantically, these do the exact same thing as the original lispy syntax: the bindings are evaluated first, and then the body is evaluated with the bindings in place. The purpose of the second variant (the *let-where*) is just readability; sometimes it looks clearer to place the body expression first, and only then explain what the symbols in it mean. From c82ee66e9a5ce72a5348f3d400597fefe26ad981 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:51:56 -0700 Subject: [PATCH 21/31] Update README.md --- macro_extras/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/macro_extras/README.md b/macro_extras/README.md index 1a575b6e..dcc47b7c 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -114,15 +114,15 @@ let[x + y + z, These syntaxes take no macro arguments; both the let-body and the bindings are placed inside the same ``[...]``.
-**Expand for further explanation: ** +Expand for further explanation -Semantically, these do the exact same thing as the original lispy syntax: the bindings are evaluated first, and then the body is evaluated with the bindings in place. The purpose of the second variant (the *let-where*) is just readability; sometimes it looks clearer to place the body expression first, and only then explain what the symbols in it mean. +>Semantically, these do the exact same thing as the original lispy syntax: the bindings are evaluated first, and then the body is evaluated with the bindings in place. The purpose of the second variant (the *let-where*) is just readability; sometimes it looks clearer to place the body expression first, and only then explain what the symbols in it mean. -These syntaxes are valid for all **expression forms** of ``let``, namely: ``let[]``, ``letseq[]``, ``letrec[]``, ``let_syntax[]`` and ``abbrev[]``. The decorator variants (``dlet`` et al., ``blet`` et al.) and the block variants (``with let_syntax``, ``with abbrev``) support only the original lispy syntax, because there the body is in any case placed differently. +>These syntaxes are valid for all **expression forms** of ``let``, namely: ``let[]``, ``letseq[]``, ``letrec[]``, ``let_syntax[]`` and ``abbrev[]``. The decorator variants (``dlet`` et al., ``blet`` et al.) and the block variants (``with let_syntax``, ``with abbrev``) support only the original lispy syntax, because there the body is in any case placed differently. -In the first variant above (the *let-in*), note the bindings block still needs the outer parentheses. This is due to Python's precedence rules; ``in`` binds more strongly than the comma (which makes sense almost everywhere else), so to make it refer to all of the bindings, the bindings block must be parenthesized. If the ``let`` expander complains your code does not look like a ``let`` form and you have used *let-in*, check your parentheses. +>In the first variant above (the *let-in*), note the bindings block still needs the outer parentheses. This is due to Python's precedence rules; ``in`` binds more strongly than the comma (which makes sense almost everywhere else), so to make it refer to all of the bindings, the bindings block must be parenthesized. If the ``let`` expander complains your code does not look like a ``let`` form and you have used *let-in*, check your parentheses. -In the second variant (the *let-where*), note the comma between the body and ``where``; it is compulsory to make the expression into syntactically valid Python. (It's however semi-easyish to remember, since also English requires the comma for a where-expression.) +>In the second variant (the *let-where*), note the comma between the body and ``where``; it is compulsory to make the expression into syntactically valid Python. (It's however semi-easyish to remember, since also English requires the comma for a where-expression.)
#### Special syntax for one binding From 4d076441bf41d48f05c5cbaf57b6dc667da85e7a Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:55:28 -0700 Subject: [PATCH 22/31] Update README.md --- macro_extras/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/macro_extras/README.md b/macro_extras/README.md index dcc47b7c..d011536e 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -117,11 +117,11 @@ These syntaxes take no macro arguments; both the let-body and the bindings are p Expand for further explanation >Semantically, these do the exact same thing as the original lispy syntax: the bindings are evaluated first, and then the body is evaluated with the bindings in place. The purpose of the second variant (the *let-where*) is just readability; sometimes it looks clearer to place the body expression first, and only then explain what the symbols in it mean. - +> >These syntaxes are valid for all **expression forms** of ``let``, namely: ``let[]``, ``letseq[]``, ``letrec[]``, ``let_syntax[]`` and ``abbrev[]``. The decorator variants (``dlet`` et al., ``blet`` et al.) and the block variants (``with let_syntax``, ``with abbrev``) support only the original lispy syntax, because there the body is in any case placed differently. - +> >In the first variant above (the *let-in*), note the bindings block still needs the outer parentheses. This is due to Python's precedence rules; ``in`` binds more strongly than the comma (which makes sense almost everywhere else), so to make it refer to all of the bindings, the bindings block must be parenthesized. If the ``let`` expander complains your code does not look like a ``let`` form and you have used *let-in*, check your parentheses. - +> >In the second variant (the *let-where*), note the comma between the body and ``where``; it is compulsory to make the expression into syntactically valid Python. (It's however semi-easyish to remember, since also English requires the comma for a where-expression.)
@@ -141,11 +141,11 @@ let[2*x, where(x, 21)] This is valid also in the *let-in* variant, because there is still one set of parentheses enclosing the bindings block. -This is essentially special-cased in the ``let`` expander. (If interested in the technical details, look at [``unpythonic.syntax.letdoutil.UnexpandedLetView``](../unpythonic/syntax/letdoutil.py), which performs the destructuring. See also [``unpythonic.syntax.__init__.let``](../unpythonic/syntax/__init__.py); MacroPy itself already destructures the original lispy syntax when the macro is invoked.) +This is essentially special-cased in the ``let`` expander. (If interested in the technical details, look at ``unpythonic.syntax.letdoutil.UnexpandedLetView``, which performs the destructuring. See also ``unpythonic.syntax.__init__.let``; MacroPy itself already destructures the original lispy syntax when the macro is invoked.) #### Multiple expressions in body -The `let` constructs can now use a multiple-expression body. The syntax to activate multiple expression mode is an extra set of brackets around the body (like in `multilambda`; see below): +The `let` constructs can now use a multiple-expression body. The syntax to activate multiple expression mode is an extra set of brackets around the body (like in [`multilambda`](#multilambda-supercharge-your-lambdas)): ```python let((x, 1), From c3e2851673a023a8048bb5f567801ed5e0aa2820 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 13:59:39 -0700 Subject: [PATCH 23/31] Update README.md --- macro_extras/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/macro_extras/README.md b/macro_extras/README.md index d011536e..9e289331 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -28,6 +28,9 @@ The `macropy3` bootstrapper takes the `-m` option, like `python3 -m mod`. The al - [``let_syntax``, ``abbrev``: syntactic local bindings](#let_syntax-abbrev-syntactic-local-bindings); splice code at macro expansion time. - [Bonus: barebones ``let``](#bonus-barebones-let): pure AST transformation of ``let`` into a ``lambda``. +[**Sequencing**](#sequencing) + - [``do`` as a macro: stuff imperative code into an expression, *with style*](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style) + [**Tools for lambdas**](#tools-for-lambdas) - [``multilambda``: supercharge your lambdas](#multilambda-supercharge-your-lambdas); multiple expressions, local variables. - [``namedlambda``: auto-name your lambdas](#namedlambda-auto-name-your-lambdas) by assignment. From bc9f088d9609989d27a3fd5a2e45cbe351e93694 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 14:00:23 -0700 Subject: [PATCH 24/31] Update README.md --- macro_extras/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/macro_extras/README.md b/macro_extras/README.md index 9e289331..3b270caf 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -22,6 +22,8 @@ The `macropy3` bootstrapper takes the `-m` option, like `python3 -m mod`. The al **Note: This document is up-to-date for version 0.14.1.** +----- + [**Bindings**](#bindings) - [``let``, ``letseq``, ``letrec`` as macros](#let-letseq-letrec-as-macros); proper lexical scoping, no boilerplate. - [``dlet``, ``dletseq``, ``dletrec``, ``blet``, ``bletseq``, ``bletrec``: decorator versions](#dlet-dletseq-dletrec-blet-bletseq-bletrec-decorator-versions) From 8e6cd2558e18894988c4209c8f9d3147d3cb5b15 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 14:12:55 -0700 Subject: [PATCH 25/31] Update README.md --- macro_extras/README.md | 64 ++++++++++++++++++++++-------------------- 1 file changed, 34 insertions(+), 30 deletions(-) diff --git a/macro_extras/README.md b/macro_extras/README.md index 3b270caf..ed4a893c 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -223,7 +223,7 @@ Hence the ``z`` in the inner scope expands to the inner environment's ``z``, whi ### ``dlet``, ``dletseq``, ``dletrec``, ``blet``, ``bletseq``, ``bletrec``: decorator versions -*Added in v0.10.4.* Similarly to ``let``, ``letseq``, ``letrec``, these sugar the corresponding ``unpythonic.lispylet`` constructs, with the ``dletseq`` and ``bletseq`` constructs existing only as macros (expanding to nested ``dlet`` or ``blet``, respectively). +Similar to ``let``, ``letseq``, ``letrec``, these sugar the corresponding ``unpythonic.lispylet`` constructs, with the ``dletseq`` and ``bletseq`` constructs existing only as macros (expanding to nested ``dlet`` or ``blet``, respectively). Lexical scoping is respected; each environment is internally named using a gensym. Nesting is allowed. @@ -332,7 +332,7 @@ else: ### ``let_syntax``, ``abbrev``: syntactic local bindings -*Added in v0.11.0.* Locally splice code at macro expansion time (it's almost like inlining functions): +Locally splice code at macro expansion time (it's almost like inlining functions): ```python from unpythonic.syntax import macros, let_syntax, block, expr @@ -388,35 +388,41 @@ with let_syntax: After macro expansion completes, ``let_syntax`` has zero runtime overhead; it completely disappears in macro expansion. -There are two kinds of substitutions: *bare name* and *template*. A bare name substitution has no parameters. A template substitution has positional parameters. (Named parameters, ``*args``, ``**kwargs`` and default values are currently **not** supported.) - -When used as an expr macro, the formal parameter declaration is placed where it belongs; on the name side (LHS) of the binding. In the above example, ``f(a)`` is a template with a formal parameter ``a``. But when used as a block macro, the formal parameters are declared on the ``block`` or ``expr`` "context manager" due to syntactic limitations of Python. To define a bare name substitution, just use ``with block as ...:`` or ``with expr as ...:`` with no arguments. - -In the body of ``let_syntax``, a bare name substitution is invoked by name (just like a variable). A template substitution is invoked like a function call. Just like in an actual function call, when the template is substituted, any instances of its formal parameters in the definition get replaced by the argument values from the "call" site; but ``let_syntax`` performs this at macro-expansion time, and the "value" is a snippet of code. - -Note each instance of the same formal parameter (in the definition) gets a fresh copy of the corresponding argument value. In other words, in the example above, each ``a`` in the body of ``twice`` separately expands to a copy of whatever code was given as the positional argument ``a``. - -When used as a block macro, there are furthermore two capture modes: *block of statements*, and *single expression*. (The single expression can be an explicit ``do[]`` if multiple expressions are needed.) When invoking substitutions, keep in mind Python's usual rules regarding where statements or expressions may appear. - -(If you know about Python ASTs, don't worry about the ``ast.Expr`` wrapper needed to place an expression in a statement position; this is handled automatically.) +
+ Expand for further explanation + +>There are two kinds of substitutions: *bare name* and *template*. A bare name substitution has no parameters. A template substitution has positional parameters. (Named parameters, ``*args``, ``**kwargs`` and default values are currently **not** supported.) +> +>When used as an expr macro, the formal parameter declaration is placed where it belongs; on the name side (LHS) of the binding. In the above example, ``f(a)`` is a template with a formal parameter ``a``. But when used as a block macro, the formal parameters are declared on the ``block`` or ``expr`` "context manager" due to syntactic limitations of Python. To define a bare name substitution, just use ``with block as ...:`` or ``with expr as ...:`` with no arguments. +> +>In the body of ``let_syntax``, a bare name substitution is invoked by name (just like a variable). A template substitution is invoked like a function call. Just like in an actual function call, when the template is substituted, any instances of its formal parameters in the definition get replaced by the argument values from the "call" site; but ``let_syntax`` performs this at macro-expansion time, and the "value" is a snippet of code. +> +>Note each instance of the same formal parameter (in the definition) gets a fresh copy of the corresponding argument value. In other words, in the example above, each ``a`` in the body of ``twice`` separately expands to a copy of whatever code was given as the positional argument ``a``. +> +>When used as a block macro, there are furthermore two capture modes: *block of statements*, and *single expression*. (The single expression can be an explicit ``do[]`` if multiple expressions are needed.) When invoking substitutions, keep in mind Python's usual rules regarding where statements or expressions may appear. +> +>(If you know about Python ASTs, don't worry about the ``ast.Expr`` wrapper needed to place an expression in a statement position; this is handled automatically.) +
**HINT**: If you get a compiler error that some sort of statement was encountered where an expression was expected, check your uses of ``let_syntax``. The most likely reason is that a substitution is trying to splice a block of statements into an expression position. -Expansion of ``let_syntax`` is a two-step process: - - - First, template substitutions. - - Then, bare name substitutions, applied to the result of the first step. - -This design is to avoid accidental substitutions into formal parameters of templates (that would usually break the template, resulting at best in a mysterious error, and at worst silently doing something unexpected), if the name of a formal parameter happens to match one of the currently active bare name substitutions. - -Within each step, the substitutions are applied **in definition order**: - - - If the bindings are ``((x, y), (y, z))``, then an ``x`` at the use site transforms to ``z``. So does a ``y`` at the use site. - - But if the bindings are ``((y, z), (x, y))``, then an ``x`` at the use site transforms to ``y``, and only an explicit ``y`` at the use site transforms to ``z``. +
+ Expansion of ``let_syntax`` is a two-step process: -Even in block templates, arguments are always expressions, because invoking a template uses the function-call syntax. But names and calls are expressions, so a previously defined substitution (whether bare name or an invocation of a template) can be passed as an argument just fine. Definition order is then important; consult the rules above. +> - First, template substitutions. +> - Then, bare name substitutions, applied to the result of the first step. +> +>This design is to avoid accidental substitutions into formal parameters of templates (that would usually break the template, resulting at best in a mysterious error, and at worst silently doing something unexpected), if the name of a formal parameter happens to match one of the currently active bare name substitutions. +> +>Within each step, the substitutions are applied **in definition order**: +> +> - If the bindings are ``((x, y), (y, z))``, then an ``x`` at the use site transforms to ``z``. So does a ``y`` at the use site. +> - But if the bindings are ``((y, z), (x, y))``, then an ``x`` at the use site transforms to ``y``, and only an explicit ``y`` at the use site transforms to ``z``. +> +>Even in block templates, arguments are always expressions, because invoking a template uses the function-call syntax. But names and calls are expressions, so a previously defined substitution (whether bare name or an invocation of a template) can be passed as an argument just fine. Definition order is then important; consult the rules above. +
-It is allowed to nest ``let_syntax``. Lexical scoping is supported (inner definitions of substitutions shadow outer ones). +Nesting ``let_syntax`` is allowed. Lexical scoping is supported (inner definitions of substitutions shadow outer ones). When used as an expr macro, all bindings are registered first, and then the body is evaluated. When used as a block macro, a new binding (substitution declaration) takes effect from the next statement onward, and remains active for the lexically remaining part of the ``with let_syntax:`` block. @@ -446,11 +452,9 @@ This was inspired by Racket's [``let-syntax``](https://docs.racket-lang.org/refe As a bonus, we provide classical simple ``let`` and ``letseq``, wholly implemented as AST transformations, providing true lexical variables but no assignment support (because in Python, assignment is a statement) or multi-expression body support. Just like in Lisps, this version of ``letseq`` (Scheme/Racket ``let*``) expands into a chain of nested ``let`` expressions, which expand to lambdas. -These are provided as a curiosity, and not designed to work together with the rest of the macros; for that, use the above ``let``, ``letseq`` and ``letrec`` from the module ``unpythonic.syntax``. - -*Changed in v0.11.0.* The barebones constructs now live in the separate module ``unpythonic.syntax.simplelet``, and are imported like ``from unpythonic.syntax.simplelet import macros, let, letseq``. - +These are provided in the separate module ``unpythonic.syntax.simplelet``, and are imported: +``from unpythonic.syntax.simplelet import macros, let, letseq``. ## Sequencing From 5e65945d733bf32b526ec70bb0058092b2881eed Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 15:46:54 -0700 Subject: [PATCH 26/31] Update README.md --- macro_extras/README.md | 172 ++++++++++++++++++----------------------- 1 file changed, 74 insertions(+), 98 deletions(-) diff --git a/macro_extras/README.md b/macro_extras/README.md index ed4a893c..883b8285 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -119,7 +119,7 @@ let[x + y + z, These syntaxes take no macro arguments; both the let-body and the bindings are placed inside the same ``[...]``.
-Expand for further explanation +Expand for Further Explanation >Semantically, these do the exact same thing as the original lispy syntax: the bindings are evaluated first, and then the body is evaluated with the bindings in place. The purpose of the second variant (the *let-where*) is just readability; sometimes it looks clearer to place the body expression first, and only then explain what the symbols in it mean. > @@ -169,7 +169,7 @@ let[[y << x + y, # v0.12.0+ (y, 2))] ``` -The let macros implement this by inserting a ``do[...]`` (see below). In a multiple-expression body, also an internal definition context exists for local variables that are not part of the ``let``; see ``do`` for details. +The let macros implement this by inserting a ``do[...]`` (see below). In a multiple-expression body, also an internal definition context exists for local variables that are not part of the ``let``; see [``do``](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style) for details. Only the outermost set of extra brackets is interpreted as a multiple-expression body; the rest are interpreted as usual, as lists. If you need to return a literal list from a ``let`` form with only one body expression, use three sets of brackets: @@ -389,7 +389,7 @@ with let_syntax: After macro expansion completes, ``let_syntax`` has zero runtime overhead; it completely disappears in macro expansion.
- Expand for further explanation + Expand for Further Explanation >There are two kinds of substitutions: *bare name* and *template*. A bare name substitution has no parameters. A template substitution has positional parameters. (Named parameters, ``*args``, ``**kwargs`` and default values are currently **not** supported.) > @@ -452,7 +452,7 @@ This was inspired by Racket's [``let-syntax``](https://docs.racket-lang.org/refe As a bonus, we provide classical simple ``let`` and ``letseq``, wholly implemented as AST transformations, providing true lexical variables but no assignment support (because in Python, assignment is a statement) or multi-expression body support. Just like in Lisps, this version of ``letseq`` (Scheme/Racket ``let*``) expands into a chain of nested ``let`` expressions, which expand to lambdas. -These are provided in the separate module ``unpythonic.syntax.simplelet``, and are imported: +These are provided in the separate module ``unpythonic.syntax.simplelet``, import them with the line: ``from unpythonic.syntax.simplelet import macros, let, letseq``. @@ -464,7 +464,7 @@ Macros that run multiple expressions, in sequence, in place of one expression. We provide an ``expr`` macro wrapper for ``unpythonic.seq.do``, with some extra features. -This essentially allows writing imperative code in any expression position. For an `if-elif-else` conditional, see `cond`; for loops, see the functions in [`unpythonic.fploop`](../unpythonic/fploop.py) (esp. `looped`). +This essentially allows writing imperative code in any expression position. For an `if-elif-else` conditional, see [`cond`](#cond-the-missing-elif-for-a-if-p-else-b); for loops, see the functions in [`unpythonic.fploop`](../unpythonic/fploop.py) (esp. `looped`). ```python from unpythonic.syntax import macros, do, local, delete @@ -483,13 +483,9 @@ y = do[local[a << 17], True] ``` -*Changed in v0.14.0.* Added ``delete[...]`` to allow deleting a ``local[...]`` binding. This uses ``env.pop()`` internally, so a ``delete[...]`` returns the value the deleted local variable had at the time of deletion. (So if you manually use the ``do()`` function in some code without macros, feel free to ``env.pop()`` in a do-item if needed.) +Local variables are declared and initialized with ``local[var << value]``, where ``var`` is a bare name. To explicitly denote "no value", just use ``None``. ``delete[...]`` allows deleting a ``local[...]`` binding. This uses ``env.pop()`` internally, so a ``delete[...]`` returns the value the deleted local variable had at the time of deletion. (So if you manually use the ``do()`` function in some code without macros, feel free to ``env.pop()`` in a do-item if needed.) -*Changed in v0.12.0* ``local(...)`` is now ``local[...]`` (note brackets), to emphasize it's related to macros. - -*Changed in v0.11.0.* ``localdef(...)`` is now just ``local(...)``. Shorter, and more descriptive, as it defines a local name, not a function. - -Local variables are declared and initialized with ``local[var << value]``, where ``var`` is a bare name. To explicitly denote "no value", just use ``None``. A ``local`` declaration comes into effect in the expression following the one where it appears, capturing the declared name as a local variable for the **lexically** remaining part of the ``do``. In a ``local``, the RHS still sees the previous bindings, so this is valid (although maybe not readable): +A ``local`` declaration comes into effect in the expression following the one where it appears, capturing the declared name as a local variable for the **lexically** remaining part of the ``do``. In a ``local``, the RHS still sees the previous bindings, so this is valid (although maybe not readable): ```python result = [] @@ -501,19 +497,20 @@ assert result == [[], [1]] Already declared local variables are updated with ``var << value``. Updating variables in lexically outer environments (e.g. a ``let`` surrounding a ``do``) uses the same syntax. -The reason we require local variables to be declared is to allow write access to lexically outer environments. - -Assignments are recognized anywhere inside the ``do``; but note that any ``let`` constructs nested *inside* the ``do``, that define variables of the same name, will (inside the ``let``) shadow those of the ``do`` - as expected of lexical scoping. - -The necessary boilerplate (notably the ``lambda e: ...`` wrappers) is inserted automatically, so the expressions in a ``do[]`` are only evaluated when the underlying ``seq.do`` actually runs. - -When running, ``do`` behaves like ``letseq``; assignments **above** the current line are in effect (and have been performed in the order presented). Re-assigning to the same name later overwrites (this is afterall an imperative tool). +
+The reason we require local variables to be declared is to allow write access to lexically outer environments. -We also provide a ``do0`` macro, which returns the value of the first expression, instead of the last. +>Assignments are recognized anywhere inside the ``do``; but note that any ``let`` constructs nested *inside* the ``do``, that define variables of the same name, will (inside the ``let``) shadow those of the ``do`` - as expected of lexical scoping. +> +>The necessary boilerplate (notably the ``lambda e: ...`` wrappers) is inserted automatically, so the expressions in a ``do[]`` are only evaluated when the underlying ``seq.do`` actually runs. +> +>When running, ``do`` behaves like ``letseq``; assignments **above** the current line are in effect (and have been performed in the order presented). Re-assigning to the same name later overwrites (this is afterall an imperative tool). +> +>We also provide a ``do0`` macro, which returns the value of the first expression, instead of the last. +
**CAUTION**: ``do[]`` supports local variable deletion, but the ``let[]`` constructs don't, by design. When ``do[]`` is used implicitly with the extra bracket syntax, any ``delete[]`` refers to the scope of the implicit ``do[]``, not any surrounding ``let[]`` scope. - ## Tools for lambdas Macros that introduce additional features for Python's lambdas. @@ -582,7 +579,7 @@ Lexically inside a ``with namedlambda`` block, any literal ``lambda`` that is as Decorated lambdas are also supported, as is a ``curry`` (manual or auto) where the last argument is a lambda. The latter is a convenience feature, mainly for applying parametric decorators to lambdas. See [the unit tests](../unpythonic/syntax/test/test_lambdatools.py) for detailed examples. -The naming is performed using the function ``unpythonic.misc.namelambda``, which will return a renamed copy with its ``__name__``, ``__qualname__`` and ``__code__.co_name`` changed. +The naming is performed using the function ``unpythonic.misc.namelambda``, which will return a modified copy with its ``__name__``, ``__qualname__`` and ``__code__.co_name`` changed; the original function object is not mutated. **Supported assignment forms**: @@ -591,14 +588,10 @@ The naming is performed using the function ``unpythonic.misc.namelambda``, which - Expression-assignment to an unpythonic environment, ``f << (lambda ...: ...)`` - Let bindings, ``let[(f, (lambda ...: ...)) in ...]``, using any let syntax supported by unpythonic (here using the haskelly let-in just as an example). + - Env-assignments are now processed lexically, just like regular assignments. Added support for let-bindings. Support for other forms of assignment might or might not be added in a future version. -*Changed in v0.13.0.* Env-assignments are now processed lexically, just like regular assignments. Added support for let-bindings. - -*Changed in v0.13.1.* Now the return value of ``namelambda``, which this uses internally, is a modified copy; the original function object is not mutated. - - ### ``quicklambda``: combo with ``macropy.quick_lambda`` To be able to transform correctly, the block macros in ``unpythonic.syntax`` that transform lambdas (e.g. ``multilambda``, ``tco``) need to see all ``lambda`` definitions written with Python's standard ``lambda``. However, the highly useful ``macropy.quick_lambda`` uses the syntax ``f[...]``, which (to the analyzer) does not look like a lambda definition. @@ -633,8 +626,6 @@ with quicklambda, tco: ### ``envify``: make formal parameters live in an unpythonic ``env`` -*Added in v0.14.0.* - When a function whose definition (``def`` or ``lambda``) is lexically inside a ``with envify`` block is entered, it copies references to its arguments into an unpythonic ``env``. At macro expansion time, all references to the formal parameters are redirected to that environment. This allows rebinding, from an expression position, names that were originally the formal parameters. Wherever could *that* be useful? For an illustrative caricature, consider [PG's accumulator puzzle](http://paulgraham.com/icad.html). The modern pythonic solution: @@ -680,8 +671,6 @@ with autoreturn, envify: The ``with`` block adds a few elements, but if desired, it can be refactored into the definition of a custom dialect in [Pydialect](https://github.com/Technologicat/pydialect). - - ## Language features To boldly go where Python without macros just won't. Changing the rules by code-walking and making significant rewrites. @@ -714,15 +703,12 @@ assert add3(1)(2)(3) == 6 - Function calls are autocurried, and run ``unpythonic.fun.curry`` in a special mode that no-ops on uninspectable functions (triggering a standard function call with the given args immediately) instead of raising ``TypeError`` as usual. -**CAUTION**: Some builtins are uninspectable or may report their arities incorrectly; in those cases, ``curry`` may fail, occasionally in mysterious ways. The function ``unpythonic.arity.arities``, which ``unpythonic.fun.curry`` internally uses, has a workaround for the inspectability problems of all builtins in the top-level namespace (as of Python 3.7), but e.g. methods of builtin types are not handled. - -*Changed in v0.13.0.* The special mode for uninspectables used to be enabled for the dynamic extent of the ``with curry`` block; the new lexical behavior is easier to reason about. Also, manual uses of the ``curry`` decorator (on both ``def`` and ``lambda``) are now detected, and in such cases the macro now skips adding the decorator. +**CAUTION**: Some built-ins are uninspectable or may report their arities incorrectly; in those cases, ``curry`` may fail, occasionally in mysterious ways. The function ``unpythonic.arity.arities``, which ``unpythonic.fun.curry`` internally uses, has a workaround for the inspectability problems of all built-ins in the top-level namespace (as of Python 3.7), but e.g. methods of built-in types are not handled. +The special mode for uninspectables used to be enabled for the dynamic extent of the ``with curry`` block; the new lexical behavior is easier to reason about. Also, manual uses of the ``curry`` decorator (on both ``def`` and ``lambda``) are now detected, and in such cases the macro now skips adding the decorator. ### ``lazify``: call-by-need for Python -*Added in v0.13.0.* - Also known as *lazy functions*. Like [lazy/racket](https://docs.racket-lang.org/lazy/index.html), but for Python. Note if you want *lazy sequences* instead, Python already provides those; just use the generator facility (and decorate your gfunc with ``unpythonic.gmemoize`` if needed). Lazy function example: @@ -1002,55 +988,57 @@ with continuations: print(fail()) ``` -Code within a ``with continuations`` block is treated specially. Roughly: - - - Each function definition (``def`` or ``lambda``) in a ``with continuations`` block has an implicit formal parameter ``cc``, **even if not explicitly declared** in the formal parameter list. - - The continuation machinery will set the default value of ``cc`` to the default continuation (``identity``), which just returns its arguments. - - The default value allows these functions to be called also normally without passing a ``cc``. In effect, the function will then return normally. - - If ``cc`` is not declared explicitly, it is implicitly declared as a by-name-only parameter named ``cc``, and the default value is set automatically. - - If ``cc`` is declared explicitly, the default value is set automatically if ``cc`` is in a position that can accept a default value, and no default has been set by the user. - - Positions that can accept a default value are the last positional parameter that has no default, and a by-name-only parameter in any syntactically allowed position. - - Having a hidden parameter is somewhat magic, but overall improves readability, as this allows declaring ``cc`` only where actually explicitly needed. - - **CAUTION**: Usability trap: in nested function definitions, each ``def`` and ``lambda`` comes with **its own** implicit ``cc``. - - In the above ``amb`` example, the local variable is named ``ourcc``, so that the continuation passed in from outside (into the ``lambda``, by closure) will have a name different from the ``cc`` implicitly introduced by the ``lambda`` itself. - - This is possibly subject to change in a future version (pending the invention of a better API), but for now just be aware of this gotcha. - - Beside ``cc``, there's also a mechanism to keep track of the captured tail of a computation, which is important to have edge cases work correctly. See the note on **pcc** (*parent continuation*) in the docstring of ``unpythonic.syntax.continuations``, and [the pictures](callcc_topology.pdf). - - - In a function definition inside the ``with continuations`` block: - - Most of the language works as usual; especially, any non-tail function calls can be made as usual. - - ``return value`` or ``return v0, ..., vn`` is actually a tail-call into ``cc``, passing the given value(s) as arguments. - - As in other parts of ``unpythonic``, returning a tuple means returning multiple-values. - - This is important if the return value is received by the assignment targets of a ``call_cc[]``. If you get a ``TypeError`` concerning the arguments of a function with a name ending in ``_cont``, check your ``call_cc[]`` invocations and the ``return`` in the call_cc'd function. - - ``return func(...)`` is actually a tail-call into ``func``, passing along (by default) the current value of ``cc`` to become its ``cc``. - - Hence, the tail call is inserted between the end of the current function body and the start of the continuation ``cc``. - - To override which continuation to use, you can specify the ``cc=...`` kwarg, as in ``return func(..., cc=mycc)``. - - The ``cc`` argument, if passed explicitly, **must be passed by name**. - - **CAUTION**: This is **not** enforced, as the machinery does not analyze positional arguments in any great detail. The machinery will most likely break in unintuitive ways (or at best, raise a mysterious ``TypeError``) if this rule is violated. - - The function ``func`` must be a defined in a ``with continuations`` block, so that it knows what to do with the named argument ``cc``. - - Attempting to tail-call a regular function breaks the TCO chain and immediately returns to the original caller (provided the function even accepts a ``cc`` named argument). - - Be careful: ``xs = list(args); return xs`` and ``return list(args)`` mean different things. - - TCO is automatically applied to these tail calls. This uses the exact same machinery as the ``tco`` macro. - - - The ``call_cc[]`` statement essentially splits its use site into *before* and *after* parts, where the *after* part (the continuation) can be run a second and further times, by later calling the callable that represents the continuation. This makes a computation resumable from a desired point. - - The continuation is essentially a closure. - - Just like in Scheme/Racket, only the control state is checkpointed by ``call_cc[]``; any modifications to mutable data remain. - - Assignment targets can be used to get the return value of the function called by ``call_cc[]``. - - Just like in Scheme/Racket's ``call/cc``, the values that get bound to the ``call_cc[]`` assignment targets on second and further calls (when the continuation runs) are the arguments given to the continuation when it is called (whether implicitly or manually). - - A first-class reference to the captured continuation is available in the function called by ``call_cc[]``, as its ``cc`` argument. - - The continuation is a function that takes positional arguments, plus a named argument ``cc``. - - The call signature for the positional arguments is determined by the assignment targets of the ``call_cc[]``. - - The ``cc`` parameter is there only so that a continuation behaves just like any continuation-enabled function when tail-called, or when later used as the target of another ``call_cc[]``. - - Basically everywhere else, ``cc`` points to the identity function - the default continuation just returns its arguments. - - This is unlike in Scheme or Racket, which implicitly capture the continuation at every expression. - - Inside a ``def``, ``call_cc[]`` generates a tail call, thus terminating the original (parent) function. (Hence ``call_ec`` does not combo well with this.) - - At the top level of the ``with continuations`` block, ``call_cc[]`` generates a normal call. In this case there is no return value for the block (for the continuation, either), because the use site of the ``call_cc[]`` is not inside a function. +
+ Code within a ``with continuations`` block is treated specially. Roughly: + +> - Each function definition (``def`` or ``lambda``) in a ``with continuations`` block has an implicit formal parameter ``cc``, **even if not explicitly declared** in the formal parameter list. +> - The continuation machinery will set the default value of ``cc`` to the default continuation (``identity``), which just returns its arguments. +> - The default value allows these functions to be called also normally without passing a ``cc``. In effect, the function will then return normally. +> - If ``cc`` is not declared explicitly, it is implicitly declared as a by-name-only parameter named ``cc``, and the default value is set automatically. +> - If ``cc`` is declared explicitly, the default value is set automatically if ``cc`` is in a position that can accept a default value, and no default has been set by the user. +> - Positions that can accept a default value are the last positional parameter that has no default, and a by-name-only parameter in any syntactically allowed position. +> - Having a hidden parameter is somewhat magic, but overall improves readability, as this allows declaring ``cc`` only where actually explicitly needed. +> - **CAUTION**: Usability trap: in nested function definitions, each ``def`` and ``lambda`` comes with **its own** implicit ``cc``. +> - In the above ``amb`` example, the local variable is named ``ourcc``, so that the continuation passed in from outside (into the ``lambda``, by closure) will have a name different from the ``cc`` implicitly introduced by the ``lambda`` itself. +> - This is possibly subject to change in a future version (pending the invention of a better API), but for now just be aware of this gotcha. +> - Beside ``cc``, there's also a mechanism to keep track of the captured tail of a computation, which is important to have edge cases work correctly. See the note on **pcc** (*parent continuation*) in the docstring of ``unpythonic.syntax.continuations``, and [the pictures](callcc_topology.pdf). +> +> - In a function definition inside the ``with continuations`` block: +> - Most of the language works as usual; especially, any non-tail function calls can be made as usual. +> - ``return value`` or ``return v0, ..., vn`` is actually a tail-call into ``cc``, passing the given value(s) as arguments. +> - As in other parts of ``unpythonic``, returning a tuple means returning multiple-values. +> - This is important if the return value is received by the assignment targets of a ``call_cc[]``. If you get a ``TypeError`` concerning the arguments of a function with a name ending in ``_cont``, check your ``call_cc[]`` invocations and the ``return`` in the call_cc'd function. +> - ``return func(...)`` is actually a tail-call into ``func``, passing along (by default) the current value of ``cc`` to become its ``cc``. +> - Hence, the tail call is inserted between the end of the current function body and the start of the continuation ``cc``. +> - To override which continuation to use, you can specify the ``cc=...`` kwarg, as in ``return func(..., cc=mycc)``. +> - The ``cc`` argument, if passed explicitly, **must be passed by name**. +> - **CAUTION**: This is **not** enforced, as the machinery does not analyze positional arguments in any great detail. The machinery will most likely break in unintuitive ways (or at best, raise a mysterious ``TypeError``) if this rule is violated. +> - The function ``func`` must be a defined in a ``with continuations`` block, so that it knows what to do with the named argument ``cc``. +> - Attempting to tail-call a regular function breaks the TCO chain and immediately returns to the original caller (provided the function even accepts a ``cc`` named argument). +> - Be careful: ``xs = list(args); return xs`` and ``return list(args)`` mean different things. +> - TCO is automatically applied to these tail calls. This uses the exact same machinery as the ``tco`` macro. +> +> - The ``call_cc[]`` statement essentially splits its use site into *before* and *after* parts, where the *after* part (the continuation) can be run a second and further times, by later calling the callable that represents the continuation. This makes a computation resumable from a desired point. +> - The continuation is essentially a closure. +> - Just like in Scheme/Racket, only the control state is checkpointed by ``call_cc[]``; any modifications to mutable data remain. +> - Assignment targets can be used to get the return value of the function called by ``call_cc[]``. +> - Just like in Scheme/Racket's ``call/cc``, the values that get bound to the ``call_cc[]`` assignment targets on second and further calls (when the continuation runs) are the arguments given to the continuation when it is called (whether implicitly or manually). +> - A first-class reference to the captured continuation is available in the function called by ``call_cc[]``, as its ``cc`` argument. +> - The continuation is a function that takes positional arguments, plus a named argument ``cc``. +> - The call signature for the positional arguments is determined by the assignment targets of the ``call_cc[]``. +> - The ``cc`` parameter is there only so that a continuation behaves just like any continuation-enabled function when tail-called, or when later used as the target of another ``call_cc[]``. +> - Basically everywhere else, ``cc`` points to the identity function - the default continuation just returns its arguments. +> - This is unlike in Scheme or Racket, which implicitly capture the continuation at every expression. +> - Inside a ``def``, ``call_cc[]`` generates a tail call, thus terminating the original (parent) function. (Hence ``call_ec`` does not combo well with this.) +> - At the top level of the ``with continuations`` block, ``call_cc[]`` generates a normal call. In this case there is no return value for the block (for the continuation, either), because the use site of the ``call_cc[]`` is not inside a function. +
#### Differences between ``call/cc`` and certain other language features - Unlike **generators**, ``call_cc[]`` allows resuming also multiple times from an earlier checkpoint, even after execution has already proceeded further. Generators can be easily built on top of ``call/cc``. [Python version](../unpythonic/syntax/test/test_conts_gen.py), [Racket version](https://github.com/Technologicat/python-3-scicomp-intro/blob/master/examples/beyond_python/generator.rkt). - The Python version is a pattern that could be packaged into a macro with MacroPy; the Racket version has been packaged as a macro. - Both versions are just demonstrations for teaching purposes. In production code, use the language's native functionality. - - Python's builtin generators have no restriction on where ``yield`` can be placed, and provide better performance. + - Python's built-in generators have no restriction on where ``yield`` can be placed, and provide better performance. - Racket's standard library provides [generators](https://docs.racket-lang.org/reference/Generators.html). - Unlike **exceptions**, which only perform escapes, ``call_cc[]`` allows to jump back at an arbitrary time later, also after the dynamic extent of the original function where the ``call_cc[]`` appears. Escape continuations are a special case of continuations, so exceptions can be built on top of ``call/cc``. @@ -1296,9 +1284,7 @@ The ``call_cc[]`` explicitly suggests that these are (almost) the only places wh Write Python almost like Lisp! -Lexically inside a ``with prefix`` block, any literal tuple denotes a function call, unless quoted. The first element is the operator, the rest are arguments. - -*Changed in v0.11.0.* Bindings of the ``let`` macros and the top-level tuple in a ``do[]`` are now left alone, but ``prefix`` recurses inside them (in the case of bindings, on each RHS). +Lexically inside a ``with prefix`` block, any literal tuple denotes a function call, unless quoted. The first element is the operator, the rest are arguments. Bindings of the ``let`` macros and the top-level tuple in a ``do[]`` are now left alone, but ``prefix`` recurses inside them (in the case of bindings, on each RHS). The rest is best explained by example: @@ -1419,9 +1405,7 @@ If you wish to omit ``return`` in tail calls, this comboes with ``tco``; just ap ### ``forall``: nondeterministic evaluation -*Changed in v0.11.0.* The previous ``forall_simple`` has been renamed ``forall``; the macro wrapper for the hacky function version of ``forall`` is gone. This change has the effect of changing the error raised by an undefined name in a ``forall`` section; previously it was ``AttributeError``, now it is ``NameError``. - -Behaves the same as the multiple-body-expression tuple comprehension ``unpythonic.amb.forall``, but implemented purely by AST transformation, with real lexical variables. This is essentially a MacroPy implementation of Haskell's do-notation for Python, specialized to the List monad (but the code is generic and very short; see ``unpythonic.syntax.forall``). +Behaves the same as the multiple-body-expression tuple comprehension ``unpythonic.amb.forall``, but implemented purely by AST transformation, with real lexical variables. This is essentially a MacroPy implementation of Haskell's do-notation for Python, specialized to the List monad (but the code is generic and very short; see ``unpythonic.syntax.forall``). ```python from unpythonic.syntax import macros, forall, insist, deny @@ -1446,6 +1430,8 @@ Assignment (with List-monadic magic) is ``var << iterable``. It is only valid at ``insist`` and ``deny`` are not really macros; they are just the functions from ``unpythonic.amb``, re-exported for convenience. +The error raised by an undefined name in a ``forall`` section is ``NameError``. + ## Convenience features @@ -1466,7 +1452,7 @@ print(answer(42)) Syntax is ``cond[test1, then1, test2, then2, ..., otherwise]``. Expansion raises an error if the ``otherwise`` branch is missing. -*Added in v0.10.0.* Any part of ``cond`` may have multiple expressions by surrounding it with brackets: +Any part of ``cond`` may have multiple expressions by surrounding it with brackets: ```python cond[[pre1, ..., test1], [post1, ..., then1], @@ -1492,7 +1478,7 @@ aif[2*21, Syntax is ``aif[test, then, otherwise]``. The magic identifier ``it`` refers to the test result while (lexically) inside the ``aif``, and does not exist outside the ``aif``. -*Added in v0.10.0.* Any part of ``aif`` may have multiple expressions by surrounding it with brackets (implicit ``do[]``): +Any part of ``aif`` may have multiple expressions by surrounding it with brackets (implicit ``do[]``): ```python aif[[pre, ..., test], @@ -1505,10 +1491,6 @@ To denote a single expression that is a literal list, use an extra set of bracke ### ``autoref``: implicitly reference attributes of an object -*Added in v0.14.0.* - -*Changed in v0.14.1.* Added support for nested autoref blocks (lookups are lexically scoped). - Ever wish you could ``with(obj)`` to say ``x`` instead of ``obj.x`` to read attributes of an object? Enter the ``autoref`` block macro: ```python @@ -1527,6 +1509,8 @@ The transformation is applied for names in ``Load`` context only, including name Names in ``Store`` or ``Del`` context are not redirected. To write to or delete attributes of ``o``, explicitly refer to ``o.x``, as usual. +Nested autoref blocks are allowed (lookups are lexically scoped). + Reading with ``autoref`` can be convenient e.g. for data returned by [SciPy's ``.mat`` file loader](https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html). See the [unit tests](../unpythonic/syntax/test/test_autoref.py) for more usage examples. @@ -1534,8 +1518,6 @@ See the [unit tests](../unpythonic/syntax/test/test_autoref.py) for more usage e ### ``dbg``: debug-print expressions with source code -*Added in v0.14.1.* - [DRY](https://en.wikipedia.org/wiki/Don't_repeat_yourself) out your [qnd](https://en.wiktionary.org/wiki/quick-and-dirty) debug printing code. Both block and expression variants are provided: ```python @@ -1586,16 +1568,12 @@ For details on implementing custom debug print functions, see the docstrings of Inspired by the [dbg macro in Rust](https://doc.rust-lang.org/std/macro.dbg.html). - - ## Other Stuff that didn't fit elsewhere. ### ``nb``: silly ultralight math notebook -*Added in v0.13.0.* - Mix regular code with math-notebook-like code in a ``.py`` file. To enable notebook mode, ``with nb``: ```python @@ -1622,8 +1600,6 @@ A custom print function can be supplied as the first positional argument to ``nb Obviously not intended for production use, although is very likely to work anywhere. - - ## Meta Is this just a set of macros, a language extension, or a compiler for a new language that just happens to be implemented in MacroPy, à la *On Lisp*? All of the above, really. From 7f1e954907c7b7d5251707c4aac73934c2a3b4ac Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 27 Aug 2019 15:51:48 -0700 Subject: [PATCH 27/31] Update README.md --- macro_extras/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/macro_extras/README.md b/macro_extras/README.md index 883b8285..b26ae164 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -987,9 +987,9 @@ with continuations: print(fail()) print(fail()) ``` - +Code within a ``with continuations`` block is treated specially.
- Code within a ``with continuations`` block is treated specially. Roughly: + Roughly: > - Each function definition (``def`` or ``lambda``) in a ``with continuations`` block has an implicit formal parameter ``cc``, **even if not explicitly declared** in the formal parameter list. > - The continuation machinery will set the default value of ``cc`` to the default continuation (``identity``), which just returns its arguments. From e1bb64e46bf9f66dfc720ea7ef911c50f9a40735 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 3 Sep 2019 17:59:46 -0700 Subject: [PATCH 28/31] Update features.md --- README.md | 11 +- doc/features.md | 225 +++++++++++++++++------------------------ macro_extras/README.md | 134 +++++++++++++----------- 3 files changed, 172 insertions(+), 198 deletions(-) diff --git a/README.md b/README.md index 46c7c49a..d19c1c95 100644 --- a/README.md +++ b/README.md @@ -8,18 +8,21 @@ Design considerations are based in simplicity, robustness, and with minimal depe **This is semantics, not syntax!** -[Strictly speaking](https://stackoverflow.com/questions/17930267/what-is-the-difference-between-syntax-and-semantics-of-programming-languages), ``True``: we just repurpose Python's existing syntax to give it new meanings. However, in the Racket reference, **a** *syntax* designates a macro, in contrast to a *procedure* (regular function). We provide syntaxes in this particular sense. The name ``unpythonic.syntax`` is also shorter to type than ``unpythonic.semantics``, less abstruse, and close enough to convey the intended meaning. +[Strictly speaking](https://stackoverflow.com/questions/17930267/what-is-the-difference-between-syntax-and-semantics-of-programming-languages), ``True``. We just repurpose Python's existing syntax to give it new meanings. However, in the Racket reference, **a** *syntax* designates a macro, in contrast to a *procedure* (regular function). We provide syntaxes in this particular sense. The name ``unpythonic.syntax`` is also shorter to type than ``unpythonic.semantics``, less obscure, and close enough to convey the intended meaning. If you want custom *syntax* proper, then you may be interested in [Pydialect](https://github.com/Technologicat/pydialect). ### Dependencies -Currently none required; [MacroPy](https://github.com/azazel75/macropy) optional, to enable the syntactic macros. +Currently none required. +[MacroPy](https://github.com/azazel75/macropy) optional, to enable the syntactic macros. ### Documentation -- [pure-Python (basic implementation)](doc/features.md) -- [syntactic macros (advanced implementation)](macro_extras/README.md): the second half of ``unpythonic``. +[pure-Python (basic implementation) documentation](doc/features.md) +[syntactic macros (advanced implementation) documentation](macro_extras/README.md): the second half of ``unpythonic``. + +[Design Notes](doc/design-notes.md): for more insight into the design choices of ``unpythonic`` ## Installation diff --git a/doc/features.md b/doc/features.md index 97cf870c..5e7dfb87 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1,68 +1,64 @@ -This README documents the pure-Python part of ``unpythonic``. See also [documentation for the macros](macro_extras/). - -**Contents**: - - - [**Bindings**](#bindings) - - [``let``, ``letrec``: local bindings in an expression](#let-letrec-local-bindings-in-an-expression) - - [Lispylet: alternative syntax](#lispylet-alternative-syntax) - - [``env``: the environment](#env-the-environment) - - [``assignonce``](#assignonce), a relative of ``env``. - - [``dyn``: dynamic assignment](#dyn-dynamic-assignment) a.k.a. parameterize, special variables, "dynamic scoping". - - - [**Containers**](#containers) - - [``frozendict``: an immutable dictionary](#frozendict-an-immutable-dictionary) - - [`cons` and friends: pythonic lispy linked lists](#cons-and-friends-pythonic-lispy-linked-lists) - - [``box``: a mutable single-item container](#box-a-mutable-single-item-container) - - [Container utilities](#container-utilities): ``get_abcs``, ``in_slice``, ``index_in_slice`` - - - [**Sequencing**](#sequencing), run multiple expressions in any expression position (incl. inside a ``lambda``). - - [``begin``: sequence side effects](#begin-sequence-side-effects) - - [``do``: stuff imperative code into an expression](#do-stuff-imperative-code-into-an-expression) - - [``pipe``, ``piped``, ``lazy_piped``: sequence functions](#pipe-piped-lazy_piped-sequence-functions) - - - [**Batteries**](#batteries) missing from the standard library. - - [**Batteries for functools**](#batteries-for-functools): `memoize`, `curry`, `compose`, `withself` and more. - - [``curry`` and reduction rules](#curry-and-reduction-rules): we provide some extra features for bonus haskellness. - - [**Batteries for itertools**](#batteries-for-itertools): multi-input folds, scans (lazy partial folds); unfold; lazy partial unpacking of iterables, etc. - - [``islice``: slice syntax support for ``itertools.islice``](#islice-slice-syntax-support-for-itertoolsislice) - - [`gmemoize`, `imemoize`, `fimemoize`: memoize generators](#gmemoize-imemoize-fimemoize-memoize-generators), iterables and iterator factories. - - [``fup``: functional update; ``ShadowedSequence``](#fup-functional-update-shadowedsequence): like ``collections.ChainMap``, but for sequences. - - [``view``: writable, sliceable view into a sequence](#view-writable-sliceable-view-into-a-sequence) with scalar broadcast on assignment. - - [``mogrify``: update a mutable container in-place](#mogrify-update-a-mutable-container-in-place) - - [``s``, ``m``, ``mg``: lazy mathematical sequences with infix arithmetic](#s-m-mg-lazy-mathematical-sequences-with-infix-arithmetic) - - - [**Control flow tools**](#control-flow-tools) - - [``trampolined``, ``jump``: tail call optimization (TCO) / explicit continuations](#trampolined-jump-tail-call-optimization-tco--explicit-continuations) - - [``looped``, ``looped_over``: loops in FP style (with TCO)](#looped-looped_over-loops-in-fp-style-with-tco) - - [``gtrampolined``: generators with TCO](#gtrampolined-generators-with-tco): tail-chaining; like ``itertools.chain``, but from inside a generator. - - [``setescape``, ``escape``: escape continuations (ec)](#setescape-escape-escape-continuations-ec) - - [``call_ec``: first-class escape continuations](#call_ec-first-class-escape-continuations), like Racket's ``call/ec``. - - [``forall``: nondeterministic evaluation](#forall-nondeterministic-evaluation), a tuple comprehension with multiple body expressions. - - - [**Other**](#other) - - [``def`` as a code block: ``@call``](#def-as-a-code-block-call): run a block of code immediately, in a new lexical scope. - - [``@callwith``: freeze arguments, choose function later](#callwith-freeze-arguments-choose-function-later) - - [``raisef``: ``raise`` as a function](#raisef-raise-as-a-function), useful inside a lambda. - - [``pack``: multi-arg constructor for tuple](#pack-multi-arg-constructor-for-tuple) - - [``namelambda``, rename a function](#namelambda-rename-a-function) - - [``timer``: a context manager for performance testing](#timer-a-context-manager-for-performance-testing) - - [``getattrrec``, ``setattrrec``: access underlying data in an onion of wrappers](#getattrrec-setattrrec-access-underlying-data-in-an-onion-of-wrappers) - - [``arities``, ``kwargs``: Function signature inspection utilities](#arities-kwargs-function-signature-inspection-utilities) - - [``Popper``: a pop-while iterator](#popper-a-pop-while-iterator) - - - [**Advanced: syntactic macros**](macro_extras/): the second half of ``unpythonic``. - - - [**Meta**](#meta) - - [Design notes](#design-notes) - - [Installation](#installation) - - [License](#license) - - [Acknowledgements](#acknowledgements) - - [Python-related FP resources](#python-related-fp-resources) - -For many examples, see the unit tests located in [unpythonic/test/](unpythonic/test/), the docstrings of the individual features, and this README. - -*This README doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out of date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this README) should agree with the unit tests. So if something fails to work as advertised, check what the tests say - and optionally file an issue on GitHub so that the documentation can be fixed.* - +# Unpythonic: Python meets Lisp and Haskell + +Documentation for the underlying pure-Python API, which can be used directly if you don't want to depend on MacroPy. See also [documentation for syntactic macros](macro_extras/). + +### Features + +[**Bindings**](#bindings) +- [``let``, ``letrec``: local bindings in an expression](#let-letrec-local-bindings-in-an-expression) + - [Lispylet: alternative syntax](#lispylet-alternative-syntax) +- [``env``: the environment](#env-the-environment) +- [``assignonce``](#assignonce), a relative of ``env``. +- [``dyn``: dynamic assignment](#dyn-dynamic-assignment) a.k.a. parameterize, special variables, "dynamic scoping". + +[**Containers**](#containers) +- [``frozendict``: an immutable dictionary](#frozendict-an-immutable-dictionary) +- [`cons` and friends: pythonic lispy linked lists](#cons-and-friends-pythonic-lispy-linked-lists) +- [``box``: a mutable single-item container](#box-a-mutable-single-item-container) +- [Container utilities](#container-utilities): ``get_abcs``, ``in_slice``, ``index_in_slice`` + +[**Sequencing**](#sequencing), run multiple expressions in any expression position (incl. inside a ``lambda``). +- [``begin``: sequence side effects](#begin-sequence-side-effects) +- [``do``: stuff imperative code into an expression](#do-stuff-imperative-code-into-an-expression) +- [``pipe``, ``piped``, ``lazy_piped``: sequence functions](#pipe-piped-lazy_piped-sequence-functions) + +[**Batteries**](#batteries) missing from the standard library. +- [**Batteries for functools**](#batteries-for-functools): `memoize`, `curry`, `compose`, `withself` and more. + - [``curry`` and reduction rules](#curry-and-reduction-rules): we provide some extra features for bonus Haskellness. +- [**Batteries for itertools**](#batteries-for-itertools): multi-input folds, scans (lazy partial folds); unfold; lazy partial unpacking of iterables, etc. +- [``islice``: slice syntax support for ``itertools.islice``](#islice-slice-syntax-support-for-itertoolsislice) +- [`gmemoize`, `imemoize`, `fimemoize`: memoize generators](#gmemoize-imemoize-fimemoize-memoize-generators), iterables and iterator factories. +- [``fup``: functional update; ``ShadowedSequence``](#fup-functional-update-shadowedsequence): like ``collections.ChainMap``, but for sequences. +- [``view``: writable, sliceable view into a sequence](#view-writable-sliceable-view-into-a-sequence) with scalar broadcast on assignment. +- [``mogrify``: update a mutable container in-place](#mogrify-update-a-mutable-container-in-place) +- [``s``, ``m``, ``mg``: lazy mathematical sequences with infix arithmetic](#s-m-mg-lazy-mathematical-sequences-with-infix-arithmetic) + +[**Control flow tools**](#control-flow-tools) +- [``trampolined``, ``jump``: tail call optimization (TCO) / explicit continuations](#trampolined-jump-tail-call-optimization-tco--explicit-continuations) +- [``looped``, ``looped_over``: loops in FP style (with TCO)](#looped-looped_over-loops-in-fp-style-with-tco) +- [``gtrampolined``: generators with TCO](#gtrampolined-generators-with-tco): tail-chaining; like ``itertools.chain``, but from inside a generator. +- [``setescape``, ``escape``: escape continuations (ec)](#setescape-escape-escape-continuations-ec) + - [``call_ec``: first-class escape continuations](#call_ec-first-class-escape-continuations), like Racket's ``call/ec``. +- [``forall``: nondeterministic evaluation](#forall-nondeterministic-evaluation), a tuple comprehension with multiple body expressions. + +[**Other**](#other) +- [``def`` as a code block: ``@call``](#def-as-a-code-block-call): run a block of code immediately, in a new lexical scope. +- [``@callwith``: freeze arguments, choose function later](#callwith-freeze-arguments-choose-function-later) +- [``raisef``: ``raise`` as a function](#raisef-raise-as-a-function), useful inside a lambda. +- [``pack``: multi-arg constructor for tuple](#pack-multi-arg-constructor-for-tuple) +- [``namelambda``, rename a function](#namelambda-rename-a-function) +- [``timer``: a context manager for performance testing](#timer-a-context-manager-for-performance-testing) +- [``getattrrec``, ``setattrrec``: access underlying data in an onion of wrappers](#getattrrec-setattrrec-access-underlying-data-in-an-onion-of-wrappers) +- [``arities``, ``kwargs``: Function signature inspection utilities](#arities-kwargs-function-signature-inspection-utilities) +- [``Popper``: a pop-while iterator](#popper-a-pop-while-iterator) + +[**Advanced: syntactic macros**](macro_extras/): the second half of ``unpythonic``. + +For many examples, see [the unit tests](unpythonic/test/), the docstrings of the individual features, and this guide. + +*This document doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out of date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this guide) should agree with the unit tests. So if something fails to work as advertised, check what the tests say - and optionally file an issue on GitHub so that the documentation can be fixed.* + +**This document is up-to-date for v0.14.1.** ## Bindings @@ -70,7 +66,7 @@ Tools to bind identifiers in ways not ordinarily supported by Python. ### ``let``, ``letrec``: local bindings in an expression -Introduce bindings local to an expression, like Scheme's ``let`` and ``letrec``. For easy-to-use versions of these constructs that look almost like normal Python, see [macros](macro_extras/). This section documents the underlying pure-Python API, which can also be used directly if you don't want to depend on MacroPy. +Introduces bindings local to an expression, like Scheme's ``let`` and ``letrec``. For easy-to-use versions of these constructs that look almost like normal Python, see [our macros](macro_extras/). In ``let``, the bindings are independent (do not see each other). A binding is of the form ``name=value``, where ``name`` is a Python identifier, and ``value`` is any expression. @@ -238,9 +234,7 @@ print(e) # empty! When the `with` block exits, the environment clears itself. The environment instance itself remains alive due to Python's scoping rules. -*Changed in v0.13.1.* ``env`` now provides the ``collections.abc.Mapping`` API. - -*Changed in v0.14.0.* ``env`` now provides also the ``collections.abc.MutableMapping`` API. +``env`` provides the ``collections.abc.Mapping`` and ``collections.abc.MutableMapping`` APIs. ### ``assignonce`` @@ -294,15 +288,18 @@ Dynamic variables are set using `with dyn.let(...)`. There is no `set`, `<<`, un The values of dynamic variables remain bound for the dynamic extent of the `with` block. Exiting the `with` block then pops the stack. Inner dynamic scopes shadow outer ones. Dynamic variables are seen also by code that is outside the lexical scope where the `with dyn.let` resides. -Each thread has its own dynamic scope stack. A newly spawned thread automatically copies the then-current state of the dynamic scope stack **from the main thread** (not the parent thread!). Any copied bindings will remain on the stack for the full dynamic extent of the new thread. Because these bindings are not associated with any `with` block running in that thread, and because aside from the initial copying, the dynamic scope stacks are thread-local, any copied bindings will never be popped, even if the main thread pops its own instances of them. +
+Each thread has its own dynamic scope stack. +A newly spawned thread automatically copies the then-current state of the dynamic scope stack **from the main thread** (not the parent thread!). Any copied bindings will remain on the stack for the full dynamic extent of the new thread. Because these bindings are not associated with any `with` block running in that thread, and because aside from the initial copying, the dynamic scope stacks are thread-local, any copied bindings will never be popped, even if the main thread pops its own instances of them. The source of the copy is always the main thread mainly because Python's `threading` module gives no tools to detect which thread spawned the current one. (If someone knows a simple solution, PRs welcome!) Finally, there is one global dynamic scope shared between all threads, where the default values of dynvars live. The default value is used when ``dyn`` is queried for the value outside the dynamic extent of any ``with dyn.let()`` blocks. Having a default value is convenient for eliminating the need for ``if "x" in dyn`` checks, since the variable will always exist (after the global definition has been executed). +
To create a dynvar and set its default value, use ``make_dynvar``. Each dynamic variable, of the same name, should only have one default set; the (dynamically) latest definition always overwrites. However, we do not prevent overwrites, because in some codebases the same module may run its top-level initialization code multiple times (e.g. if a module has a ``main()`` for tests, and the file gets loaded both as a module and as the main program). -See also the methods of ``dyn``; particularly noteworthy are ``asdict`` and ``items``, which give access to a live view to dyn's contents in a dictionary format (intended for reading only!). The ``asdict`` method essentially creates a ``collections.ChainMap`` instance, while ``items`` is an abbreviation for ``asdict().items()``. The ``dyn`` object itself can also be iterated over; this creates a ``ChainMap`` instance and redirects to iterate over it. +See also the methods of ``dyn``; particularly noteworthy are ``asdict`` and ``items``, which give access to a live view to dyn's contents in a dictionary format (intended for reading only!). The ``asdict`` method essentially creates a ``collections.ChainMap`` instance, while ``items`` is an abbreviation for ``asdict().items()``. The ``dyn`` object itself can also be iterated over; this creates a ``ChainMap`` instance and redirects to iterate over it. ``dyn`` also provides the ``collections.abc.Mapping`` API. To support dictionary-like idioms in iteration, dynvars can alternatively be accessed by subscripting; ``dyn["x"]`` has the same meaning as ``dyn.x``, so you can do things like: @@ -314,11 +311,6 @@ Finally, ``dyn`` supports membership testing as ``"x" in dyn``, ``"y" not in dyn For some more details, see [the unit tests](unpythonic/test/test_dynassign.py). -*Changed in v0.13.0.* The ``asdict`` and ``items`` methods previously returned a snapshot; now they return a live view. - -*Changed in v0.13.1.* ``dyn`` now provides the ``collections.abc.Mapping`` API. - - ## Containers We provide some additional containers. @@ -327,8 +319,6 @@ The class names are lowercase, because these are intended as low-level utility c ### ``frozendict``: an immutable dictionary -*Added in v0.13.0.* - Given the existence of ``dict`` and ``frozenset``, this one is oddly missing from the standard library. ```python @@ -451,15 +441,7 @@ Iterators are supported to walk over linked lists (this also gives sequence unpa Python's builtin ``reversed`` can be applied to linked lists; it will internally ``lreverse`` the list (which is O(n)), then return an iterator to that. The ``llist`` constructor is special-cased so that if the input is ``reversed(some_ll)``, it just returns the internal already reversed list. (This is safe because cons cells are immutable.) -Cons structures can optionally print like in Lisps: - -```python -print(cons(1, 2).lispyrepr()) # --> (1 . 2) -print(ll(1, 2, 3).lispyrepr()) # --> (1 2 3) -print(cons(cons(1, 2), cons(3, 4)).lispyrepr()) # --> ((1 . 2) . (3 . 4)) -``` - -However, by default, they print in a pythonic format suitable for ``eval`` (if all elements are): +Cons structures, by default, print in a pythonic format suitable for ``eval`` (if all elements are): ```python print(cons(1, 2)) # --> cons(1, 2) @@ -467,7 +449,13 @@ print(ll(1, 2, 3)) # --> ll(1, 2, 3) print(cons(cons(1, 2), cons(3, 4)) # --> cons(cons(1, 2), cons(3, 4)) ``` -*Changed in v0.11.0.* In previous versions, the Lisp format was always used for printing. +Cons structures can optionally print like in Lisps: + +```python +print(cons(1, 2).lispyrepr()) # --> (1 . 2) +print(ll(1, 2, 3).lispyrepr()) # --> (1 2 3) +print(cons(cons(1, 2), cons(3, 4)).lispyrepr()) # --> ((1 . 2) . (3 . 4)) +``` For more, see the ``llist`` submodule. @@ -475,15 +463,13 @@ For more, see the ``llist`` submodule. There is no ``copy`` method or ``lcopy`` function, because cons cells are immutable; which makes cons structures immutable. -(However, for example, it is possible to `cons` a new item onto an existing linked list; that's fine because it produces a new cons structure - which shares data with the original, just like in Racket.) +(However, for example, it is possible to ``cons`` a new item onto an existing linked list; that's fine because it produces a new cons structure - which shares data with the original, just like in Racket.) In general, copying cons structures can be error-prone. Given just a starting cell it is impossible to tell if a given instance of a cons structure represents a linked list, or something more general (such as a binary tree) that just happens to locally look like one, along the path that would be traversed if it was indeed a linked list. The linked list iteration strategy does not recurse in the ``car`` half, which could lead to incomplete copying. The tree strategy that recurses on both halves, on the other hand, will flatten nested linked lists and produce also the final ``nil``. -*Added in v0.13.0.* We provide a ``JackOfAllTradesIterator`` as a compromise that understands both trees and linked lists. Nested lists will be flattened, and in a tree any ``nil`` in a ``cdr`` position will be omitted from the output. - -*Changed in v0.13.1.* ``BinaryTreeIterator`` and ``JackOfAllTradesIterator`` now use an explicit data stack instead of implicitly using the call stack for keeping track of the recursion. Hence now all ``cons`` iterators work for arbitrarily deep cons structures without causing Python's call stack to overflow, and without the need for TCO. +We provide a ``JackOfAllTradesIterator`` as a compromise that understands both trees and linked lists. Nested lists will be flattened, and in a tree any ``nil`` in a ``cdr`` position will be omitted from the output. ``BinaryTreeIterator`` and ``JackOfAllTradesIterator`` use an explicit data stack instead of implicitly using the call stack for keeping track of the recursion. All ``cons`` iterators work for arbitrarily deep cons structures without causing Python's call stack to overflow, and without the need for TCO. ``cons`` has no ``collections.abc`` virtual superclasses (except the implicit ``Hashable`` since ``cons`` provides ``__hash__`` and ``__eq__``), because general cons structures do not fit into the contracts represented by membership in those classes. For example, size cannot be known without iterating, and depends on which iteration scheme is used (e.g. ``nil`` dropping, flattening); which scheme is appropriate depends on the content. @@ -492,10 +478,6 @@ The linked list iteration strategy does not recurse in the ``car`` half, which c ### ``box``: a mutable single-item container -*Added in v0.12.0.* - -*Changed in v0.13.0.* The class and the data attribute have been renamed to ``box`` and ``x``, respectively. - No doubt anyone programming in an imperative language has run into the situation caricatured by this highly artificial example: ```python @@ -965,7 +947,7 @@ because ``(g, x, y)`` is just a tuple of ``g``, ``x`` and ``y``. This is by desi - `scanl` is suitable for infinite inputs. - Iteration stops after the final result. - For `scanl`, this is what `foldl` would have returned (if the fold terminates at all, i.e. if the shortest input is finite). - - *Changed in v0.11.0.* For `scanr`, **ordering of output is different from Haskell**: we yield the results in the order they are computed (via a linear process). + - For `scanr`, **ordering of output is different from Haskell**: we yield the results in the order they are computed (via a linear process). - Multiple input iterables and shortest/longest termination supported; same semantics as in `foldl`, `foldr`. - One-input versions with optional init are provided as `scanl1`, `scanr1`. Note ordering of arguments to match `functools.reduce`, but op is still the rackety `op(elt, acc)`. - `rscanl`, `rscanl1` reverse each input and then left-scan. This syncs the **right** ends. @@ -1000,11 +982,11 @@ because ``(g, x, y)`` is just a tuple of ``g``, ``x`` and ``y``. This is by desi - `partition` from `itertools` [recipes](https://docs.python.org/3/library/itertools.html#itertools-recipes). - `rev` is a convenience function that tries `reversed`, and if the input was not a sequence, converts it to a tuple and reverses that. The return value is a `reversed` object. - `scons`: prepend one element to the start of an iterable, return new iterable. ``scons(x, iterable)`` is lispy shorthand for ``itertools.chain((x,), iterable)``, allowing to omit the one-item tuple wrapper. - - `inn`: contains-check (``x in iterable``) with automatic termination for monotonic divergent infinite iterables. *Added in v0.13.1.* + - `inn`: contains-check (``x in iterable``) with automatic termination for monotonic divergent infinite iterables. - Only applicable to monotonic divergent inputs (such as ``primes``). Increasing/decreasing is auto-detected from the first non-zero diff, but the function may fail to terminate if the input is actually not monotonic, or has an upper/lower bound. - - `iindex`: like ``list.index``, but for a general iterable. Consumes the iterable, so only makes sense for memoized inputs. *Added in v0.13.1.* - - `prod`: like the builtin `sum`, but compute the product. Oddly missing from the standard library. *Added in v0.13.1.* - - `window`: sliding length-n window iterator for general iterables. Acts like the well-known [n-gram zip trick](http://www.locallyoptimal.com/blog/2013/01/20/elegant-n-gram-generation-in-python/), but the input can be any iterable. *Added in v0.14.1.* + - `iindex`: like ``list.index``, but for a general iterable. Consumes the iterable, so only makes sense for memoized inputs. + - `prod`: like the builtin `sum`, but compute the product. Oddly missing from the standard library. + - `window`: sliding length-n window iterator for general iterables. Acts like the well-known [n-gram zip trick](http://www.locallyoptimal.com/blog/2013/01/20/elegant-n-gram-generation-in-python/), but the input can be any iterable. Examples: @@ -1109,9 +1091,7 @@ assert tuple(curry(clip, 5, 10, range(20)) == tuple(range(5, 15)) ``` -### ``islice``: slice syntax support for ``itertools.islice`` - -*Added in v0.13.1.* +### ``islice``: slice syntax support for ``itertools.islice` Slice an iterable, using the regular slicing syntax: @@ -1238,7 +1218,7 @@ The only differences are the name of the decorator and ``return`` vs. ``yield fr ### ``fup``: Functional update; ``ShadowedSequence`` -We provide ``ShadowedSequence``, which is a bit like ``collections.ChainMap``, but for sequences, and only two levels (but it's a sequence; instances can be chained). See its docstring for details. +We provide ``ShadowedSequence``, which is a bit like ``collections.ChainMap``, but for sequences, and only two levels (but it's a sequence; instances can be chained). Use ``ShadowedSequence`` for slicing (read-only), equality comparison, ``str`` and ``repr``. Out-of-range read access to a single item emits a meaningful error, like in ``list``. See its docstring for details. The function ``fupdate`` functionally updates sequences and mappings. Whereas ``ShadowedSequence`` reads directly from the original sequences at access time, ``fupdate`` makes a shallow copy (of the same type as the given input sequence) when it finalizes its output. The utility function ``fup`` is a specialization of ``fupdate`` to sequences, and adds support for the standard slicing syntax. @@ -1273,7 +1253,7 @@ Slicing supports negative indices and steps, and default starts, stops and steps When ``fupdate`` constructs its output, the replacement occurs by walking *the input sequence* left-to-right, and pulling an item from the replacement sequence when the given replacement specification so requires. Hence the replacement sequence is not necessarily accessed left-to-right. (In the last example above, ``tuple(range(5))`` was read in the order ``(4, 3, 2, 1, 0)``.) -The replacement sequence must have at least as many items as the slice requires (when applied to the original input). Any extra items in the replacement sequence are simply ignored (so e.g. an infinite ``repeat`` is fine), but if the replacement is too short, ``IndexError`` is raised. (*Changed in v0.13.1.* This was previously ``ValueError``.) +The replacement sequence must have at least as many items as the slice requires (when applied to the original input). Any extra items in the replacement sequence are simply ignored (so e.g. an infinite ``repeat`` is fine), but if the replacement is too short, ``IndexError`` is raised. It is also possible to replace multiple individual items. These are treated as separate specifications, applied left to right (so later updates shadow earlier ones, if updating at the same index): @@ -1347,15 +1327,8 @@ The notation follows the ``unpythonic`` convention that ``<<`` denotes an assign The ``fup`` call is essentially curried. It takes in the sequence to be functionally updated. The object returned by the call accepts a subscript to specify the index or indices. This then returns another object that accepts a left-shift to specify the values. Once the values are provided, the underlying call to ``fupdate`` triggers, and the result is returned. -*Changed in v0.13.1.* Added support to ``ShadowedSequence`` for slicing (read-only), equality comparison, ``str`` and ``repr``. Out-of-range read access to a single item emits a meaningful error, like in ``list``. The utility ``fup`` was previously a macro; now it is a regular function, with slightly changed syntax to accommodate. - - ### ``view``: writable, sliceable view into a sequence -*Added in v0.14.0.* Added the read-only cousin ``roview``, which behaves the same except it has no ``__setitem__`` or ``reverse``. This can be useful for giving read-only access to an internal sequence. The constructor of the writable ``view`` now checks that the input is not read-only (``roview``, or a ``Sequence`` that is not also a ``MutableSequence``) before allowing creation of the writable view. - -*Added in v0.13.1.* - A writable view into a sequence, with slicing, so you can take a slice of a slice (of a slice ...), and it reflects the original both ways: ```python @@ -1391,10 +1364,10 @@ Getting/setting an item (subscripting) checks whether the index cache needs upda The ``unpythonic.collections`` module also provides the ``SequenceView`` and ``MutableSequenceView`` abstract base classes; ``view`` is a ``MutableSequenceView``. +There is the read-only cousin ``roview``, which behaves the same except it has no ``__setitem__`` or ``reverse``. This can be useful for giving read-only access to an internal sequence. The constructor of the writable ``view`` checks that the input is not read-only (``roview``, or a ``Sequence`` that is not also a ``MutableSequence``) before allowing creation of the writable view. -### ``mogrify``: update a mutable container in-place -*Added in v0.13.0.* +### ``mogrify``: update a mutable container in-place Recurse on given container, apply a function to each atom. If the container is mutable, then update in-place; if not, then construct a new copy like ``map`` does. @@ -1434,12 +1407,6 @@ For convenience, we introduce some special cases: ### ``s``, ``m``, ``mg``: lazy mathematical sequences with infix arithmetic -*Added in v0.13.0.* - -*Added in v0.13.1:* ``primes`` and ``fibonacci``. - -*Added in v0.14.0:* ``mg``, a decorator to mathify a gfunc, so that it will ``m()`` the generator instances it makes. Combo with ``imemoize`` for great justice, e.g. ``a = mg(imemoize(s(1, 2, ...)))``. - We provide a compact syntax to create lazy constant, arithmetic, geometric and power sequences: ``s(...)``. Numeric (``int``, ``float``, ``mpmath``) and symbolic (SymPy) formats are supported. We avoid accumulating roundoff error when used with floating-point formats. We also provide arithmetic operation support for iterables (termwise). To make any iterable infix math aware, use ``m(iterable)``. The arithmetic is lazy; it just plans computations, returning a new lazy mathematical sequence. To extract values, iterate over the result. (Note this implies that expressions consisting of thousands of operations will overflow Python's call stack. In practice this shouldn't be a problem.) @@ -1448,6 +1415,8 @@ The function versions of the arithmetic operations (also provided, à la the ``o We provide the [Cauchy product](https://en.wikipedia.org/wiki/Cauchy_product), and its generalization, the diagonal combination-reduction, for two (possibly infinite) iterables. Note ``cauchyprod`` **does not sum the series**; given the input sequences ``a`` and ``b``, the call ``cauchyprod(a, b)`` computes the elements of the output sequence ``c``. +We also provide ``mg``, a decorator to mathify a gfunc, so that it will ``m()`` the generator instances it makes. Combo with ``imemoize`` for great justice, e.g. ``a = mg(imemoize(s(1, 2, ...)))``. + Finally, we provide ready-made generators that yield some common sequences (currently, the Fibonacci numbers and the prime numbers). The prime generator is an FP-ized sieve of Eratosthenes. ```python @@ -2316,7 +2285,7 @@ Note [the grammar](https://docs.python.org/3/reference/grammar.html) requires a ### ``@callwith``: freeze arguments, choose function later -*Added in v0.11.0.* If you need to pass arguments when using ``@call`` as a decorator, use its cousin ``@callwith``: +If you need to pass arguments when using ``@call`` as a decorator, use its cousin ``@callwith``: ```python from unpythonic import callwith @@ -2435,11 +2404,7 @@ assert tuple(myzip(lol)) == ((1, 3, 5), (2, 4, 6)) ### ``namelambda``, rename a function -*Changed in v0.13.0.* Now supports renaming any function object (``isinstance(f, (types.LambdaType, types.FunctionType))``), and will rename a lambda even if it has already been named. - -*Changed in v0.13.1.* Now the return value is a modified copy; the original function object is not mutated. - -For those situations where you return a lambda as a closure, call it much later, and it happens to crash - so you can tell from the stack trace *which* of the *N* lambdas in the codebase it is. +For those situations where you return a lambda as a closure, call it much later, and it happens to crash - so you can tell from the stack trace *which* of the *N* lambdas in the codebase it is. The return value is a modified copy; the original function object is not mutated. You can rename any function object (``isinstance(f, (types.LambdaType, types.FunctionType))``), and it will rename a lambda even if it has already been named. For technical reasons, ``namelambda`` conforms to the parametric decorator API. Usage: @@ -2472,8 +2437,6 @@ The inner lambda does not see the outer's new name; the parent scope names are b ### ``timer``: a context manager for performance testing -*Added in v0.13.0.* - ```python from unpythonic import timer @@ -2492,8 +2455,6 @@ The auto-print mode is a convenience feature to minimize bureaucracy if you just ### ``getattrrec``, ``setattrrec``: access underlying data in an onion of wrappers -*Added in v0.13.1.* - ```python from unpythonic import getattrrec, setattrrec @@ -2574,8 +2535,6 @@ Inspired by various Racket functions such as ``(arity-includes?)`` and ``(proced ### ``Popper``: a pop-while iterator -*Added in v0.14.1.* - Consider this highly artificial example: ```python diff --git a/macro_extras/README.md b/macro_extras/README.md index b26ae164..d041b2cf 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -2,13 +2,15 @@ Our Python language extensions, as syntactic macros, are built on [MacroPy](https://github.com/azazel75/macropy), from the PyPI package ``macropy3``. If you want to take language extension a step further, see our sister project [Pydialect](https://github.com/Technologicat/pydialect). -The unit tests that contain usage examples (located in [unpythonic/syntax/test/](../unpythonic/syntax/test/)) cannot be run directly because macro expansion occurs at import time. Instead, run them via the included [generic MacroPy3 bootstrapper](macropy3). For convenience, ``setup.py`` installs this bootstrapper. +The [unit tests that contain usage examples](../unpythonic/syntax/test/) cannot be run directly because macro expansion occurs at import time. Instead, run them via the included [generic MacroPy3 bootstrapper](macropy3). For convenience, ``setup.py`` installs this bootstrapper. + +### Set Up To use the bootstrapper, run: `./macropy3 -m some.module` (like `python3 -m some.module`); see `-h` for options. -The tests use relative imports; invoke them from the top-level directory of ``unpythonic`` as e.g.: +The tests use relative imports. Invoke them from the top-level directory of ``unpythonic`` as e.g.: ``macro_extras/macropy3 -m unpythonic.syntax.test.test_curry`` @@ -16,61 +18,62 @@ The tests use relative imports; invoke them from the top-level directory of ``un There is no abbreviation for ``memoize(lambda: ...)``, because ``MacroPy`` itself already provides ``lazy`` and ``interned``. -!! **Currently** (12/2018) this requires the latest MacroPy from git HEAD. !! +**Currently (12/2018) this requires the latest MacroPy from git HEAD.** The `macropy3` bootstrapper takes the `-m` option, like `python3 -m mod`. The alternative is to specify a filename positionally, like ``python3 mod.py``. In either case, the bootstrapper will import the module in a special mode that pretends its `__name__ == '__main__'`, to allow using the pythonic conditional main idiom also in macro-enabled code. -**Note: This document is up-to-date for version 0.14.1.** +*This document doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out of date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this guide) should agree with the unit tests. So if something fails to work as advertised, check what the tests say - and optionally file an issue on GitHub so that the documentation can be fixed.* + +**This document is up-to-date for v0.14.1.** ------ +### Features [**Bindings**](#bindings) - - [``let``, ``letseq``, ``letrec`` as macros](#let-letseq-letrec-as-macros); proper lexical scoping, no boilerplate. - - [``dlet``, ``dletseq``, ``dletrec``, ``blet``, ``bletseq``, ``bletrec``: decorator versions](#dlet-dletseq-dletrec-blet-bletseq-bletrec-decorator-versions) - - [``let_syntax``, ``abbrev``: syntactic local bindings](#let_syntax-abbrev-syntactic-local-bindings); splice code at macro expansion time. - - [Bonus: barebones ``let``](#bonus-barebones-let): pure AST transformation of ``let`` into a ``lambda``. +- [``let``, ``letseq``, ``letrec`` as macros](#let-letseq-letrec-as-macros); proper lexical scoping, no boilerplate. +- [``dlet``, ``dletseq``, ``dletrec``, ``blet``, ``bletseq``, ``bletrec``: decorator versions](#dlet-dletseq-dletrec-blet-bletseq-bletrec-decorator-versions) +- [``let_syntax``, ``abbrev``: syntactic local bindings](#let_syntax-abbrev-syntactic-local-bindings); splice code at macro expansion time. +- [Bonus: barebones ``let``](#bonus-barebones-let): pure AST transformation of ``let`` into a ``lambda``. [**Sequencing**](#sequencing) - - [``do`` as a macro: stuff imperative code into an expression, *with style*](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style) +- [``do`` as a macro: stuff imperative code into an expression, *with style*](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style) [**Tools for lambdas**](#tools-for-lambdas) - - [``multilambda``: supercharge your lambdas](#multilambda-supercharge-your-lambdas); multiple expressions, local variables. - - [``namedlambda``: auto-name your lambdas](#namedlambda-auto-name-your-lambdas) by assignment. - - [``quicklambda``: combo with ``macropy.quick_lambda``](#quicklambda-combo-with-macropyquick_lambda) - - [``envify``: make formal parameters live in an unpythonic ``env``](#envify-make-formal-parameters-live-in-an-unpythonic-env) +- [``multilambda``: supercharge your lambdas](#multilambda-supercharge-your-lambdas); multiple expressions, local variables. +- [``namedlambda``: auto-name your lambdas](#namedlambda-auto-name-your-lambdas) by assignment. +- [``quicklambda``: combo with ``macropy.quick_lambda``](#quicklambda-combo-with-macropyquick_lambda) +- [``envify``: make formal parameters live in an unpythonic ``env``](#envify-make-formal-parameters-live-in-an-unpythonic-env) [**Language features**](#language-features) - - [``curry``: automatic currying for Python](#curry-automatic-currying-for-python) - - [``lazify``: call-by-need for Python](#lazify-call-by-need-for-python) - - [Forcing promises manually](#forcing-promises-manually) - - [Binding constructs and auto-lazification](#binding-constructs-and-auto-lazification) - - [Note about TCO](#note-about-tco) - - [``tco``: automatic tail call optimization for Python](#tco-automatic-tail-call-optimization-for-python) - - [TCO and continuations](#tco-and-continuations) - - [``continuations``: call/cc for Python](#continuations-callcc-for-python) - - [Differences between ``call/cc`` and certain other language features](#differences-between-callcc-and-certain-other-language-features) (generators, exceptions) - - [``call_cc`` API reference](#call_cc-api-reference) - - [Combo notes](#combo-notes) - - [Continuations as an escape mechanism](#continuations-as-an-escape-mechanism) - - [What can be used as a continuation?](#what-can-be-used-as-a-continuation) - - [This isn't ``call/cc``!](#this-isnt-callcc) - - [Why this syntax?](#why-this-syntax) - - [``prefix``: prefix function call syntax for Python](#prefix-prefix-function-call-syntax-for-python) - - [``autoreturn``: implicit ``return`` in tail position](#autoreturn-implicit-return-in-tail-position), like in Lisps. - - [``forall``: nondeterministic evaluation](#forall-nondeterministic-evaluation) with monadic do-notation for Python. - - [**Convenience features**](#convenience-features) - - [``cond``: the missing ``elif`` for ``a if p else b``](#cond-the-missing-elif-for-a-if-p-else-b) - - [``aif``: anaphoric if](#aif-anaphoric-if), the test result is ``it``. - - [``autoref``: implicitly reference attributes of an object](#autoref-implicitly-reference-attributes-of-an-object) - - [``dbg``: debug-print expressions with source code](#dbg-debug-print-expressions-with-source-code) - - *Changed in v0.13.1.* The ``fup[]`` macro is gone, and has been replaced with the ``fup`` function, with slightly changed syntax to accommodate. - +- [``curry``: automatic currying for Python](#curry-automatic-currying-for-python) +- [``lazify``: call-by-need for Python](#lazify-call-by-need-for-python) + - [Forcing promises manually](#forcing-promises-manually) + - [Binding constructs and auto-lazification](#binding-constructs-and-auto-lazification) + - [Note about TCO](#note-about-tco) +- [``tco``: automatic tail call optimization for Python](#tco-automatic-tail-call-optimization-for-python) + - [TCO and continuations](#tco-and-continuations) +- [``continuations``: call/cc for Python](#continuations-callcc-for-python) + - [Differences between ``call/cc`` and certain other language features](#differences-between-callcc-and-certain-other-language-features) (generators, exceptions) + - [``call_cc`` API reference](#call_cc-api-reference) + - [Combo notes](#combo-notes) + - [Continuations as an escape mechanism](#continuations-as-an-escape-mechanism) + - [What can be used as a continuation?](#what-can-be-used-as-a-continuation) + - [This isn't ``call/cc``!](#this-isnt-callcc) + - [Why this syntax?](#why-this-syntax) +- [``prefix``: prefix function call syntax for Python](#prefix-prefix-function-call-syntax-for-python) +- [``autoreturn``: implicit ``return`` in tail position](#autoreturn-implicit-return-in-tail-position), like in Lisps. +- [``forall``: nondeterministic evaluation](#forall-nondeterministic-evaluation) with monadic do-notation for Python. + +[**Convenience features**](#convenience-features) +- [``cond``: the missing ``elif`` for ``a if p else b``](#cond-the-missing-elif-for-a-if-p-else-b) +- [``aif``: anaphoric if](#aif-anaphoric-if), the test result is ``it``. +- [``autoref``: implicitly reference attributes of an object](#autoref-implicitly-reference-attributes-of-an-object) +- [``dbg``: debug-print expressions with source code](#dbg-debug-print-expressions-with-source-code) + [**Other**](#other) - - [``nb``: silly ultralight math notebook](#nb-silly-ultralight-math-notebook) - - [**Meta**](#meta) - - [The xmas tree combo](#the-xmas-tree-combo): notes on the macros working together. +- [``nb``: silly ultralight math notebook](#nb-silly-ultralight-math-notebook) + +[**Meta**](#meta) +- [The xmas tree combo](#the-xmas-tree-combo): notes on the macros working together. ## Bindings @@ -102,7 +105,7 @@ The bindings are given as macro arguments as ``((name, value), ...)``, the body #### Alternate syntaxes -The following Haskell-inspired, perhaps more pythonic alternate syntaxes are now available: +The following Haskell-inspired, perhaps more pythonic alternate syntaxes are also available: ```python let[((x, 21), @@ -119,9 +122,9 @@ let[x + y + z, These syntaxes take no macro arguments; both the let-body and the bindings are placed inside the same ``[...]``.
-Expand for Further Explanation - ->Semantically, these do the exact same thing as the original lispy syntax: the bindings are evaluated first, and then the body is evaluated with the bindings in place. The purpose of the second variant (the *let-where*) is just readability; sometimes it looks clearer to place the body expression first, and only then explain what the symbols in it mean. +Semantically, these do the exact same thing as the original lispy syntax: + +>The bindings are evaluated first, and then the body is evaluated with the bindings in place. The purpose of the second variant (the *let-where*) is just readability; sometimes it looks clearer to place the body expression first, and only then explain what the symbols in it mean. > >These syntaxes are valid for all **expression forms** of ``let``, namely: ``let[]``, ``letseq[]``, ``letrec[]``, ``let_syntax[]`` and ``abbrev[]``. The decorator variants (``dlet`` et al., ``blet`` et al.) and the block variants (``with let_syntax``, ``with abbrev``) support only the original lispy syntax, because there the body is in any case placed differently. > @@ -150,7 +153,7 @@ This is essentially special-cased in the ``let`` expander. (If interested in the #### Multiple expressions in body -The `let` constructs can now use a multiple-expression body. The syntax to activate multiple expression mode is an extra set of brackets around the body (like in [`multilambda`](#multilambda-supercharge-your-lambdas)): +The `let` constructs can now use a multiple-expression body. The syntax to activate multiple expression mode is an extra set of brackets around the body ([like in `multilambda`](#multilambda-supercharge-your-lambdas)): ```python let((x, 1), @@ -169,9 +172,9 @@ let[[y << x + y, # v0.12.0+ (y, 2))] ``` -The let macros implement this by inserting a ``do[...]`` (see below). In a multiple-expression body, also an internal definition context exists for local variables that are not part of the ``let``; see [``do``](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style) for details. +The let macros implement this by inserting a ``do[...]`` (see below). In a multiple-expression body, also an internal definition context exists for local variables that are not part of the ``let``; see [``do`` for details](#do-as-a-macro-stuff-imperative-code-into-an-expression-with-style). -Only the outermost set of extra brackets is interpreted as a multiple-expression body; the rest are interpreted as usual, as lists. If you need to return a literal list from a ``let`` form with only one body expression, use three sets of brackets: +Only the outermost set of extra brackets is interpreted as a multiple-expression body. The rest are interpreted as usual, as lists. If you need to return a literal list from a ``let`` form with only one body expression, use three sets of brackets: ```python let((x, 1), @@ -334,6 +337,8 @@ else: Locally splice code at macro expansion time (it's almost like inlining functions): +#### ``let_syntax`` + ```python from unpythonic.syntax import macros, let_syntax, block, expr @@ -389,9 +394,9 @@ with let_syntax: After macro expansion completes, ``let_syntax`` has zero runtime overhead; it completely disappears in macro expansion.
- Expand for Further Explanation - ->There are two kinds of substitutions: *bare name* and *template*. A bare name substitution has no parameters. A template substitution has positional parameters. (Named parameters, ``*args``, ``**kwargs`` and default values are currently **not** supported.) + There are two kinds of substitutions: + +>*Bare name* and *template*. A bare name substitution has no parameters. A template substitution has positional parameters. (Named parameters, ``*args``, ``**kwargs`` and default values are currently **not** supported.) > >When used as an expr macro, the formal parameter declaration is placed where it belongs; on the name side (LHS) of the binding. In the above example, ``f(a)`` is a template with a formal parameter ``a``. But when used as a block macro, the formal parameters are declared on the ``block`` or ``expr`` "context manager" due to syntactic limitations of Python. To define a bare name substitution, just use ``with block as ...:`` or ``with expr as ...:`` with no arguments. > @@ -403,11 +408,12 @@ After macro expansion completes, ``let_syntax`` has zero runtime overhead; it co > >(If you know about Python ASTs, don't worry about the ``ast.Expr`` wrapper needed to place an expression in a statement position; this is handled automatically.)
+

**HINT**: If you get a compiler error that some sort of statement was encountered where an expression was expected, check your uses of ``let_syntax``. The most likely reason is that a substitution is trying to splice a block of statements into an expression position.

- Expansion of ``let_syntax`` is a two-step process: + Expansion of this macro is a two-step process: > - First, template substitutions. > - Then, bare name substitutions, applied to the result of the first step. @@ -421,11 +427,14 @@ After macro expansion completes, ``let_syntax`` has zero runtime overhead; it co > >Even in block templates, arguments are always expressions, because invoking a template uses the function-call syntax. But names and calls are expressions, so a previously defined substitution (whether bare name or an invocation of a template) can be passed as an argument just fine. Definition order is then important; consult the rules above.
+

Nesting ``let_syntax`` is allowed. Lexical scoping is supported (inner definitions of substitutions shadow outer ones). When used as an expr macro, all bindings are registered first, and then the body is evaluated. When used as a block macro, a new binding (substitution declaration) takes effect from the next statement onward, and remains active for the lexically remaining part of the ``with let_syntax:`` block. +#### `abbrev` + The ``abbrev`` macro is otherwise exactly like ``let_syntax``, but it expands in the first pass (outside in). Hence, no lexically scoped nesting, but it has the power to locally rename also macros, because the ``abbrev`` itself expands before any macros invoked in its body. This allows things like: ```python @@ -464,7 +473,7 @@ Macros that run multiple expressions, in sequence, in place of one expression. We provide an ``expr`` macro wrapper for ``unpythonic.seq.do``, with some extra features. -This essentially allows writing imperative code in any expression position. For an `if-elif-else` conditional, see [`cond`](#cond-the-missing-elif-for-a-if-p-else-b); for loops, see the functions in [`unpythonic.fploop`](../unpythonic/fploop.py) (esp. `looped`). +This essentially allows writing imperative code in any expression position. For an `if-elif-else` conditional, [see `cond`](#cond-the-missing-elif-for-a-if-p-else-b); for loops, see [the functions in `unpythonic.fploop`](../unpythonic/fploop.py) (esp. `looped`). ```python from unpythonic.syntax import macros, do, local, delete @@ -508,6 +517,7 @@ Already declared local variables are updated with ``var << value``. Updating var > >We also provide a ``do0`` macro, which returns the value of the first expression, instead of the last.

+

**CAUTION**: ``do[]`` supports local variable deletion, but the ``let[]`` constructs don't, by design. When ``do[]`` is used implicitly with the extra bracket syntax, any ``delete[]`` refers to the scope of the implicit ``do[]``, not any surrounding ``let[]`` scope. @@ -579,7 +589,7 @@ Lexically inside a ``with namedlambda`` block, any literal ``lambda`` that is as Decorated lambdas are also supported, as is a ``curry`` (manual or auto) where the last argument is a lambda. The latter is a convenience feature, mainly for applying parametric decorators to lambdas. See [the unit tests](../unpythonic/syntax/test/test_lambdatools.py) for detailed examples. -The naming is performed using the function ``unpythonic.misc.namelambda``, which will return a modified copy with its ``__name__``, ``__qualname__`` and ``__code__.co_name`` changed; the original function object is not mutated. +The naming is performed using the function ``unpythonic.misc.namelambda``, which will return a modified copy with its ``__name__``, ``__qualname__`` and ``__code__.co_name`` changed. The original function object is not mutated. **Supported assignment forms**: @@ -590,7 +600,7 @@ The naming is performed using the function ``unpythonic.misc.namelambda``, which - Let bindings, ``let[(f, (lambda ...: ...)) in ...]``, using any let syntax supported by unpythonic (here using the haskelly let-in just as an example). - Env-assignments are now processed lexically, just like regular assignments. Added support for let-bindings. -Support for other forms of assignment might or might not be added in a future version. +Support for other forms of assignment may or may not be added in a future version. ### ``quicklambda``: combo with ``macropy.quick_lambda`` @@ -628,7 +638,9 @@ with quicklambda, tco: When a function whose definition (``def`` or ``lambda``) is lexically inside a ``with envify`` block is entered, it copies references to its arguments into an unpythonic ``env``. At macro expansion time, all references to the formal parameters are redirected to that environment. This allows rebinding, from an expression position, names that were originally the formal parameters. -Wherever could *that* be useful? For an illustrative caricature, consider [PG's accumulator puzzle](http://paulgraham.com/icad.html). The modern pythonic solution: +Wherever could *that* be useful? For an illustrative caricature, consider [PG's accumulator puzzle](http://paulgraham.com/icad.html). + +The modern pythonic solution: ```python def foo(n): @@ -987,7 +999,7 @@ with continuations: print(fail()) print(fail()) ``` -Code within a ``with continuations`` block is treated specially. +Code within a ``with continuations`` block is treated specially.

Roughly: @@ -1405,7 +1417,7 @@ If you wish to omit ``return`` in tail calls, this comboes with ``tco``; just ap ### ``forall``: nondeterministic evaluation -Behaves the same as the multiple-body-expression tuple comprehension ``unpythonic.amb.forall``, but implemented purely by AST transformation, with real lexical variables. This is essentially a MacroPy implementation of Haskell's do-notation for Python, specialized to the List monad (but the code is generic and very short; see ``unpythonic.syntax.forall``). +Behaves the same as the multiple-body-expression tuple comprehension ``unpythonic.amb.forall``, but implemented purely by AST transformation, with real lexical variables. This is essentially a MacroPy implementation of Haskell's do-notation for Python, specialized to the List monad (but the code is generic and very short; see ``unpythonic.syntax.forall``). ```python from unpythonic.syntax import macros, forall, insist, deny From dc7758c0e12e7db6f82becfd169683837360bda5 Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 17 Sep 2019 17:01:42 -0700 Subject: [PATCH 29/31] Documentation changes 9/10 of requested changes made --- README.md | 21 +++++++-------------- doc/design-notes.md | 6 +++--- doc/features.md | 8 +++++--- macro_extras/README.md | 16 ++++++++++++---- 4 files changed, 27 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index d19c1c95..037fd1b5 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,21 @@ # Unpythonic: Python meets Lisp and Haskell -In the spirit of [toolz](https://github.com/pytoolz/toolz), we provide missing features for Python, mainly from the list processing tradition, but with some Haskellisms mixed in. We place a special emphasis on **clear, pythonic syntax**. +In the spirit of [toolz](https://github.com/pytoolz/toolz), we provide missing features for Python, mainly from the list processing tradition, but with some Haskellisms mixed in. We place a special emphasis on **clear, pythonic syntax**. These features make up the pure-Python core of `unpythonic`, and are meant to be used directly. We also provide extensions to the Python language as a set of [syntactic macros](https://en.wikipedia.org/wiki/Macro_(computer_science)#Syntactic_macros) that are designed to work together. Each macro adds an orthogonal piece of functionality that can (mostly) be mixed and matched with the others. -We also provide extensions to the Python language as a set of [syntactic macros](https://en.wikipedia.org/wiki/Macro_(computer_science)#Syntactic_macros) that are designed to work together. The macros provide features such as *automatic* currying, *automatic* tail-call optimization, call-by-need (lazy functions), continuations (``call/cc``), lexically scoped ``let`` and ``do`` with lean syntax, implicit return statements, and easy-to-use multi-expression lambdas with local variables. Each macro adds an orthogonal piece of functionality that can (mostly) be mixed and matched with the others. +The macros provide an extension to the pure-Python layer, and offers features such as *automatic* currying, *automatic* tail-call optimization, lexically scoped ``let`` and ``do`` with lean syntax, and implicit return statements. Some of these macro features, like call-by-need (lazy functions), continuations (``call/cc``), and easy-to-use multi-expression lambdas with local variables, are not available in the pure-Python layer. Additionally, some pure-Python features like batteries for itertools do not have a macro layer equivalent. Check the [documentation](#documentation) for the full sets of features. -Design considerations are based in simplicity, robustness, and with minimal dependencies. See our [design notes](doc/design-notes.md) for more information. - -**This is semantics, not syntax!** - -[Strictly speaking](https://stackoverflow.com/questions/17930267/what-is-the-difference-between-syntax-and-semantics-of-programming-languages), ``True``. We just repurpose Python's existing syntax to give it new meanings. However, in the Racket reference, **a** *syntax* designates a macro, in contrast to a *procedure* (regular function). We provide syntaxes in this particular sense. The name ``unpythonic.syntax`` is also shorter to type than ``unpythonic.semantics``, less obscure, and close enough to convey the intended meaning. - -If you want custom *syntax* proper, then you may be interested in [Pydialect](https://github.com/Technologicat/pydialect). +The design considerations of `unpythonic` are based in simplicity, robustness, and with minimal dependencies. See our [design notes](doc/design-notes.md) for more information. ### Dependencies Currently none required. -[MacroPy](https://github.com/azazel75/macropy) optional, to enable the syntactic macros. +[MacroPy](https://github.com/azazel75/macropy) optional, to enable the syntactic macro layer. ### Documentation -[pure-Python (basic implementation) documentation](doc/features.md) -[syntactic macros (advanced implementation) documentation](macro_extras/README.md): the second half of ``unpythonic``. - -[Design Notes](doc/design-notes.md): for more insight into the design choices of ``unpythonic`` +[Pure-Python feature set](doc/features.md) +[Syntactic macro feature set](macro_extras/README.md): the second half of ``unpythonic``. +[Design notes](doc/design-notes.md): for more insight into the design choices of ``unpythonic`` ## Installation diff --git a/doc/design-notes.md b/doc/design-notes.md index 188cae39..4a793630 100644 --- a/doc/design-notes.md +++ b/doc/design-notes.md @@ -4,7 +4,7 @@ - [Python is Not a Lisp](#python-is-not-a-lisp) - [Assignment Syntax](#assignment-syntax) - [TCO Syntax and Speed](#tco-syntax-and-speed) -- [Comboability](#comboability) +- [Comboability of Syntactic Macros](#comboability-of-syntactic-macros) - [No Monads?](#no-monads) - [Further Explanation](#further-explanation) - [Notes on Macros](#notes-on-macros) @@ -78,8 +78,8 @@ For other libraries bringing TCO to Python, see: - [ActiveState recipe 474088](https://github.com/ActiveState/code/tree/master/recipes/Python/474088_Tail_Call_Optimization_Decorator), based on ``inspect``. - ``recur.tco`` in [fn.py](https://github.com/fnpy/fn.py), the original source of the approach used here. - [MacroPy](https://github.com/azazel75/macropy) uses an approach similar to ``fn.py``. - -### Comboability + +### Comboability of Syntactic Macros Making macros work together is nontrivial, essentially because *macros don't compose*. [As pointed out by John Shutt](https://fexpr.blogspot.com/2013/12/abstractive-power.html), in a multilayered language extension implemented with macros, the second layer of macros needs to understand all of the first layer. The issue is that the macro abstraction leaks the details of its expansion. Contrast with functions, which operate on values: the process that was used to arrive at a value doesn't matter. It's always possible for a function to take this value and transform it into another value, which can then be used as input for the next layer of functions. That's composability at its finest. diff --git a/doc/features.md b/doc/features.md index 5e7dfb87..4b4233ba 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1,6 +1,8 @@ # Unpythonic: Python meets Lisp and Haskell -Documentation for the underlying pure-Python API, which can be used directly if you don't want to depend on MacroPy. See also [documentation for syntactic macros](macro_extras/). +Documentation for the underlying pure-Python API, which provides the non-macro features that can be used directly. This API acts as the core for the macro layer, see the [documentation for syntactic macros](macro_extras/) for more about those features. + +Please note that there are features that appear in both the pure-Python layer and the macro layer, as well as features that only exist in the pure-Python layer. If you don't want to depend on MacroPy, feel free to use the features as defined below (though, this may be less convenient). ### Features @@ -56,7 +58,7 @@ Documentation for the underlying pure-Python API, which can be used directly if For many examples, see [the unit tests](unpythonic/test/), the docstrings of the individual features, and this guide. -*This document doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out of date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this guide) should agree with the unit tests. So if something fails to work as advertised, check what the tests say - and optionally file an issue on GitHub so that the documentation can be fixed.* +*This document doubles as the API reference, but despite maintenance on a best-effort basis, may occasionally be out-of-date at places. In case of conflicts in documentation, believe the unit tests first; specifically the code, not necessarily the comments. Everything else (comments, docstrings and this guide) should agree with the unit tests. So if something fails to work as advertised, check what the tests say - and optionally file an issue on GitHub so that the documentation can be fixed.* **This document is up-to-date for v0.14.1.** @@ -289,7 +291,7 @@ Dynamic variables are set using `with dyn.let(...)`. There is no `set`, `<<`, un The values of dynamic variables remain bound for the dynamic extent of the `with` block. Exiting the `with` block then pops the stack. Inner dynamic scopes shadow outer ones. Dynamic variables are seen also by code that is outside the lexical scope where the `with dyn.let` resides.
-Each thread has its own dynamic scope stack. +Each thread has its own dynamic scope stack. There is also a global dynamic scope for default values, shared between threads. A newly spawned thread automatically copies the then-current state of the dynamic scope stack **from the main thread** (not the parent thread!). Any copied bindings will remain on the stack for the full dynamic extent of the new thread. Because these bindings are not associated with any `with` block running in that thread, and because aside from the initial copying, the dynamic scope stacks are thread-local, any copied bindings will never be popped, even if the main thread pops its own instances of them. The source of the copy is always the main thread mainly because Python's `threading` module gives no tools to detect which thread spawned the current one. (If someone knows a simple solution, PRs welcome!) diff --git a/macro_extras/README.md b/macro_extras/README.md index d041b2cf..b7e61155 100644 --- a/macro_extras/README.md +++ b/macro_extras/README.md @@ -2,7 +2,15 @@ Our Python language extensions, as syntactic macros, are built on [MacroPy](https://github.com/azazel75/macropy), from the PyPI package ``macropy3``. If you want to take language extension a step further, see our sister project [Pydialect](https://github.com/Technologicat/pydialect). -The [unit tests that contain usage examples](../unpythonic/syntax/test/) cannot be run directly because macro expansion occurs at import time. Instead, run them via the included [generic MacroPy3 bootstrapper](macropy3). For convenience, ``setup.py`` installs this bootstrapper. +The [unit tests that contain usage examples](../unpythonic/syntax/test/) cannot be run directly because macro expansion occurs at import time. Instead, run them via the included [generic MacroPy3 bootstrapper](macropy3). For convenience, ``setup.py`` installs this bootstrapper.\ + +Please note that there are features that appear in both the pure-Python layer and the macro layer, as well as features that only exist in the macro layer. + +**This is semantics, not syntax!** + +[Strictly speaking](https://stackoverflow.com/questions/17930267/what-is-the-difference-between-syntax-and-semantics-of-programming-languages), ``True``. We just repurpose Python's existing syntax to give it new meanings. However, in the Racket reference, **a** *syntax* designates a macro, in contrast to a *procedure* (regular function). We provide syntaxes in this particular sense. The name ``unpythonic.syntax`` is also shorter to type than ``unpythonic.semantics``, less obscure, and close enough to convey the intended meaning. + +If you want custom *syntax* proper, then you may be interested in [Pydialect](https://github.com/Technologicat/pydialect). ### Set Up @@ -598,7 +606,7 @@ The naming is performed using the function ``unpythonic.misc.namelambda``, which - Expression-assignment to an unpythonic environment, ``f << (lambda ...: ...)`` - Let bindings, ``let[(f, (lambda ...: ...)) in ...]``, using any let syntax supported by unpythonic (here using the haskelly let-in just as an example). - - Env-assignments are now processed lexically, just like regular assignments. Added support for let-bindings. + - Env-assignments are processed lexically, just like regular assignments. Support for other forms of assignment may or may not be added in a future version. @@ -717,7 +725,7 @@ assert add3(1)(2)(3) == 6 **CAUTION**: Some built-ins are uninspectable or may report their arities incorrectly; in those cases, ``curry`` may fail, occasionally in mysterious ways. The function ``unpythonic.arity.arities``, which ``unpythonic.fun.curry`` internally uses, has a workaround for the inspectability problems of all built-ins in the top-level namespace (as of Python 3.7), but e.g. methods of built-in types are not handled. -The special mode for uninspectables used to be enabled for the dynamic extent of the ``with curry`` block; the new lexical behavior is easier to reason about. Also, manual uses of the ``curry`` decorator (on both ``def`` and ``lambda``) are now detected, and in such cases the macro now skips adding the decorator. +Manual uses of the `curry` decorator (on both `def` and `lambda`) are detected, and in such cases the macro skips adding the decorator. ### ``lazify``: call-by-need for Python @@ -1296,7 +1304,7 @@ The ``call_cc[]`` explicitly suggests that these are (almost) the only places wh Write Python almost like Lisp! -Lexically inside a ``with prefix`` block, any literal tuple denotes a function call, unless quoted. The first element is the operator, the rest are arguments. Bindings of the ``let`` macros and the top-level tuple in a ``do[]`` are now left alone, but ``prefix`` recurses inside them (in the case of bindings, on each RHS). +Lexically inside a ``with prefix`` block, any literal tuple denotes a function call, unless quoted. The first element is the operator, the rest are arguments. Bindings of the ``let`` macros and the top-level tuple in a ``do[]`` are left alone, but ``prefix`` recurses inside them (in the case of bindings, on each RHS). The rest is best explained by example: From 781424ca2e73e00f025e337f6544f47eab8477e4 Mon Sep 17 00:00:00 2001 From: aisha w <50159272+aisha-w@users.noreply.github.com> Date: Tue, 17 Sep 2019 17:03:33 -0700 Subject: [PATCH 30/31] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 037fd1b5..15f6e097 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Currently none required. ### Documentation [Pure-Python feature set](doc/features.md) -[Syntactic macro feature set](macro_extras/README.md): the second half of ``unpythonic``. +[Syntactic macro feature set](macro_extras/README.md): the second half of ``unpythonic``. [Design notes](doc/design-notes.md): for more insight into the design choices of ``unpythonic`` ## Installation From 2bab53f58b41808da36c72607cc200ab312f2f6b Mon Sep 17 00:00:00 2001 From: aisha-w <50159272+aisha-w@users.noreply.github.com> Date: Wed, 18 Sep 2019 07:19:52 -0700 Subject: [PATCH 31/31] Update features.md --- doc/features.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/doc/features.md b/doc/features.md index 4b4233ba..aac8f56b 100644 --- a/doc/features.md +++ b/doc/features.md @@ -1509,10 +1509,6 @@ Tools related to control flow. ### ``trampolined``, ``jump``: tail call optimization (TCO) / explicit continuations -**v0.10.0**: ``fasttco`` has been renamed ``tco``, and the exception-based old default implementation has been removed. See also [macros](macro_extras/) for an easy-to-use solution. - -**v0.11.1**: The special jump target ``SELF`` (keep current target) has been removed. If you need tail recursion in a lambda, use ``unpythonic.fun.withself`` to get a reference to the lambda itself. See example below. - Express algorithms elegantly without blowing the call stack - with explicit, clear syntax. *Tail recursion*: