Introduction

I am finishing up this post on the train back from the Autumn 2015 Darcs sprint. Today (Sept 20, Sun) was a very fun day full of darcs chatting and coding. By the end of the day we’ve heard a number of presentations

• Ganesh described his work on "stash" command for darcs (naming subject to change!). It involves some refactoring of the rebase code. I hope we would hear more from him on that, because the internal workings are actually quite interesting — I believe it’s the first time singleton types and DataKinds are used in the darcs codebase;
• Florent Becker gave a presentation about Pijul and the theory behind it — A Categorical Theory of Patches by Samuel Mimram and Cinzia Di Giusto, see arXiv:1311.3903;
• Vinh Dang talked about his improvements on the darcs wiki (it’s about time to organize the website), his goal was to make it more accessible to the newcomers;
• Yours truly gave a small presentation, outline of which you will find below:

Looking back

I have spent this summer hacking on DarcsDen as part of the Google Summer of Code program.

My basic goal was to create a "local" version of darcsden. It was not a trivial task to install darcsden (and probably installation is still not very easy!). It uses a third-party software like Redis and CouchDB. During my coding process I modifed darcsden such that it now can be a good choice for local (or lightweight single user) darcs UI. The local darcsden version can be used without any databases, tracking the repositories in the local file system. This way darcsden can be used by a developer on her local computer, like darcsum, (for working with/comparing repositories) as well as a replacement for darcsweb/cgit — a single user web front for darcs repositories.

Besides that a user of a local version can use darcsden’s interactive UI for recording new patches, as well as a command-line tool den for a quick way of browsing the repositories.

Installing darcsden-local is currently not as easy as I want to it be, but I hope that soon you will be able to install it just by running cabal install darcsden or brew install darcsden. As for now, one could do the following:

1. darcs get --lazy http://hub.darcs.net/co-dan/darcsden-local
2. cabal install . or stack install

This should install the darcsden binary and all the related css/js files. You can start darcsden by running darcsden --local. If you open your web browser you should see a list of repositories in the current directory.

However, you might have repositories scattered all over the place, and scanning your whole system for darcs repositories is just inefficient. For this purposes darcs keeps a list of repositories in a file inside your ~/.darcs directory. You can manage that list either by hand, or using the command-line den tool:

• den $PATH — add $PATH to the list of repositories in ~/.darcs/darcsden_repos (if it’s not already present there), start darcsden server in the background and launch the browser pointing to $PATH;
• den — the same as den .;
• den --add $PATH — add $PATH to the list of repositories in ~/.darcs/darcsden_repos;
• den --remove $PATH — remove $PATH from the list of repositories in ~/.darcs/darcsden_repos.

In order to further customize darcsden, one can tweak the configuration file located at ~/.darcs/darcsden.conf. Apart from the usual darcsden settings one may pay attention to the following variables:

• homeDir (default .), points to the "root" directory with repositories. If the list file ~/.darcs/darcsden_repos is not present darcsden will recursively search repositories in that directory
• unLocal, pwLocal: the username and the password of the "local" user

The user/password credentials are required for editing the repositories and recording new patches. However, the den binary should automatically pick them up from the config file and log you in.

Once you are logged in, and you have unrecorded changes in the repository, you can use darcsden UI to record a new patch.

DarcsDen record

Below you can see an example of recording and merging patches from a branch.

DarcsDen merge

Darsden allows you to create forks/branches of your repositories, and it keeps track of the patch dependencies in your branches.

More "internal" changes:

• Instead of having to specify some parts of the configuration in DarcsDen.Settings, darcsden now uses runtime flags: –hub for using hub-specific modifications, –local for using the local backend and no flag for default behaviour
• The flag actually choose what are called instances — something that a bit less fine-grained than settings. Instances allow you to pick backend, overwrite settings, modify the looks of the front page.
• HTTP-testing using wreq. The previous test suite used selenium and it got bit-rotten. The wreq-based is easier to run and perhaps slightly easier to maintain.
• HTTP auth, which is used as part of the local instance; the den tool utilizes it to log the user in automatically.
• Support for repositories inside directories and nested repositories.
• All the backend code that is used for handling repositories and meta-data on the file system.
• Functionality for downloading zipped dist archives of darcs repositories.
• Assorted mini-fixes

What now?

During the sprint I hacked together some code for viewing suspended patches along the regular ones. The next step would be to have a similar interface for managing the suspended patches.

We have also discussed the possibility of adding rewrite rules implementing short-cut fusion for the directed types in Darcs. In order to see if it’s really worth it we would have to bring back to life the benchmarking suite (or at least check on it!).

It was a really exciting weekend for me and I was delighted to meet some of my IRC friends. As it turns out, it is a small world and despite being from different parts of it we have a bunch of common IRL friends, professors. As the French would (probably not) say, très bien. The next darcs sprint will probably be in January, and probably in Europe, again.

One of the things that I particularly like about the Darcs codebase is that you can see that the developers were not shy about using intermediate-slash-advanced Haskell/GHC features to help achieving type safety. You can see GADTs, associated types, phantom types, existential types actively used.

In this post I would like to discuss the representation of patches and the use of type witnesses in darcs.

A word about patches and contexts

This post is intended for people interested in patch theory and its implementation in darcs. A passing familiarity with patches, contexts, inverses, etc is useful, but not required, as we restate some basic definitions in this section.

A primitive patch is a basic unit in patch theory. It constitutes an atomic change in the repository. The definition of a primitive patch may vary, but usually the following changes are considered primitive patches:

• Removing a file from the repository
• Adding file to the repository
• Changing a line n in a file in the repository
• Identity patch, i.e. an empty/non-existing change that does not modify the state of the repository at all

Every primitive patch has a pre-context and a post-context. Roughly, you can think of a pre-context as the full state of the repository before the change was made, and of the post-context as the full state of the repository after the change was applied. We write (x)-A->(y) for a patch A with a pre-context x and a post-context y.

If a primitive patch A has a pre-context a, a post-context o, and a primitive patch B has a pre-context o, a post-context b, then we can combine two patches to obtain a sequential patch AB with the pre-context a and the post-context b.

Every primitive patch (x)-A->(y) has an inverse (y)-A^{-1}->(x), such that (x)-A->(y)-A^{-1}->(x) is equivalent to the identity patch (x)-1->(x).

In the next sections we will see how those things are implemented in darcs.

Primitive patches and witnesses

A primitive patch, which constitutes a single fine grained change, can be represented as a (G)ADT:

data Prim where
    Move :: FileName -> FileName -> Prim
    RmFile :: FileName -> Prim
    AddFile :: FileName -> Prim
    Hunk :: FileName -> Int -> ByteString -> ByteString -> Prim
    RmDir :: FileName -> Prim
    …

We can represent complex patches as sequences of primitive patches:

data Patch where
    NilP :: Patch
    PrimP :: Prim -> Patch
    SeqP :: Patch -> Patch -> Patch

This seems reasonable enough. But if we implement our patch theory this way we seem to be missing something — patches have (pre- and post-) contexts. Having contexs allows us to enforce patch composition on the level of type system. Consider the following definition, which uses phantom types as type witnesses for contexts.

data Prim x y where
    Move :: FileName -> FileName -> Prim x y
    RmFile :: FileName -> Prim x y
    AddFile :: FileName -> Prim x y
    Hunk :: FileName -> Int -> ByteString -> ByteString -> Prim x y
    RmDir :: FileName -> Prim x y
    …

data Patch x y where
    NilP :: Patch x x
    PrimP :: Prim x y -> Patch x y
    SeqP :: Patch x y -> Patch y z -> Patch x z

We call the types with witnesses representing pre- and post-contexts directed types. Intuitively, the directed type D x y has a “direction” from x to y, written as (x)->(y). The Prim datatype looks pretty much like the type actually used in Darcs. The Patch datatype, however, is completely artificial. We will see in the next sections how Darcs really models complex patches.

Directed programming

FL & RL

Two particularly useful directed types used in darcs are directed lists: forwards lists of the type FL a x y and reverse lists RL a x y. Forward lists “go” from the head to the tail; reverse lists “go” from the tail to the head. The lists are polymorphic over a just like regular lists.

data FL a x y where
    NilFL :: FL a x x    {- The empty list stays at (x) -}
    (:>:) :: a x y -> FL a y z -> FL a x z
infixr 5 :>:

For myself, I visualise forward lists like this:

(x) —a—> (y) —b—> (z) ——Nil——> (z)
a :: Patch x y
b :: Patch y z
(a :>: b :>: NilFL) :: FL Patch x z

The reverse lists are “going” from tail to head¹

data RL a x y where
    NilRL :: RL a x x
    (:<:) :: RL a x y -> a y z -> RL a x z
infixl 5 :<:

(Mnemonically, the the head is always “greater” than the tail)

The reason I used the word “go” inside quotation marks so far is the following. Reverse lists and forward lists represent the same concept: a sequence of patches (or a sequence of directed things for that matter). They only differ in the associativity of the elements. Forward lists associate to the right, but reverse lists associate to the left.

p = Move "foo" "bar"

-- [[(x) --Nil--> (x) -p-> (y)] -id-> (y)] -p^{-1}-> (x)
example :: RL Patch x x
example = NilRL :<: p :<: Identity :<: inverse p

-- (x) -p-> [(y) -p^{-1}-> (x) --Nil--> (x)]
example2 :: FL Patch x x
example2 = p :>: inverse p :>: NilFL

The right-associated/reverse lists provide easy access to the last element of the sequence; the left-associated/forward lists provide easy access to the first element of the sequence. Therefore, if we view a repository as a directed sequence of patches, right-associated lists are useful for operations that work on the “latest” patches in the repository (such as record/unrecord), and left-associated lists are useful for commands that scan the repository from the beginning (like clone).

We can reassociate the lists easily, and verify that the two representations are isomoprhic:²

reverseRL :: RL a wX wY -> FL a wX wY
reverseRL = r NilFL
-- the type signature of r basically gives us an invariant
-- wZ is slowly "decreasing" reaching the point where
-- wZ = wX; at that point the first argument is of type FL a wX wY
  where r :: FL a wZ wY -> RL a wX wZ -> FL a wX wY
        r a NilRL = a
        r a (xs :<: x) = r (x :>: a) xs

For example,

-- [[(x) --Nil--> (x) -p-> (y1)] -q-> (y2)] -r-> (z)

Turns into

-- (x) -p-> [(y1) -q-> (y2) -r-> [(z) --Nil--> (z)]]

Exercise: write a function reverseFL :: FL a wX wY -> RL a wX wY

We can write a lot of directed analogues of familiar list functions. For example, here is a directed append:

infixl 5 +<+
(+<+) :: RL a wX wY -> RL a wY wZ -> RL a wX wZ
xs +<+ NilRL = xs
xs +<+ (ys :<: y) = (xs +<+ ys) :<: y

Exercise: write directed append for forward lists: (+>+) :: FL a wX wY -> FL a wY wZ -> FL a wX wZ

Type witnesses

So we wrote a bunch of standard list functions for directed lists; what about some of the other functions? Can we, for example, implement filter for directed lists. Can it look like filterFL :: (a wX wY -> Bool) -> FL a wX wY? Well, we can try writing

filterFL :: (a wX wY -> Bool) -> FL a wX wY
filterFL p NilFL = NilFL
filterFL p (x :>: xs) | p x = filterFL p xs
                      | otherwise = x :>: filterFL p xs

However, under closer scrutiny we realize that it does not typecheck! In the second clause of filterFL we have the following information:

x :: a x y, xs :: FL a y z
filterFL xs :: FL a y z

Thus, in the first case (in which p x holds) we try to return something of the type FL a wY wZ, when FL a wX wZ was expected. It is clear that generally we can do this only if x :: a wX wX, i.e. wY = wX. But a simple predicate of the type p :: a wX wZ -> Bool won’t tell us anything about that. We need an additional type witness in our system telling us that if p x holds, then x :: a wX wX. For that purpose we introdue the EqCheck datatype.

data EqCheck wX wY where
    IsEq :: EqCheck wX wX
    NotEq :: EqCheck wX wY

then the type of a predicate would be

type Pred a = forall wX wY. a wX wY -> EqCheck wX wY

If (p x) = IsEq, then the typechecker will know that x :: a wX wX. We can then finally write

filterFL :: Pred a -> FL a wX wY -> FL a wX wY
filterFL p NilFL = NilFL
filterFL p (x :>: xs) | IsEq <- p x = filterFL p xs
                      | otherwise   = x :>: filterFL p xs

EqCheck is used this way in the darcs source code to e.g., filter our internal patches. Sometimes darcs stores information — like suspended patches — in the so called internal patches. Every patch type implements the internal patch checker (code slightly adapted):

-- |Provides a hook for flagging whether a patch is "internal" to the repo
-- and therefore shouldn't be referred to externally, e.g. by inclusion in tags.
-- Note that despite the name, every patch type has to implement it, but for
-- normal (non-internal) types the default implementation is fine.
-- Currently only used for rebase internal patches.
class MaybeInternal p where
    -- | @maybe (const NotEq) (fmap isInternal patchInternalChecker) p@
    -- returns 'IsEq' if @p@ is internal, and 'NotEq' otherwise.
    -- The two-level structure is purely for efficiency: 'Nothing' and 'Just (InternalChecker (const NotEq))' are
    -- semantically identical, but 'Nothing' allows clients to avoid traversing an entire list.
    -- The patch type is passed as an 'FL' because that's how the internals of named patches are stored.
    patchInternalChecker :: Maybe (forall wX wY . FL p wX wY -> EqCheck wX wY)
    patchInternalChecker = Nothing

When the user runs darcs tag in the repository, darcs creates a dummy patch that explicitly depends on all the previous patches — apart from the internal ones of course. Thus, the tag command uses the following function (slightly adapted):

filterNonInternal :: MaybeInternal p => PatchSet p wX wY -> PatchSet p wX wY
filterNonInternal =
  case patchInternalChecker of
     Nothing -> id
     Just f -> \l -> PatchSet (filterRL (f . patchcontents . hopefully) (unPatchSet l))

where a PatchSet is the list of PatchInfoAnd patches — patches together with the meta-information.

It is worth noting that EqCheck x y is isomorphic Maybe (x :~: y), but the propositional equality datatype has only been added to base since 4.7.0.0. In the future versions darcs will probably switch to using Data.Type.Equality.

Conclusion

We’ve briefly touched upon patch representation in darcs and talked about directed types and directed programming.

A good if a bit outdated reference is Jason Dagit’s master thesis (specifically the bits from chapter 4). The wiki is currently lacking in material, but I hope to improve the situation eventually.

Next time we will probably discuss either directed pairs and their use in darcs, or sealed datatypes, or both.

A repository can change it’s state from rebase-in-progress back to normal if there are no suspended patches left. However, be aware that you cannot unsuspend a patch¹ if you have unrecorded changes in the repository. In light of this, I suggest recording a temporary patch with current changes

darcs record -am DRAFT

You can suspend that patch at the beginning of your rebase process and apply it at the end.

General overview of rebase

darcs rebase is an operation (or, rather, a family of operations) that allows one to make changes “deep down” in the repository history. One of the crucial things that allows for rebase to work is the fact that since darcs 2.10 patches can be suspended. When one performs any of the darcs rebase commands, the repository moves to a special rebase-in-progress state. In this state repository contains a pristine, a set of patches, a working copy, and — in addition to all the usual stuff — a set of suspended patches. Suspended patches are not active in the repository — that is, they are not applied.

Let’s go over the rebase subcommands

rebase log/rebase changes

This is simple: list the suspended patches

rebase suspend

Moves selected patches into the suspended state. Once the patch is suspended it is no longer active in the repository.

Note: once you suspend a patch, it changes its identity. That means that even if you suspend a patch and unsuspend it immediately, you will get a different repository that you have started with. Let this be a good reason (one of a couple!) for doing rebase on a separate branch.

> cat file
test
123
> darcs rebase suspend
	patch 64523bc4622fad02a4bdb9261887628b7997ebdd
Author: Daniil Frumin 
Date:   Thu Jul 23 18:49:30 MSK 2015
  * 123
Shall I suspend this patch? (1/5)  [ynW…], or ? for more options: y
patch cc54d7cf4b9e3d13a24ce0b1b77b76581d98d75d
Author: Daniil Frumin 
Date:   Thu Jul 23 18:43:53 MSK 2015
  * Test
Shall I suspend this patch? (2/5)  [ynW…], or ? for more options: d
Rebase in progress: 1 suspended patches
> darcs rebase log
patch 64523bc4622fad02a4bdb9261887628b7997ebdd
Author: Daniil Frumin 
Date:   Thu Jul 23 18:49:30 MSK 2015
  * 123
Shall I view this patch? (1/?) [yN…], or ? for more options: y
[123
Daniil Frumin **20150723154930
 Ignore-this: 43e09e6503ac74688e74441dc29bce25
] hunk ./file 2
+123
Rebase in progress: 1 suspended patches
> cat file
test

rebase unsuspend

Does the opposite of suspend: applies a suspended patch to the repository and changes its state to normal.

> darcs rebase unsuspend                                                                                       
patch 64523bc4622fad02a4bdb9261887628b7997ebdd
Author: Daniil Frumin 
Date:   Thu Jul 23 18:49:30 MSK 2015
  * 123
Shall I unsuspend this patch? (1/1)  [ynW…], or ? for more options: y
Do you want to unsuspend these patches? [Yglqk…], or ? for more options: y
Rebase finished!

rebase apply

Rebase apply takes a patch bundle and tries to apply all the patches in the bundle to the current repository. If a patch from the bundle conflicts with a local patch, then the local patch gets suspended. You will thus have a chance to resolve the conflict by amending your conflicting patches, at a price of.. well, changing the identity of your local patches.

rebase pull

Sort of like rebase apply, but instead of a patch bundle it obtains the patches from a remote repository.

Specifically, rebase pull applies all the remote patches, one-by-one, suspending any local patches that conflict. We will see more of rebase pull in the second example.

Example 1: suspending local changes

Imagine the following situation: at point A you add a configuration file to your repository, then you record a patch B that updates the settings in the configuration file. After that you make some more records before you realize that you’ve included by accident your private password in patch A! You want to get rid of it in your entire history, but you can’t just unrecord A, because B depends on A, and possibly some other patches depend on B.

The contents of the configuration file after patch A:

port = 6667
host = irc.freenode.net
password = awesomevcs 

Patch B, diff:

@@ -1,3 +1,4 @@
-port = 6667
+port = 6697
+usessl = True
host = irc.freenode.net
password = awesomevcs

You cannot just amend patch A, because the patch B depends on A:

> darcs amend                                                                                                         
patch 1925d640f1f3180cb5b9e64260c1b5f374fce4ca
Author: Daniil Frumin 
Date:   Tue Jul 21 13:23:07 MSK 2015
  * B

Shall I amend this patch? [yNjk…], or ? for more options: n

Skipping depended-upon patch:
patch 22d7c8da83141f8b1f80bdd3eff02064d4f45c6b
Author: Daniil Frumin 
Date:   Tue Jul 21 13:22:24 MSK 2015
  * A

Cancelling amend since no patch was selected.

What we will have to do is temporarily suspend patch B, amend patch A, and then unsuspend B.

> darcs rebase suspend
patch 1925d640f1f3180cb5b9e64260c1b5f374fce4ca
Author: Daniil Frumin 
Date:   Tue Jul 21 13:23:07 MSK 2015
  * B

Shall I suspend this patch? (1/2)  [ynW…], or ? for more options: y
patch 22d7c8da83141f8b1f80bdd3eff02064d4f45c6b
Author: Daniil Frumin 
Date:   Tue Jul 21 13:22:24 MSK 2015
  * A

Shall I suspend this patch? (2/2)  [ynW…], or ? for more options: d
Rebase in progress: 1 suspended patches

At this point, the state of our repository is the following: there is one (active) patch A, and one suspended patch B.

> darcs rebase changes -a
patch 4c5d45230dc146932b21964aea938e2a978523eb
Author: Daniil Frumin 
Date:   Tue Jul 21 13:28:58 MSK 2015
  * B

Rebase in progress: 1 suspended patches
> darcs changes -a
patch 21f56dfb425e4c49787bae5db4f8869e96787fb2
Author: Daniil Frumin 
Date:   Tue Jul 21 13:28:49 MSK 2015
  * A

Rebase in progress: 1 suspended patches
> cat config
port = 6667
host = irc.freenode.net
password = awesomevcs
> $EDITOR config # remove the password bit
> darcs amend
patch 22d7c8da83141f8b1f80bdd3eff02064d4f45c6b
Author: Daniil Frumin 
Date:   Tue Jul 21 13:22:24 MSK 2015
  * A

Shall I amend this patch? [yNjk…], or ? for more options: y
hunk ./config 3
-password = awesomevcs
Shall I record this change? (1/1)  [ynW…], or ? for more options: y
Do you want to record these changes? [Yglqk…], or ? for more options: y
Finished amending patch:
patch 21f56dfb425e4c49787bae5db4f8869e96787fb2
Author: Daniil Frumin 
Date:   Tue Jul 21 13:28:49 MSK 2015
  * A

Rebase in progress: 1 suspended patches

Now that we’ve removed the password from the history, we can safely unsuspend patch B (in this particular situation we actually know that applying B to the current state of the repository won’t be a problem, because B does not conflict with our modified A)

> darcs rebase unsuspend
patch 1925d640f1f3180cb5b9e64260c1b5f374fce4ca
Author: Daniil Frumin 
Date:   Tue Jul 21 13:23:07 MSK 2015
  * B

Shall I unsuspend this patch? (1/1)  [ynW…], or ? for more  options: y
Do you want to unsuspend these patches? [Yglqk…], or ? for more options: y
Rebase finished!

And that’s done!

> cat config
port = 6697
usessl = True
host = irc.freenode.net

You may use this rebase strategy for removing sensitive information from the repository, for removing that 1GB binary .iso that you added to your repository by accident, or for combining two patches into one deep down in the patchset.

Example 2: developing against a changing upstream – rebase pull

Imagine you have a fork R’ of a repository R that you are working on. You are implementing a feature that involves a couple of commits. During your work you record a commit L1 that refractors some common datum from modules A.hs and B.hs. You proceed with your work recording a patch L2. At this point you realise that after you forked R, the upstream recorded two more patches U1 and U2, messing with the common datum in A.hs. If you just pull U1 into your fork R’, you will have a conflict, that you will have to resolve by recording another patch on top.

      S
     / \
    /   \
   L1   U1

Note: if you run darcs rebase pull in R’, then the only patches that will be suspended are the ones which are already in R’. Because suspended patches gain new identity, make sure that you do not have other people’s conflicting patches present in R’.

The way to solve this would be to first do darcs rebase pull, which would suspend the conflicting patches, and then start unsuspending the patches one by one, making sure that you fix any conflicts that may arise after each unsuspend.

Consider a concrete example with two repositories rep1 and rep1_0.

rep1_0 > darcs changes 
patch ebaccd5c36667b7e3ee6a49d25ef262f0c7edf2b
Author: Daniil Frumin 
Date:   Mon Jul 27 20:56:25 MSK 2015
  * commit2

patch a7e0d92a53b0523d0224ef8ffae4362adf854485
Author: Daniil Frumin 
Date:   Mon Jul 27 20:56:25 MSK 2015
  * commit1
	rep1_0 > darcs diff —from-patch=commit2
patch ebaccd5c36667b7e3ee6a49d25ef262f0c7edf2b
Author: Daniil Frumin 
Date:   Mon Jul 27 20:56:25 MSK 2015
  * commit2
diff -rN -u old-rep1_0/dir1/file2 new-rep1_0/dir1/file2
— old-rep1_0/dir1/file2	1970-01-01 03:00:00.000000000 +0300
    +++ new-rep1_0/dir1/file2	2015-07-28 12:25:54.000000000 +0300
@@ -0,0 +1 @@
+double whatsup
	rep1_0 > cd ../rep1
rep1 > darcs changes
patch e3df0e23a3915910a81eb8181d7b3669e8f270a9
Author: Daniil Frumin 
Date:   Tue Jul 28 12:27:55 MSK 2015
  * commit2’

patch a7e0d92a53b0523d0224ef8ffae4362adf854485
Author: Daniil Frumin 
Date:   Mon Jul 27 20:56:25 MSK 2015
  * commit1
rep1 > darcs diff —from-patch=“commit2’”
patch e3df0e23a3915910a81eb8181d7b3669e8f270a9
Author: Daniil Frumin 
Date:   Tue Jul 28 12:27:55 MSK 2015
  * commit2’
diff -rN -u old-rep1/dir1/file2 new-rep1/dir1/file2
— old-rep1/dir1/file2	1970-01-01 03:00:00.000000000 +0300
+++ new-rep1/dir1/file2	2015-07-28 12:28:39.000000000 +0300
@@ -0,0 +1 @@
+touch file2
\ No newline at end of file
diff -rN -u old-rep1/file1 new-rep1/file1 — old-rep1/file1	2015-07-28 12:28:39.000000000 +0300 +++ new-rep1/file1	2015-07-28 12:28:39.000000000 +0300
@@ -1 +1 @@
-whatsup
\ No newline at end of file
+double whatsup
\ No newline at end of file

The patch commit2 from rep1_0 conflicts with commit2’ from rep1.

rep1 > darcs rebase pull ../rep1_0
patch ebaccd5c36667b7e3ee6a49d25ef262f0c7edf2b
Author: Daniil Frumin 
Date:   Mon Jul 27 20:56:25 MSK 2015
  * commit2
Shall I pull this patch? (1/1)  [ynW…], or ? for more options: y
Do you want to pull these patches? [Yglqk…], or ? for more options: y
The following local patches are in conflict:
patch e3df0e23a3915910a81eb8181d7b3669e8f270a9
Author: Daniil Frumin 
Date:   Tue Jul 28 12:27:55 MSK 2015
  * commit2’
Shall I suspend this patch? (1/1)  [ynW…], or ? for more options: y
Do you want to suspend these patches? [Yglqk…], or ? for more options: y
Finished pulling.
Rebase in progress: 1 suspended patches

Now we have one patch — commit2’ — in the suspended state. We want to resolve the conflict by amending commit2’. We will do that by unsuspending it and manually editing out the conflicting lines. This will also make it depend on commit2.

rep1 > darcs rebase unsuspend
patch e3df0e23a3915910a81eb8181d7b3669e8f270a9
Author: Daniil Frumin 
Date:   Tue Jul 28 12:27:55 MSK 2015
  * commit2’
Shall I unsuspend this patch? (1/1)  [ynW…], or ? for more options: y
Do you want to unsuspend these patches? [Yglqk…], or ? for more options: d
We have conflicts in the following files:
./dir1/file2

Rebase finished!
rep1 > cat dir1/file2
v v v v v v v

=============
double whatsup
*************
touch file2
^ ^ ^ ^ ^ ^ ^
rep1 > $EDITOR dir1/file2
rep1 > darcs amend -a
patch 40b3b4123c78dba6a6797feb619572072654a9cd
Author: Daniil Frumin 
Date:   Tue Jul 28 12:32:56 MSK 2015
  * commit2’
Shall I amend this patch? [yNjk…], or ? for more options: y
Finished amending patch:
patch c35867259f187c1bc30310f1cacb34c1bb2cce41
Author: Daniil Frumin 
Date:   Tue Jul 28 12:34:30 MSK 2015
  * commit2’
rep1 > darcs mark-conflicts
No conflicts to mark.

Another repository saved from conflicting patches, yay!

Citing the official release notes

> The darcs team is pleased to announce the release of darcs 2.10.1 !
> ..
> 
> # What's new in 2.10.1 (since 2.10.0) #
> 
> - generalized doFastZip for darcsden support
> - support terminfo 0.4, network 2.6, zlib 0.6, quickcheck 2.8 and
> attoparsec 0.13
> - errorDoc now prints a stack trace (if profiling was enabled) (Ben Franksen)
> - beautified error messages for command line and default files (Ben Franksen)
> - fixed the following bugs:
>       - issue2449: test harness/shelly: need to handle
> mis-encoded/binary data (Ganesh Sittampalam)
>       - issue2423: diff only respecting --diff-command when a diff.exe
> is present (Alain91)
>       - issue2447: get contents of deleted file (Ben Franksen)
>       - issue2307: add information about 'darcs help manpage' and
> 'darcs help markdown' (Dan Frumin)
>       - issue2461: darcs log --repo=remoterepo creates and populates
> _darcs (Ben Franksen)
>       - issue2459: cloning remote repo fails to use packs if cache is
> on a different partition (Ben Franksen)
> 
> # Feedback #
> 
> If you have an issue with darcs 2.10.0, you can report it on
> http://bugs.darcs.net/ . You can also report bugs by email to
> bugs at darcs.net, or come to #darcs on irc.freenode.net.

I’ve updated Mac OS to version 2.10.1. You can install it with

brew install http://darcs.covariant.me/darcs.rb

Server side

Transient storage

A lot of darcsden code is (especially the parts that are closer to the users’ web browser — handlers, pages, so on) is written around sessions. Sessions are stored in a special storage — implemented by the DarcsDen.Backend.Transient, but if we abstract away from the details we have a Session datatype. Authorization and authentication information is handled by sessions using functions setUser :: (BTIO bt) => Maybe User -> Session -> Snap Session, notice :: (BTIO bt) => String -> Session -> Snap () (display a message to the user) and others. The BTIO bt part is just a synonym for

type BTIO bt = (BackendTransient bt, ?backendTransient :: bt, MonadIO (BackendTransientM bt))

Which basically says that we are operating with a transient backend that supports all of necessary operations, and we can also do IO in it. Right now there are only two transient backends (read: two ways of storing sessions): Redis and in-process memory.

Running sessions

If we have a piece of Snap code that we want to “sessionify” we use the following interface:

withSession :: (BTIO bt) => (Session -> Snap ()) -> Snap ()

What this does is it basically checks for a cookie — in case it is present it grabs the session information from the storage (in accordance with the cookie); if the cookie is not present it creates a new session and stores it in a cookie.

If we have a page of a type Session -> Snap (), we might want to give user an option to do HTTP authentication on that page. We introduce another function

withBasicAuth :: (BP bp, BTIO bt) 
              => (Session -> Snap ()) 
              -> (Session -> Snap ())
withBasicAuth act s = do
    rq  do
          rawHeader <- maybe throwChallenge return $ getHeader “Authorization” rq
          let (Just (usr,pw)) = getCredentials rawHeader
          c  errorPage “Unknown user”
            Just u -> if checkPassword (fromBS pw) u
                         then doLogin (fromBS usr)
                         else errorPage “Bad password”
      _ -> act s

So, what is going on in here? First of all, we check if the “login” parameter is set to “true”. If it does, we try to get the “Authorization” header, de-encode it, and check whether the credentials are good.

throwChallenge :: Snap a
throwChallenge = do
    modifyResponse $ (setResponseStatus 401 “Unauthorized”) .
                     (setHeader “WWW-Authenticate” “Basic realm=darcsdenrealm”)
    getResponse >>= finishWith

If the response header is present, it is of a form Basic x, where x is a base64 encoded string user:password. We can extract the credentials from the header like this:

import qualified Data.ByteString          as B
import qualified Data.ByteString.Base64   as B
…
getCredentials :: B.ByteString  — ^ Header
               -> Maybe (B.ByteString, B.ByteString) — ^ Possibly (username, password)
getCredentials header =
    if (isInfixOf “Basic “ header)
      then fmap extract (hush (B.decode (B.drop 6 header)))
      else Nothing
  where
    extract cred = case (B.breakByte (c2w ‘:’) cred) of
                     (usr, pw) -> (usr, safeTail pw)

On the client

The tests that I am currently writing for darcsden are all of the same form: I use wreq to do requests to the darcsden server, then, if necessary, I run taggy to extract information from the webpage, and compare it to the “canonical” information.

As it turns out, doing HTTP Basic Auth is very easy with wreq! First of all, we define a function for doing a GET request that will do some exception handling for us:

getSafeOpts :: Options -> String -> IO (Either Status (Response ByteString))
getSafeOpts opts url = fmap Right (getWith opts url) `catch` hndlr
  where
    hndlr (StatusCodeException s _ _) = return (Left s)
    hndlr e = throwIO e

This way, we won’t get a runtime exception when accessing a non-existing page or getting a server error. Doing a GET request with HTTP Basic Auth is now very easy:

getWithAuth :: (String, String) -> String -> IO (Either Status (Response ByteString))
getWithAuth (username,pw) url = getSafeOpts opts url
  where
    opts = defaults & auth ?~ basicAuth (toBS username) (toBS pw)
                    & param “login” .~ [“true”]

In that snippet we use lenses to set up an auth header and an HTTP parameter (?login=true).

Finally, after obtaining a Response ByteString, we can parse it with taggy-lens:

parsed :: Fold (Response L.ByteString) Text.Taggy.Node
parsed = responseBody 
       . to (decodeUtf8With lenientDecode)
       . html

We can then play with it in GHCi

*Main> Right r  r ^.. parsed
[NodeElement (Element {eltName = “DOCTYPE”, eltAttrs = fromList [(“html”,””)], eltChildren = [NodeElement (Element {eltName = “html”, eltAttrs = fromList [], eltChildren = [NodeElement (Element {eltName = “head”, eltAttrs = fromList [], eltChildren = [NodeElement (Element {eltName = “title”, eltAttrs = fromList [], eltChildren = [NodeContent “localhost”]}),NodeElement (Element {eltName = “link”, eltAttrs = fromList [(“href”,”http://localhost:8900/public/images/favicon.ico”),…

If we want to check that we are indeed logged in correctly, we should look for the “log out” button. Taggy does all the heavy lifting for us, we just have to write down a lens (more precisely, a fold) to “extract” a logout button from the page

logoutButton :: HasElement e => Fold e Text
logoutButton = allAttributed (ix “class” . only “logout”)
             . allNamed (only “a”)
             . contents

logoutButton searches for <div class=“logout”> and returns the text in the link inside the div. There might be many such links inside the node, hence we use a fold.

*Main> r ^.. parsed . logoutButton
[“log out”]

In this case, since we only care if such link is present, we can use a preview

*Main> r ^? parsed . logoutButton
Just “log out”

Conclusion

Well, that’s about it for now. I regret taking way too much time writing this update, and I hope to deliver another one soon. Meanwhile, some information regarding the darcsden SoC project can be found on the wiki.

brew install http://darcs.covariant.me/darcs.rb

The formula contains a context (--exact-version) and it is a static binary.

This is my first post from the (hopefuly fruitful!) series of blog posts as part of my Haskell SoC project. I will spend a great chunk of my summer hacking away on DarcsDen; in addition, I will document my hardships and successes here. You can follow my progress on my DarcsHub.

This particular post will be about my working environment.

The problem

Hoogle is an amazing tool that usually needs no introduction. Understandably, the online version at haskell.org indexes only so many packages. This means that if I want to use hoogle to search for functions and values in packages like darcs and darcsden, I will have to set up a local copy.

Cabal sandboxing is a relatively recent feature of the Cabal package manager, but I don’t think it is reasonable in this day to install from the source (let alone develop) a Haskell package without using sandboxing.

The problem seems to be that the mentioned tools do not play well together out of the box, and some amount of magic is required. In this note I sketch the solution, on which I’ve eventually arrived after a couple of tries.

Using hoogle inside a Cabal sandbox

The presumed setup: a user is working on a package X using the cabal sandboxes. The source code is located in the directory X and the path to the cabal sandbox is X/.cabal-sandbox.

Step 1: Install hoogle inside the sandbox. This is simply a matter of running cabal install hoogle inside X. If you want to have a standard database alongside the database for your packages in development, now is the time to do .cabal-sandbox/bin/hoogle data.

Step 2: Generate haddocks for the packages Y,Z you want to use with hoogle. In my case, I wanted to generate haddocks for darcs and darcsden. This is just a matter of running cabal haddock --hoogle in the correct directory.

Step 3: Convert haddocks to .hoo files. Run the following commands in X/:

.cabal-sandbox/bin/hoogle convert /path/to/packageY/dist/doc/html/*/*.txt

You should see something like

Converting /path/to/packageY/dist/doc/html/Y/Y.txt
Converting Y... done

after which the file Y.hoo appears in /path/to/packageY/dist/doc/html/Y/

Step 4: Moving and combining databases. The hoogle database should be stored in .cabal-sandbox/share/*/hoogle-*/databases. Create such directory, if it’s not present already. Then copy the ‘default’ database to that folder:

cp .cabal-sandbox/hoogle/databases/default.hoo .cabal-sandbox/share/*/hoogle-*/databases

Finally, you can combine your Y.hoo with the default database.

.cabal-sandbox/bin/hoogle combine /path/to/packageY/dist/doc/html/*/*.hoo .cabal-sandbox/share/*/hoogle-*/databases/default.hoo
mv default.hoo .cabal-sandbox/share/*/hoogle-*/databases/default.hoo

And you are done! You can test your installation

$ .cabal-sandbox/bin/hoogle rOwner
DarcsDen.State.Repo rOwner :: Simple Lens (Repository bp) String

For additional usability, consider adding .cabal-sandbox/bin to your $PATH.

Happy holidays, everyone!

I would like to announce a new version of the Hastache library, version 0.6.1. Some interesting and useful changes, as well as improvements and bugfixes are included in the release. See below for an extended changelog.

Hastache is a Haskell implementation of the mustache templating system.

Quick start

cabal update
cabal install hastache

A simple example:

import Text.Hastache 
import Text.Hastache.Context 
import qualified Data.Text.Lazy.IO as TL

main = hastacheStr defaultConfig (encodeStr template) (mkStrContext context)
    >>= TL.putStrLn

template = "Hello, {{name}}!\n\nYou have {{unread}} unread messages." 

context "name" = MuVariable "Haskell"
context "unread" = MuVariable (100 :: Int)

Read Mustache documentation for template syntax; consult README for more details.

Whats’s new in 0.6.1?

Most of the new features in this release deal with generic contexts.

Context merging

composeCtx is a left-leaning composition of contexts. Given contexts c1 and c2, the behaviour of (c1 <> c2) x is following: if c1 x produces ‘MuNothing’, then the result is c2 x. Otherwise the result is c1 x. Even if c1 x is ‘MuNothing’, the monadic effects of c1 are still to take place.

Generic contexts for more datatypes

The mkGenericContext function now supports additional datatypes like Maybe (with Nothing being an empty/null value) and Either.

Context modifiers and renaming

The new mkGenericContext' is a generalized version of mkGenericContext and it takes two addition arguments. The first one, of type (String -> String) is simply a renaming function, similar to fieldLabelModifier of aeson. To see this feature in action, consider the following example:

{-# LANGUAGE DeriveDataTypeable #-}

import Text.Hastache 
import Text.Hastache.Context 

import qualified Data.Text.Lazy as TL 
import qualified Data.Text.Lazy.IO as TL 

import Data.Data (Data, Typeable)
import Data.Decimal
import Data.Generics.Aliases (extQ)

data Test = Test {f :: Int}
     deriving (Data, Typeable)

val1 :: Test
val1 = Test 1

val2 :: Test
val2 = Test 2

r "f" = "foo"
r x   = x

example :: Test -> IO TL.Text
example v = hastacheStr defaultConfig
                        (encodeStr template)
                        (mkGenericContext' r defaultExt v)

template = "An integer: {{foo}}"

main = do
  example val1 >>= TL.putStrLn
  example val2 >>= TL.putStrLn

In the example we use the renaming function r to rename a field “f” to “foo”.

The second additional argument is a query extension, of type Ext:

type Ext = forall b. (Data b, Typeable b) => b -> String

A query extension is a way of turning arbitrary datatypes into strings. This might come in very handy, if you want to generate mustache contexts from records/datatypes that contain non-primitive datatypes (from non-base modules) that you want to display. Before 0.6.1, if you had a record that contained, for example, a Decimal field, and you wanted to convert it to a context and access that field, you were simply out of luck. With this release you can basically extend the mkGenericContext' function to support any datatypes you want! Once again, I believe an example is worth a thousand words, so let us consider a slightly modified version of the example above:

{-# LANGUAGE DeriveDataTypeable #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE StandaloneDeriving #-}

-- Custom extension function for types that are not supported out of
-- the box in generic contexts
import Text.Hastache 
import Text.Hastache.Context 

import qualified Data.Text.Lazy as TL 
import qualified Data.Text.Lazy.IO as TL 

import Data.Data (Data, Typeable)
import Data.Decimal
import Data.Generics.Aliases (extQ)

data DecimalOrInf = Inf | Dec Decimal deriving (Data, Typeable)
deriving instance Data Decimal
data Test = Test {n::Int, m::DecimalOrInf} deriving (Data, Typeable)


val1 :: Test
val1 = Test 1 (Dec $ Decimal 3 1500)

val2 :: Test
val2 = Test 2 Inf

query :: Ext
query = defaultExt `extQ` f
  where f Inf = "+inf"
        f (Dec i) = show i

r "m" = "moo"
r x   = x

example :: Test -> IO TL.Text
example v = hastacheStr defaultConfig
                        (encodeStr template)
                        (mkGenericContext' r query v)

template = concat [ 
     "An int: {{n}}\n",
     "{{#moo.Dec}}A decimal number: {{moo.Dec}}{{/moo.Dec}}",
     "{{#moo.Inf}}An infinity: {{moo.Inf}}{{/moo.Inf}}"
     ] 

main = do
  example val1 >>= TL.putStrLn
  example val2 >>= TL.putStrLn

As you can see, the query extensions are combined using the extQ function from Data.Generics, and the “unit” of this whole thing is defaultExt function.

Links

• hackage
• source code
• issue tracker

Acknowledgments

This release would not have been possible without Tobias Florek, Edsko de Vries, Janne Hellsten, @clinty on Github, Stefan Kersten, Herbert Valerio Riedel, and other people who were submitting issues, patches, and requests.

Hastache is a Haskell implementation of the mustache templating system.

Quick start

cabal update
cabal install hastache

A simple example:

import Text.Hastache 
import Text.Hastache.Context 
import qualified Data.Text.Lazy.IO as TL

main = hastacheStr defaultConfig (encodeStr template) (mkStrContext context)
    >>= TL.putStrLn

template = "Hello, {{name}}!\n\nYou have {{unread}} unread messages." 

context "name" = MuVariable "Haskell"
context "unread" = MuVariable (100 :: Int)

Read Mustache documentation for template syntax; consult README for more details.

Whats’s new in 0.6?

The interface of the library has been switched from ByteString to (lazy) Text. That means, for example, that the type of hastcheStr function is now the following:

hastacheStr
:: MonadIO m => MuConfig m	-> Text	-> MuContext m -> m Text

The generic context generation (mkGenericContext) now supports functions of the types Text -> Text and Text -> m Text, as well as types with multiple constructors. That is, given a Haskell datastructure

data A = A { str :: String }
       | B { num :: Int }

it is possible to write a template like this:

{{#A}}
A : {{str}}
{{/A}}
{{#B}}
B : {{num}}
{{/B}}

Please take a look at the multiple constructors example if you are interested.

Additionally, a couple of new MuVar instances has been added.

Links

• hackage
• source code
• issue tracker

Acknowledgments

Special thanks to Mark Lentczner, Alexander Voikov and others who reported issues, tested the library and provided feedback.

Representable functors are functors that are isomorphic to $Hom(A, -)$ or $Hom(-, A)$ for some object $A$. In the note I give examples of some simple representable functors and prove that Maybe is not representable.

The result of checking for the sandbox is cached, which is probably unnecessary; it updates only when the user performs a cabal command or changes a directory.

After you place the contents of the script in your .zshrc file (or in a similar location), you should update your $PROMPT to use $(sandbox_prompt). The prompt I am using, by the way, is

HOST_COLOR=%{$fg_bold[yellow]%}
local ret_status="%(?:%{$fg_bold[green]%}:%{$fg_bold[red]%})%?%{$reset_color%}"
PROMPT=$'\n$(ssh_connection)$HOST_COLOR%n@%m%{$reset_color%}$(my_git_prompt)$(sandbox_prompt) : %~\n[${ret_status}] %# '

and it is based on the oh-my-solarized theme.

0.2

Keeping up with the Scotty 0.6 updating. The introduction of the custom exception types is reflected in the scotty-hastache package as well. The main types are now polymorphic over an exception type

type ScottyH e = ScottyT e HState
type ActionH e = ActionT e HState

In addition, the specialized (to Text) types and runners are introduced

type ScottyH' = ScottyH Text
type ActionH' = ActionH Text
scottyH'     :: Port -> ScottyH' () -> IO ()
scottyHOpts' :: Options -> ScottyH' () -> IO ()

Thanks to Kirill Zaborsky for pointing out the incompatibility with 0.6.

As always, the scotty-hastache library is available on Hackage and on GitHub

Annotation

Computer scientists have been paying attention concurrency and parallelism since the dawn of the computing as science. In this talk I would like to discuss the topic called true concurrency – the concurrency in presence of parallelism, as opposed to concurrency defined by interleaving.

Carl Adam Petri played a crucial role in the area of true concurrency, he was interested in analyzing the relations and connections between the abstract machines (that of Turing, Church and others) and their physical counterparts. Petri observed that the assumption of the global state is unrealistic in big systems and he proposed to replace the “execution semantics’’ with the semantics of”nonsequential proceses’’, the notion which gave rise to the theory of occurrence nets and unfoldings. That corpus of work has be introduced originally to solve the very theoretical problem: to give precise semantics to concurrent systems. Goltz, Reisig, Nielsen, Plotkin, Winskel, Engelfreit, Montanari are among the people who have contributed to the topic.

Later on, with the rise of model checking techniques, it has been discovered that partial-order semantics (unfoldings) may be used in practice for formally verifying systems. McMillan, Esparza, Romer, Vogler, Khomenko and others have pioneered what is now can be called a “partial order approach to model checking”.

True concurrency is still a thriving field, and in my talk I hope to touch the foundations, as well as pragmatic applications of the approaches derived from the theoretical foundations of the subject.

Slides

.PDF link

• diagrams-ghcjs finally got text support

Big thanks to Joel Burget for the implementation.

The testsuite demo for the package can be found at http://co-dan.github.io/ghcjs/diagrams-ghcjs-tests/Main.jsexe/ as usual.

• New interactive widgets

We got rid of the old wizard-like widgets in favour of “all-in-one” style widgets. Thanks to Brent and Luite who came up with the type trickery to get this done.

• New flat design

In order to match your slick & flat iOS7 style I’ve rolled out an update to Bootstrap 3.

• Ability to paste code with errors. (It’ll ask you to make sure you didn’t make a mistake by accident)
• Ability to quickly include a number of imports. By checking the “import standard modules” checkbox you’ll automatically bring several useful modules into the scope:
  • Diagrams.Prelude
  • Diagrams.Backend.SVG (or Diagrams.Backend.GHCJS if you are making an interactive widget)
  • Data.Maybe
  • Data.Tuple
  • Data.List
  • Data.Char
• Other minor UI fixes such as documentation improvement

Introduction

The project is live at http://paste.hskll.org. The source code can be found at http://github.com/co-dan/interactive-diagrams.

I would like to say that I do plan to continue working on the project (and on adjacent projects as well if possible).

Interactive diagrams

Interactive diagrams is a pastebin and a set of libraries used for dynamically compiling, interpreting and rendering the results of user inputted code in a secure environment.

The user inputs some code and the app compiles and renders it. Graphical output alongside with code can be useful for sharing the experiments, teaching beginners an so on. If the users inputs a code that can not be rendered on the server (i.e.: a function), the app produces an HTML5/JS widget that runs the corresponding code.

The produced libraries can be used in 3rd party services/programs.

Screenshot

Technology used

The pastebin is powered by Scotty and scotty-hastache, the access to PosgreSQL db is done via the excellent persistent library. The compilation is done using GHC and GHCJS inside the workers processes powered by the restricted-workers library.

You can read some my previous report on this project which is still pretty relevant.

I plan on updating the documents on the wiki sometime soon.

Progress

The bad news is that I don’t think I was able to 100% complete what I originally envisioned. The good news is that I seem to know, almost exactly, what do I want to improve and how to do that. As I’ve mentioned I plan on continuing with the project and I hope that the project will grow and improve.

One thing that I felt was annoying is the (technical) requirement to use GHC HEAD. Because of that a lot of packages required updates and fixes. Due to changes in GHC and bugfixes in GHCJS I had to perform the tiring and not so productive procedure of updating all the necessary tools, rebuilding everything and so on. But I guess that’s just how computers work and I am sure that in the future (with the release of GHC 7.8 and a new Haskell Platform) the maintenance and installation will be much easier. Another thing that took a lot of my time was configuring the system and setting up the necessary security measures, which was also necessary.

Other stuff that kinda slowed thing down include: the lack of a good build system, in some cases non-American timezone (actually I think that the fact that my mentor, Luite Stegeman, was quite close to me in terms of timezones allowed us to communicated very frequently, as we did), the lack of knowledge of the tools I used (although you can think of it this way: I had an ability to learn exciting new things ;] ).

Among the grand things I plan to do: release a library for manipulating Haskell AST at the GHC level; make an IRC bot using the eval-api and restricted-workers; continue writing my notes/tutorials about GHC API (I have a few drafts laying around).

Some code refactoring should come along and a number of features for the pastebin should be implemented.

Feelings

When the end of the program was approaching I predicted that I would have that sort of conflicted feelings that you usually get when you finish reading a good book – one part of you feels happy because you had an enjoyable experience, yet another part of you doesn’t feel so giddy, because the thing that you enjoyed is over. Well, I didn’t get this with GSoC. I did feel happy, but I didn’t get this touch of sadness. GSoC was a way for me to get into collaborating with people on real-world open source projects, and the end of GSoC for me is a beginning of something else. I can use my experience now to write better code, write more code and write useful code.

Concluding

I had a very exciting summer and I would positively recommend anyone eligible to participate in the Google Summer of Code program. There is, however, a thing to remember. Programmers are known to be the kind of people who set ambitious goals. Reach for something inspiring, ambitious, yet realistic. Make sure to find something in between, that way you’ll have a concrete target that you know that you are able to achieve, but you also have a room for improvement.

PS. Acknowledgments

I would like to thank various people who helped me along the summer: Luite Stegeman, Brent Yorgey, Carter Schonwald, Daniel Bergey, Andrew Farmer; everyone in #diagrams, everyone in the #haskell channel who patiently answered my question; everyone on GitHub who responded to my comments, questions and pull requests. The Haskell community is lucky to have a huge amount of friendly and smart people.

Example code:

{-# LANGUAGE OverloadedStrings #-}
module Main where

import Text.Hastache
import Web.Scotty.Trans as S
import Web.Scotty.Hastache

main :: IO ()
main = scottyH 3000 $ do
  setTemplatesDir "templates"
  -- ^ Setting up the director with templates
  get "/:word" $ do
    beam <- param "word"
    setH "action" $ MuVariable (beam :: String)
    -- ^ "action" will be binded to the contents of 'beam'
    hastache "greet.html"

This is possible due to the recent changes introduced into the latest scotty-0.5 release, in particular the switch to monad transformers. You can read the scotty-0.5 release notes here.

The scotty-hastache library is available on Hackage and on GitHub

I’ve picked up the development of the diagrams-ghcjs backend for the infamous diagrams library. This backend is what we use for the interactive-diagrams pastebin and it renders diagrams on an HTML5 Canvas using the canvas bindings for ghcjs. The diagrams-ghcjs is a fork of (unmaintained?) diagrams-canvas backend by Andy Gill and Ryan Yates.

The library is still WIP and it requires bleeding edge versions of ghcjs, ghcjs-canvas and ghcjs’ shims to function.

The library is scheduled to be released together with ghcjs.

Demo

The current demo can be found at http://co-dan.github.io/ghcjs/diagrams-ghcjs-tests/Main.jsexe/. It is supposed to work in both Firefox and Chrome.

Problems

• Text is not implemented. Some work is being done in the text-support branch. Generally, it has been proven hard to implement good text support, even diagrams-svg backend text support is limited;
• Firefox still does not support line dashing via ‘setLineDash’ and ‘lineDashOffset’. As a result we need to have additional shims.

Intro

It’s hard to get into writing code that uses GHC API. The API itself is and the number of various functions and options significantly outnumber the amount of tutorials around.

In this series of blog posts I’ll elaborate on some of the peculiar, interesting problems I’ve encountered during my experience writing code that uses GHC API and also provide various tips I find useful.

I have built for myself a small layer of helper functions that helped me with using GHC API for the interactive-diagrams project. The source can be found on GitHub and I plan on refactoring the code and releasing it separately.

Today I would like to talk about a different ways of bringing contents of Haskell modules into scope, a process that is necessary for evaluating/interpreting bits of code on-the-fly.

Many of the points I make in this post are actually trivial, but nevertheless I made all of the mistakes I mentioned in this, perhaps post due to my naive approach of quickly diving in and experimenting, instead of reading into the documentation and source code. Now I actually realize that this post should been the first in the series, since it probably deals with more basic (and fundamental) stuff than the previous two posts. But anyway, here it is.

Interpreted modules

Imagine the following situation: we have a Haskell source file with code we want to load dynamically and evaluate. That is a basic task in the GHC API terms but nevertheless there are some caveats. We start with the most basics.

Let us have a file ‘test.hs’ containing the code we want to access:

module Test (test) where
test :: Int
test = 123

The basic way to get the ‘test’ data would be to load ‘Test’ as an interpreted module:

import Control.Applicative
import DynFlags
import GHC
import GHC.Paths
import GhcMonad            (liftIO) -- from ghc7.7 and up you can use the usual
    -- liftIO from Control.Monad.IO.Class
import Unsafe.Coerce

main = defaultErrorHandler defaultFatalMessager defaultFlushOut $ do
    runGhc (Just libdir) $ do
        -- we have to call 'setSessionDynFlags' before doing
        -- everything else
        dflags <- getSessionDynFlags
        -- If we want to make GHC interpret our code on the fly, we
                  -- ought to set those two flags, otherwise we
                  -- wouldn't be able to use 'setContext' below
        setSessionDynFlags $ dflags { hscTarget = HscInterpreted
                                    , ghcLink   = LinkInMemory
                                    }
        setTargets =<< sequence [guessTarget "test.hs" Nothing]
        load LoadAllTargets
        -- Bringing the module into the context
        setContext [IIModule $ mkModuleName "Test"]

        -- evaluating and running an action
        act <- unsafeCoerce <$> compileExpr "print test"           
        liftIO act

The reason that we have to use HscInterpreted and LinkInMemory is that otherwise it would compile test.hs in the current directory and leave test.hi and test.o files, which we would not be able to load in the interpreted mode. setContext, however will try to bring the code in those files first, when looking for the module ‘Test’

dan@aquabox
[0] % ghc --make target.hs -package ghc
[1 of 1] Compiling Main             ( target.hs, target.o )
Linking target ...

dan@aquabox
[0] % ./target
123

Let’s try something fancier like printing a list of integers, one-by-one.

main = defaultErrorHandler defaultFatalMessager defaultFlushOut $ do
    runGhc (Just libdir) $ do
        dflags <- getSessionDynFlags
        setSessionDynFlags $ dflags { hscTarget = HscInterpreted
                                    , ghcLink   = LinkInMemory
                                    }
        setTargets =<< sequence [guessTarget "test.hs" Nothing]
        load LoadAllTargets
        -- Bringing the module into the context
        setContext [IIModule $ mkModuleName "Test"]

        -- evaluating and running an action
        act <- unsafeCoerce <$> compileExpr "forM_ [1,2,test] print"
        liftIO act

But when we try to run it..

dan@aquabox
[0] % ./target
target: panic! (the 'impossible' happened)
  (GHC version 7.6.3 for x86_64-apple-darwin):
        Not in scope: `forM_'

Please report this as a GHC bug:
http://www.haskell.org/ghc/reportabug

Hm, it looks like we need to bring Control.Monad into the scope.

This brings us to the next section.

Package modules

Naively, we might want to load Control.Monad in a similar fashion as we did with loading test.hs

main = defaultErrorHandler defaultFatalMessager defaultFlushOut $ do
    runGhc (Just libdir) $ do
        dflags <- getSessionDynFlags
        setSessionDynFlags $ dflags { hscTarget = HscInterpreted
                                    , ghcLink   = LinkInMemory
                                    }
        setTargets =<< sequence [ guessTarget "test.hs" Nothing
                                , guessTarget "Control.Monad" Nothing]
        load LoadAllTargets
        -- Bringing the module into the context
        setContext [IIModule $ mkModuleName "Test"]

        -- evaluating and running an action
        act <- unsafeCoerce <$> compileExpr "forM_ [1,2,test] print"
        liftIO act

Our attempt fails:

dan@aquabox
[0] % ./target
target: panic! (the 'impossible' happened)
  (GHC version 7.6.3 for x86_64-apple-darwin):
        module `Control.Monad' is a package module

Please report this as a GHC bug:
http://www.haskell.org/ghc/reportabug

Huh, what? I thought guessTarget works on all kinds of modules.

Well, it does. But it doesn’t “load the module”, it merely sets it as the target for compilation, basically it (together with load LoadAllTargets) does what ghc --make does. And surely it doesn’t make much sense to ghc --make Control.Monad when Control.Monad is a module from the base package. What we need to do instead is to bring the compiled Control.Monad module into scope. Luckily it’s not very hard to do with the help of the simpleImportDecl :: ModuleName -> ImportDecl name:

main = defaultErrorHandler defaultFatalMessager defaultFlushOut $ do
    runGhc (Just libdir) $ do
        dflags <- getSessionDynFlags
        setSessionDynFlags $ dflags { hscTarget = HscInterpreted
                                    , ghcLink   = LinkInMemory
                                    }
        setTargets =<< sequence [ guessTarget "test.hs" Nothing ]
        load LoadAllTargets
        -- Bringing the module into the context
        setContext [ IIModule . mkModuleName $ "Test"
                   , IIDecl
                     . simpleImportDecl
                     . mkModuleName $ "Control.Monad" ]

        -- evaluating and running an action
        act <- unsafeCoerce <$> compileExpr "forM_ [1,2,test] print"
        liftIO act

And we can run it

dan@aquabox
[0] % ./target
1
2
123

Compiled modules

What we have implemented so far corresponds to the :load* command in GHCi, which gives us the full access to the source code of the program. To illustrate this let’s modify our test file:

module Test (test) where

test :: Int
test = 123

test2 :: String
test2 = "Hi"

Now, if we try to load that file as an interpreted module and evaluate test2 nothing will stop us from doing so.

dan@aquabo
[0] % ./target-interp
(123,"Hi")

To use the compiled module we have to bring Test into the context the same way we dealt with Control.Monad

main = defaultErrorHandler defaultFatalMessager defaultFlushOut $ do
runGhc (Just libdir) $ do
    dflags <- getSessionDynFlags
    setSessionDynFlags $ dflags { hscTarget = HscInterpreted
                                , ghcLink   = LinkInMemory
                                }
    setTargets =<< sequence [ guessTarget "Test" Nothing ]
    load LoadAllTargets
    -- Bringing the module into the context
    setContext [ IIDecl $ simpleImportDecl (mkModuleName "Test")
               , IIDecl $ simpleImportDecl (mkModuleName "Prelude")
               ]
    printExpr "test"
    printExpr "test2"

printExpr :: String -> Ghc ()
printExpr expr = do
    liftIO $ putStrLn ("-- Going to print " ++ expr)
    act <- unsafeCoerce <$> compileExpr ("print (" ++ expr ++ ")")
    liftIO act

Output:

dan@aquabox : ~/snippets/ghcapi
[0] % ./target
-- Going to print test
123
-- Going to print test2
target: panic! (the 'impossible' happened)
  (GHC version 7.6.3 for x86_64-apple-darwin):
	Not in scope: `test2'
Perhaps you meant `test' (imported from Test)

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

Note: I had to bring the Prelude into context this time, like a regular module. I tried setting the ideclImplicit option in ImportDecl, but it didn’t work for some reason. Maybe it actually supposed to do not what I think it supposed to do, but something else.

Outro

So, that is it, we have managed to dynamically load Haskell source code and evaluate it. I can only refer you to the GHC haddocs for specific functions that we used in this post, most of them contain way more options that we used and they might prove to be useful to you.