Nov 6, 2014
On September 28 we released 3.1 of the FP Complete public
services. We've discussed this a bit already, but in this blog
post, we want to share some of the more technical details of what
we've added.
Laundry list of features
We've added a whole bunch of new features to FP Haskell Center. Here are some of the most notable:
You can now access all features of the IDE for free
It's faster
More file type support: Literate Haskell, hs-boot, and C files
Git Shell allows you to run arbitrary Git commands on a fully interactive command line
In addition to GHCi runs of code, you can build and run binaries inside the IDE
Ability to pass arbitrary flags to deployment builds
View usages of an identifier
Better support for Template Haskell-included files (like Hamlet)
Modified data files immediately appear to your application (good for modifying a CSS file in a web app, for example)
Better session management, including logging out of all sessions
School of Haskell supports autorun and Disqus comments
Keep reading for a drill down of some of this list. And for some
lower-level, GHC-facing bits of our work, jump
ahead to the "faster" sections.
Open Publish model
As
announced previously, we've decided to make FP Haskell Center
even more open. Since our 3.1 release, all IDE features are turned
on for all users, with and without a paid account.
The new FPHC terms & conditions no longer restrict the free
community version to non commercial use. Even anonymous users can
access the full spread of features in the IDE, from type checking
to executable generation, and even Git repositories (though you'll
definitely want to log in to be able to use that properly).
There are two distinctions between our community and paid licenses now:
Commercial licenses provide more compute power.
Under community licenses, all commits made in the IDE will be
published. Similar to the GitHub model, free projects are open
projects. You'll need a commercial license for private projects,
which can still be acquired by contacting FP
Complete Sales. So now when you commit, you'll see this
message:
Committed changes are public
Any new commits are now made public and will make your project
public. Existing private projects will not be published until you
make a new commit.
NOTE: If you have an existing project with commits, it will remain private until your next commit.
Additionaly, there are now no restrictions on how you license
your published code. This means that although you give others the
right to view your published code, they receive no implicit license
to distribute your code or use it in derived works. You can still
explicitly license your code under an open source license if you
choose.
Autorun active code samples
Based on feedback (mainly from Edward Kmett), there's a new
directive in the code fence syntax in School of Haskell posts,
which will auto-run the code snippet when the page loads. This will
be especially nice for showing graphical web pages or to begin an
interactive process. The code is written as:
School of Haskell improvements
The School of Haskell now supports adding Disqus comments to your own tutorials (yet again suggested by Ed).
To enable this, go into your account settings under “Profile” and you can choose:
☑ Use site default (currently: no comments)
☐ Disable comments
☐ Use FP Complete's Disqus account
☐ Use your own Disqus account
Disqus account ID:
[ ]
To use your own Disqus account, hit the “Use your own Disqus
account” checkbox and put your account id in the box. Then go into
your Disqus site settings under the “Advanced” tab and add
fpcomplete.com under the “Trusted Domains” text box.
GHC 7.8, new libraries
We've updated our default compiler version to GHC 7.8. We still
provide a 7.4 environment, but it is considered deprecated, and we
recommend users upgrade to 7.8. In addition, as usual, we've
updated our library set to the newest versions of upstream
packages, and will begin releasing unstable package snapshots next
month.
Defer type errors
Together with GHC 7.8 come some new compiler features. For the
most part, these are simply available to you without any IDE
support. We added one nifty feature though: You can now defer type
errors using the checkbox in the Messages tab:
☑ Enable suggestions ☑ Enable warnings ☑ Defer type errors
This will change type errors into warnings which are instead
thrown at runtime. A very useful way to keep getting type
information when you have a simple type error elsewhere in the
code.
This plays in very nicely with the new Type Holes feature. Now you can program Python in Haskell!
Improved root & filtering
In the “Root & Filtering” tab in Settings, there is a way to
change the root directory of your project, this is similar to doing
a cd in your terminal. This is useful if you have
several projects in one repository and you want to just compile the
modules within one directory, rather than everything in the whole
repository.
You can also ignore certain files using the blacklist input, e.g. src/Main.hs will hide that file from the list. This can be useful for hiding things from compilation.
This feature is still considered experimental.
Faster
You should notice a definite improvement in responsive in the
IDE. We've actually implemented many different changes to make this
happen. Our network communication layer, for example, has less
overhead involved, by removing some unnecessary parsing from our
reverse proxy machines. We've also refactored quite a bit of our
async architecture to provide better concurrency for backend
actions.
However, the biggest change comes to how we interact with GHC. We use GHC for a few different things in the IDE:
Type checking code
Provide information on identifier locations, types, usages, etc
Generating and running bytecode
Those first two bullets dovetail together rather nicely, though
the second can take more time than the first. However, the third
point doesn't play so nicely with the other two. The issue is that,
while GHC is running the bytecode, it can't continue to
compile new code. And this is definitely something we want to
allow. For example, a common use case it running a web application
in the IDE, while continuing to hack on it while it's running in
the background.
Before the 3.1 release, our solution was to have three copies of
GHC running for each project: one to do a quick typecheck, one to
extract type information, and one for running code. This meant you
would get error messages quickly, but couldn't always get type
information right away. It also meant your project required much
more memory, which overall slowed things down.
Our new architecture involves just a single GHC process running
for each project. We load code into this process, and it
typechecks, extracts information, and generates bytecode, all at
the same time. The trick is what happens when you press run? We
would like to use the bytecode already available in the current GHC
process, without tying that process up on just running the
bytecode.
The solution we ended up at is calling forkProcess.
However, it wasn't quite as simple as that. To quote
forkProcess's documentation:
forkProcess comes with a giant warning: since any other running
threads are not copied into the child process, it's easy to go
wrong: e.g. by accessing some shared resource that was held by
another thread in the parent.
Sure enough, we ran into exactly this bug almost immediately.
And fortunately, we have some very good news to report: it's fixed!
You can see more information in GHC trac tickets 9295 and 9296
(and check Phabricator for some
great reaction gifs). These changes have been merged upstream into
GHC. Since writing those patches, we've stress tested our IDE
significantly, and have no longer been able to reproduce any
forkProcess bugs, which looks incredibly encouraging.
Note, though, that there are still some issues around forkProcess. If you're planning on using it in your
own code, you should be aware that you can still run into problems,
especially with multithreaded code.
One other change we made was switching to a dynamically linked
GHC process. We'd wanted to do this for a while due to improved
memory usage. Ultimately, we had to make the switch due to a bug in C static initializers. Now we get the nice benefit that separate
processes on the same machine can share memory space for their
dynamic libraries.