The Commercial Haskell SIG members want to help people adopt

Haskell. What would help? Data beats speculation, so FP Complete

recently emailed surveys to over 16000 people interested in

Haskell. The questions were aimed at identifying needs rather than

celebrating past successes, and at helping applied users rather

than researchers.

Over 1240 people sent detailed replies, spending over 250 person-hours to provide their answers.

This rich data set includes extensive information on Haskell

user needs. We are open-sourcing the entire anonymized data set,

downloadable by clicking here [.zip]. There are numeric ratings and extensive textual

comments. Feel free to analyze the data -- or just read the

summaries -- and share your most helpful and actionable conclusions

with the community. We will too.

First Results

Although we have completed only basic analysis, here are some of

our first findings -- those so clear that they show up on even the

most basic examination of the aggregate data.

• Satisfaction with the language, the compiler, and the community are high.

• Among non-students, 58% would recommend Haskell for a project

  at their workplace, but only 26% actually use it at work -- partly

  due to colleagues unfamiliar with Haskell who see it as requiring

  skills that are hard to obtain, or who need to see more success

  stories. Would improvement to colleagues' perceptions make a

  difference in the team's choice of Haskell for a project? 33% of

  respondents rated this "crucial" and another 26% said it would be

  "important", while only 16% said it would be a "slight help" or no

  help.

• Package management with cabal is the single worst aspect of

  using Haskell. Asked if improvements to package management would

  make a difference to their future choice of Haskell for a project,

  38% said it would be "crucial" and a further 29% said it would be

  "important". Comments connected cabal with words like hell, pain,

  awful, sucks, frustrating, and hideous. Only this topic showed such

  grave dissatisfaction.

• Documentation improvements are a very high priority. For

  example, users need more concrete tutorials and templates showing

  them exactly what kinds of problems Haskell is good at solving, and

  exactly how to implement such programs completely. 65% of

  respondents said improvements to documentation and learning

  resources would be crucial or important, and a further 23% said

  they would be helpful. However, comments did not begin to approach

  the level of concern seen with cabal.

• Skills are a priority. Users need to see that people with

  Haskell skills are readily available, and that Haskell skills are

  quite feasible to learn. A majority of respondents said an

  improvement in availability of skilled personnel would make an

  important or crucial difference to them, and many also expressed

  concern about their or colleagues' abilities to learn the needed

  concepts and skills.

We have started deeper statistical analysis of the data, and we

hope that some readers of this post will perform -- and share --

even better analyses than we can. New issues may become clearer by

clustering or segmenting users, or through other statistical

techniques. Also, we may find more clarity about needs through

deeper study of textual responses. Follow-up studies are also a

possibility.

We propose that the community, given this large and detailed

data set, should set some of its priorities in a data-driven manner

focused on user-expressed needs. This effort should be

complementary to the ongoing research on issues of historic Haskell

strength such as language design and runtime architecture.

Areas for Further Work

We request that useful findings or insights derived from the

open-sourced data set be shared with the community, including

attribution of the source of the data and the methods of analysis

used.

The data collected strongly invites clustering or segmentation,

so as to identify needs of different sub-populations. FP Complete

has already begun one such study.

The data collected includes extensive textual remarks which

should be studied by knowledgeable people for insights. Automated

text analysis methods may also be applicable.

Cost-benefit analysis seems worthwhile: based on identified

needs, what improvements would help the most people, to the

greatest degree, at the least cost and/or the least delay? A method

to match volunteer contributors with identified high-payoff

projects also seems worthwhile.

It would be useful to merge the data from versions 0.1 and 0.2

with version 1.0 of the survey, since 1.0 includes only 71% of the

total answers received. Differences between the questions and

scales make this a nontrivial, but still realistic, goal.

If important new hypotheses require testing, or if further

detail is needed, we intend to conduct follow-up research at some

future date among users who volunteered their email addresses for

follow-up questions.

A future repeat survey could determine which improvement efforts are successful.

Methodology Notes

This was not a survey of the general public, but of a population

motivated to provide feedback on Haskell. Invitees included 16165

non-opted-out email addresses gathered from FP Complete's website,

in randomized order. Due to privacy considerations this list will

not be published, but FP Complete was able to use it to contact

these users since the survey was directly related to their

demonstrated interest in Haskell. The high quality of the list is

reflected in the extremely high response rate (7.7%), the low

bounce rate (1.9%), and the low unsubscribe rate (also 1.9%).

Surveys were conducted using SurveyGizmo.com, with an email

inviting each participant to click a link to a four-page Web-based

survey. Survey form 0.1 invitations went to 1999 users of whom 190

completed the survey. Survey form 0.2, incorporating some edits,

went to 2000 users of whom 170 completed the survey. Survey form

1.0, incorporating further edits, went to 12166 users of whom 894

completed the survey.

Form 0.2 incorporated edits to eliminate questions yielding

little information about how to help users, either because

satisfaction was very high (the language itself, the compiler, the

community) or because two questions were redundant. Also, new

questions inspired by textual responses to form 0.1 were

included.

Form 1.0 incorporated further such edits. Also, the rating scale

was changed to ask about helping the user's (and team's) future

choice of Haskell rather than current usefulness/difficulty. The

ratings questions were displayed under the heading "Would

improvements help you and your group to choose Haskell for your

future work?"

Responses were processed anonymously, but users were given the

option to fill in their email address if they would accept

follow-up questions, and the option to name their

company/organization. Users were informed that the survey results,

without these fields, would be shared with the community.

Acknowledgments

We are grateful to the many, many people who spent their

valuable time and expertise completing and returning their survey

forms. Thanks to Dr. Tristan Webb and Ms. Noelle McCool-Smiley,

both of FP Complete, for their material help in formulating and

conducting the survey. Thanks to FP Complete's corporate customers

for providing the revenues that allow us to fund this and other

community projects. Thanks to the Commercial Haskell SIG for

providing the motivation behind this project. Thanks to the many

volunteers who've spent absolutely huge amounts of time and

expertise making Haskell as good as it is today, and who continue

to make improvements like those requested by the survey

participants. Thanks to the companies that allow some of their

staff to spend company time making such contributions to the common

good. Special thanks to the late Professor Paul Hudak; may we all

strive to live up to his example.