Markov Chain Monte Carlo Sampling (3)

How to manage NixOS systems using Nix flakes.

The second call for projects for the Tweag Open Source Fellowship is open and brings financial support and mentorship for Open source contributors.

Announcement of the upcoming QualifiedDo language extension.

How to setup a Buildkite-based CI for Nix projects with workers running on GCP.

How we ensured that the implementation of random v1.2 produces higher quality random numbers than v1.1.

How Nix flakes enable caching of evaluation results of Nix expressions.

Looking back at the journey which brought us there, and forward to what still lies ahead.

How Nix is collaborating with Software Heritage for long-term software reproducibility.

Benchmarks of inline-java and a discussion of its performance trade-offs.

Tweag introduces our first cohort of Open Source Fellows.

An introduction to Nix flakes and a tutorial on how to use them.

How to build a complete Haskell project with Bazel.

How to derive instances via arbitrary isomorphisms.

Imprecise, synchronous, asynchronous exceptions: what do they mean, and what to do with them.

How to set up a remote Bazel cache, and why it is a good idea.

This blog post describes the tradeoffs of the choice between eager and lazy instantiation of type variables in GHC.

During my internship, I implemented the explicit specificity proposal to make GHC type signatures complete.

Peirce divides reasoning into three complementary processes: deduction, abduction and induction. In this post, I will show how these logical processes relate to software.

In this blog post series, we're going to lead you through Bayesian modeling in Haskell with the monad-bayes library. In the third part of the series, we setup a simple Bayesian neural network.

A primer on the interaction between linear types and exceptions in Haskell.

The first call for the new Tweag Open Source Fellowship is open. It offers financial support and mentorship to open source contributors.

In this post about inline-java I aim to walk you through the upcoming safe interface of the library, which allows detecting memory management mistakes at compile time using linear types.

This post describes a profiling technique for Haskell codebases that yields faithful results and is well adapted to large repositories.

Haskell's Data and Control module hierarchies have always bugged me. Now, I understand that the intuition behind the Data/Control separation is rooted in a deep technical justification.

In this second post of Tweag's four-part series, we discuss Gibbs sampling, an important MCMC-related algorithm which can be advantageous when sampling from multivariate distributions. Two different examples and, again, an interactive Python notebook illustrate use cases and the issue of heavily correlated samples.

Asterius compiles Haskell code into WebAssembly code to be executed in a browser or in Node.js. It has reached a new milestone by being able to compile the diagrams library and its dependencies.

How to use the Makam metalanguage to implement an executable formal semantics for a simple language.

Learn how Nix's new post-build hooks feature can automatically sign and upload artifacts to a binary cache in a trusted way.

Here's Part 2 in Tweag's Series about Bayesian modeling in Haskell with the monad-bayes library.

We're happy to announce the first release of Porcupine, an open source framework to express portable and customizable data pipelines.

In this first post of Tweag's four-part series on Markov chain Monte Carlo sampling algorithms, you will learn about why and when to use them and the theoretical underpinnings of this powerful class of sampling methods. We discuss the famous Metropolis-Hastings algorithm and give an intuition on the choice of its free parameters. Interactive Python notebooks invite you to play around with MCMC yourself and thus deepen your understanding of the Metropolis-Hastings algorithm.

We're happy to announce the first release of Ormolu, a formatter for Haskell source code.

Bazel gets native support for third-party Haskell libraries and building Cabal packages since the 0.10 release of rules_haskell.

I got the opportunity to work on Bazel's Persistent Worker Mode for GHC during my internship at Tweag. My goal was to improve the mode of communication between Bazel and the Haskell GHC compiler.

In this blog post series, we're going to lead you through Bayesian modeling in Haskell with the monad-bayes library. In the first part of the series, we introduce two fundamental concepts of `monad-bayes`: `sampling` and `scoring`.

I got the opportunity to work on Asterius, a new Haskell to WebAssembly compiler, during my internship at Tweag. My task was to get anything numerics-related stabilized in its compiled code.

People keep asking me, “What's the point of Haskell, why do you folks even use it?”. How can you answer this?

We visualize large collections of Haskell and Python source codes as 2D maps using methods from Natural Language Processing (NLP) and dimensionality reduction and find a surprisingly rich structure for both languages. Clustering on the 2D maps allows us to identify common patterns in source code which give rise to these structures. Finally, we discuss this first analysis in the context of advanced machine learning-based tools performing automatic code refactoring and code completion.

Every day we write repetitive code. A lot of it is boilerplate that you write only to satisfy your compiler/interpreter. But how do languages differ in their boilerplate content? We explore these questions using data sets of Python and Haskell code.

Edsger Dijkstra took issue with the “unbridled” use of the programming construct in 1968. He noted that our already limited ability to reason about the dynamic behaviour of imperative programs was nigh impossible in the presence of -jumps to arbitrary labels. With , control flow…

If we think of the average Haskeller, who do we see? A poor, broken person. Tired head falls into the open palms, sobbing, deprived of the basic human right for automated source code formatting. Is it at all conceivable that so many attempts were made and none quite succeeded…

Tweag.io has a bit of a history with language interop. By this point, we created or collaborated with others in the community on HaskellR, inline-c, inline-java, and now inline-js. The original idea for this style of interop was realized in language-c-inline by Manuel Chakravarty…

Inspired by the Event Horizon Telescope images, we develop a quick exploratory study about future possibilities of this technology called the Sneakernet: Could massive data transfer give a new live to the homing pigeon industry? How about using transportation means that are optimized to carry incredible amounts of weight? Or transportation means that are designed to be fast as a bullet?

I want to present you a Terraform plugin for securely managing secrets that was written for Digital Asset, who kindly allowed me to open source it. The general idea of this plugin is to protect secrets by making use of Terraform’s state. Terraform maintains the state of the world…

Today we’re excited to announce lorri, a new tool for NixOS, Linux, and macOS that makes developing with Nix even nicer: When people try lorri, we often hear that it is more magical than they expected. What is lorri? lorri is a nix-shell replacement for project development…

There has been recent online discussions about impredicative encoding (aka final encoding) of free monads. Capabilities-as-type-classes, as embodied by the capability library, are essentially the same thing.

The time has come again to invite applications for Tweag I/O’s paid internship programme this summer. For the past few years Tweag has taken on a number of students interested in developing their experience working on projects of benefit to the wider programming community. This…

In a recent project I was tasked with creating a local testing environment for a Kubernetes cluster the client was actively working on. The main requirements were: Cross platform: It should work on Linux and macOS. Ease of use: The setup should be easy to use. Suitable for CI…

Millions of Jupyter notebooks are spread over the internet - machine learning, astrophysics, biology, economy, you name it. What a great age for reproducible science! Or that's what you think until you try to actually run these notebooks. Then you realize that having understandable high-level code alone is not enough to reproduce something on a computer. JupyterWith is a solution to this problem.

Have you ever needed to execute in another computer a binary that you have built, only to find that it doesn’t work because it depends on shared libraries that aren’t installed? Or even worse, you don’t have admin rights on the machine, so you cannot install the correct…

Haskell is in a way a unique language. It is, to be clear, a language people and businesses use for building “serious stuff”, yet it is also a language which remains a platform for experimentation with functional programming. This means that the teams which develop in Haskell…

The repositories of distributions such as Debian and Nixpkgs are among the largest collections of open source (and some unfree) software. They are complex systems that connect and organize many interdependent packages. In this blog post I'll try to shed some light on them from the perspective of Nixpkgs, mostly with visualizations of its complete dependency graph.

Introduction Haskell and data science - on first sight a great match: native function composition, lazy evaluation, fast execution times, and lots of code checks. These sound like ingredients for scalable, production-ready data transformation pipelines. What is missing then? Why…

Getting your browser to print “Hello World!” is fun, and a milestone for compiler writers targeting the web. Soon expectations change. Can you call existing code? Can you write a moderately complex web app? How fast does it run? Previously, we showed how Haskell can call…

If you’re a conscientious developer like I am my boss is, you probably have a benchmark suite for the programs and libraries you develop. This allows you to see the impact of your changes on the performance of your applications. However, the temporal aspect of these benchmark…

About a year ago, Michael Snoyman made a blog post about the ReaderT pattern. Ostensibly, it’s a blog post about preferring to encode state as -s when you can. But that’s not how we read it. What we saw first and foremost is a story about using extensional type classes describing…

Note: since Mar 19, 2020, we’ve changed the JavaScript import syntax: the -th argument is now instead of . The code snippets in this post have been adjusted accordingly. Previously, we announced the Asterius compiler, a new GHC-backend that translates Haskell to WebAssembly…

Funflow is a workflow management tool. It turns out that workflow management tools and build tools are closely related. So if you’re more familiar with the latter, this post might be of interest to you. We’ll be illustrating some of Funflow’s features by implementing Make on top…

In an earlier blog post, we discussed how streaming libraries help writing composable programs without lazy I/O. We showed, for a simple program, how using a streaming library helps reducing the amount of unaided bookkeeping that the programmer needs to do in order to make it…

Asterius is an experimental WebAssembly backend for GHC. We aim to support most GHC features while making the emitted WebAssembly code as small and fast as possible. The project recently reached an exciting milestone: for the first time, simple Haskell functions can be compiled…

Funflow is a system for building and running workflows. It’s a system we’ve been working on for the last few months with a small number of clients. In this blog post we’ll talk about what workflows are, why we built Funflow, and what we’d like to do with it in the future. What is…

We are happy to announce that Stackage HEAD is now functional. For those who had missed the original blog post, or already forgot it, let’s refresh the idea behind Stackage HEAD. Each Stackage Nightly is a build plan that includes a large set of real-world packages that are known…

We’re hiring multiple software engineers to join our team. Our mission: deliver correct high-performance software faster. We have found functional programming in particular to be an enabling factor for us on some of the most ambitious projects we’ve faced for our clients…

This is the second post in our series about building polyglot projects using Bazel. You might find it useful to start with our first installmentruleshaskellpost]. A is an extraordinary thing. You specify a set of targets, you tell Make what the dependencies are between these…

In the “all about reflection” post, we introduced the data type. Any sorting function should have the type . However, this doesn’t actually guarantee that is indeed a sorting function: it only says that is a sorted list, but not that it is actually a sort of . For example is…

Publishing code to the world is easy these days: take your code, tack some extra metadata onto it, call that a package and upload it to npmjs.com / crates.io / Hackage / etc. It’s also easy for other developers to include your code as part of their project: build tools are…

Tweag I/O is inviting applications from students for our paid internship programme this summer. If you’re a student excited about working with Haskell, Nix and similar things, then get in touch! In previous years, our interns have worked on a variety of projects across research…

The utility of free monads can show up in surprising places. One of the projects I’m working on is an AI, and part of the strategy that it uses for responding to user input is quite simple: it generates many possible responses, and then evaluates them. Most of the computations it…

An important device in the tool belt I carry around everyday is type class reflection. I don’t reach for it often, but it can be very useful. Reflection is a little known device. And for some reason it is often spoken of with a hint of fear. In this post, I want to convince you…

Foreign function interfaces (FFI) allow fast interop between languages. Unlike other approaches, like performing RPC calls between different components written in different languages, using the FFI allows for all manner of data to be shared between each language runtime, in the…

This is the fifth post in a series about array programming in Haskell — you might be interested in the first, second, third, and fourth, too. A recurring theme in array programming is performance. After all, many algorithms in numerical computing and data science are…

Nix on Windows: does it run yet? That’s the question I wondered about while testing the latest NixOS release, version 17.09. To that end, I had the idea of running the Nix installation process from inside the Windows Subsystem for Linux (WSL) see if it worked. And it worked…

If Haskell was a god, often would he be depicted with the ravens Modularity and Abstraction flying above him, hovering the world and reporting to him every detail of our whereabouts. Haskell would sit on the Throne of Purity and look upon the world with an eye full of wisdom. And…

A recent development in Haskell land is the formation of the GHC DevOps Group, which was the topic of last week’s blog post. The group is a community of parties committed to the future of GHC. Tweag I/O is one such party. We are helping the group achieve its goals with concrete…

As Haskell is increasingly used in production environments, the Haskell toolchain is of critical importance to a growing number of people and organisations. At the heart of this toolchain is GHC (The Glasgow Haskell Compiler). Conceived nearly thirty years ago as a research…

This is the fourth post in a series about array programming in Haskell — you might be interested in the first, second, and third, too. In the previous post of this series, we explored the basic, down-to-earth, index-based array interfaces that have their roots in the Haskell…

In our first blog post about streaming, we discussed how streaming libraries strengthen types to catch more errors. However, when we strengthen types, we need to be careful to not hinder program composition. After all, precise type information can make it more onerous to convince…

This is the third post in a series about array programming in Haskell — you might be interested in the first and second, too. In the previous post of this series, we discussed commonly used vector and matrix routines, which are available in highly-optimised implementations in…

Previously, we discussed how to use inline-java to call any Java function from Haskell. The reverse is also possible, though that will be a topic for a future post. In this post, we’ll peek underneath the hood to talk a little about how inline-java does its deed. You might find…

Our introductory post for inline-java showed that we could call methods written in Java (or indeed any JVM function) from Haskell. Much in the style of other packages, it is moreover possible to do using java syntax, so examples from Java API documentation can be reused as-is. In…

In this post I’ll show you how to see the performance of your software project improve over the entire lifetime of the project. That is, how to get something like the below picture. For that, you’ll need to systematically track and store benchmark results over time. We’ll use…

This is the second post in a series about array programming in Haskell — you might be interested in the previous post, too. Matrices are the bread and butter of most scientific and numeric computing. It is not surprising then that there is a range of standard libraries and…

We saw last time that with linear types, we could precisely capture the state of sockets in their types. In this post, I want to use the same idea of tracking states in types, but applied to a more unusual example from our paper: sending rich structured data types across the…

Haskell is an awesome language, but we need to remember that it is not very useful in isolation. In almost any realistic application, Haskell has to coexist with other languages, even if only to call existing C libraries or to make use of operating system services. In actual…

Array programming in Haskell is attractive, but it is also somewhat confusing. Haskell’s functional syntax is close to the mathematical notation that is often used to develop and explain the algorithms on which array programs are based, which leads to great clarity of expression…

At the time of our first post on linear types, we were fresh out of the design phase to extend GHC with linear types. We had a prototype implementation, but it was just a proof of concept: there was precious little you could do with it. A few months down the line, we are now in a…

I have had a passion for programming ever since I wrote my first piece of code as a kid. I am not sure why. Maybe it is the absolute, god-like control over the behaviour of the machine, or losing myself in the details of the code. In any case, it is intriguing. When teenage play…

In school, we’re taught that I/O is a simple affair: read data in, write data out. Rinse, repeat. But then as practitioners we realize matters are often more complicated. For one, I/O is slow, so we probably want operations to overlap (i.e. be processed asynchronously…

A guest post by Sigrid Keydana telling us the backstory behind the very cool trading app notebook she put together… Keras, ggplot2, Haskell and R all in one Jupyter notebook! Post originally appeared here. Earlier today, I presented at UseR! 2017 about HaskellR: a great piece of…

Théophane is a summer intern here at Tweag I/O. Thanks also in part to the financial help from the Nix community, he’s spending six months on devising a type system for the Nix language. Nix, the purely functional package manager Nix is a cross platform tool for managing the…

Our mission: deliver correct high-performance software faster. To do that, we recruited a team of experimentalists who use their formal training in a variety of theories and their experience to continuously improve our methodology and publish new tech. Theories that are…

We’re extending GHC with linear types. Ever since Jean-Yves Girard discovered linear logic in 1986, researchers around the world have been going “wow! resource tracking, this must be useful for programming languages”. After all, any real computation on a real machine takes…

By now, Haskell has first class support for seamlessly embedding foreign code into source files and casually call anything in C (via inline-c) or R (via inline-r), let alone that whole programs can also be compiled down to in-browser JavaScript, thanks to GHCJS. Today the…

Maintaining a compute cluster for batch or stream data processing is hard work. Connecting it up to storage facilities and time sharing resources across multiple demands even more so. Fortunately cloud service providers these days typically upscale their offering to not just…

Large scale distributed applications are complex: there are effects at scale that matter far more than when your application is basked in the warmth of a single machine. Messages between any two processes may or may not make it to their final destination. If reading from a memory…

Stack and Cabal are great tools for incremental development. But before even starting to think about what your first commit will be about on a freshly checked out project, what you have to do is get a working development environment, by which we mean: see if it builds! Stack…

Most good data stories start with a interesting question. If the average request latency went down by a further 100ms, by how much could we expect user engagement to increase? How can we detect evidence of corruption of government officials given a list of all bids nationwide for…