Tweag
Technical groups
Dropdown arrow
Open source
Careers
Research
Blog
Contact
Consulting services
Technical groups
Dropdown arrow
Open source
Careers
Research
Blog
Contact
Consulting services

Converting a polyglot project build to Bazel: part 1

20 October 2022 — by Karol Czułkowski

Bazel is a great tool for describing monorepo builds. This is especially the case when one has to deal with more than one technology or language within the same repository. In this blog post I am going to demonstrate how to convert the Makefile build of example-servant-elm to Bazel. Well… maybe not all at once. In the first post of this series, I’d like to focus only on the Haskell components of the example-servant-elm project. The following post will cover the Bazel build for the front-end (Elm) component.

This blog post is, after a fashion, an extension of the talk given by Andreas Herrmann during ZuriHac 2021. I want to slightly modify Andreas’ approach here, especially the part which relates to the Haskell component (backend server), and utilize a feature that was recently added to the gazelle_cabal project—support for the Cabal sub-libraries. All the code from this blog post can be found here.

example-servant-elm project structure

Before we start, it will be helpful to analyze the actual structure of the example-servant-elm project:

$ tree
.
├── assets
├── client
│   ├── elm.json
│   ├── GenerateElm.hs
│   ├── Makefile
│   ├── src
│   │   └── Main.elm
│   └── tests
│       └── Tests.elm
├── example-servant-elm.cabal
├── Makefile
├── package.yaml
├── README.md
├── server
│   ├── src
│   │   ├── Api.hs
│   │   ├── App.hs
│   │   └── Main.hs
│   └── test
│       ├── AppSpec.hs
│       └── Spec.hs
├── stack.yaml
└── stack.yaml.lock
  1. package.yaml file - hpack description for Haskell components
  2. example-servant-elm.cabal file - generated by hpack from the package.yaml file
  3. server directory - backend code written in Haskell
  4. client directory - frontend code written in Elm with an additional Haskell file
  5. assets directory - a place where all the web components end up (e.g. the index.html file)

It is clear that the Haskell code is split into two logically separated components— the server and the client—but there is only a single .cabal file in this project. The Haskell code inside the client directory generates Elm bindings that call the Haskell API code from the Elm frontend application. Therefore, server/src/Api.hs is a dependency for both client/GenerateElm.hs and server/src/Main.hs.

Dependency graph

With this in mind, the Makefile-to-Bazel conversion process will involve first creating a sub-library for server/src/Api.hs in the server component, and subsequently reusing it for both the Main.hs and GenerateElm.hs binaries. We can automatically generate the Bazel build definition by utilizing the gazelle_cabal extension, which will create all the rules_haskell entries for us.

Haskell components—The server and… the client

Adjusting hpack/Cabal definitions

In order to use gazelle_cabal, the client/GenerateElm.hs component must be taken into account by Cabal, which is not yet the case. Since the Cabal file is generated by hpack, package.yaml must be extended. This is achieved by adding a shared library for the API, and another executable for the Elm generation, resulting in:

name: example-servant-elm

dependencies:
  - ...

internal-libraries:
  api:
    source-dirs:
      - server/src
    exposed-modules: Api
    other-modules: []

executables:
  server:
    main: Main.hs
    source-dirs:
      - server/src
    dependencies: api
    other-modules: [App]
  generate-elm:
    main: GenerateElm.hs
    source-dirs:
      - client/
    dependencies: api
    other-modules: []

Running hpack against the updated package.yaml gives us what we expect—an example-servant-elm.cabal file with two executables depending on a single internal library:

$ hpack && cat example-servant-elm.cabal
cabal-version: 2.0

-- This file has been generated from package.yaml by hpack version 0.34.4.
--
-- see: https://github.com/sol/hpack

name:           example-servant-elm
version:        0.0.0
build-type:     Simple

library api
  exposed-modules:
      Api
  hs-source-dirs:
      server/src
  build-depends:
      ...
  default-language: Haskell2010

executable generate-elm
  main-is: GenerateElm.hs
  hs-source-dirs:
      client/
  build-depends:
        ...
      , api
        ...
  default-language: Haskell2010

executable server
  main-is: Main.hs
  other-modules:
      App
  hs-source-dirs:
      server/src
  build-depends:
        ...
      , api
        ...
  default-language: Haskell2010

...

The api component is called an internal library (or sub-library) in Cabal’s nomenclature, and the relevant section of the cabal file can be additionally annotated with the visibility parameter, indicating whether the library can be used only in this package, or also in other Cabal packages. By default the visibility is private. This is taken into account by gazelle_cabal, which will generate suitable rules_haskell entries.

The generation of BUILD.bazel files will have the following flow:

generation-flow

We can alias this build command as follows:

$ alias gen-build-files="hpack && bazel run :gazelle"

Playing with gazelle_cabal

Now we can create a WORKSPACE file in the example-servant-elm root directory, and add all the preambles required for gazelle_cabal, rules_haskell and other dependencies.

workspace(name = "example-servant-elm")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "io_tweag_gazelle_cabal",
    strip_prefix = "gazelle_cabal-main",
    url = "https://github.com/tweag/gazelle_cabal/archive/main.zip",
    sha256 ="49c9bda91ae2867fa45dbc6f5c04d0ee0146dd433ba7ab9694ed2914a092e817",
)

...

http_archive(
    name = "rules_haskell",
    sha256 = "851e16edc7c33b977649d66f2f587071dde178a6e5bcfeca5fe9ebbe81924334",
    strip_prefix = "rules_haskell-0.14",
    urls = ["https://github.com/tweag/rules_haskell/archive/v0.14.tar.gz"],
)

load("@rules_haskell//haskell:repositories.bzl", "rules_haskell_dependencies")
load("@rules_haskell//haskell:cabal.bzl", "stack_snapshot")

rules_haskell_dependencies()

...

stack_snapshot(
    name = "stackage",
    packages = [
        "path", #keep
        "path-io", #keep
        "aeson" #keep
    ],
    snapshot = "lts-18.12",
)

Following gazelle_cabal’s README, we add the required Gazelle targets to the build file at the root of the example-servant-elm directory:

$ cat BUILD.bazel
load("@bazel_gazelle//:def.bzl", "gazelle")
load(
    "@bazel_gazelle//:def.bzl",
    "DEFAULT_LANGUAGES",
    "gazelle_binary",
)

gazelle(
    name = "gazelle",
    data = ["@io_tweag_gazelle_cabal//cabalscan"],
    gazelle = ":gazelle_binary",
)

gazelle_binary(
    name = "gazelle_binary",
    languages = DEFAULT_LANGUAGES + ["@io_tweag_gazelle_cabal//gazelle_cabal"],
)

gazelle(
    name = "gazelle-update-repos",
    command = "update-repos",
    data = ["@io_tweag_gazelle_cabal//cabalscan"],
    extra_args = [
        "-lang",
        "gazelle_cabal",
        "stackage",
    ],
    gazelle = ":gazelle_binary",
)

Finally, it’s time to use gazelle_cabal to automatically generate all the rules_haskell targets in our repository. On the very first run it will fail due to of Paths_* modules being required by the spec test suite. This requirement is automatically inserted by hpack (and the Paths_* files are automatically generated by Cabal). In this project, we don’t actually need the Paths_* files. We can get rid of the requirement by setting other-modules in the spec target section of package.yaml explicitly.

$ cat package.yaml
...
tests:
  spec:
    ...
    other-modules: [AppSpec, Api, App]
...
$ #################################################################
$ gen-build-files
generated example-servant-elm.cabal
INFO: Analyzed target //:gazelle (1 packages loaded, 3 targets configured).
INFO: Found 1 target...
Target //:gazelle up-to-date:
  bazel-bin/gazelle-runner.bash
  bazel-bin/gazelle
INFO: Elapsed time: 0.161s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
$ #################################################################
$ cat BUILD.bazel
...
# rule generated from example-servant-elm.cabal by gazelle_cabal
haskell_library(
    name = "api",
    srcs = ["server/src/Api.hs"],
    compiler_flags = ["-DVERSION_example_servant_elm=\"0.0.0\""],
    version = "0.0.0",
    visibility = ["//visibility:public"],
    deps = [
        "@stackage//:aeson",
        ...
        "@stackage//:warp",
    ],
)

# rule generated from example-servant-elm.cabal by gazelle_cabal
haskell_binary(
    name = "generate-elm",
    srcs = ["client/GenerateElm.hs"],
    compiler_flags = ["-DVERSION_example_servant_elm=\"0.0.0\""],
    version = "0.0.0",
    visibility = ["//visibility:public"],
    deps = [
        ":api",
        ...
    ],
)

# rule generated from example-servant-elm.cabal by gazelle_cabal
haskell_binary(
    name = "server",
    srcs = [
        "server/src/App.hs",
        "server/src/Main.hs",
    ],
    compiler_flags = ["-DVERSION_example_servant_elm=\"0.0.0\""],
    version = "0.0.0",
    visibility = ["//visibility:public"],
    deps = [
        ":api",
        ...
    ],
)

# rule generated from example-servant-elm.cabal by gazelle_cabal
haskell_test(
    name = "spec",
    srcs = [
        "server/src/Api.hs",
        "server/src/App.hs",
        "server/src/Main.hs",
        "server/test/AppSpec.hs",
        "server/test/Spec.hs",
    ],
    compiler_flags = ["-DVERSION_example_servant_elm=\"0.0.0\""],
    version = "0.0.0",
    visibility = ["//visibility:public"],
    deps = [
        ...
    ],
)

So far, so good. It turns out that gazelle_cabal has generated a bunch of rules_haskell targets within the BUILD.bazel file. It created exactly what was expected: two binary targets—generate-elm and server—which depend on the api target. Moreover, there’s also the spec target, which is responsible for test execution.

Building the :api, :generate-elm and :server targets

Our next step is to run/build each one of the generated targets to verify that all of them work as expected. We will start with the :api target. It fails on the first run, so we have to iterate and fix the errors.


$ bazel build :api
ERROR: example-servant-elm/BUILD.bazel:33:16: no such target '@stackage//:elm-bridge': target 'elm-bridge' not declared in package '' defined by /home/kczulko/.cache/bazel/_bazel_kczulko/0544c71977bc4fd7972aa4a6b74ea529/external/stackage/BUILD.bazel and referenced by '//:api'
ERROR: example-servant-elm/BUILD.bazel:33:16: no such target '@stackage//:servant': target 'servant' not declared in package '' defined by /home/kczulko/.cache/bazel/_bazel_kczulko/0544c71977bc4fd7972aa4a6b74ea529/external/stackage/BUILD.bazel and referenced by '//:api'
...
INFO: Elapsed time: 0.579s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (3 packages loaded, 0 targets configured)

Essentially, we need to add all the dependencies from package.yaml under the dependencies section of the stack_snapshot repository defined in the WORKSPACE file. Fortunately, we don’t have to add those manually but can instead invoke gazelle_update_repos to generate the imports. Running the command below automatically populates stack_snapshot with all the required dependencies like wai, warp, aeson etc:

$ bazel run :gazelle-update-repos && cat WORKSPACE
...
stack_snapshot(
    name = "stackage",
    ...
    packages = [
        "aeson",  #keep
        "base",
        "containers",
        "elm-bridge",
        "hspec",
        "hspec-discover",
        "http-client",
        "http-types",
        "path",  #keep
        "path-io",  #keep
        "servant",
        "servant-client",
        "servant-elm",
        "servant-server",
        "text",
        "transformers",
        "wai",
        "warp",
    ],
    snapshot = "lts-18.12",
)
...

At this point we can update the gen-build-files alias to include :gazelle-update-repos as the last step of BUILD.bazel files generation:

$ alias gen-build-files="hpack && \
  bazel run :gazelle && \
  bazel run :gazelle-update-repos"

The next step is to add precise versions to the dependencies which require them. All of these dependencies can be found in the stack.yaml file - elm-bridge, servant-elm, wai-make-assets. To prevent gazelle-update-repos from removing them, we have to mark those with gazelle’s built-in #keep directive. This is basically a comment that will be interpreted by the gazelle engine, which we have to add next to the dependency name inside the stack_snapshot bazel workspace rule. Those two actions should populate the packages array in stack_snapshot as follows:

stack_snapshot(
    name = "stackage",
    packages = [
        "aeson",  #keep
        ...
        "elm-bridge",
        "elm-bridge-0.6.0", #keep
        ...
        "servant-elm",
        "servant-elm-0.7.2", #keep
        ...
        "wai-make-assets-0.2", #keep
        "warp",
    ],
    snapshot = "lts-18.12",
)

Evidently, some of those dependencies are quasi-duplicated (e.g. elm-bridge and elm-bridge-0.6.0). In this case, rules_haskell will unify all quasi-duplicates and fetch the specified version from Hackage. We can use unversioned labels in rules_haskell targets, like haskell_library or haskell_binary, to refer to these libraries. So, instead of having to use elm-bridge-0.6.0 in the build definition, we can simply refer to elm-bridge.

The next problem we encounter is quite typical for Haskell builds:

$ bazel build :api

...

Setup.hs: Missing dependency on a foreign library:
* Missing (or bad) header file: zlib.h
* Missing (or bad) C library: z
...

We need zlib. Since this setup is using nix, the easiest way to include zlib in this build is to copy the configuration from the gazelle_cabal WORKSPACE file (for example), and insert it under the stack_snapshot repository rule.


stack_snapshot(
    ...
    extra_deps = {"zlib" : ["@zlib.dev//:zlib"]},
    ...
)

...

nixpkgs_package(
   name = "nixpkgs_zlib",
   attribute_path = "zlib",
   repository = "@nixpkgs",
)

nixpkgs_package(
   name = "zlib.dev",
   build_file_content = """

   ...

   """,
   repository = "@nixpkgs",
)

Now the :api target should build without errors:

$ bazel build :api
/nix/store/7whk47dml5f1dpvy6cgvaf80ll1h7pkd-zlib-1.2.11-dev
/nix/store/bl1kc9hw119ly7if07m2pbqak2yhars8-zlib-1.2.11
INFO: Analyzed target //:api (6 packages loaded, 4303 targets configured).
INFO: Found 1 target...
Target //:api up-to-date:
  bazel-bin/libHSapi.a
  bazel-bin/libHSapi-ghc8.10.4.so
INFO: Elapsed time: 611.149s, Critical Path: 551.46s
INFO: 196 processes: 125 internal, 71 linux-sandbox.
INFO: Build completed successfully, 196 total actions

The next target we’ll work on is :generate-elm, which should build successfully after removing the redundant imports from GenerateElm.hs.

The :server target builds successfully without any manual intervention. Thank you gazelle_cabal! Almost all automatically generated targets work without manual intervention. However, when we run this target, it immediately terminates. We’ll return to that problem later.

$ bazel run :server
...
INFO: Build completed successfully, 1 total action
server: missing directory: 'client/'
Please create 'client/'.
(You should put sources for assets in there.)

Building the :spec target

The last Bazel target to evaluate is :spec. On the first run it’s going to fail with the following error:

Use --sandbox_debug to see verbose messages from the sandbox
ghc: could not execute: hspec-discover
Target //:spec failed to build

This means that we have to add hspec-discover to the build-tool-depends section of example-servant-elm.cabal file, just like in this example. Since the Cabal file is automatically generated, we are required to add a build-tools section to package.yaml, and to regenerate BUILD.bazel.

$ cat package.yaml
...
tests:
  spec:
    ...
    build-tools:
      - hspec-discover
  ...

$ gen-build-files && cat BUILD.bazel
...
haskell_test(
    name = "spec",
    ...
    compiler_flags = [
        "-DVERSION_example_servant_elm=\"0.0.0\"",
        "-DHSPEC_DISCOVER_HSPEC_DISCOVER_PATH=$(location @stackage-exe//hspec-discover)",
    ],
    tools = ["@stackage-exe//hspec-discover"],
    ...
)

gazelle_cabal has introduced the HSPEC_DISCOVER_HSPEC_DISCOVER_PATH variable, so we can reuse it in the server/Spec.hs file to let GHC know where to find hspec-discover. This is a good step toward hermeticity in comparison to what the server/Spec.hs file contained previously. Finally, server/Spec.hs should contain the following:

{-# LANGUAGE CPP #-}
{-# OPTIONS_GHC -F -pgmF HSPEC_DISCOVER_HSPEC_DISCOVER_PATH #-}

There is also some additional cleaning up required in server/test/AppSpec.hs, otherwise compilation will fail due to name shadowing and unused-imports warnings. After that, we can finally try to run the :spec target:

$ gen-build-files && bazel run :spec
...
Executing tests from //:spec
-----------------------------------------------------------------------------

App
  app
    /api/item
      returns an empty list FAILED [1]
      /api/item/:id
        returns a 404 for missing items FAILED [2]
      POST
        allows to create an item FAILED [3]
        lists created items FAILED [4]
        lists 2 created items FAILED [5]
      DELETE
        allows to delete items FAILED [6]

Failures:

  server/test/AppSpec.hs:30:7:
  1) App.app./api/item returns an empty list
       uncaught exception: ErrorCall
       missing directory: 'client/'
       Please create 'client/'.
       (You should put sources for assets in there.)
...

There are two things worth observing now:

  1. The target compiles—that’s obviously great!
  2. All the tests fail—let’s search for where “client” is used.

With regard to the second observation, it turns out that the string “client” is used in the server/src/App.hs options method:

$ bat server/src/App.hs
...
  1819options :: Options
  20options = Options "client"
  21...

Indeed, options is used by the server function, and the server function is used by the app function, and the app function is imported and used in server/test/AppSpec.hs. Whew… we’ve got it! It turns out to be used to generate and load assets with the serveAssets function. The serveAssets function calls make to generate those assets, but our assets are actually static, so let’s use the simpler wai-app-static package instead:

$ git diff server/src/App.hs
diff --git a/server/src/App.hs b/server/src/App.hs
index 203a6c7..e0a674e 100644
--- a/server/src/App.hs
+++ b/server/src/App.hs
@@ -7,6 +7,7 @@ import           Control.Monad.IO.Class
 import           Data.Map
 import           Network.Wai
-import           Network.Wai.MakeAssets
+import           Network.Wai.Application.Static (staticApp, defaultFileServerSettings)
 import           Servant

 import           Api
@@ -16,15 +17,14 @@ type WithAssets = Api :<|> Raw
 withAssets :: Proxy WithAssets
 withAssets = Proxy

-options :: Options
-options = Options "client"
+assets :: Application
+assets = staticApp $ defaultFileServerSettings "assets"

 app :: IO Application
 app = serve withAssets <$> server

 server :: IO (Server WithAssets)
 server = do
-  assets <- serveAssets options
   db     <- mkDB
   return (apiServer db :<|> Tagged assets)

Of course, we also need to add wai-app-static under the dependencies list within the package.yaml file.

...
tests:
  spec:
    ...
    dependencies:
      ...
      - http-types
      - wai-app-static

This also means that we can remove the wai-make-assets-0.2 #keep entry from the stack_snapshot packages list, as well as from the package.yaml file. Let’s regenerate the build definitions and check whether the tests pass…

$ gen-build-files && bazel test :spec
...
INFO: Build completed successfully, 2 total actions
//:spec                                                                  PASSED in 0.0s

Executed 1 out of 1 test: 1 test passes.

The “client” error sounds familiar—we encountered it before in the case of the :server target. Let’s check whether :server still terminates:

$ bazel run :server
...
INFO: Build completed successfully, 10 total actions
listening on port 3000...

We’re home and dry. It looks like getting rid of wai-make-asset did the job. The :spec and :server targets work, and all the tests passed.

Summary

We’ve shown how quickly you can switch your build from the combination of Cabal/make/hpack to Bazel. We now have one tool, and one approach to maintain the build process of example-servant-elm. Thanks to gazelle_cabal it was almost fully automated—especially for a few simple Cabal build definitions.

In the next post we’re going to “bazelify” the Elm part of this repository. Stay tuned!

About the author

Karol Czułkowski

If you enjoyed this article, you might be interested in joining the Tweag team.

This article is licensed under a Creative Commons Attribution 4.0 International license.

Company

AboutOpen SourceCareersContact Us

Connect with us

© 2024 Modus Create, LLC

Privacy PolicySitemap