Supercharge tests with circle-ci & crystalball [Part 1]

2 minute read

We will explore how to optimize your test suite using Crystalball test selection library, to reduce the test run time, down to a minute. This blog post is an attempt to document my own experience of setting up crystalball alongside circle-ci parallel runs.

This is Part 1 of this series.

You can find all the links to this series here


For large codebases, handling a huge test suite becomes challenging soon.

  • The journey of a test suite starts small, finishing up within a couple of minutes in a local machine.
  • Gradually the codebase starts to gain some mileage. Rspec runs notching > 10 mins in the CI.
  • Flaky tests start to show up. CI runs retried multiple times. The alarm bell starts ringing. Sounds familiar?!

One of the approaches to solve this problem is to use predictive test-selection. Thankfully in ruby world, there is already a ready-made solution available.

Aaron Patterson has written about this, and the good folks over toptal have built the library Crystalball :heart:.

Here are some references to get to know about crystalball

  1. The documentation is pretty detailed.
  2. This presentation at ruby kaigi is a great demonstration of the features.

Setup Circle-CI config

Circle-CI provides parallel runs based on automatic test splitting based on multiple strategies. Their documentation provides details on this.

IMO the most useful test-splitting strategy is --split-by=timings.

We can set-up parallel runs in the circle-ci config.

version: 2.1
    executor: rspec-executor
    parallelism: 5
      - run:
            name: Run rspec in parallel
            command: |
              TESTFILES=$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)
              bundle exec rspec --format progress \
                                --format RspecJunitFormatter \
                                --out ~/test-results/rspec/rspec.xml \
      - store_test_results:
          path: ~/test-results

    - rspec:

Setup Crystalball

Out in the wild, one of the prominent OSS codebase, which uses Crystalball is Gitlab. They have documented their usage of crystalball in their CI flow here.

IMO that is a good direction to follow. Using that we can come up with a workflow. :point_down:

Workflow Diagram


To setup crystalball, first, we will need to install the gem

gem install crystalball

Since we are running our specs in parallel, multiple mapping files will be generated. Hence it is important to store all of them in a folder. That can be defined in config/crystalball.yml file.

# Default: `tmp/crystalball_data.yml`
execution_map_path: crystalball
diff_from: origin/master

In the spec/spec_helper.rb, we will define all the strategies

if ENV['CRYSTALBALL'] == 'true'
  require 'crystalball'
  require 'crystalball/rails'

  Crystalball::MapGenerator.start! do |config|
    config.register ['Object'])

Once all of this is done, we will need to make Crystalball play nice with Circle-CI.

That will be a code heavy post, which I will take up in Part-2 of this series.