Decode irregular JSON with Circe

The popular CI server Jenkins has a JSON API to access builds, but its JSON has a rather strange shape: Relevant information sits in a generic actions array containing objects of various different shapes (even empty objects for whatever reason):

The JSON

The array looks as follows (converted to YAML and stripped of some irrelevant parts):

actions:
  - parameters:
    - name: SERVICE_BUILD
      value: '2840'
    # […]
    - name: GIT_COMMIT
      value: 922cc937eb9c9142ebf0d8672a2b13f5fd28ae3e
  - causes: # …
  - {}
  - buildsByBranchName:
      refs/remotes/origin/master:
        # …
    lastBuiltRevision:
      SHA1: 922cc937eb9c9142ebf0d8672a2b13f5fd28ae3e
      branch:
      - SHA1: 922cc937eb9c9142ebf0d8672a2b13f5fd28ae3e
        name: refs/remotes/origin/master
    scmName: ''
  - {}

We are interested in lastBuiltRevision.SHA1.

Decode actions into a Scala ADT

A Circe Decoder for lastBuiltRevision is straigt-forward:

import io.circe._

final case class LastBuiltRevision(sha1: String)

object LastBuiltRevision {
  implicit val lastBuiltRevisionDecoder
    : Decoder[LastBuiltRevision] =
    Decoder.forProduct1("SHA1")(LastBuiltRevision(_))
}

For actions we create an ADT modelling all shapes we’re interested in and define an actionDecoder which simplies tries all possible shapes until one fits, via Decoder.or:

sealed trait Action

object Action {
  final case class Git(lastBuiltRevision: LastBuiltRevision)
      extends Action
  object Git {
    implicit val gitDecoder: Decoder[Git] =
      Decoder.forProduct1("lastBuiltRevision")(Git(_))
  }

  final case class Parameter(name: String, value: String)
  object Parameter {
    implicit val parameterDecoder: Decoder[Parameter] =
      Decoder.forProduct2("name", "value")(Parameter(_, _))
  }

  final case class Parameters(parameters: List[Parameter])
      extends Action
  object Parameters {
    implicit val parametersDecoder: Decoder[Parameters] =
      Decoder.forProduct1("parameters")(Parameters(_))
  }

  implicit val actionDecoder: Decoder[Action] = {
    import cats.syntax.functor._
    Decoder[Parameters].widen
      .or(Decoder[Git].widen)
  }
}

We need to explicitly widen each decoder to Decoder[Action]—Circe decoders are invariant.

Skip over unknown actions

We gracefully ignore unknown objects in actions with another level of indirection: An ADT that either represents a known action or the raw JSON of an unknown object:

sealed trait MaybeAction
object MaybeAction {
  final case class Known(action: Action) extends MaybeAction
  final case class Unknown(contents: Json)
      extends MaybeAction

  implicit val maybeActionDecoder: Decoder[MaybeAction] =
    Decoder[Action]
      .map(Known)
      .or(Decoder[Json].map(Unknown))
}

Decode a build

We can now decode an entire build into a Build case class and collect all known actions:

final case class Build(actions: List[MaybeAction]) {
  def knownActions: List[Action] = actions.collect {
    case MaybeAction.Known(action) => action
  }
}
object Build {
  implicit val buildDecoder: Decoder[Build] =
    Decoder.forProduct1("actions")(Build(_))
}

Decode coproducts of actions

The actionDecoder lists all variants explicitly. With Shapeless we can reduce the boilerplate and define a generic decoder over all variants of Action: The trait is sealed, so all variants are known at compilation time and can be introspected with type-level programming.

The Action trait is equivalent to the following shapeless.Coproduct:

Action.Git :+: Action.Parameters :+: shapeless.CNil

Which reads as as ”either an Action.Git or an Action.Parameters”. A value of this type, eg, an Action.Parameters value, looks as follows:

Inr(Inl(Parameters(List(…))))

Which reads as “skip the first product alternative (Inr) and move on to the next alternative which has a value (Inl)”.

We can recursively define a decoder for a Coproduct of Actions:

private implicit val cnilDecoder: Decoder[CNil] =
  Decoder.failed(DecodingFailure("CNil", List.empty))

private implicit def cconsActionDecoder[H <: Action, T <: Coproduct](
    implicit decodeH: Decoder[H],
    decodeT: Decoder[T]
): Decoder[H :+: T] =
  decodeH.map(Inl[H, T]).or(decodeT.map(Inr[H, T]))

CNil is the recursion anchor; it serves as a base case for this inductive definition, however a coproduct will never have a CNil value at runtime. In the cconsActionDecoder step we try to decode to the type of the current coproduct position or fall back to decode the remaining coproduct positions, and the lift the result back to a Coproduct by applying the Inl and Inr constructors to both cases. We define the implicit privately to not let them leak into other scopes and wreck havoc of implicit resultions.

Decode actions from coproducts

We still need to go from a coproduct of action alternatives to the actual Action type, by means of shapeless.Generic:

private def genericActionDecoder[Repr <: Coproduct](
    implicit genericAction: Generic.Aux[Action, Repr],
    decodeRepr: Decoder[Repr]
): Decoder[Action] = decodeRepr.map(genericAction.from)

implicit val actionDecoder: Decoder[Action] =
  genericActionDecoder

As said above the generic representation of a sealed trait is a coproduct; we can seamlessly convert between a sealed trait and coproduct of values of that trait; in this case we use Generic to move from a coproduct to the action type itself. Again, the decoder is private to not leak it into other scopes.