Decode irregular JSON with Circe
The popular CI server Jenkins has a JSON API to access
builds, but its JSON has a rather strange shape: Relevant information sits in a
generic actions
array containing objects of various different shapes (even
empty objects for whatever reason):
The JSON
The array looks as follows (converted to YAML and stripped of some irrelevant parts):
actions:
- parameters:
- name: SERVICE_BUILD
value: '2840'
# […]
- name: GIT_COMMIT
value: 922cc937eb9c9142ebf0d8672a2b13f5fd28ae3e
- causes: # …
- {}
- buildsByBranchName:
refs/remotes/origin/master:
# …
lastBuiltRevision:
SHA1: 922cc937eb9c9142ebf0d8672a2b13f5fd28ae3e
branch:
- SHA1: 922cc937eb9c9142ebf0d8672a2b13f5fd28ae3e
name: refs/remotes/origin/master
scmName: ''
- {}
We are interested in lastBuiltRevision.SHA1
.
Decode actions into a Scala ADT
A Circe Decoder
for lastBuiltRevision
is straigt-forward:
import io.circe._
final case class LastBuiltRevision(sha1: String)
object LastBuiltRevision {
implicit val lastBuiltRevisionDecoder
: Decoder[LastBuiltRevision] =
Decoder.forProduct1("SHA1")(LastBuiltRevision(_))
}
For actions
we create an ADT modelling all shapes we’re interested in and
define an actionDecoder
which simplies tries all possible shapes until one
fits, via Decoder.or
:
sealed trait Action
object Action {
final case class Git(lastBuiltRevision: LastBuiltRevision)
extends Action
object Git {
implicit val gitDecoder: Decoder[Git] =
Decoder.forProduct1("lastBuiltRevision")(Git(_))
}
final case class Parameter(name: String, value: String)
object Parameter {
implicit val parameterDecoder: Decoder[Parameter] =
Decoder.forProduct2("name", "value")(Parameter(_, _))
}
final case class Parameters(parameters: List[Parameter])
extends Action
object Parameters {
implicit val parametersDecoder: Decoder[Parameters] =
Decoder.forProduct1("parameters")(Parameters(_))
}
implicit val actionDecoder: Decoder[Action] = {
import cats.syntax.functor._
Decoder[Parameters].widen
.or(Decoder[Git].widen)
}
}
We need to explicitly widen
each decoder to Decoder[Action]
—Circe decoders
are invariant.
Skip over unknown actions
We gracefully ignore unknown objects in actions
with another level of
indirection: An ADT that either represents a known action or the raw JSON of an
unknown object:
sealed trait MaybeAction
object MaybeAction {
final case class Known(action: Action) extends MaybeAction
final case class Unknown(contents: Json)
extends MaybeAction
implicit val maybeActionDecoder: Decoder[MaybeAction] =
Decoder[Action]
.map(Known)
.or(Decoder[Json].map(Unknown))
}
Decode a build
We can now decode an entire build into a Build
case class and collect all
known actions:
final case class Build(actions: List[MaybeAction]) {
def knownActions: List[Action] = actions.collect {
case MaybeAction.Known(action) => action
}
}
object Build {
implicit val buildDecoder: Decoder[Build] =
Decoder.forProduct1("actions")(Build(_))
}
Decode coproducts of actions
The actionDecoder
lists all variants explicitly. With Shapeless we can
reduce the boilerplate and define a generic decoder over all variants of
Action
: The trait is sealed
, so all variants are known at compilation time
and can be introspected with type-level programming.
The Action
trait is equivalent to the following shapeless.Coproduct
:
Action.Git :+: Action.Parameters :+: shapeless.CNil
Which reads as as ”either an Action.Git
or an Action.Parameters
”.
A value of this type, eg, an Action.Parameters
value, looks as follows:
Inr(Inl(Parameters(List(…))))
Which reads as “skip the first product alternative (Inr
) and move on to the
next alternative which has a value (Inl
)”.
We can recursively define a decoder for a Coproduct of Action
s:
private implicit val cnilDecoder: Decoder[CNil] =
Decoder.failed(DecodingFailure("CNil", List.empty))
private implicit def cconsActionDecoder[H <: Action, T <: Coproduct](
implicit decodeH: Decoder[H],
decodeT: Decoder[T]
): Decoder[H :+: T] =
decodeH.map(Inl[H, T]).or(decodeT.map(Inr[H, T]))
CNil
is the recursion anchor; it serves as a base case for this inductive
definition, however a coproduct will never have a CNil
value at runtime. In
the cconsActionDecoder
step we try to decode to the type of the current
coproduct position or fall back to decode the remaining coproduct positions, and
the lift the result back to a Coproduct
by applying the Inl
and Inr
constructors to both cases. We define the implicit privately to not let them
leak into other scopes and wreck havoc of implicit resultions.
Decode actions from coproducts
We still need to go from a coproduct of action alternatives to the actual
Action
type, by means of shapeless.Generic
:
private def genericActionDecoder[Repr <: Coproduct](
implicit genericAction: Generic.Aux[Action, Repr],
decodeRepr: Decoder[Repr]
): Decoder[Action] = decodeRepr.map(genericAction.from)
implicit val actionDecoder: Decoder[Action] =
genericActionDecoder
As said above the generic representation of a sealed trait is a coproduct; we
can seamlessly convert between a sealed trait and coproduct of values of that
trait; in this case we use Generic
to move from a coproduct to the action
type itself. Again, the decoder is private to not leak it into other scopes.