Check out ScalaPBC.
ScalaPB code generator can be invoked in your Maven build through the protobuf-maven-plugin. See example project.
The relevant parts are marked with "Add protobuf-maven-plugin..."
The example maven project invokes ScalaPBC. To get these ScalaPB features, you need to pass a generator parameter to ScalaPBC. See the supported generator parameters and how to use them in ScalaPBC documentation.
If you are using sbt-protoc and importing protos like
or common protocol buffers like
Add the following to your
sbt-protoc to extract protos from this jar (and all its
dependencies, which includes Google's common protos), and make them available
in the include path that is passed to protoc.
If you are not using sbt (for example, spbc), then you need to make those files available on the file system.
For some proto packages, we may already provide pre-compiled generated code in ScalaPB's Common Protos project. If what you are looking for is not there, consider adding it by following the instructions on the project's page.
To build it yourself using SBT: include the jar as a
protobuf dependency in your libraryDependencies:
This will tell sbt-protoc to extract the protos from that jar into
target/scala-2.vv/protobuf_external_src and add them both to the import
search path and the set of sources to generate code for (
intransitive modifier makes it not unpack the dependencies of this
library to the same directory since you may not want to generate classes for them
as well. Many common libraries depend on
protobuf-java, and generated
classes for them are already shipped with ScalaPB's
intransitive(), you should ensure all the dependencies are provided,
protobuf (depending if you want to compile them
or just import them).
You may find other protos under
protobuf_external_src that you do not wish to
compile. You can exclude them by adding an
See full example here.
Why message fields are wrapped in an
Option in proto3?#
For a proto like this:
The generated case class for
B will have a field
a: Option[A]. The reason
is that in the proto3 format, it is valid for an encoded message of type
B to not
contain a value for the field
a. Using the
Option[A] type lets us distinguish
between the case where a value for
A was not provided and the case where
was explictly set to a certain value (even if that value is the default value
A). The case where
a is not set, and the case that
A is set to its
default value have two distinct binary representations (in both proto2 and
You can set a certain message type or a field to not be wrapped in an
How do I represent
Option[T] in proto3 for scalar fields?#
Scalar fields are the various numeric types,
everything except of messages. In the proto2 wire format, there is a distinction between
not setting a value (
None), or setting it to its default value (
In proto3, this distinction has been removed in the wire format. Whenever the value
of a scalar type is zero, it does not get serialized. Therefore, the parser is not
able to distinguish between
None. The semantics is that a zero
has been received.
Optional message types are still wrapped in
Option since there is a
distinction in the wire format between leaving out a message (which is
None or sending one, even if all its fields are unset (or
assigned default values). If you wish to have
Option around scalar
types in proto3, you can use this fact to your advantage by using primitive wrappers
It turns out that global generator parameters that affect source code compatibility are something that we would like to avoid, as it is creating issues that are tricky to resolve. For example, if:
- package A provides proto files and generate source code with one value of a generator parameter,
- package B is compiled separately with a different value of this generator parameter, and imports protos from package A, as well as its generated classes.
then the code generator for B has no way of knowing that package A has code generated with a different parameter, and that it needs to account for that in the way it references it. This leads to compilation errors in ScalaPB or in user-provided code generators.
In version 0.8.2, we introduced package-scoped options which let you set file-level options for an entire package at once.
Easy! Check out giter8 template for writing new code generators.
Easy! See this example:
Easy! See this example:
Generated code would be:
You can use field-transformations to match on third-party types and apply arbitrary field-level options. See example here.
Using Spark, I am getting
No encoder found for ... or
Unable to find encoder for type#
When using ScalaPB case classes with Spark datasets or dataframes, you need to be using sparksql-scalapb:
Make sure the version of ScalaPB, sparksql-scalapb and ScalaPB match according to this table.
In the scope where the exception occurs, make sure you import
scalapb.spark.Implicits._. Do not import
If you use an interactive notebook such as Databricks or spark-shell, there is a default import of
spark.implicits._that is executed prior to your own code. There is currently no way to disable this behavior. The workaround is to manually pass the encoder explicitly in the locations where an encoder is not found, for example:import scalapb.spark.Implicits.typedEncoderToEncoderdf.as[MyMessageType](typedEncoderToEncoder[MyMessageType])spark.createDataset(myList, typedEncoderToEncoder[MyMessageType])
This error is very common for newcomers to ScalaPB and Spark. If the guidance above did not resolve your issue and you would like to get support over Gitter, Github or Stackoverflow, please clearly indicate that you have read this FAQ, and provide the protos and code that triggers the exception, including the relevant imports in scope. Ideally, provide a reproducible example. The easiest way to do it is to fork this repo, adjust so it reproduces the issue you have and provide a link to your fork in your problem report.
You can use ScalaPB with the official Protobuf Plugin for Gradle.
protobuf section should look like this:
See full example here.