Golang. Start using Protobuf.
The current chapter is the continuation of Golang related series started with the previous one:
The input for the current chapter is here. The project has REST API endpoint implementation to support usual CRUD operations. It utilizes Golang’s generics to create a common-purpose Router and Repository and uses it to create concrete pair to serve Brand entities CRUD. The entities are stored in memory, so with every server stop all data get lost. This chapter will fix it by storing data in the file and reading from it at startup.
Protobuf is a binary serialization format playing an important role in gRPC technology (both originated by Google). It is even easier to use Protobuf on its own to transmit in-memory objects into byte array and save it into binary file.
There are two things to install to start using Protobuf:
- Protobuf compiler which is CLI to create language specific ready to use code file from Protobuf schema file (not only for that). This should be in your machine paths to allow call protoc from any place.
- Go Protobuf plugin to target the generation process to Golang file (there are many other languages can benefits from using Protobuf and each has corresponding plugin. Please run following:
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
The plan is to put Protobuf-related stuff into the subfolder protobuf and the schema file (named brand.proto) is the starting point:
#1: use the up to date version. Please read through language guide to get familiar with it.
#2..#3: name package with a unique name to avoid clashing with similar message/service names and specify Protobuf Go generated file location.
#6..#10: define how Brand message (Protobuf version) looks like. The prefix Proto is added to all Brands-related parts to distinguish it from REST API Brand entity. Here, not only field names and types are important, but also indexes, which Protobuf uses during serialization, so indexes must be unique.
#11: As the repository stores many Brands, the array of Brands (repeated is the keyword in Protobuf terminology) is used in parent message. In this way, one parent message transfers many child.
#5..#12: Parent message ProtoBrandRepo itself is defined.
The next step is to generate Golang-specific files with the help of protoc compiler and Golang plugin installed before. Here is the command:
The schema file is protobuf/brand.proto and the output folder is protobuf. The result of the command invocation is file protobuf/golang_protobuf_brand/brand.pb.go (it is in specified by go_out parameter folder, the full path is taken from brand.proto#3 and the name is an original file name with .go extension as this is Golang file).
Please take a look at generated file here. The plan is to use the defined ProtoBrandRepo Golang struct, fill it with data, serialize it to byte array, and store it in to file. Also, the whole way back.
Golang runtime Protobuf package is here. So, it needs to be installed:
Final step is extend BrandRepo struct which maintains Storage layer of CRUD for Brand, with persisting Brands into binary file with the help of generated Protobuf schema and Protobuf Golang package. The final version can be found here, but the changes are one const and two new functions:
#2: file name to store serialized messages.
#4..#29: Serialize and store messages to file.
#6..#8: Create pointer to Protobuf-generated ProtoBrandRepo struct with empty Brand array inside.
#10..#16: fill ProtoBrandRepo instance with all Brands repository has.
#18..#21: Marshal ProtoBrandRepo instance to byte array.
#23..#26: Rewrite file content with byte array.
#31..#58: Load byte array from file, unmarshal it to ProtoBrandRepo instance and iterate over deserialized Brands to transmit to Brands repository array.
Newly created function loadFromFileStorage() should be call from NewBrandRepo() function to read stored in file Brands before new instance is returned. Thus, Brands are survive through REST API server restart. All repository modified CRUD operations (Create(…), Update(…) and DeleteOne(…)) should have saveToFileStorage() function call. Thus, just after the content of repository has changed, current state stored into file.
The final version of repository Go file is here.
Time to run REST API server and see serialization in action:
Open in VS Code (with REST Client plugin installed) file brand.rest to try create couple Brands and observe newly created file brands-storage.pb with Protobuf serialized messages.
Clicking brands-storage.pb file in VS Code shows that file is not text one, but still somehow reveals the internals:
Protobuf compiler (protoc) helps to view inside file:
And, of course, when REST API server restarts, GET request return all Brands from previous session.
The final version of the project is in here.
Protobuf is optimized data serialization protocol which is around quite a long and still has territory to conquer. This protocol is supported in many programming languages (Protobuf compiler plus target language plugin) giving ability to have server and client written with different languages. Protobuf binary nature is somewhat prevents him from getting popularity in comparison to JSON. Still it is a core stone of gRPC, which, in turns, is gaining popularity in microservice architecture world.
