The most common secret in the Storj ecosystem is an access grant, which is used in client applications to access objects. At a first glance, an access grant looks to be long random text:
But under the hood it has an internal structure and stores important information. This post explains the content internals of the magical secret grant and explains how the underlying information can be parsed out of it.
To understand the internals of a grant, first we need to decode the string with a base58 decoder. Base58 is similar to base64, it’s just a binary-to-text encoding which can make printable strings from any binary.
When we have the raw binary data, we can start to investigate it further. The first byte here is the version (today it’s always 0). The last four bytes is a checksum (first four bytes of sha256 sum which is applied twice: sha256(sha256(data)))
Let’s drop the version and checksum information and focus on the data part, which is serialized with protobuf.
Protobuf is a widely used encoding, and usually the encoding/decoding logic is generated based on a type descriptor specified in “proto” files. But under the hood it uses a simple encoding:
- Integers are encoded with variants (simple byte values, but first bits helps to decide the length of the full number)
- Strings and byte arrays are encoded with the length first and the raw value.
In our case the protobuf encoded data has three parts:
As we can see the first part is a string. Because string is encoded as (varint + string) the content can be read without any encoding just with decoding the grant with base58.
(Note: Please don’t try it at home! Copy-pasting your access grant to online decoders is not safe)
The Satellite address is just the host and port of a Satellite. The only addition is the “node id.” This id is a unique identifier, and each node has one in the Storj ecosystem. But under the hood, it’s the hash of the CA cert used for SSL connection. Adding it to the address helps to avoid man-in-the-middle attacks, as only the original Satellite can provide certificates signed with the generated certificate authority.
The next part is the API key which is slightly more complicated as it is a macaroon:
Macaroon is a layered cookie and a layered secret structure. It has three parts:
- The head is just a unique random value
- The caveat is a list of permission restrictions (some Storj examples: DisallowList, NotAfter, AllowedPaths)
- The signature is a cumulative signature, which makes it possible to add more caveats after the first creation (any holder of macaroon can further restrict it by adding more caveats and updating the signature)
The caveat restriction is based on the specific way how the signatures are generated:
Satellite API (the original issuer of the API key) generates the unique random header and the secret. They are saved to the Satellite database under a specific name (name of the API key).
With this signature schema, it’s possible to further restrict any API key any time without updating the database. A Satellite can validate all the derived macaroons thanks to the saved secret, and when the record gets removed from the database, all the derived macaroons become unusable.
And finally, we can have a look at the encryption access.
An API key is enough to access a Satellite and get the metadata but to decrypt the object names and object content, a secret is required. This secret is also included in the grant, and it’s derived from the secret password to have a predictable length. The algorithm is simple:
The secret is the input that you type in the web UI or an uplink.
The created DefaultKey is stored in the grant (together with the used path and content cipher and block size) and used to encrypt the path of the objects.
One API key (macaroon) can restrict access only to multiple sub-path (prefixes). In this case, the DefaultKey shouldn’t be shared in the grant, but multiple patch specific keys can be included (in the Store entries).
As a summary: the access grant includes all the information which is required to upload/delete/use objects in Storj DCS.
- Address and identity of the Satellite
- API key which gives permission
- Encryption key which makes it possible to decrypt path and object content
As a bonus: the internal structure of an API key also can be checked with the uplink CLI. Here’s an example access grant:
