Looking for review and comment on the following:
The context for which this is designed for is described here but that document is not complete, it is waiting on the code being finished.
I presented a part of the Mesh to a closed audience yesterday. Their response was 'this is blockchain for encryption'.
Now as many of you know, I avoid the term 'blockchain' because, well there is a huge amount of fraud in the crypto-currency world and one of the biggest frauds is the security claims being made. And there is a huge price bubble that is inevitably going to burst some time. And there is that business of using 0.5% of global energy to support the system.
The audience in question specializes in information engagement and they told me that what I was describing was blockchain encryption and either I described it as such or people won't understand it.
The DARE container format is an append only log file with integrity checks via a Merkle Tree and incremental encryption. Any record in the chain can be encrypted under the (salted) result of a key exchange presented earlier in the chain.
So one application could be incremental encryption of server log files. Attempting to encrypt a log file by performing an RSA encryption of each entry is obviously nonsense from the performance and data volume point of view. Curve25519 is a little more practical but is still more overhead than is likely to be tolerable.
The DARE structure uses a public key exchange to establish a master key. Individual records can then be encrypted under a key generated from that master key by means of HKDF Key Generation with a unique per entry salt.
The net result is that the DARE format can be used to encrypt arbitrary numbers of entry or arbitrary size bounded only by the limitations of the encryption algorithm (2^39 for AES), data length (2^63) and nonce size (2^125 for the UDF scheme). These limitations could be lifted if people think there is a good argument to do so. The only one I think might be an issue is the one imposed by AES.
As you would expect from me, the format supports use of split decryption keys so that decryption of the data can be gated on a decryption key share in the cloud that does not have decryption capability by itself.
The container format supports rapid traversal in either direction by the inclusion of frame length indicators at the start AND end of each frame (always), Merkle tree indexing (optional) and direct indexing (defined but not yet implemented).
The Merkle tree can be signed to provide non repudiable authentication for the entire tree.
Besides GDPR compliant server logs, I can think of many, many applications. I use the structure extensively in the Mesh to manage sets and lists of entries. One of the applications I have always had in mind is end-to-end secure Web services including chat rooms and comment forums/mailing lists.
So is there anything I must add?
Right now, I have not published my MetaNotary protocol because I haven't implemented it. I have published the basis for the analysis though in the trust paper.
A MetaNotary is simply any notary that includes the outputs of other notaries. Thus it is possible to prove quite easily that no other scheme can ever present a higher work factor than a MetaNotary since if such a notary existed, the MetaNotary can simply consume its output as an input and present at least equal the work factor.
The incentives for participating in a MetaNotary are very interesting and very similar to the arguments I made when I invented the referer field in HTTP as a mechanism to support advertising.
Let us say that there are notaries A and B who both serve separate communities. The communities rate their notaries according to the work factor that they present. Let us also suppose that notary C would like to enter the market.
Notary C can offer a higher work factor than A or B by consuming both as inputs. A and B can however match the outputs of C by agreeing to consume each other as input.
Simulations show that the optimal strategy for notaries would be to consume as many other notaries outputs as possible so as to increase their work factor as much as possible and also to provide an incentive to other notaries to consume their outputs as input.
The net is that I expect to see a notary structure emerge that is similar in structure to the PGP keyserver infrastructure that has emerged based on Brian LaMacchia's original MIT key server. Essentially, any notary will be able to gain notarization of their output and fixing of their input by contributing to and consuming from one or more public metanotaries.
In short, the system will hold together through reciprocity. Since metanotary.com
was taken, I call this the emergent reciprocal meta-system the internotary.
While there are still arguable advantages of the Satoshi consensus mechanism in terms of performance, it is an algorithm that cannot be made efficient because it is the inefficiency of the algorithm that makes the Satoshi consensus work. The internotary offers a different but compelling compromise between partitioning, availability and consistency at negligible cost. Furthermore, the internotary consensus does not require any consumption of resources of any type.