Back to the Basics
The ditching of PARSEC is a big milestone of the SAFE project, and nobody says it better than Irvine himself on July 28, 2020:
We are on track now and making great progress. No to a network that did not follow the original design principles but to the actual SAFE network as envisaged so many years ago and sold to investors in 2014. That network is lighter, faster and much more natural than the network pushed for in the last few years. We now are on the verge of the actual SAFE network and that is great. It’s been a very exhausting ride and the last 12-18 months were horribly difficult, but in the end the network can only work in the way we are now building it.
We will have parsec gone soon, no more false and unnatural network invariant. This is now the real deal, the natural network based on natural design where errors happen and are recoverable as opposed to a network where an error brings the whole thing down as it broke an invariant.
A huge difference is the team, now we are heavily weighted with community first Engineers. That passion is essential to success and we have that a plenty now. The passion to launch and launch everything at once is the biggest driver we have had. No more debates over absolute must have total order for everything before acting. Now we have a network where things happen, order is guaranteed, but over time and where infrastructure knowledge is transferred with every section message, we have farming started, safecoin in place and much more, things that were thought to be years and years after Fleming.
So this is it folks, the real deal, not a side show or partial view of the network, but Fleming is coming with batteries included. It won’t upgrade but that last part will happen very soon. Then we can debate algorithms for costs/distributions and much more, but we can have those debates with a live network and not a dream network.
It took time and we had so many side streets it was dizzy, but we are hard and fast on track to a solid roadmap here. That roadmap is really simple for Fleming, Launch everything, leave out nothing. This means what was previously seen as years of work, being compressed into the last few months. It will happen, but it will happen correctly.
By "unnatural" Irvine was referring to Feynman's remark that if you can't find something (a design) in nature, then it is wrong. Earlier on May 29, in a response to a criticism from a forum member that "the problem is not about the trial and error, the problem is the premature promotion", referring to PARSEC, Irvine wrote:
I can only agree here actually. There was too much promotion and way too much time making an individual crate “production ready”. Parsec is still a great achievement in ABFT. It was believed to be the answer to many things by ordering everything. The problem is centralised data structure to order all other data structures.
In any case we tried it and we tried to make it work with data and it was problematic. I was not a fan of spending more time finding ways of pruning the graph or sorting the malice issue and so on. It’s not required to order as parsec orders as it does now, like template/generic spread through all aspects of the system.
With data it made sense (as it did in 2015 with data chains) to allow data to find it’s own order and with causal order via CRDT types then that is a solved issue. This left parsec handling routing membership and it does that fine, it is right now, but with a caveat, the graph keeps growing and there are naive steps to prune with 10 1GB logs kept, this is out of line with “work in as many machines as possible”. As membership can be handled with group consensus (dkg) and, if required (I think not) a bft crdt orswat (we have POC of that and it will be published soon, but not by us).
All in all I am happier to write all that off and move at the pace we currently are, I cannot afford an 18 month fix period for the order everything approach. It is not how SAFE was envisaged, if it had worked and been a panacea, then great, but it’s just not. Unfortunately a case of tunnel vision there I think and not considering the network as a whole. These days we have meets where all we discuss is the whole network. It’s a different proposition and pace to launch is way beyond any other period in our history.
tl;dr Tried it and really tried to make it work, now back to SAFE and focus on provably consistent highly concurrent data types. It is so much simpler and we will get working on smaller machines as has been my goal for … well forever really :wink:
In a subsequent post, he emphasized his principal approach to launching the network in this critical phase of the project:
If you recall Fleming was headed to be a routing only network mostly. So focus on membership changes. That would have been a disaster IMO and I argued for full data handling, even safecoin. I felt that, otherwise we would be misleading folk and mostly ourselves. It did cause a lot of internal debate and strife, but this all shows why Fleming has to be much more than a view of a bit of the network with no function. It has to be a point where we can say “all this works” and the network function is proven now. Lets add upgrades, some clean up and efficiency improvements and call that Maxwell and then just get to beta. No more side tracks, full steam ahead with all functionality, even when it’s brittle.
...
There’s a lot of reasons to be comfy here, we will all understand all the code. No super genius required and no complexity that is not needed. So secure and efficient is the goal.
Clearly the "kids" in the team, esp. those working on PARSEC, perceived the project as a new R&D opportunity, so you work on each module, optimize and test it as thoroughly as you can, before even thinking about putting all the pieces together. Although they all knew that SAFE was a decade-old project, they did not have the visceral understanding of this fact. It's easy to think that to redesign and reimplement the internet, you will need the resources of all the tech giants combined, and thus as a small team, you can't be impatient. But Irvine is the last person you call impatient. He just has a much more integrative and evolutionary mindset in making things work. This is indeed related to the notion of premature optimization, as you can easily overlook or entirely miss the important constraints the system imposes when you make individual components in isolation, and this is apparently what happened during the PARSEC endeavor: they didn't even bother to pause and check if this wonder tool would actually work within the network as a whole. A great lesson learned.
Data Structures
New CRDT to rule them all:
Up until now we’ve been exposing two different Safe native data types, a Sequence (already being CRDT) and a Map (which we haven’t yet converted into CRDT). They both used to allow mutations to the Policy set for them, i.e. the permissions to access and mutate them.
After more research and tests made on the Sequence data type, we came to the conclusion that being able to handle all different types of scenarios of concurrent Policy mutations, combined with concurrent mutation of the data itself, was making the logic and code very complex. We also realised we were trying to go against the CRDT nature itself. This is why we moved away from that path and we now have a
RegisterCRDT data type which doesn’t allow mutations on the Policy - once the content is stored on the network the Policy is enforced on each message and while the policy cannot be mutated.This all means that all our client side abstractions, e.g. FilesContainers and NRS containers, are now based on this single CRDT data type.
Data API
A vast proportion of today's web applications run on relational DBMS servers, understandably, because a powerful query language makes lives easier. Can the Safe Network compete with that developer experience, in a new world where server is not a thing?
Irvine in Feb 2019 commented:
I think the SAFE network at launch will not be anything like a SQL database. It is a huge discussion, to replicate SQL on a server is unlikely, to replicate the function provided by SQL on a server is. SQL on a sever is generally faster as you do not worry about security and can use data locality etc. The cost of that is security and scalability. So then you look at Amazon etc. they do not use SQL servers, but decentralised systems like dynamo, it is more CRDT like as opposed to consensus driven ordering (PARSEC), but works at huge scale and secured behind a firewall. We do not need the firewall as SAFE has secured data. So yes this can be done and done at scale with te security of SAFE, but it will not be SQL, but it can provide the same end user results SQL does.