EOSCommunity.org Forums

EOSIO Reference Authenticator and EOSIO tooling - how we got to where we are today

This is largely an opinion piece focused on the EOSIO Reference Authenticator, UAL, and the tooling surrounding these projects. All were created with great intentions, but the execution didn’t work out.

It’s been a long while since I have looked at the reference authenticator, but from what I remember: the payload specification wasn’t going to work well (it was all hex encoded JSON, and a lot of it), the eosio.assert contract was very cumbersome (both on-chain and for the APIs/wallets), and the manifest specification (while nice and related to the conversations today in Telegram) had many quirks. The SDKs (Swift and Java) used in the reference authenticator, while not directly related, were also heavily opinionated by the choices made within the other components listed above.

UAL on the other hand survived and has seen some adoption. I think that’s largely because the reference authenticator was abstracted away from it enough and other wallets could build plugins for it. As soon as a second active protocol emerged (ESR/Anchor coming in to a Scatter/protocol dominated market) that need was realized. The execution on UAL is ended up in a similar situation as the authenticator itself, and I wrote details about the shortcomings of UAL in this thread and touch on it briefly when posting about whether developers should be using a wallet abstraction library or not.

We have put a lot of effort into our SDKs, which integrate into UAL, and now see areas where some components should be elevated out of Anchor-specific code and into wallet-agnostic code for others to use - the newest of those being Resource Providers. This is incredibly hard with how UAL was initially designed since the interfaces themselves are what need to change. That’ll require redoing everything most likely and at that point, it might as well be a project built from scratch and reimagined for todays needs.

To get even further in the tool chain - UAL is based on eosjs which brings along another set of limitations. UAL operations largely depends on how eosjs functions… which is basically a black box. You push in A to a function and you get B in response. There’s no room for interpretation in how the code goes from A to B, which is a problem when you want to go from A (create transaction) to C (cosign transaction) to B (sign transaction). There are many instances where this same situation plays out while using eosjs and it’s not possible to deal with unless you copy chunks of the eosjs code and reimplementing it in your application.

From top to bottom the components in this stack had design problems. Seeing as how we’re on the frontier of technology, this isn’t unexpected and should almost always be expected. The problem however was that B1 (as the creator of these libraries) didn’t identify these problems in the design and we (the community of developers) hadn’t yet experienced the issues to help shape it. As an outsider, I can only imagine this was not the developers fault but instead the general nature of priorities within a company. However, with how interconnected all of these components were, it ended up being very hard to affect change without breaking other things, leading to a situation where the community really couldn’t make much use of these things.

A new generation of tooling

So with the history outlined and some of the problems touched upon, this is where I’m going to start talking more about what our team has been doing for the past two years related to this, and hopefully illustrate the importance of this effort. Right now I am more interested in getting other developers engaged than shilling our work, but it’s probably going to sound like shilling.

So starting with the authenticator and its related components… everything used in the authenticator was depended on one another, the payload spec required the assert contract, the manifest spec, and even the SDKs to some degree. You couldn’t use the authenticators approach unless you accepted all of the components and each of their individual pros and cons. We believed the cons of the entire stack outweighed the pros of using any individual piece, and the changing any individual piece required changing multiple parts of the stack.

So we started our approach by creating individual components that didn’t depend on one another. This is where the EOSIO Signing Request proposal (EEP-7) (ESR) came into play serving as the basic building block that defined a transaction. The payloads from ESR are encoded using the same ABI encoding techniques the blockchain uses, which solves many issues with size and compatibility that the reference authenticators protocols would have run into. It also cuts out all of the excess data, that while useful for some requests, don’t need to be required by all requests. Instead it allows those extra components to exist without rigidly requiring them.

As for the reference authenticator app itself, the reference authenticator was a UI built on a specific payload/assert/manifest spec, so it wasn’t really reusable. We started prototyping Anchor mobile, heavily inspired by the reference authenticator’s approach, but changing the internals to use ESR. For our iOS prototype, this also led us to implement the ESR specification in Swift, and while we were at it we wrote our own library to handle everything Swift related and ditch the B1 library. We stopped using the B1 library for the same type of reasons I outlined above related to eosjs, it’s difficult to use and serves mostly as a black box.

Anchor at this point could now take in an ESR payload and sign a transaction. The difficulty though was getting the transaction payload from an application to the signer - which led to the creation of another protocol on top of ESR, the Anchor Link protocol (which should have a more generic name since it’s capable of more than just supporting Anchor … but it has stuck). This is another building block, one which is capable of establishing a session between an ESR compatible authenticator and an external ESR compatible app/dapp.

If it breaks down or is unavailable, the ESR payload with an ESR authenticator still works - it’s just not as easy to send them from point A to point B (you need a URI handler or QR code or something). It also introduced encryption, identity requests to serve as a handshake mechanism, and a number of other convenience features that help improve the user and developer experience.

It’s also completely replaceable - someone could take the ESR specification and build an entirely different way of relaying the requests without how we designed Link or the transport.

Anchor Link was also intentionally designed to accept modular transports, allowing the creation of different user interfaces and ways to use these sessions. We created the first one we just generically called the browser transport. This transport is what we plug in to wallet abstraction layers like UAL or Transit to provide the experience users experience today. This is also where we have integrated Fuel to provide both free and paid transactions to the end user of an application, though this logic should realistically exist in the wallet abstractions themselves.

In developing both Anchor Link, the transport, and the plugins to distribute this tech - that’s when we hit the wall with eosjs. So just like we did with Swift, we built a powerful javascript library that serves as a replacement for the B1 code that we simply call @greymass/eosio (and sometimes refer to as eosio-core). This library serves as another stand alone building block, which now Anchor Link and the transport are based upon (along with a lot of other projects). It allows us as developers to overcome the challenges we experienced much more easily and allows us to do things that just aren’t possible with eosjs.

Today

It’s been a couple years now and despite all this effort there’s still a lot more to do. Just based on what was described above, I can easily rattle off a few issues discussed above that we haven’t touched yet:

  • The problems that the manifest/assert components originally tried to solve still haven’t been resolved yet. We have some ideas, which were the ones that sparked this entire conversation in Telegram.
  • UAL hasn’t been improved upon and has severe limitations on how it can be used. This is an area I’d really like to tackle since I think we have some great experience and insight working with Anchor Link and the transport - but there’s little incentive to do so.
  • The Java SDK from B1 has many of the issues that the Swift SDK from B1 did. We have a replacement for the Swift SDK, but no replacement for the Java one. It would be worthwhile to revisit and redevelop it for Java developers using EOSIO.
  • Some of the new solutions presented above are not complete and many have no real documentation, so its hard for other developers to adopt them. This again is largely due to prioritization issues and lack of hands on keyboards to generate it.

The future is bright though, and I really believe what we have started and continued working on, and would like to get others working on with us, is the start of the EOSIO 2.0 application stack. The EOSIO 2.0 backend launched a while back and had significant improvements, and the backend keeps improving - but the application side of the equation as it is presented to developers today is very rough around the edges.

Things like ESR, the Link protocol, transports, the JS/Swift/Java SDKs, a UAL replacement library, and clear documentation around all of it are what EOSIO development is going to look like in the future.

11 Likes

Some of what went wrong came from my own design decisions, so it may be helpful to go over them as a way to avoid repeating past mistakes.

Eosjs 20 started small. It was a set of independant pieces that could come together to go from a JSON request all the way through to pushing a signed transaction to nodeos. Since those initial pieces could come together in many ways to cover many possible flows, and since I assumed most UI devs wouldn’t want to learn to piece those together, I created an entry point to cover the most common flow at the time. I intended that entry to also function as a sample of how to bring the pieces together so apps which needed diffent flows could see how to customize it. This double use was a mistake.

Over time, more flows became necessary, and outside dev objection to bringing the low-level pieces together themselves was even stronger than I anticipated. The flows also grew in complexity. In response, the entry point eventually became an object and ended up doing a lot more than just gluing low-level pieces together. It wasn’t very customizable beyond a fixed set of flows and options, and was no longer suitable as an example of piecing the other components together. I don’t like what the object became.

Another problem: is a SignatureProvider’s job to sign something, or to present a UI? At the time, the two seemed unseparable to support UAL. I now believe failing to separate these contributes to eosjs’s problems.

4 Likes

I absolutely agree it’s helpful and very much appreciate you chiming in on this topic. Being able to look back and see what did or didn’t work as intended is the best way to figure out how to move forward and improve. I’m preaching to the choir here… but figured it was worth saying for anyone else reading along.

I’m assuming we’re talking about the transact flow. Even today, it’s still probably the most common flow and will likely remain that way for a while. Most applications, being on the more simple side, have really benefited from the existence of this flow and the straight forward path it gives them to create and submit a transaction to the blockchain.

I don’t want to discount that part of eosjs at all, since without that initial work, things like eos-voter (and then Anchor) could not have existed.

It did serve this purpose for us for a long while and we still use a couple different variations of these pieces tied together for some of our flows. It became unwieldy though around the time we started using Fuel and ONLY_BILL_FIRST_AUTHORIZER to start cosigning transactions. With that seeming to be the inevitable future of how these chains operated is when we seriously started considering “what do we need to do to make this easier?”.

With how dynamic this entire space is, I don’t think any of us saw this coming. We didn’t know what we needed until we literally used the software for a couple years. You summed this all up pretty well in your response about the increase in flow complexity and eventually it not becoming all that customizable. I can’t tell you how many pieces of code I’ve written that have ended up in the same situation.

Yeah - it got pretty blurry here. I think somewhere between when a developer calls transact and a SignatureProvider responds required a few other steps that could have been injected between. The UI and any sort of modifications to the transaction being requested could have happened here.

I suspect since eosjs really has no notion of the UI, this ends up making it very difficult to inject other operations in this mix. I know from our end, being able to stop the transact flow from an app and prompt the user to accept a fee would have been pretty difficult without changes there. I know UAL struggles with this as well which is why our UI prompts all exist as a layer living on top of the UAL UI.

Thanks again for jumping in on this topic. It’s safe to say that without the work you (and others) put in on these technologies, who knows what the EOSIO ecosystem would look like today.

5 Likes

Thanks for this update. Your team and efforts are a bright spot. I intend to have my team investigate and learn from this work.

2 Likes