EOSCommunity.org Forums

Greymass Roadmap - APIs

Greymass
#1

This post will serve as a wiki-style roadmap on the development efforts by Greymass to build and maintain APIs for the EOSIO ecosystem and all chains using it. These services are the backbone of our overall roadmap, which can be found here:

While this content here will be technical, it is our hope that it illustrates the inner working of the Greymass team and the contributions we have made while advancing these new technologies.

EOSIO APIs - who, what, why?

What are EOSIO APIs? EOSIO APIs are specialized servers that websites, applications, and wallets use to send and receive information from EOSIO-based blockchains. Different types of APIs perform different functions on the networks and offer varying benefits to developers looking to read and interact with the blockchain.

Why are more EOSIO APIs needed? The existing APIs offered by various organizations do not meet the needs and requirements of all projects within the space, including some of the projects led by Greymass. We have set out to build public APIs that meet our needs, which other organizations and their projects will likely benefit from.

Who is behind the creation of EOSIO APIs?: Members of the Greymass team started working to improve EOSIO APIs project in late 2018 with the introduction of the “Light History API”. Greymass, as an organization building on top of EOSIO, needed more scalable and improved APIs to expand and enhance the other projects we work on. @aaron, @johan, and Scott from the Greymass team are all involved in the development of these projects.

Progress and Roadmap for APIs built by Greymass

We will be posting regular updates in response to this post with changes as they occur, as well as maintaining this original post with an overall status of this effort.

Sub-project definitions:
Since we currently are building our APIs ad-hoc and don’t have dedicated names for many, listed below are what we refer to these projects as alongside a brief definition of what purpose it serves.

  • Custom History Solution: An unnamed custom EOSIO history API engine which matches the v1 specification but no longer runs the deprecated v1 plugin.
  • Fuel: An API service that can cover resource costs for user transactions.
  • Sextant: Another unnamed series of new generic EOSIO API endpoints Greymass is developing primarily to service wallets and other account related services.
  • History API: The live history APIs that Greymass operates behind all of its public API servers.
  • Light History API: The light history APIs that Greymass developed based on the v1 standard.

Summer 2020:

In Progress

  • EOSIO API: Sextant (Account Creation)
    • MVP Development complete, entering testing phase using Jungle 3 testnet.
    • Support for EOS, Telos, Proton and WAX integrated but not enabled until testing is complete.
    • Premium gm account name on supported networks has been registered for use.
    • Integration into Anchor (iOS) complete and available on TestFlight.
  • Fuel: Work has begun on version 2 of the Fuel API software, which is a complete rebuild that will now use eosio-core and eosio-signing-request rev3.
  • Custom History Solution: Deploy production version of new history solution to replace v1 plugin.
  • Custom History Solution: Release alpha version that others are able to test and run.
  • Light History API: Update to 2.0.7
  • ESR + OAuth2/JWT Server: Experimental new API layer to provide blockchain-based identities to off-chain applications.
  • HyperBuoy: A cloudflare worker implementation of buoy (ESR callback layer) that can be geodistributed for lower latency.

Completed:

  • Sextant (Account Creation):
    • Administrative interfaces
    • App Store Integration
    • API layer for wallet integration
    • Sybil protection on iOS
    • Allow our wallets the ability to create new accounts on predefined networks.
  • Custom History Solution: We have now deployed our custom history solution onto a 2nd EOSIO network, WAX, and are now serving all traffic through it instead of the v1 plugin. It is available publicly at wax.greymass.com. No issues reported thus far and the API remains completely reverse compatible with the v1 specification.
  • Custom History Solution: Optimizations and Improvements
    • Resolved a case sensitivity issue with get_transaction calls.
    • Added new caching mechanism for scripts which scrape history for accounts with long history. The caching mechanism caches recently decompressed chunks of history to prevent excessive CPU usage during subsequent lookups.
    • Optimized key/account lookup index to support accounts with massive amounts of child accounts (e.g. the wam account on WAX).
  • Merged the fixes from Block.one related to the new key lookup APIs into our 2.0.6 production branch (greymass-v2.0.6-hl-hotfix) and deployed to both EOS and Jungle 3.
  • Upgraded all WAX API servers to 2.0.x version.
  • Upgraded all FIO API servers to 2.0.0 version.
  • History API: Discovered and fixed a new bug involving the forking logic that only occurs when a producer is double producing.
  • History API: Optimizations within the filesystem have been made to avoid periods where the API would stall as it was processing heavy blocks.
  • All APIs: Improved reporting mechanisms to detect individual endpoints going offline, as opposed to servers.
  • EOSIO API (Key Lookups): Utilize the new key lookup APIs in nodeos 2.0.6
  • History API: Deploy new redundant servers for EOS history behind eos.greymass.com
  • Light History API: Update to 2.0.6
  • EOSIO API (Key Lookups): Create a new API endpoint that’s able to lookup keys across multiple EOSIO blockchains.
  • Fuel API: Create an API endpoint which allows the retrieval of a signature before submitting a transaction.
    • Available on our API endpoints though /v1/cosigner/sign by submitting an ESR request as request and a permission as signer.
  • Custom History Solution: Match key lookup API calls to v1 specification.
  • Custom History Solution: Add historic information about keys during lookup as additional fields.

Early 2020 and earlier:

Completed

  • Fuel API: Create cosigning solution that can intelligently cosign requests to cover resource costs of users.
  • Custom History Solution: Exact match of all the v1 plugins API calls and responses
  • Custom History Solution: Deploy new history solution to production
  • Custom History Solution: Move account history, key associations, and controlled account indices external to nodeos.
  • Custom History Solution: Move trace data out of shared memory and into an external data set.
  • Custom History Solution: Build associated API calls to access new external data.
  • Custom History Solution: Develop an event system to deal with forking logic.
  • Custom History Solution: Move trace retrieval process and compilation out of nodeos.
  • EOSIO API (Routing): Create unified endpoint to allow swapping of blockchains by specifying a chain ID in the request header.
  • Light History API: Update to 2.0.4

Unscheduled

  • History API: Start drafting a new API specification to improve on call structures, while maintaining v1 reverse compatibility.

Additional Information

Metrics - Adoption and Usage

For the sake of privacy of our users, we do not perform any long term metrics tracking of our API infrastructure or its usage. All logs are deleted within 24 hours of being recorded.

What we can share is live metrics of how many requests are made to access our APIs. We will make an attempt to update this post with periodic snapshots based on our averages, starting today.

  • September 16th, 2020, our APIs servers average between around ~146 million per day (~1700 requests/sec).
    • Traffic Breakdown by Network
      • EOS represents ~1350 requests/sec
      • FIO represents ~50-100 requests/sec
      • Telos represents ~10-50 requests/sec
      • Proton represents ~1-5 requests/sec
      • WAX represents ~200 requests/sec
    • Traffic Breakdown by Type
      • ~1500 requests/sec towards general state APIs
      • ~200 requests/sec towards history APIs
  • July 17th, 2020, our APIs servers average between around ~77 million per day (~900 requests/sec).
    • Traffic Breakdown by Network
      • EOS represents ~750 requests/sec
      • FIO represents ~25-50 requests/sec
      • Telos represents ~5-10 requests/sec
      • Proton represents ~1-5 requests/sec
      • WAX represents ~100 requests/sec
    • Traffic Breakdown by Type
      • ~800 requests/sec towards general state APIs
      • ~100 requests/sec towards history APIs
1 Like
The 3 aspects of the Greymass Roadmap
#3

February 2020

Since our January update we have continued to add additional components to our custom history solution. We set our sights on matching the v1 specification while focusing in on performance, scalability, and stability - all on reasonably priced servers. This new platform is built in golang and uses custom indexers to build data sets external to nodeos using BadgerDB.

1 Like
#4

July 2020

4 NEW, 6 COMPLETED

  • NEW (In Progress) - EOSIO API (Account Creation): Establish an account creation API service.
  • NEW (In Progress) - Custom History Solution: Deploy production version of new history solution to replace v1 plugin on multiple networks.
  • NEW (In Progress) - Custom History Solution: Release alpha version that others are able to test and run.
  • NEW (In Progress) - Light History API: Update to 2.0.7
  • UPDATE (Complete) - History API: Deploy new redundant servers for EOS history behind eos.greymass.com
  • UPDATE (Complete) - Light History API: Update to 2.0.6
  • UPDATE (Complete) - EOSIO API (Key Lookups): Create a new API endpoint that’s able to lookup keys across multiple EOSIO blockchains.
  • UPDATE (Complete) - Fuel API: Create an API endpoint which allows the retrieval of a signature before submitting a transaction.
  • UPDATE (Complete) - Custom History Solution: Match key lookup API calls to v1 specification.
  • UPDATE (Complete) - Custom History Solution: Add historic information about keys during lookup as additional fields.
1 Like
#6

August 2020

6 NEW, 1 UPDATE, 4 COMPLETED

  • UPDATE (Complete) - EOSIO API (Key Lookups): Utilize the new key lookup APIs in nodeos 2.0.6
  • NEW (In progress) - EOSIO API (Account Creation): API service layer now renamed to Sextant
    • Administrative interfaces created
    • App Store Integration
    • API layer for wallet integration
    • Sybil protection on iOS
  • NEW (In progress) - ESR + OAuth2/JWT Server: Experimental new API layer to provide blockchain-based identities to off-chain applications.
  • NEW (In progress) - HyperBuoy: A cloudflare worker implementation of buoy (ESR callback layer) that can be geodistributed for lower latency.
  • NEW (Complete) History API: Discovered and fixed a new bug involving the forking logic that only occurs when a producer is double producing.
  • NEW (Complete) History API: Optimizations within the filesystem have been made to avoid periods where the API would stall as it was processing heavy blocks.
  • NEW (Complete) All APIs: Improved reporting mechanisms to detect individual endpoints going offline, as opposed to servers.
1 Like
#7

September 2020

6 NEW, 1 UPDATE, 5 COMPLETED

NOTE: The API category is now divided into two sections, one for the development work that Greymass is doing to build API custom infrastructure and the other for more generic updates. For the purpose of the OIG, the Greymass APIs category is our work in the realm of APIs, where as the rest is generic maintenance we do as part of our operations.

Greymass APIs

  • NEW (Complete) - Custom History Solution: We have now deployed our custom history solution onto a 2nd EOSIO network, WAX, and are now serving all traffic through it instead of the v1 plugin. It is available publicly at wax.greymass.com. No issues reported thus far and the API remains completely reverse compatible with the v1 specification.
  • NEW (Complete) - Custom History Solution: Optimizations and Improvements
    • Resolved a case sensitivity issue with get_transaction calls.
    • Added new caching mechanism for scripts which scrape history for accounts with long history. The caching mechanism caches recently decompressed chunks of history to prevent excessive CPU usage during subsequent lookups.
    • Optimized key/account lookup index to support accounts with massive amounts of child accounts (e.g. the wam account on WAX).
  • NEW (In progess) - Fuel: Work has begun on version 2 of the Fuel API software, which is a complete rebuild that will now use eosio-core and eosio-signing-request rev3.
  • UPDATE (In progress) - EOSIO API: Sextant (Account Creation)
    • MVP Development complete, entering testing phase using Jungle 3 testnet.
    • Support for EOS, Telos, Proton and WAX integrated but not enabled until testing is complete.
    • Premium gm account name on supported networks has been registered for use.
    • Integration into Anchor (iOS) complete and available on TestFlight.

Generic APIs

1 Like
#8

November 2020

1 NEW, 2 UPDATE

NOTE: The API category is now divided into two sections, one for the development work that Greymass is doing to build API custom infrastructure and the other for more generic updates. For the purpose of the OIG, the Greymass APIs category is our work in the realm of APIs, where as the rest is generic maintenance we do as part of our operations.

Greymass APIs

  • NEW (In progress) - EEP-X: EOSIO Resource Provider Specification
    • This new specification outlines how to create a resource provider API endpoint to cosign transactions and cover the resource costs of users on EOSIO networks.
    • A functional business-logic free tech demo will be released alongside the specification.
    • The new wallet abstraction layer project we’re developing will follow this new standard for API design.
  • UPDATE (In progess) - Fuel: Work is nearing completion on version 2 of the Fuel API software, which is a complete rebuild that will now use eosio-core and eosio-signing-request rev3.
    • Beta version will be deployed to live servers soon.
    • The cosigning backend process has been completed with new features added.
    • The manager processes to track transactions and state have been rewritten to support the new cosigners.
  • UPDATE (In progress) - EOSIO API: Sextant (Account Creation)
    • Testing continues as we resolve bugs both on our end and those that have been introduced in iOS 14

Generic APIs

  • No updates, all running optimally.
1 Like
#9

February 2021

Greymass APIs

  • NEW (Complete) - Fuel has been deployed to the WAX network. We can now provide CPU, NET, and RAM to users of this network through our new APIs.
  • UPDATE (Complete) - Fuel version 2 is now released.
    • Version 2 of the APIs has been deployed to all networks we currently provide Fuel on.
    • The preflight request endpoint has moved to /v1/resource_provider/request_transaction with its parameters changed to match that of the new resource model. It can accept an ESR payload, serialized transaction, or a packed transaction.
    • Transaction fees are now implemented for CPU, NET, and RAM when a user either doesn’t have resources of their own or has exceeded their free usage quota.
    • RAM can now be provided to accounts for a fee.
  • UPDATE (Complete) - EOSIO API: Sextant (Account Creation)
    • Our account creation APIs are now live for EOS, Telos, and WAX. Access to these APIs is granted through the iOS version of Anchor, with more integrations coming soon.
  • UPDATE (In progress) - EEP-X: EOSIO Resource Provider Specification
    • Revision 1 is in progress at this point. The prototype of the specification has been established in Fuel v2 and is performing well. The specification itself is being based on our experiences building Fuel and will provide a solid foundation to present a standardized approach.

Generic APIs

  • We have added some rate limits on our WAX APIs that should not affect normal users and their use cases, but will prevent bots from spamming thousands of requests per second.
1 Like
#10

April 2021

  • NEW (Complete) - The callback service for Anchor (buoy) has been updated to be distributed.
    • 3 regional servers (North America, Europe, and Asia/Pacific) have been deployed to decrease latency for both apps and users.
    • MQTT has been implemented within the software to do peer to peer routing between regions, should for whatever reason the client and wallet be connecting from different regions.
  • UPDATE (Complete) - EOSIO API: Sextant (Account Creation)
    • Improved reliability of account creation for all networks.
    • Increased notifications the watchdog service is able to relay for better monitoring.
    • Abstracting payment providers to open up support for additional platforms.
  • UPDATE (Complete) - EOSIO API: Lighthouse (Account Lookups)
    • Added support for v1/chain/get_accounts_by_authorizers to improve lookup performance.
  • UPDATE (In progress) - EEP-X: EOSIO Resource Provider Specification
3 Likes