The Friction Points of Storing Data on Solana and how $SHDW solves them

What’s this article about?

In this article we discuss what are known as “Points of Friction” in the two largest decentralized storage providers: Arweave and FileCoin. As we examine these points of friction, we’ll discuss the ways that GenesysGo’s upcoming Shadow Drive and $SHDW token address these points of friction as we launch the only decentralized storage solution optimized specifically for Solana.

The goal of this article is to ensure that potential participants in the $SHDW IDO (Jan 3rd, 2022 at 2pm UTC) understand the existing landscape for decentralized storage on Solana and why the Shadow Drive and $SHDW token will be integral in the future scaling and success of Solana.

In case you’ve missed what’s been going on…

Hopefully by now you’ve heard about the GenesysGo IDO, the $SHDW token, and the work we’ve begun on our Solana optimized decentralized storage solution… The Shadow Drive.

If not, here are some resources that can read/watch which will add some context to some of the things in this article. With that in mind, much of this article is written specifically for those who have already read/watched the resources below…

https://www.youtube.com/results?search_query=genesysgo — For starters, this is just a list of YouTube videos that show up when you search “GenesysGo”

Below, is a list of slightly more curated resources!

Knox Trades and I have a long robust conversation about GenGo, $SHDW, Solana, and infrastructure as a whole. If you have no idea where to start, I’ve been told that this is probably one of the best videos to ELI5 your way into why the $SHDW IDO is something you should be a part of!

A more recent video that does a really great job of hitting the highpoints and doing it in a really quick and concise manner.

Finally, I wrote this Medium article to act as a full guide to everything we’re working on. In the meantime, let’s move on to some of the biggest friction points when it comes to utilizing decentralized storage solutions on the Solana Network and how GenesysGo’s Shadow Drive solves them!

What are “Points of Friction?”

It’s important to make sure we’re all on the same page when it comes to terminology. Points of Friction describe the points in the use of a particular project, service, etc negatively impacts the user experience and creates user friction.

The more “Points of Friction” you have in your system, the higher “User Friction” becomes, the less likely you are to see users consistently use said system.

A system which addresses points of friction and lowers overall user friction has a high likelihood of gaining mass adoption by stealing over market share from a high friction system.

The ethos behind building the Shadow Drive

Arweave and Filecoin are expensive, difficult to use, difficult to access, and have proven to be unreliable when it comes to serving the ecosystem of the Solana blockchain.

It’s important to remember, this article is written with the Solana blockchain in mind. We’re not saying that we’re trying to build the Arweave or Filecoin replacement… or, even really competing for their market share (…yet…).

What we’re saying is that Arweave and Filecoin have proven to be an unreliable solution to those building on Solana.

With this in mind, the ethos behind the Shadow Drive is simple…

  1. The solution should be cost effective and easy to access.
  2. It should be optimized for speed, reliability, and to handle Solana’s throughput.
  3. The user experience should be uncomplicated and seamless

With all that said, let’s examine some of the ways that Shadow Drive seeks to improve the decentralized storage experience on Solana.

Ease of Access

If you are a Solana builder and want to use Arweave or Filecoin to store your data then you’ll need to hold some of the native token, $AR or $FIL respectively.

So, you’re a Solana builder and you need storage for your project’s data. You’ve heard that Arweave stores data forever and so you plan to store it there. Well, you’ll need $AR tokens in order to do so.

You go check all your favorite Solana DEXs and Swaps but can’t find $AR listed anywhere. So, you go check out FTX and just find AR-PERP.

So, then you head to Google and ask…

Oh… well then. Guess you need to go figure that out…

What about Filecoin? Well, same problem because it’s not listed on any Solana DEXs or Swaps. Let’s try FTX again?

PERPs and Futures contracts… what can you tell us Google?

Neither $AR nor $FIL are SPL (Solana) native, meaning you cannot just scoop up some $AR or $FIL into your Phantom wallet. You have to go pick them up on other blockchain networks, which forces you to have to wade through potential headaches around high gas fees and low txn speeds.

The $SHDW token will be the first Solana native storage token that is traded on multiple DEXs, you can hold in your Solana wallet (i.e. Phantom Wallet), and can transact with in the safety of Solana’s low txn fee ecosystem.

The three DEXs where $SHDW will list immediately after IDO (more DEXs to be added)

Ease of access is incredibly important to how we think about the Shadow Drive’s ability to gain user adoption. By being a Solana native token (and the only decentralized storage utility token in the Solana ecosystem), the $SHDW token (and the Shadow Drive) has instantly removed a huge point of friction around decentralized storage.

Reliability

As the largest RPC provider on Solana, we have powered well over 100 NFT drops. When a NFT project connects to our network, we often get asked, “Is your network having issues? Because I’m trying to upload my NFT meta-data to Arweave and it is telling me that my transactions keep failing.”

Nope, it’s not the Solana RPC servers that are having problems… it’s Arweave’s. Since the root cause to the issues in this section are shared by both Filecoin and Arweave, we’ll focus specifically on Arweave in an effort to keep the article length somewhat in check.

So are these Arweave transactions failing? You see, Arweave has a pretty low max speed when you compare it to Solana…

5k TPS isn’t gonna cut it for Solana…

Compare that to Solana…

Starting to see where I’m going?

When a blockchain gets behind and can’t build blocks fast enough to handle the amount of transaction requests thrown at it then those transactions start failing and have to resubmitted.

Arweave uses its own home grown consensus mechanism (Proof of Access) and all data must be passed through that consensus mechanism to be stored on Arweave. The issue is that this significantly slows down Arweave’s ability to process transactions and keep up with Solana. This is because, ultimately nothing can keep up with Solana, except Solana.

This results in data uploads to the Arweave network from Solana having multiple failed transactions that still charge you the $AR tokens despite failing. Imagine trying to upload the NFT meta-data for your 5,000 NFT collection but, because of transaction failures, you have to pay as if you uploaded 10,000.

To wrap your head around how the Shadow Drive addresses this issue, I’d refer you back to where I wrote, “Nothing can keep up with Solana, except Solana.” The Shadow Drive doesn’t take information that has already been through the Solana consensus mechanism and then try to feed it through a second one… why do that when the Proof of History mechanism already exists and is already cutting edge?

Instead, the Shadow Drive focuses on what we call, “maintaining the integrity of the consensus.” Our experience is deeply rooted into the Solana validator/RPC infrastructure and, as such, the architecture for the Shadow Drive is built to take advantage of the data streams that Solana themselves has already built.

Solana validator nodes already communicate with one another using completely secure UDP port connections. The data in these ports is already hashed (encrypted), sharded (broken up into copies), and already been verified as valid data. We’ve spent the past nine months deep in the tunnels and pipes where this data flows and already have the infrastructure in place (via our RPC network) to begin storing it.

The cause at the root of Arweave and Filecoin’s reliability issues when it comes to storing Solana data is they fundamentally have the wrong architecture to integrate into Solana.

Meanwhile, the Shadow Drive isn’t so much built on top of the Solana network, as it is built directly into the Solana network at the most foundational level. This would completely remove the point of friction around reliability.

Flexibility of the Use Case and Cost Model

The last point of friction we’ll discuss is around the way that both Arweave and Filecoin are designed to be used. In this section, we’ll start with Arweave and then move on to Filecoin.

So what is Arweave’s use case? Arweave was built to store data forever and in a permanent fashion. Once data is uploaded to Arweave it lives there forever and can never be changed.

Imagine if those embarrassing spring break pictures suddenly became immortal…

Now, in many cases this is great and can have many use cases. However, what if you want to store data that you might want or need to go back an edit later? Or, what if you want to store data but don’t want to store it forever? Well, unfortunately, that is not what Arweave was designed to do.

Additionally, because there is no flexibility in this model there is also no flexibility in the cost either. Arweave charges their users by having them pay for 200 years of storage up front and they have done a significant amount of math to forecast where they think the cost of storage is going to go.

Because storage on Arweave is designed to be forever, Arweave requires that you pay approximately $10,217 worth of $AR per terabyte of storage (this price fluctuates and the price shown below was the cost of storage as of the timing of this writing).

Now, this is where we get into Arweave’s very specific and, in our opinion, somewhat narrow use case… because if you want to store data literally forever (or, at least for the 200 years that you’ve paid for) and have the data stored in such a way where it can never be altered then Arweave is pretty great! Well… until you consider just how long of a period 200 years actually is… for context… 200 years ago, the year was 1821 and the United States had purchased Florida for $5,000,000.

Just 84 years ago data storage looked like this…

Would the addition of a third person make for the beginnings of a consensus mechanism?

In order for the exceptionally high cost of storing data on Arweave to be worthwhile, you need your data to be stored on Arweave for the full 200 years. Arweave’s yellow-paper addresses this notion as well…

The last few sentences feel a little “hand wavey” to me

Ultimately, there is a very high chance that Arweave users will have paid many many years of storage costs that will never get used and they have no choice but to pay for 200 years of storage regardless of whether they need/want it or not.

Meanwhile, Filecoin has taken the opposite approach and has built a use case and cost structure that is more similar to big tech cloud storage. Filecoin users pay “rent” in order to have their data stored and the rent charged is variable depending on the rates being charged by miners (those operating Filecoin nodes).

Currently, the points of friction in the use case and cost structure of Filecoin are significantly less than Arweave’s and we have found a lot of value in modeling some aspects of the Shadow Drive off of Filecoin. The Filecoin ICO and current market cap are arguably closer to what $SHDW is capable of in our opinion, however we do see points of friction emerging over time for Filecoin as well. This is because of the buy/sell marketplace style setup that Filecoin has.

Users must find a node operator who is willing to store their data and the user + miner must “agree” on the cost of storage to be paid. This starts to sound somewhat reminiscent to users who are attempting to have a transaction validated on Ethereum and must find a miner willing to validate the transaction for an agreed upon price. Gas fees are something the Solana ecosystem is especially cognizant of and rising storage costs simply due to miners increasing their storage fees is a potential hurdle that Filecoin cannot address due to the marketplace structure being built into the fundamental structure of their architecture.

The Shadow Drive provides users with the flexibility to choose how long to store the data, is the data editable or immutable, but strips away the “gas fees” style storage marketplace in favor of a fixed cost structure that still provides storage at a cheaper rate than Arweave, Filecoin, or even big tech cloud storage.

The team and I are able to improve upon the designs of both Arweave and Filecoin and create less points of friction because we are building with a true web3 mentality in mind. Arweave and Filecoin are both inhibited because they are built upon legacy business models which focus exclusively on the exchange of a service for a currency.

As I recently wrote in a Twitter thread, web3 has enabled the creation of many new and creative ways to power an infrastructure and empower node operators (Shadow Operators as we like to call them) with ways to contribute compute, share in the cost of expenses, and help drive adoption of the service provided.

By building the Shadow Drive with the best parts of Arweave and the best parts of Filecoin in mind, we’ve created a use case and cost structure that allows for much more creativity, innovation, and flexibility.

For the specific numbers on what that cost structure looks like, please see the “Comprehensive Guide to GenesysGo” Medium article which is linked at the top of the page.

Tying it all together

As stated, the original goal of this article was to “Points of Friction” in both Arweave and Filecoin and to review why neither of these solutions has made any meaningful efforts to integrate into the Solana ecosystem.

There are fundamental design differences in how both Arweave and Filecoin work that will prevent them from being able to serve a significant amount of the Solana ecosystem. Without designing systems specifically for Solana, neither decentralized storage provider will be able to handle the throughput or the amount of data produced by the Solana blockchain.

Thus, you have multiple points of friction which ultimately contribute to very high user friction and creates a huge opportunity for a decentralized storage solution which removes them.

The Shadow Drive and $SHDW token are designed specifically to handle Solana’s throughput by directly integrating with the Proof of History consensus mechanism. Additionally, the ease of access to the $SHDW token will allow new users, who have never tried to use Arweave/Filecoin due to the difficult in acquiring $AR/$FIL, to easily access and use the Shadow Drive.

Removing these points of friction the Shadow Drive’s architectural and economic models will allow the Solana ecosystem access to data and information in a way that has not been previously possible and will ensure that the Solana blockchain is able to scale for decades to come.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
GenesysGo

GenesysGo

GenesysGo is a Blockchain Service Provider. We provide robust and secure infrastructure to give builders, stakers, and users the performance they deserve.