Storj informed B&F on its decentralized storage offering, revealing why Web3 storage decentralizers use cryptocurrency and blockchain, and how Storj makes decentralized storage fast.
Decentralized storage (dStorage) takes the virtual equivalent of an object storage system and stores chunks of data on globally distributed nodes owned by independent operators who are paid using cryptocurrency, with storage I/O transactions recorded using the blockchain. You realize how strange this is when you notice that the nodes of an on-premises object storage system do not use the blockchain to record their transactions and do not require any cryptocurrency payment. But it’s Web3, so cool?
Storj offers decentralized cloud object storage (DCS) software, formerly known as Tardigrade, which uses MinIO and is S3 compatible. John Gleeson is the COO of Storj and he answered the question about cryptocurrency: “It’s about cross-border payments. Cryptocurrency allows us to pay what is currently around 9,000 operators, operating 16,000 storage nodes in 100 different countries in a way that I can’t see any other way to do it. Really, the cryptocurrency component is not an innovation.
If Storj paid its storage node providers – its operators – in fiat currency, then it would have to manage exchange rates and transactions. Using cryptocurrency allows Storj to avoid currency exchange complications – the operator is paid under the cryptocurrency scheme and everyone takes care of the conversion into their local real money.
blockchain
The use of blockchain is inherent in cryptocurrencies but is also typically used in dStorage to confirm and verify that data I/O has occurred and that storage capacity is present and available in the dStorage network.
A blockchain is a decentralized register of transactions on a peer-to-peer network and is only needed because peers in the network cannot be natively trusted to be present and functioning as they should. In an on-premises multi-node storage system, nodes can be physically seen and system controllers know of their presence and activity. Indeed, if a node fails, systems can detect it, for example via a missing heartbeat signal, and fail over to a functioning node.
This is not the case in a general peer-to-peer network, where there are no controllers. A node in Guatemala can go down without any other peer system noticing. Blockchain technology is used to verify that mining nodes are active and working properly. For example, in the Filecoin storage network, blockchain verification relies on operators providing proof of replication and proof of space-time (PoSt).
Storj only uses blockchain for its internal cryptocurrency transactions, as Gleeson said: “We use blockchain only for payment. Our token is an ERC-20 compatible token built on the Ethereum blockchain. And that is the extent of blockchain technology in our product.
Technology versus philosophy
Several dStorage vendors believe that the World Wide Web has become too centralized in the hands of large corporations, such as Amazon, Google, and Microsoft. They believe in the wisdom of crowds, the computing ability of vendors and users to interact with and exploit an Internet that is not dominated by large corporations. For them, blockchain is the technology – the golden door – opening the internet to freedom, with self-policing users and operators working in a cryptocurrency environment protected by the technological magic that is blockchain.
But companies want to store data securely and cost-effectively. They have no philosophical desire to overthrow Amazon because Amazon’s existence is somehow just plain wrong. Indeed, most would like be Amazon. For them, storing data should be fast – I/O performance is important, and dStorage is usually only fast enough to store archival data. It’s slow, in other words.
For example, in the Filecoin network, it may take five to ten minutes for a 1 MiB (1.1 MB) file from the beginning (transaction acceptance) to the end of the download process (recording the transaction block chain, i.e. appearing on the chain) .
Storage Features
Gleeson said Storj is different and doesn’t use blockchain for storage I/O transactions — agreements in which operators make storage capacity available to customers, accept and store and then retrieve data from clients.
He said: “Blockchain is computationally intensive, it tends to have very high latency. And if you have a storage company and you have an hour and a half to save a file and retrieve a file in a way synchronous isn’t something a lot of people can use well but if you have sub-second latency and high throughput then you have a product that caters for a wide range of use cases use”, and not just for archiving.
Storj wondered, Gleeson said, “Could you actually use some of the primitives of decentralized systems but without a distributed ledger, without a real blockchain, without a power-intensive proof-of-work component? Simply tap into what is ultimately a system of thousands and millions of hard drives all over the world that are 10 or 20 percent full, and pool that underutilized capacity so you can reap some of the benefits of a proven distributed distribution. systems like Gluster and Ceph? »
You would need to build an opt-in incentive system with zero-trust architecture and strong encryption layers, and have easy-to-use, yet powerful access management capabilities to deliver such a cloud storage product.
And that’s what Gleeson says Storj did. “You just took a fundamentally different approach, which is the Airbnb approach. Right? You can and try to build hotels or, or you can pool excess room capacity from people all over the world.
He said, “We provide enterprise-grade service. We offer that we are the only decentralized project offering an SLA: 11 nine durability, 99.97% uptime. »
Cost
Storage cost is $0.004/GB/month and Bandwidth (egress) costs $0.007/GB. Multi-region capability is included at no cost. It’s far cheaper than AWS, and also Wasabi and Backblaze.
Gleeson said, “Because we’re not building buildings and stuffing them with servers and hard drives, we’re able to really capitalize on the unit economy here and deliver a product that represents 1 /5 to 1/40th of Amazon’s price depending on your use case.
Storj “taps into any data center, any computer anywhere that has excess hard drive capacity. And when you can share that with the network, we’re able to aggregate all that capacity into a single logical object store and present it to applications to store data. It allows individuals to monetize their idle capacity. …And that gives us the advantage in terms of safety, price and performance.
The client applications communicate via an uplink with so-called satellite nodes which, in turn, are linked to the nodes providing the storage capacity. Storage node operators are paid for storage and egress. We could think of a node as a virtual storage drive and a satellite as the rough equivalent of a file (metadata) controller, which knows which drives (nodes) store which data fragments.
Incoming data files or objects are split into more than 80 sections (fragments), encrypted, erasure coded, and distributed across a subset of available nodes. 29 fragments can be used to reconstruct lost data.
Performance and testing
Storj delivers better performance than S3 for many workloads, measured in milliseconds, and claims it’s a class of content delivery network. It writes data and presents data to clients (read I/O) using parallel fetches to/from shard storage nodes and claims that a typical laptop can achieve transfer speeds of 1 Gbit/ s with a compatible Internet connection, while more powerful servers can exceed 5 Gbit. /sec download (reads) and more than 5Gbit/sec download (writes). Extremely powerful servers on strong networks can achieve transfer speeds of large data sets in excess of 10 Gbps.
An academic from the University of Edinburgh, Professor Antonin Portelli, wrote a Storj performance article when storing HPC data. A 128GB simulated HPC file filled with random data was uploaded to Storj’s decentralized cloud storage (DCS) from the University of Edinburgh‘s DiRAC Tursa supercomputer. The server used was powered by a dual-socket AMD EPYC 7H12 processor with 128 cores and 1TB of DRAM.
The files were then downloaded from a number of supercomputing centers in the United States and the download speed was measured. All upload and download sites had several GB/s internet access. Here are the results:
Download rates were between 400 and 600 MB/s with upload rates between 280 and 320 MB/s. Manual file splitting achieved a download rate of 700 MB/s (purple line) and a download rate of download > 600MB/s. Portelli comments: “It’s safe to say that these rates are quite impressive overall, especially since thanks to the decentralized nature of the network, there is no need to search for a network path optimal between source and destination.
This is a higher level of performance than Filecoin or other blockchain-centric dStorage networks can achieve.
Storj claims its performance is fast enough to be used for video storage and streaming, cloud-native apps, software and large file distribution, and as a backup target. Gleeson said performance should improve further in 2023 due to better geographic placement and search algorithms, and Reed Solomon (erasure coding) adjustments.
Comments are closed.