Multi-HD setup for a single node

  • Guest
  • Feb 2 2019
  • Will not implement
  • Attach files
  • david kasta commented
    2 Jul 12:29am
    Running virtual machines uses a lot of resources and energy, multiple nodes in the same operating system would be better.
  • Mike Kirk commented
    29 Mar 02:38pm

    It's common for people to add storage to their system, and for that space to appear as a new drive letter or mount point. Because Storj is casting the widest of nets to attact SNOs, most of them aren't going to be people who configure RAID or other OS-level options to merge multiple drives into one logical block of space: so the Storj client should be able to handle more than one drive/mount/directory.

    Brandon's comment on Sept 9 2019 doesn't make sense. If I had one Storj client managing 4 x 2TB, and one drive died, I still only lost 25% of the files, even if running only one Storj node on top of them. If a node can read from 3 locations but falls-on-it's-face when getting read errors from a 4th... that's a software issue and would be logged as a bug. Storj beautifully handles recovery from entire nodes disappearing from the Internet: I have confidence your node devs can prevent simple I/O errors from taking down the entire app (especially if 3 other managed storage locations are still happy and healthy).

    I can understand this not being a priority. But spinning things to sound like a node couldn't easily and gracefully handle one-of-many storage locations going down is misleading. Without this feature at some point, Storj will be limiting how quickly SNOs can scale to meet demand (if interest is strong: hope for the best!)

  • vla dro commented
    1 Mar 10:24am

    Brandon- its a good idea but u will spend x4 resources for it... most of them will be spent for operation system, not for node operations

  • Admin
    Brandon Iglesias commented
    9 Sep, 2019 05:54pm

    Hey everyone Brandon the Product Manager @storj here. Thanks for submitting this idea, Chris VdG has a point about the redundancy between raid and erasure coding. Our recommendation is to run one node per HD. This will actually spread your overall risk of a HD failing and losing the withholdings for that specific node ID. If you have 4 2 TB HDs then the best thing to do is run 1 node per HD and if one of those fails then at most you would only lose 25 percent of your overall held amount amongst all 4 nodes. Instead of using raid to backup one of your nodes its may be in your best interest to just allocate that extra raid space to the network and profit off of it instead.

  • Xiao Lu commented
    10 Apr, 2019 07:57pm
    It is a good idea,hope it could be achieve soon.
  • Chris VdG commented
    5 Apr, 2019 02:39pm
    Maybe if the NAT traversal implementation allows it, we could have several storage node containers on a host each serving a different disk?
  • Chris VdG commented
    5 Apr, 2019 01:28pm
    Using erasure coding (Storj) on top of RAID (node) with parity is like throwing salt in the Sea... IMHO if someone is really concerned about keeping their reputation high (e.g.: a more professional farmer) then sure, go for it. But using even more disk space for RAID parity on top of erasure coding parity does not seem like efficient use of disk space to me. I'm active in cloud tech and we do not use erasure coding on top of RAID, only erasure coding for our storage back-end. This would also allow as suggested here for less reputation loss to have one disk fail on a node instead losing a whole node.
  • Aleksei Leonov commented
    3 Apr, 2019 06:58pm
    If you lose data on any of this "RAID" it could lead the disqualification of the node and losing reputation. If you will use a normal RAID with parity, the one failed drive will not leads of losing data, reputation and escrow. So, I think it's better to have the RAID with parity outside of the node's software
  • Nicolas Hugo commented
    16 Mar, 2019 07:50pm
    A Multi-HD setup for a single node could "transfer" the complexity to setup the multiple hardisks (Raid0, Raid5, linear, logical volumes, ...) to the storage node software. It could also avoid storage node disqualification if one drive fails (in raid0 by exemple).
  • H3 commented
    8 Mar, 2019 09:13am
    Workaround: mhddfs (file system for unifying several mount points into one).
  • adam lentz commented
    24 Feb, 2019 04:30am
    Better management of multiple drives available on a single node. That's the idea to improve storage of unused drives. If storj accessed multiple drives on a single node, would make better management. If a single drive fails the data could be recovered and all other drive data would still be valid. Storj is already a form of raid with the nodes. The point is node operators may have multiple drives of different sizes and want to share this unused space, thus the definition of storj and decentralization. If a single drive in a multi drive node fails, the drive could be replaced and recovered with minimal impact on reputation instead of losing an entire node to a single disk failure. A Node on RAID0 will fail completely if one drive fails. On other RAID formats less space is available for redundancy if a drive fails. If this is too much then it's back to one node per drive. Node operators want to share as much free unused space as possible.
  • Chris Stevens commented
    18 Feb, 2019 08:33pm
    Not sure why this would be an enhancement/feature of Storj. I understand we're dealing with Storage, but Storj itself is a software layer that makes use of available hardware. This hardware *should* be controlled by the OS or even better, a hardware raid. Look into RAIDs. Software Raid on Windows, you can use Storage Spaces/Storage Pools (Windows 10, Server 2012+), or StableBit DrivePool.