64 Open Source Enterprise Storage Solutions
As data storage needs continue to grow and many organizations move toward software-defined infrastructure, more enterprises are using open source software to meet some of their storage needs. Projects like Hadoop, Ceph, Gluster and others have become very common at large enterprises.
Home users and small businesses can also benefit from open source storage software. These applications can make it possible to set up your own NAS or SAN device using industry-standard hardware without paying the high prices vendors charge for dedicated storage appliances. Open source software also offers users the option to set up a cloud storage solution where they have control over security and privacy, and it can also offer affordable options for backup and recovery.
The list below features 64 open source storage projects that are among the best options available for enterprises, SMBs and individual users. Please note that this is not a ranking. Entries are organized into categories and then alphabetized within the categories.
If you know of additional open source storage applications that you believe should be on our list, feel free to note them in the Comments section below.
NAS/SAN Software
1. Ceph
Red Hat’s Ceph offers unified object and block storage capabilities. It’s a distributed storage solution that boasts excellent performance, scalability and reliability. Well-known users include Cisco, DreamHost, CERN, Bloomberg, and Deutsche Telekom. Operating System: Linux
2. CryptoNAS
This project aims to simplify the process of setting up an encrypted fileserver. It comes in a live-CD package or a server package that adds a web front-end. Operating System: Linux
3. ESOS
Short for Enterprise Storage OS, ESOS is a Linux distribution for setting up a storage array on your own hardware. Commercial support is available. Operating System: Linux
4. FreeNAS
FreeNAS claims to be the “the World’s #1 storage OS with over 10+ million downloads.” It counts the United Nations, the University of Florida, the Salvation Army, Reuters, Michigan State University, the Department of Homeland Security and many other organizations among its users. It can be installed on nearly any hardware to turn it into a network attached storage (NAS) device. Paid, supported enterprise solutions based on the same technology are available under the TrueNAS brand name. Operating System: FreeBSD
5. NAS4Free
Another option for do-it-yourself NAS, NAS4Free claims to be “the simplest and fastest way to create a centralized and easily-accessible server for all kinds of data.” Key features include the ZFS file system, software RAID (levels 0, 1 or 5) and disk encryption. Operating System: FreeBSD
6. Openfiler
As a unified storage solution, Openfiler aims to combine the best features of NAS storage with the best features on SAN storage. Key features include high availability/failover, block replication and Web-based management. Its users include Motorola, Pratt & Whitney, Bill Me Later and the London Metropolitan Police. A paid commercial edition is available. Operating System: Linux
Based on Debian Linux, OpenMediaVault describes itself as a “next-generation network attached storage (NAS) solution.” It was designed to provide NAS for home users or small offices. It offers a Web-based administration console and includes software RAID capabilities. Operating System: Linux
The Turnkey Linux project offers images for setting up Linux-based servers for a variety of purposes, including an image for creating a simple NAS device. It includes support for SMB, SFTP, NFS, WebDAV and rsync file transfer protocols. Operating System: Linux
Filesystems
9. Btrfs
A joint project supported by Facebook, Fujitsu, Intel, the Linux Foundation, Netgear, Novell, Oracle, Red Hat and others, Btrfs is a copy on write (CoW) filesystem for Linux. It focuses on “fault tolerance, repair and easy administration.” Operating System: Linux
10. Ext4
Included in most popular Linux distributions, Ext4 supports file systems up to 1 EB in size with up to 16 TB per file. Other key features include extents, multiblock allocation, delayed allocation, Fast fsck, journal checksumming, “no journaling” mode, online defragmentation and more. Operating System: Linux
11. GlusterFS
A Red Hat project, GlusterFS is a highly scalable file system built for applications like media streaming and big data analytics. Professional support is available through third-party vendors. It has a large and active user community, and the website includes links to many Gluster-related blogs. Operating System: Linux
12. Lustre
Built to handle the needs of high-performance computing (HPC) environments, Lustre is a scalable parallel file system. It was first developed at Carnegie Mellon University, and its first users included various U.S. Department of Energy National Laboratories. The latest version, released in April, supports Data on MDT, file-level redundancy, lock ahead and more. Operating System: Linux
13. ZFS
Also incorporated into many Linux distributions, including Arch, Debian, Fedora, Ubuntu and others, ZFS is another highly scalable file system. It includes compression, protection against data corruption, snapshots, RAID support and more. Operating System: Linux, macOS, FreeBSD
RAID
14. DRBD
DRBD is an open source solution for building high-availability storage clusters based on RAID-1. Commercial products, including software-defined storage, disaster recovery and high availability solutions based on the same technology, are available through project sponsor Linbit. Operating System: Linux
15. Mdadm
Built into the Linux kernel, mdadm makes it easy to create, manage and monitor storage arrays. It can also move spares between RAID arrays. More information is also available on the Linux RAID wiki. Operating System: Linux
16. Raider
Raider is a bash shell script that allows users convert any Linux disk into a RAID array with software RAID 1, 4, 5, 6 or 10. It works with many of the most popular Linux distributions, including Ubuntu, Debian, OpenSUSE, Fedora, Linux Mint and others. Operating System: Linux
17. RaidEye
RaidEye isn’t so much a tool for creating RAID devices as a utility for monitoring RAID volumes. It works with the RAID capabilities built into macOS and notifies users of problems via a sound alarm, popup dialog and email. Operating System: macOS
18. SnapRAID
SnapRAID arrays can recover from up to six disk failures. The tool was built for home media servers or other environments with a lot of big files that rarely change. Key features include data hashing, the ability to recover deleted files and no lock-in. Operating System: Windows, Linux, macOS
Backup and Synchronization
19. AMANDA
The Advanced Maryland Automatic Network Disk Archiver, or AMANDA, is a popular network backup solution that can save data from Linux, Unix or Windows systems to hard drives, tape or optical media. It was last updated in December 2017. Zmanda, which sponsors the project, offers commercial products based on the same technology. Operating System: Windows, Linux, macOS.
20. Areca Backup
Designed for personal use, Area is a simple but versatile backup solution. Key features include delta backup, compression, encryption, filters, as-of-date recovery and more. Operating System: Windows, Linux
21. BackupPC
Robust enough for enterprise use, BackupPC backs up data from Linux and Windows systems to disk. Noteworthy features include a unique pooling scheme, optional compression, a web interface and support for mobile devices. It claims to be highly configurable and easy to install and maintain. Operating System: Windows, Linux
22. Bacula
Another option for enterprises, Bacula is a network backup solution that aims to be easy to use and very efficient. It claims to be the most popular open source backup program. Commercial support and services for the solution are available through Bacula Systems. Operating System: Windows, Linux, macOS
23. Bareos
Forked from Bacula, Bareos is a popular open source backup option that is under very active development with the latest version released in February 2018. The Bareos.com website offers paid support and services for the tool. Operating System: Windows, Linux, macOS
24. Box Backup
This “completely automatic” backup solution creates backups continuously and can also create snapshots when desired. It includes encryption and optional RAID capabilities, and old file versions and deleted files remain available. Operating System: Windows, Linux
25. BURP
Short for “BackUp And Restore Program,” BURP is a network backup solution. It offers a choice of two different protocols: one based on librsync (see below) and one that uses variable length chunking for inline deduplication. It is designed to be easier to configure than some other open source solutions, and it can do delta backups. Operating System: Windows, Linux
26. Clonezilla
Designed to replace Acronis True Image or Norton Ghost, Clonezilla is useful for both system deployment and backup and recovery. It comes in two flavors: live for standalone systems and SE for network backup or cloning multiple systems at once. The SE version can clone 40 or more systems at once. Operating System: Linux
Powerful but lightweight, this backup tool takes up only 220KB of space on your hard drive. It supports multiple languages, has an intuitive interface and includes a scheduler. It is also helpful for syncing photos, music and other files across devices. Operating System: Windows
28. DAR
Disk Archive, a.k.a. DAR, is an older command-line tool for backup, but it is still being updated regularly with the most recent version released in April 2018. For those who prefer a GUI, one is available through the DarGUI project. Operating System: Windows, Linux, macOS
29. DirSync Pro
This “small but powerful,” utility offers incremental backup, filtering and scheduling capabilities. It also boasts an intuitive interface, and it offers the ability to analyze two sets of files or folders and detect the changes between them. It also includes a helpful real-time synchronization option. Operating System: Windows
30. Duplicati
Duplicati works with cloud storage services like AWS S3, Microsoft OneDrive, Google Drive and Box to create backups with AES-256 encryption. It does a full backup on first use and incremental backups after that; it also offers data deduplication capabilities. Operating System: Windows, Linux, macOS
31. FOG
FOG offers cross-platform cloning and imaging capabilities plus remote management for networks of any size. It offers support through forums and a wiki. Operating System: Linux, Windows, macOS.
32. FreeFileSync
A tool for standalone systems, FreeFileSync aims to save users time when setting up and running backups. It is cross-platform and includes 64-bit support. Tutorials and a manual are available on the website. Operating System: Linux, Windows, macOS
33. FullSync
Although it was designed to help web developers push updates to their sites, FullSync can also be used by anyone to create backups. Key features include multiple modes, flexible rules, buffered filesystems, support for multiple file transfer protocols and more. Note that development on this project has slowed somewhat as it hasn’t been updated since April 2016. Operating System: Linux, Windows, macOS
34. Grsync
Grsync takes the older rsync synchronization tool and adds an easy-to-use GUI. Noteworthy features include unlimited sessions, highlighted errors, batch capabilities, simulations, support for multiple languages and more. Operating System: Linux, Windows, macOS
35. Mondo Rescue
For Linux and FreeBSD only, Mondo Rescue is a disaster recovery solution that supports tape, disk, network or optical media backups. According to its website, its users include “Lockheed-Martin, Nortel Networks, Siemens, HP, IBM, and dozens of smaller companies.” The most recent update was released in April 2016. Operating System: Linux, Free BSD
36. Partimage
This tool saves partitions of drives as image files, making it useful for backup or installing the same image on multiple systems. It can run across networks or on a standalone PC. It can also be used to create a SystemRescueCD. Operating System: Linux
37. Redo
Redo boasts that its bare-metal restore capabilities can get a crashed system back up and running in as little as 10 minutes. It’s very easy to use and can also recover deleted images and files. Operating System: Windows, Linux
38. Rsync
Rsync is a Unix-based file-transfer utility with synchronization capabilities that make it suitable for creating backups or mirroring. It’s a useful tool but is best used by advanced users. The most recent version was released in January 2018. Operating System: Linux, Windows, macOS
39. Synkron
While this app is focused primarily on synchronization, it can be used for creating backups as well. Key features include analysis capabilities, blacklisting, restores and cross-platform support. Documentation is available in both German and English. Operating System: Windows, Linux, macOS
40. Unison
Like Synkron, Unison is a file synchronization tool. It can copy files between any two systems connected to the internet, and it has features in common with source code management tools as well as with backup utilities. Its advantage over some other synchronization tools is that it can combine two sets of files where both sets have undergone changes. Operating System: Windows, Unix
41. UrBackup
This client-server backup solution does both image and file backups. It promises “both data safety and a fast restoration time.” It makes backups while the system is in use without interrupting normal operation. Operating System: Windows, Linux
42. Weex
The Weex developers intended it primarily as a tool for pushing content to websites, but it can also be used to synchronize or backup files. It supports FTP file transfer and uses caching to speed data transfer. Operating System: Windows, Linux
Online/Cloud Data Storage
43. CloudStack
This Apache Foundation project is open source cloud computing platform that includes cloud storage capabilities. Noteworthy features include compute orchestration, network as a service capabilities, user and account management, resource accounting and support for multiple hypervisors. Operating System: Windows, Linux
44. CloudStore
This Dropbox alternative synchronizes data between a system and online storage. It promises strong encryption, password-less authentication, flexible synchronization, fast setup and auto-resumes for interrupted data transfers. Operating System: Linux
45. Cozy
Cozy is both an open source project for storing personal data online and a free service for managing and securing sensitive data. Note that the free hosting is for up to 5GB of data; additional storage will require a fee. Operating System: Linux
46. FTPbox
Want to set up your own cloud storage server? FTPbox makes it easy to be your own cloud provider, with all files transferred via FTP. Operating System: Windows
47. OpenStack
Probably the best-known open source cloud computing platform, OpenStack offers a complete operating system for controlling compute, networking and, of course, storage in the cloud. It incorporates three sub-projects related to storage: Cinder, Swift and Manila. Operating System: Windows
48. Perkeep
Formerly known as Camlistore, Perkeep describes itself as “a set of open source formats, protocols, and software for modeling, storing, searching, sharing and synchronizing data in the post-PC era.” It’s still under very active development and will require some technical knowhow to use. Operating System: Linux
49. Pydio
Downloaded more than a million times, Pydio counts the University of Cambridge, Seagate, Guitar Center, Washington State University and Nikon among its users. It offers cloud-based file management and sharing. A paid enterprise distribution is available. Operating System: Windows, Linux (Android and iOS clients available)
50. Rockstor
Rockstor makes it possible to create your own NAS or cloud storage solution based on Linux and BTRFS. It comes in both personal and SMB versions. Operating System: Linux
51. SeaFile
SeaFile describes itself as “an enterprise file hosting platform with high reliability and performance.” You can download the code for free or use the paid pro edition that includes support. Operating System: Windows, Linux, macOS, Android, iOS
52. SparkleShare
SparkleShare creates a special folder on your system that is automatically synchronized with a host folder stored on your server or in the cloud. It includes encryption capabilities, and it is a good option for collaborating on documents that change frequently. Operating System: Windows, Linux, macOS
53. StackSync
Create your own scalable personal cloud with StackSync. It encrypts all data on the client side and works with cloud storage services or your own server. Operating System: Windows, Linux
54. Syncthing
Like many of the other projects in this category, Syncthing offers an alternative to Dropbox. It keeps data completely private with encryption and authentication requirements. Operating System: Windows, Linux, macOS
Storage Management
55. Libvirt Storage Management
Libvirt is an API for creating storage pools and volumes on a host system. It supports a wide variety of storage pool types including directory, filesystem, network filesystem, logical volume, disk, iSCSI, SCSI, Gluster, ZFS and more. Operating System: Linux
56. openAttic
This tool offers management and monitoring capabilities for the Ceph distributed storage platform. It offers a dashboard, as well as tools for managing pools, block devices, iSCSI, NFS, Ceph Object Gateway and Ceph nodes. Operating System: Linux
Distributed Storage/Big Data Tools
57. Alluxio
Alluxio (Formerly known as Tachyon) describes itself as “open-source memory-speed virtual distributed storage.” It works with tools like Spark, Hadoop, Flink, Zeppelin and Presto to speed performance on big data queries. Operating System: Linux, macOS
58. Hadoop
Nearly synonymous with big data, Hadoop is a widely used open source distributed storage platform for processing data. It is an Apache Foundation project, and the organization also oversees dozens of related projects. Operating System: Windows, Linux, macOS
59. HPCC
This Hadoop alternative also offers distributed storage and massive scalability. Paid enterprise services are available. Operating System: Linux
60. Sheepdog
The Sheepdog website describes this project as “a distributed object storage system for volume and container services and manages the disks and nodes intelligently.” It supports snapshotting, cloning and thin provisioning, and it is compatible with OpenStack Swift and Amazon S3. Operating System: Linux
Compression
61. 7-zip
This archiving utility can compress files 30 to 70 percent smaller than WinZip can, and 2 to 10 percent smaller than most other zip programs. It includes encryption capabilities, self-extracting files, a powerful file manager and support for a wide variety of compression formats. Operating System: Windows, Linux, macOS
62. ArcThemALL
ArcThemALL can compress to UPX, ZIP or 7Z formats and extracts 33 other types of archive files. It can also create self-extracting archives and includes AES-256 encryption capabilities. Operating System: Windows
63. Keka
For macOS, Keka is a port of 7-zip. It also compresses to Zip, Tar, Gzip, Bzip2, DMG and ISO formats. Operating System: macOS
64. PeaZip
Amazingly versatile, PeaZip can open more than 180 different types of archived files, and it writes to seven, including ZIP and 7Z. It also includes security features like strong encryption, encrypted password manager, two-factor authentication, secure delete and file hashing. Operating System: Windows, Linux, macOS.
Original article appeared at: https://www.enterprisestorageforum.com/storage-technology/open-source-storage-64-applications.html
0 Comments