Network-Guides
| main | DNS | DHCP | Storage | Repo |
Storage
| Knowledge Base | ZFS & Proxmox | ISCSI & Proxmox & SCST |
Knowledge Base
this section is highly motivated by the reddit thread Can a second computer join a ZFS pool?
Replication in Computing
Replication in computing involves sharing information to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.
- Data Replication: Where the same data is stored on multiple storage devices.
- Computation Replication: Where the same computing task is executed many times. Computational tasks may be:
Race Conditions and Locking
- this is not really network related but we are facing similar issues in networking!
Race conditions occur when two or more threads access shared data concurrently, and the outcome of the execution depends on the particular order in which the access takes place.
To prevent race conditions, synchronization mechanisms such as locks (also known as mutexes) are used.
When a thread wants to execute code within a critical section, it must first acquire the lock associated with that section. If the lock is already held by another thread, the requesting thread will block until the lock becomes available.
| Mutex | Semaphore | Read/Write Lock |
|---|---|---|
| A mutual exclusion object that prevents simultaneous access to a resource | A variable or abstract data type used to control access to a common resource by multiple processes in a concurrent system such as a multitasking operating system. | Allows concurrent read-only access but requires exclusive access for write operations. |
can two computers boot from the same drive ?
- well that depends, theoretically yes, but it doesnt come right of the box
Yes, you can boot multiple systems from NFS (typically a read-only mount). No, there won’t generally be conflicts, as long as that image doesn’t have any host-unique configuration. For example, the root that you use should not have any configuration file with a fixed IP address in it, and each host should get address information from DHCP rather than from configuration files in the shared root.
- posted by gordonmessmer in this reddit-thread
- the other option is to make us of replication
- this either requires some sort of locking algorithm, daemon or manager
- but we will learn more about this later
- see distributed storage and cloud file system
SAN (Storage Area Network) vs NAS (Network Attached Storage)
Both SAN and NAS are methods of storing data in a network environment, but they serve different purposes and are used in different scenarios.
| SAN | NAS | |
|---|---|---|
| Definition | A Storage Area Network (SAN) is a dedicated network that provides block-level data storage to servers so they appear as locally attached devices. SANs are primarily used to enhance storage devices, such as disk arrays and tape libraries, accessible to servers |
A Network Attached Storage (NAS) is a dedicated file-level data storage device that operates on a computer network. It allows multiple users and heterogeneous client devices to retrieve data from centralized disk capacity. |
| Use Cases | SANs are ideal for applications requiring high-speed, low-latency access to storage, such as databases, email servers, and virtualization environments. | NAS is suitable for file sharing, collaboration, and backup purposes, where file-level access is preferred over block-level access. |
| Architecture | SANs typically use Fibre Channel (FC) or Internet Small Computer System Interface (iSCSI) protocols for connectivity | NAS devices connect to the network using TCP/IP networking protocols and offer file-level access to the networked computers |
| Comparison | ||
| Access Level | SAN offers block-level access, making it faster for certain types of applications, while NAS provides file-level access, which is more suited for general-purpose file sharing. | |
| Protocol | SAN commonly uses FC or iSCSI, whereas NAS uses NFS (for Unix/Linux) or SMB/CIFS (for Windows). | |
| Purpose | SAN is focused on high-performance storage for critical applications, while NAS is designed for efficient file sharing and collaboration. | |
| Cost | SAN solutions can be more expensive due to the need for specialized hardware and cabling, whereas NAS devices are generally less costly and can leverage existing network infrastructure. | |
NFS
NFS (Network File System) is a distributed file system protocol that allows a system to share directories and files with others over a network. It enables users to access files on remote systems as if they were local.
|
||||||||||||||
iSCSI
iSCSI (Internet Small Computer System Interface) is a protocol that allows for the transport of block-level data over IP networks. It enables the creation of storage area networks (SANs) by allowing remote servers to access storage as if it were locally attached.
|
||||||||||||||
ZFS
ZFS (Zettabyte File System) is a combined file system and logical volume manager designed by Sun Microsystems. It stands out for its advanced features such as snapshotting, replication, automatic repair, and data compression.
|
||||||||||
High Availability (HA) in Networking
High Availability (HA) refers to systems designed to continue operating without interruption during the failure of one or more components. It ensures that critical services remain available and operational under various conditions, enhancing system reliability and minimizing downtime.
Components of HA Systems
- Redundant Hardware: Duplicate components to ensure that if one fails, another can take over seamlessly.
- Load Balancing: Distributes traffic across multiple servers to prevent overload and increase efficiency.
- Clustering: Groups servers together so they can share workloads and resources, improving performance and resilience.
- Failover Mechanisms: Automatically switch operations to a standby component when a primary component fails.
Types of HA Solutions
| Active-Passive HA | Active-Active HA | N+M Redundancy |
|---|---|---|
| In an active-passive setup, one server actively handles requests while the other remains idle until it takes over in case of a failure. | Active-active HA involves distributing workload between multiple active servers, increasing capacity and reducing single points of failure. | N+M redundancy involves having N active servers and M standby servers ready to take over in case of failures, providing a higher level of availability. |
Suse-Cloud Storage Protection: https://documentation.suse.com/de-de/sle-ha/15-SP2/html/SLE-HA-all/cha-ha-storage-protect.html
Distributed Storage
Distributed storage systems spread data across multiple nodes in a cluster to enhance scalability, reliability, and fault tolerance. This document focuses on Distributed Block Storage, where data is stored in blocks and managed independently.
Types of Distributed Storage
Object Storage
Object storage is designed for storing unstructured data, such as images, videos, and backups. Data is organized into objects, each with metadata, allowing for efficient retrieval and management.
File Storage
File storage systems manage data as files within a hierarchical namespace. They are optimized for file-based access patterns but can also support block-level access through protocols like NFS or SMB.
Distributed Block Storage
Block storage divides data into fixed-size blocks, which are managed independently.
Each block can be stored on a separate physical drive, allowing for flexible scaling and high performance.
- Examples
| Name | Description | |
| DRBD | Mirrors blockstorage so multiple nodes can use it safely (slow). | |
| Ceph | A highly scalable, open-source software-defined storage platform. Supports object, block, and file storage modes. |
|
| GlusterFS | A scalable network filesystem that allows for the creation of large, distributed storage solutions. | |
| OpenEBS | Provides container-native block storage solutions for Kubernetes environments. | |
| Amazon S3 | A widely-used object storage service that provides scalable storage for data objects. | |
| Google Cloud Storage | Similar to Amazon S3, offering durable, secure, and scalable object storage. | |
| Hadoop HDFS | Designed for storing very large files across multiple machines, providing high aggregate bandwidth through data parallelism. | |
Distributed Storage comparison
</tr>
Type UseCase Example Object Storing unstructured data, backups, and media content Amazon S3 File Managing structured data in a hierarchical manner Network Attached Storage (NAS) Block Providing raw block-level storage for databases, virtual machines, and containers SAN (Storage Area Network) Distributed Block Storage* Scalable, high-performance storage for cloud-native applications and big data analytics Ceph, GlusterFS, OpenEBS
Clustered File System
A clustered file system is a type of file system that is designed to operate across a cluster of computers, allowing them to share storage resources efficiently. These systems are built to handle the challenges of distributed computing environments, offering features such as scalability, high availability, and fault tolerance.
Key Features
- Scalability: Easily expand storage capacity by adding more nodes to the cluster.
- High Availability: Ensures that data remains accessible even if individual nodes fail.
- Fault Tolerance: Protects against data loss by replicating data across multiple nodes.
- Consistency: Maintains data consistency across the cluster, preventing conflicts and inconsistencies.
- Performance: Optimizes data access and transfer speeds across the network.
How It Works
Clustered file systems operate by presenting a unified view of the storage resources available across the cluster. When a file or directory is accessed, the system determines the optimal location for the data based on factors such as load balancing and data replication strategies. This process enables efficient data sharing and collaboration among users and applications running on different nodes.
Examples
- GFS (Global File System): Developed by Red Hat, GFS is designed for Linux clusters and supports shared access to block devices across multiple nodes.
- OCFS2 (Oracle Cluster File System): A high-performance file system developed by Oracle, OCFS2 is designed for real-time applications and supports clustering across multiple platforms.
- Ceph FS: Part of the Ceph storage platform, Ceph FS is a POSIX-compliant file system that provides object, block, and file storage in one unified system.
- GFS (Global File System): Developed by Google, GFS is designed for Linux clusters, supporting shared access to block devices across multiple nodes.
Understanding Distributed Storage and Clustered File Systems
Distributed storage and clustered file systems are two concepts that, while related, serve different purposes in the realm of data management and storage. This section aims to clarify the distinctions and provide examples relevant to both.
Distributed Storage
Distributed storage systems distribute data across multiple nodes in a network to achieve scalability, reliability, and fault tolerance. Unlike traditional storage architectures, distributed storage does not rely on a central server; instead, data is replicated across several nodes, ensuring that the system remains operational even if some nodes fail.
Characteristics
- Scalability: Easy expansion by adding more nodes.
- Reliability: Data redundancy reduces the risk of data loss.
- Availability: Continuous service despite node failures.
- Performance: High throughput and low latency due to parallel processing.
- Flexibility: Supports both file and block-level access, catering to a wide range of applications.
Clustered File Systems
Clustered file systems allow multiple servers to share a common file system, enabling them to access the same set of files as if they were a single system. This approach enhances data sharing and collaboration among servers in a cluster.
Characteristics
- Shared Access: Multiple servers can access the same files simultaneously.
- Location Independence: Files appear to be located in a single place, abstracting away the underlying distribution.
- Redundancy: Can provide redundancy through mirroring or striping techniques.
- Unified Namespace: Simplifies data management by providing a single namespace for all storage resources.
- Data Replication: Enhances data protection and availability through automatic replication across nodes.
- Load Balancing: Improves performance by automatically distributing read and write operations across the cluster.
Bridging the Gap
While distributed storage focuses on the architecture of storing data across multiple nodes, clustered file systems concentrate on how multiple servers interact with a shared file system. Both concepts aim to enhance scalability, reliability, and performance but address different aspects of data management.
- Distributed Storage is about the physical distribution of data across a network to ensure scalability and reliability.
- Clustered File Systems are about logical file system organization across multiple servers, facilitating shared access and collaboration.
Understanding these distinctions helps in selecting the appropriate technology for specific use cases, whether it’s the need for scalable storage or efficient data sharing across a cluster of servers.
Gluster FS
GlusterFS is designed with modularity in mind and supports multiple operational modes:
|
||||||||||||||||||
Ceph
Ceph is a sophisticated storage manager designed to handle data across a cluster of machines, offering features such as data redundancy and distributed storage management.
Storage Manager
- Ceph serves as a storage manager, facilitating the storage of data across various storage resources like HDDs and SSDs.
- It supports a wide range of storage media, including HDDs, SSDs, magnetic tapes, floppy disks, punched tapes, Hollerith-style punch cards, and magnetic drum memories.
Clustered Storage Manager
- Ceph is a clustered storage manager, meaning it operates across multiple interconnected machines, forming a cohesive system.
- This architecture enables enhanced scalability and resilience compared to single-machine setups.
Distributed Storage Manager
- Beyond being a clustered manager, Ceph is also a distributed storage manager. This implies that data and the supporting infrastructure are dispersed across numerous machines, avoiding centralization.
- Unlike iSCSI, which presents a single logical disk over the network in a 1:1 ratio, Ceph distributes data across multiple nodes, enhancing fault tolerance and scalability.
Data Redundancy
- Ceph incorporates data redundancy by maintaining copies of data in separate locations. This strategy protects against data loss due to hardware failure or other disruptions.
- Ensuring data redundancy is crucial for maintaining data integrity and availability in a distributed storage environment.
DRBD Overview
DRBD (Distributed Replicated Block Device) is a distributed storage system that provides block devices over a network. It is designed to allow multiple hosts to access the same block device simultaneously, with data mirrored across the participating nodes.
DRBD devices are usually readable and writable from only one node at a time, promoting a Primary/Secondary model. This model is beneficial for database workloads and virtual machine root disks. Ceph, on the other hand, supports concurrent access to the same file system from many hosts, making it suitable for disk-image stores and large-file-sized data.
|
||||||||||||||||
Split Brain Scenario
The Split Brain Scenario refers to a situation in distributed systems where two or more nodes believe they are the sole coordinator or master of the system, leading to potential data inconsistencies and conflicts. This can occur in high-availability (HA) clusters, particularly when using shared storage technologies like SAN, iSCSI, or FCoE, where network partitions may cause nodes to lose communication with each other.
Key Characteristics
- Isolation of Nodes: In a Split Brain Scenario, nodes become isolated from each other, often due to network failures or misconfigurations, leading to a state where each node believes it is the only active node.
- Data Inconsistency: Without proper coordination or fencing mechanisms, these isolated nodes may continue to accept write operations independently, resulting in duplicate entries or lost updates, compromising data integrity.
- Risk of Data Loss or Corruption: Over time, this can lead to significant data loss or corruption, as the system’s state diverges between the nodes.
Mitigation Strategies
- Fencing Mechanisms: Implementing fencing mechanisms, such as STONITH (Shoot The Other Node In The Head) or SBD (STONITH Block Device), to physically isolate malfunctioning nodes from the shared storage, preventing them from accepting write operations.
- Quorum Algorithms: Utilizing quorum algorithms to determine the majority state of the cluster, ensuring that only the majority of nodes agree on the current state, thereby preventing conflicting operations.
- Watchdog Timers: Employing watchdog timers on each node to detect and remove nodes that fail to respond, ensuring that only responsive nodes participate in the cluster operation.
Importance in High-Availability Systems
In high-availability systems, mitigating the Split Brain Scenario is crucial for maintaining data integrity and system stability. Proper configuration of fencing mechanisms, along with vigilant monitoring and timely intervention, are essential to prevent this scenario from occurring and to quickly recover from it if it does.