Apache Ozone is a distributed, scalable and high-performance object storage that can be used with the Clouderah Data Platform (CDP) and can be scaled to billions of objects of different sizes. It is designed as native object storage that provides extremely high scale, performance, and reliability to handle multiple analytical workloads using the S3 API or the traditional Hadoop API.
Today’s platform owners, business owners, data developers, analysts and engineers create new applications on the Cloudera data platform CDP, and they have to decide where and how to store that data. Structured data (such as name, date, ID, etc.) will be stored in the regular SQL database , such as the Hive or the Impala database. There are also newer AI/ML applications that require data storage, optimized for unstructured data using developer-friendly paradigms such as Python Boto API.
Apache Ozone meets these two storage use cases in various vertical industries, including:
- manufacturing, the data they generate can provide new business opportunities in addition to improving operational efficiency, such as predictive maintenance of
- retail, using big data at all stages of the retail process—from product development, pricing, demand forecasting to inventory optimization in stores.
- Healthcare, big data is used to improve profitability, conduct genomic research, improve patient experience and save lives.
Similar use cases exist in all other verticals such as insurance, finance and telecommunications.
In this blog post, we will discuss a single Ozone cluster with Hadooph core file system (HCFS) and object storage such as Amazon S3. A unified storage architecture that stores files and objects simultaneously and provides flexible, scalable and high-performance systems. In addition, data stored in Ozone can be accessed for various use cases through different protocols, eliminating the need for data duplication, reducing risks and optimizing resource utilization.
diversity of workloads
Today's fast-growing data-intensive workloads drive analytics, machine learning, AI and intelligent systems, requiring a storage platform that is both flexible and efficient. Apache Ozone native provides endpoints compatible with Amazon S3 and Hadoop file systems, designed to work seamlessly with enterprise-level data warehouses, batch processing, machine learning, and streaming workloads. Depending on the nature of integration with storage services, Ozone supports a variety of workloads, including the following prominent storage use cases:
- Ozone as pure S3 object storage semantics
- Ozone as an alternative file system for HDFS to solve the scalability problem
- Ozone as a Hadoop compatible file system ("HCFS") with limited S3 compatibility. For example, for the critical path with "/" in it, the interoperability of the same data for multiple workloads of
- is created: Multi-protocol access
Below are the main aspects of big data workloads that require HCFS semantics.
- Apache Hive: Delete table queries, delete managed Impala tables, recursive directory deletion and directory movement operations are faster and highly consistent, and no partial results will appear in any failure situation. For more details on Ozone's performance benefits and atomicity guarantee, see our earlier Cloudera blog.
- these operations are also very effective, and do not require O(n) to RPCh calls to the namespace server, where "n" is the number of file system objects in the table. Job submitters of big data analysis tools such as Hive, Apache Impala, Apache Spark, and traditional MapReduce often rename their temporary output files to the final output location at the end of the job to be publicly visible. The performance of the job is directly affected by the speed of the rename operation completion.
Concentrate files and objects under one roof
Unified design represents files, directories and objects stored in a single system.Apache Ozone achieves this important functionality using some novel architectural choices by introducing bucket types in the metadata namespace server. This allows a single Ozone cluster to function as both Hadoop Core File System (HCFS) and object storage (such as Amazon S3) by efficiently storing files, directories, objects, and buckets. It eliminates the need to port data from object storage to the file system so that analytics applications can read it. The same data can be read as objects or files.
Bucket type
Apache Ozone Object Storage recently implemented multi-protocol-aware bucket layout functionality in HDDS-5672, available in the CDP-7.1.8 release. The idea here is to classify Ozone's Buckets for based on storage use cases.
FILE_SYSTEM_OPTIMIZED bucket ("FSO")
- has a hierarchical file system namespace view of directories and files similar to HDFS.
- provides high-performance namespace metadata operations similar to HDFS.
- provides read/write functionality using the S3 API*.
OBJECT_STORE bucket ("OBS")
- provides a flat namespace (key-value) similar to Amazon S3.
The legacy buckets
- represent existing pre-created Ozone buckets for smooth upgrades from previous Ozone versions to new Ozone versions.
Use the Ozone shell command to create the FSO/OBS/LEGACY bucket. The user can specify the bucket type in the layout parameters.
$ozone sh bucket create --layout FILE_SYSTEM_OPTIMIZED /s3v/fso-bucket$ozoneshbucketcreate--layoutOBJECT_STORE/s3v/obs-bucket$ozoneshbucketcreate--layoutLEGACY/s3v/bucketBucketLayout Feature Demo introduces Ozone shell, Ozone FS and aws cli operations.
Ozone's Namespace Namespace Overview
The following is a quick overview of how Ozone manages its metadata namespace and handles client requests from different workloads based on bucket type. In addition, the bucket type concept is architecturally designed in an extensible manner to support future multi-protocols such as NFS, CSI, etc.
Ranger policy
Ranger policy enables authorized access to Ozone resources (volumes, buckets, and keys). The Ranger policy model captures the following details:
- Resource type, hierarchy, support for recursive operations, case sensitive, support for wildcards etc.
- Permissions/actions to perform on specific resources, such as read, write, delete and list
- Allow, deny or exception permissions granted to users, groups and roles
Similar to HDFS, using FSO resources, Ranger Supports authorization of rename and recursive directory deletion operations and provides a performance-optimized solution regardless of the large number of subpaths (directories/files) contained within them.
Workload migration or replication across clusters
The Tiered File System ("FILE_SYSTEM_OPTIMIZED") feature makes it easy to migrate workloads from HDFS to Apache Ozone without significant performance changes. In addition, Apache Ozone seamlessly integrates with Apache data analysis tools such as Hive, Spark, and Impala, while preserving Ranger policies and performance characteristics.
Data interoperability: multi-protocol client access
Users can store their data into the Apache Ozone cluster and access the same data through different protocols: Ozone S3 API*, Ozone FS, Ozone shell commands, etc.
For example, users can use the Ozone S3 API* to ingest data into Apache Ozone and can access the same data using the Apache Hadoop-compatible file system interface, and vice versa.
Basically, this multi-protocol capability is attractive for systems that are primarily file system oriented (like workloads) but want to add some object storage feature support.This can improve the efficiency of the user platform through local object storage. Additionally, data stored in Ozone can be shared across a variety of use cases, eliminating the need for data replication, reducing risks and optimizing resource utilization.
summary
Apache Ozone cluster provides a unified architecture on CDP, which can effectively store files, directories and objects through multi-protocol access. With this capability, users can store their data into a single Ozone cluster and access the same data for various use cases using different protocols (Ozone S3 API*, Ozone FS), eliminating the need for data replication, reducing risks and optimizing resource utilization.
In short, combining file and object protocols into one Ozone storage system can bring the benefits of efficiency, scale, and high performance. Now users have greater flexibility in how to store data and how to design applications.
S3 API* – refers to the Amazon S3 implementation of the S3 API protocol.
Original author: Aryan Gupta, and Sailaja Polavarapu
Original link: https://blog.cloudera.com/a-flexible-and-efficient-storage-system-for-diverse-workloads/
