Introduction: As the next generation of object storage is not enough, replacing HDFS is the goal of Ozone. Ozone is a new object storage system in the current Apache Hadoop ecosystem. Ozone has a deep relationship with HDFS. In terms of design, many places have also referenced HDFS and made many improvements to the shortcomings of HDFS. Many companies value not only Ozone's object storage capabilities, but also Ozone claims to be the next generation of HDFS. We seized this point and made a more thorough HDFS on Ozone architecture design and implementation than the open source community Ozone Filesystem solution, achieving phased results. This article will introduce you how to make Ozone the next generation distributed storage system for HDFS. The main contents include:
- Ozone Introduction to
- Namenode on HDDS
- Tencent
- Contribution of Tencent
- Ozone's future
1Ozone Introduction to
First, let's take a brief look at Ozone together; then, let's focus on a cross-eracial thing we are doing NameNode on HDDS.. HDFS organizational structure
Before introducing Apache Ozone, let’s first review HDFS and see what happened in HDFS. In the big data ecosystem, why is the distributed storage system that has not changed for so many years replaced? What exactly is wrong with it? Let’s first understand the composition architecture of HDFS, you can see the picture below.
HDFS adopts the Master/Slave architecture, and the master node is the NameNode node, which uniformly manages the metadata in the file system namespace, and provides a highly centralized Master service that combines file system metadata services, data block management services, node management, redundant storage management, heartbeat management and other services. In addition, there are global FSlocks and DirLocks inside NameNode, which are global locks. These designs make the HDFS NameNode architecture extremely simple, which are some of its advantages. These designs are a double-edged sword that not only creates HDFS but also limits HDFS.
and NameNode's HA (high availability High Availability) also seems particularly complicated. At least three zookeeper nodes, three JournalNode nodes, and two ZKFC nodes are required to introduce, and the deployment and maintenance are relatively complicated. The inode information and block information of the file system and the location information of the block are all maintained in the NameNode memory, which makes NameNode very memory requirements very high. Machines with customized large memory can carry a larger amount of metadata.
JD.com’s NameNode memory is 512GB, and the NameNode machine of a certain factory has 1TB of memory. In addition, the NameNode has a huge heap allocation, and JD.com's NameNode requires a 360G heap, so the requirements for GC are particularly high. Under constant tuning and modification, JD.com's customized JDK11 and JDK G1 GC perform good performance. However, small-scale companies generally do not have the ability to maintain JDK, and the solutions are not universal. ByteDance modified NameNode to C++ version, so that allocating and freeing memory are controlled by the program, which also achieves good performance. This solution is still not universal, because developing and maintaining such a C++ version of NameNode implementation also requires a considerable team. Regardless, the scalability of metadata (scalability) is a fatal disadvantage due to physical memory size.
Everyone hears about it HDFS is not friendly to small files, but why? In fact, it is not fundamentally because the file is small, but because many small files and blocks are generated, resulting in more memory objects in NameNode under the same data storage capacity, and these memory objects must be managed by NameNode, which will lead to a large amount of metadata when the data storage volume is not large.Even if there is no small file problem, when the data volume reaches hundreds of PB, NameNode needs to accept most FullBlockReports to exit safemode (safe mode), so the startup time will reach several hours. The conventional FullBlockReport, Decommision datanode, and balancer block of the
cluster will also affect the performance of the cluster metadata service. The root cause is that DataNode needs to report all blocks to NameNode, and almost all operations must obtain global locks.
Generally speaking, when 5,000 HDFS clusters have about 200PB, there will be about 400 million metadata. In terms of 200 million files and 200 million blocks, these files and block information will be maintained on the NameNode heap, occupying expensive memory and putting pressure on GC. In addition, such a large amount of data will reduce the available time of the overall cluster during startup and subsequent FullBlockReport, limiting the throughput of the HDFS cluster.
Summary, the problem with HDFS is that the metadata is all in memory, causing the problem of NameNode scalability being subject to physical memory limitations and GC disaster problems. As well as global lock and block reporting storms, resulting in poor throughput and slow NameNode startup problems.
. Ozone
In recent years, Ozone has stood out among many emerging open source distributed storage systems. Since Ozone is also produced by the Apache Open Source Foundation and is also jointly designed by many PMC and Committer of the HDFS community, it not only avoids many design defects in HDFS in terms of architectural design, but also refers to the excellent design functions of HDFS that have been tested over the years. Therefore, Ozone strives to become the next generation of distributed storage system for HDFS. There are many companies that use Ozone and participate in Ozone development. I know, including Tencent, JD , Clouderah , Google , Google , 360, and some other companies.
Among them, Tencent is heavily involved in the development, and there are also many companies waiting and watching, hoping to apply it to the company after it matures. This may be because some operational and maintenance teams are not so passionate about developing new systems. I hope everyone will not wait and see. There are many Ozone Committers in China. I hope everyone can actively join the Ozone community and push Ozone forward together. In terms of community activity, Ozone and Ratis are much more active than HDFS. You can compare the number of members of the hdfs channel and ozone channel of slack. HDFS has 111 members, Ozone already has 124 members, and Ratis is also the one that is necessary for Ozone to write data. There are more than 100 people, and there are more than 200 people in total. From the comparison of the number of Pull Requests in the entire Hadoop project last month, we can see that the activity has changed a lot, and there may be some errors, because some Hadoop patches are submitted through JIRA patches. You can participate in the code contribution of the Ozone community. Compare that, the basic facilities of the Ozone community are still very complete, such as integrating checkstyle, findbug, Test Coverage, UT, Integrity Test, as well as some tests in docker. The infrastructure is well built, and the entry threshold for developers will be relatively low.
does not need to be compiled and installed Protocol Buffers, and there is no need to worry that some message definitions in Protocol will be forward incompatible. The maven plug-in used by Ozone will automatically help you download your corresponding version of the protoc tool, and will also automatically check that the Protocol message definition is forward incompatible. The Ozone community tycoon service is also very thoughtful. During his busy schedule, I recorded a series of video tutorials for everyone and put them on YouTube. I also wrote a script that ran Ozone in Intellij idea with one click, a script that ran Ozone in docker with one click, and a script that started Ozone in kubernet with one click. There is also a synchronous meeting of the Ozone community every Friday at noon, where everyone can attend. I think these are one of the key points for Ozone to become the next generation of distributed storage systems.
Although there are some problems with HDFS, although the Ozone community is relatively active and open, why will Ozone become the next generation of distributed storage? There are many reasons. First, from the perspective of Ozone's access interface, Ozone provides many APIs, which allows various applications to access Ozone using various mainstream APIs. For example, Ozone implements hadoop-compatible file system API, which allows systems in the big data ecosystem such as spark, presto, MapReduce, Hive, Alluxio, etc. to access Ozone. In addition, we contributed Ozone underlying storage modules to Alluxio, which makes Alluxio natively support Ozone as the underlying storage for Alluxio.
Ozone also provides the S3 gateway service, making Ozone suitable for S3 scenarios.
In addition, Ozone realizes the ability to hang up the k8s through Goofys and Ozone CSI. Therefore, it is very convenient to transfer the storage of your application to Ozone.
. Advantages of Ozone design
SCM and DataNode form a common storage layer HDDS, with the light yellow part in the above picture. HDDS is the abbreviation of Hadoop Distributed Data store. If you add Ozone manager to HDDS, it is Ozone, which is an object storage service provided to the outside world. On top of HDDS, we can also build an HDDS NameNode. In this way, it is like an HDFS, which can provide file services. If we are on HDDS, we start a blockstorage service, so that it can provide the ability to hang up the disk at a high speed. A layered design like Ozone is quite clever. Let’s take a look at Ozone Manager. One of the biggest design differences between Ozone and HDFS is that Ozone is an object storage and does not maintain a file system number. The object semantic operations do not have a directory and file relationship, so it can achieve a very high throughput. Ozone Manager can also reach bucket-level concurrent read and write. Another excellent design is that both OM and SCM use RocksDB for metadata management. Ozone's metadata is not stored in memory like HDFS's NameNode, but is placed in RocksDB. Regardless of OM's metadata or Container information in SCM, the need to use the heap storage, theoretically, metadata can be expanded infinitely. There is also a very clever design, which introduces the concept of Storage Container. The block in the Container is managed by DataNode. SCM does not need to manage blocks. It only needs to manage the Container. It’s like the national leadership only needs to manage every city, and the city’s leaders manage the people in the city. Each Storage Container has a default 5G capacity, where the block status is managed by the container, so it greatly reduces the amount of data managed by SCM, thereby improving SCM's service performance and scaling capabilities. Whether it is full block reporting, incremental block reporting, as well as add and delete copies, and block balance clusters, these will not have a big impact on the performance of SCM, because blocks account for the majority, but there are relatively few containers.
. Challenges faced by Ozone
Ozone Although there are so many access methods and so many design advantages, HDFS has been leading the way for decades and is not so easy to be replaced. It requires many challenges to replace it. Let's explain it one by one.
- Ozone does not support the operations of Append and truncate, but this disadvantage is that Tencent’s two Ozone Committers have been implemented internally and are pushing into the open source repository. However, the scenarios using hflash are not yet supported, so we need to work harder later.
- In addition, Ozone writes keys without writing them, which is invisible. This is because Ozone will not submit blocks to the OM during the process of writing keys, but will submit all block information to the OM at one time after writing all the data. Therefore, the key is invisible during the process of writing. This is also because we can make up for this shortcoming through subsequent changes. The RPCh link of
- Ozone is longer than the RPC link of HDFS. This is also a disadvantage for everyone at first glance, because it splits the service into two, and its RPC access will inevitably go from client to OM, and then from OM to SCM. This is also very normal, but if you think about it carefully, is metadata taking more time to take, or data taking more time to operate the file system? If you want to open a file to read its content, or create a file to write content inside, the access to these metadata is very small. But there are some like getblockLocation, etc., which are pure metadata operations, which will definitely be slow.
- Another is Ozone. Currently, there is no metadata like a folder, so it cannot obtain folder owner, modificationTime and other information. Like I did an experiment before, which was to run the JobHistoryServer of MapReduce on Ozone. There are some jobs of yarn that can be placed in Ozone. When you look at it, you will find that some logs of newly generated jobs cannot be scanned. The reason is that we looked at the code JobHistory, which will determine whether a new subdirectory will be generated based on whether the modificationTime of the folder changes. Ozone's modificationTime is always zero now, so there will be no change and no new jobs can be scanned out. The underlying data writing of
- Ozone is based on ratios, and currently the ratio writing only supports one copy and three copy. Tencent is contributing to the open source community to support any copy of the Strorage cluster framework based on .
- SCM HA is also a big problem, and it has not yet. However, Tencent is actively leading the feature and is pushing related changes to our open source community. It is expected that this feature will be included in the launch conference of the next major version of Ozone.
- Ozone also lacks a container balancer function, which is used to balance storage and enables the storage of the cluster to be evenly distributed. For example, we had 1,000 machines at the beginning, and later we launched 10 new machines that were empty, so we needed to balance the storage to machines with lower usage. Tencent is also developing this function.
- Ozone currently lacks the DataNode disk reservation function. Ozone's write performance is being optimized through the feature of ratios stream. Our current internal tests have greatly improved the performance of some features. An essential change is to make it submit some information about the metadata of the block and the stream operation of the data block content during the writing process to separate it and can be on different nodes. The write performance of
- Ozone is being optimized through RATIS Streaming feature. Currently, our internal tests have a huge improvement in performance with this Feature.
2NameNode on HDDS In fact, most of us can improve these missing functions through subsequent development, but the support of folder metadata and the performance of speeding up folder operations are actually not easy to implement, and these are precisely the scenarios of the semantics of the big data file system and the necessary functions and performance requirements. So what to do? It seems that if we do nothing and rely on the improvement of existing Ozone components, it seems that we cannot be able to meet these scenarios of HDFS file semantics, so we started a NameNode service based on HDDS on top of HDDS.
We can create a NameNode service on HDDS to replace Ozone Manager, so that it provides a file system service to the outside world, which can fundamentally solve the problems of poor Ozone file semantics and missing folder metadata.
. The difference between file system semantics and Ozone object storage
We have been talking about file system semantics and Ozone object storage. What is the difference between them?
can be seen simply from the above figure. object storage is like the KV method on the left to manage object metadata, which does not need to manage the relationship between metadata. The file system is like the right side, and it is an additional need to use a tree-like structure as an index to manage the relationship between metadata, which is its essential difference. These differences make object storage and file systems have their own advantages and disadvantages and their respective applicable scenarios.
object storage, a URL path, is a key, and there is no concept of folders. But we have to force the object storage to use as a file system, so we can only rely on the forward slash in the key to cut out the concept of folders. In fact, there is no folder node, so there will be no metadata information for folders.
list is also slow because the list operation needs to be in rocksdb and swipe the table by prefix. rename folder operation requires modifying each key in the folder. For the file system, the list operation directly returns the children of the folder. For the rename folder, you only need to modify the name of the folder node.
understands the principle, and we have confirmed one thing: OzoneManager is an object storage service. No matter how you optimize it, it cannot match HDFS.
. Ozone's storage layered design
So what to do? Let’s review the storage hierarchical design of Ozone. Ozone was originally designed as an object storage, but the abstract HDDS layer makes the storage layer and the metadata layer separate, just like Ozone is to separate Ozone Manager and the HDDS below, so that HDDS NameNode or Block Storage can be added above the HDDS layer. Or in addition, we can also create a new other object storage on HDDS. You may need such a customized object storage in your system, and you can also implement another node similar to Ozone Manager. Our focus is to introduce the design based on Ozone layered storage. We can implement an HDFS NameNode on the HDDS layer to undertake the semantics of file system in big data scenarios.
Base is our foundation. We did not implement it from 0. We started our development work based on HDSF client, Namenode and Ozone's HDDS. The goal of
Basic is to implement a HddsClient and HddsNameNode, plus an unmodified HDDS cluster. A HDDS-based Filesystem can be implemented, which we call OZONE-DFS. This implements a file system service that is almost the same as HDFS, and can access big data workloads.
st stage 2, we want to be better than HDFS and improve throughput and performance. Therefore, we have made drastic efforts to do things that HDFS has not done for many years on the NameNode of OZONE-DFS (HDDSFS). Lock optimization, also known as fine-grained lock transformation. We have learned from Meituan's NameNode lock optimization ideas and carefully analyzed the implementation of Alluxio fine-grained locks. These are very helpful to us. After this stage is completed, our OZONE-DFS has surpassed HDFS.
This is not enough. We need to design and implement an OZONE-DFS that far exceeds HDFS. Therefore, in the third stage, we will KV transform the metadata of NameNode and adopt a two-layer metadata management method. The upper layer is memory and the lower layer is rocksdb, which not only realizes infinite expansion of metadata, but also maximizes the performance without losing too much performance. In addition, using Apache Ratis to implement NameNode HA no longer requires HA-related services such as ZK, ZKFC, JournalNode, etc. It is enough to form a RAFT Group for 3 HDDS NameNodes.
We are secondary development based on the HDFS source code:
- implements the communication between Client and HDDS Datanode, which allows normal block reading and writing.
- We have also modified the protocol from Client to HDDSNN, so that Client can apply for block to SCM through HDDS NameNode.
- does not require any changes to block-independent operations, such as createFile, mkdir, rename, etc. In this figure of
- , we can see that we have added HDDSNN as a file system metadata management operation. HddsNN can communicate with SCM to perform block application and delete operations.
- also adds HddsNNClient to access HddsNN at the same time to request file system metadata, and to access HddsDataNode to read and write data blocks. The two figures above show the specific read and write process respectively. The left side is the writing process. For example, when you want to create a file, you first have an RPC request to create the file, which is sent from the client to HDDSNN, and then HDDSNN creates a metadata. Next, the HDDSFS Client writes data to the output stream. During the data writing process, it will find that there is no block, so it will send an allocateBlock request to HDDSNN. HDDSNN does nothing and sends the allocateBlock request to SCM. SCM is responsible for allocating the Block to allocate and then returning it to the HDDSFS Client. Next, you can execute block writing logic. Client communicates with DataNode, first write some chunks, and finally when a block is reached, a putBlock request will be sent to complete the process of writing a block. The entire process is completed, and finally a complete RPC request will be sent. HDFSNN will mark the file as completed when receiving the RPC request. This is a writing process. The reading process is roughly similar, so I won't go into details. While
also realizes the functions, we also have a higher-end goal:
a set of codes, which can start HDDSFS or HDFS, and distinguish them by a configuration item.
This is because we abstracted BlockManager, DatanodeManager, and heartbeatManager, respectively, with HDDS version and HDFS version implementations.
We also implemented the Hadoop-compatible file system implementation class corresponding to odfs schema - HddsFilesystem. The same client like
can support both reading and writing hdfs and hddsfs. The class dependency diagram on the right of
, the red box is the newly added class. Most of them inherit or extend from existing HDFS classes. The specific implementation details of the writing block are in HddsOutputStream.
Our goal at this stage is very clear and clear, which is to implement a Client and NameNode, which can provide file system services with the help of HDDS storage capabilities.
HDDSFS is separated from block management, node management, heartbeat management and other functions, and stores metadata of the same scale, saving 40+% of memory.
Because HDDSFS avoids block reporting storms, the restart speed is 4 times that of HDFS for 10 million small files.
On the other hand, because HDDSFS has a metadata tree structure with file semantics, it has obvious advantages in folder operation performance compared to Ozone.
, and the performance improvements in the 100,000 child node directory are 100+ times.(Asynchronous deletion function, improve higher)
related operations based on folder attributes ( modification Time, set mode, set owner ), OzoneManager does not support it at all, and HDDSFS is perfectly supported. In the following stage, our goal is to optimize high concurrent throughput and improve cluster performance by implementing fine-grained locks.Students who have some understanding of the principle of HDFS should know that HDFS requires locking FsLock and FsDirLock before they can do specific operations. Now we have turned the FsLock part of the writelog into readlock, because we have connected the Block Manager function to the SCM. There are some changes, so that we have some operations that can be executed concurrently. At the same time, we changed the lock in FSDirectory to a fine-grained lock, which greatly improves the throughput of the cluster. Our change plan is to modify the readwritelock in FSDirectory to a locklist-based hierarchical lock. Let's look at the table on the right of the picture above to explain it in detail. For example, if you want to list a folder /a/b, it turns out that you might want to add a global read lock and then do the operation. Now instead of adding a global lock, you add a read lock to such a root, a add a read lock b adds a read lock, and then the following child node list will be returned back. This way you can tell that it is a locklist. Let me give you another example of writing. For example, createFile to create a c.txt under the /a/b folder, add a read lock to c, d, and then add a c.txt below. The newly created file is to add a write lock. Students who know the inside story may find that this is wrong. We are now a directory, and the following is a Children, an array list, which is equivalent to a thread-unsafe data structure. If several threads create at the same time, thread insecure problems will occur. That's right, we just stepped on the pit and changed children to a line-safe data structure. The advantage is that its parent node does not need to add a write lock. By designing a fine-grained lock in this way, it can not interfere with each other's operations. Compared with HDFS deletion or a DU directory, the performance of the entire cluster will be significantly reduced. Through the optimization of particle locking, this performance will be significantly improved, and its throughput will also be significantly improved overall.
You can imagine, if an Inode is associated with a ReentrantReadWriteLock, how much memory will increase? The memory is already very tight, and adding too much memory is definitely an unacceptable solution.
Therefore, we borrowed the concept of LockPool introduced by Alluxio 2.0. What is LockPool? That is, there is a resource pool, and lock is a lock resource in the resource pool. Each Inode no longer associates a lock. Instead, when the lock needs to be locked, it will go to the resource pool to apply for a lock. At the same time, the reference count will increase. When the unlock is used up, the reference count will decrease.
LockPool has a hard-working LockEvcitor that will evict when Lock reaches high watermark until it is lowered to low watermark. The flowchart above
can also help us understand the working principle of lockPool asynchronous lock Evictor. On the left side above is the process of a thread obtaining a lock corresponding to a key from lockPool, and judging whether the key exists in lockPool. This is the first step. If it does not exist, a lock will be created. If it exists, its refCount will be added one and it will be returned directly. Then he will determine whether the lockPool's size has reached a high water level. If it exceeds the high water level, it will send an Evictor signal. On the right is a thread that will keep checking whether the lockPool's size exceeds the high water level. If it does not exceed the high water level, it will await wait for the signal to trigger it. When it is found that the high water level has exceeded, an Evictor operation will be performed to kick out the uncited lock until the low water level is reached.Regarding fine-grained locks, you can also distinguish specific write operations such as creation, deletion, and modification in the future, and add different locks to maximize the throughput. If you can not lock, you will not lock, and if you can use a read lock, you will not need to write locks.
NN on HDDS—Lock Guard
https://www.com/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/s/ After adding, we can receive some alerts for some alarms, and at the same time print some warning logs into the log. After receiving the alarm, we can log in to the machine to see which lock has a problem and some context information of the lock can be typed out. Of course, this function will increase the loss of memory overhead performance, so it can be switched dynamically. After receiving the alarm information, you can use the command line tool to analyze the lock occupation situation using the lock ID you obtained.NN on HDDS——Tired Metadata Management
The third stage goal is to achieve infinite expansion of metadata through RocksDB. In memory space, we can easily manage files and folders with a tree, but RocksDB is also a KV storage. We need to split the metadata into two tables. The first table is placed inode. Here inode includes inodeFile and inodeDirectory, and use inodeId as the key. The second table is the relationship between inode. The key is a string aggregated by parentId and childName. Its form is, "", and value is the inodeID of child.
Let’s give an example, for example, what should you do if you want to obtain the metadata information of “/dir/file”? First of all, there is no need to look for the root. The root ID is 0. Next, it is to look for /dir. So go to the Edge table to find the record with the key "0, dir" and you can find it. The value is 1, which means that the ID of "dir" is 1. Then look for "1, file", which can be found in the Edge table, which means that the ID of "file" is also 2. Finally, we look for the value of 2 in the inode table and we can find the metadata of "/dir" and "/file".
NN on HDDS——Tired Metadata Management
This operation is obviously much faster in memory than in RocksDB. Therefore, hierarchical metadata management is needed, such as adding a cache layer. What is hierarchical metadata management? What are the benefits?
In a storage system, not only the data files it stores are important, but the metadata management of its storage system is also very important. Because when it comes to data access operations of the storage system, it will go through the storage system, metadata query or update operations. If there is a performance bottleneck on the operation on the metadata side, it will also lead to slow user access to data. In this article, let’s talk about how storage systems generally perform efficient metadata management, which involves a variety of different metadata management methods.
In fact, there is an evolutionary process for the metadata management of the storage system. The first generation of metadata management, the metadata is stored in external db, and then the master service and db interact with the data. Memory-managed metadata management, the master service loads external metadata db files into memory after initial startup. Partitioned metadata management, partitioning the metadata according to the given rules, and then starting multiple master services to manage their respective metadata that should be maintained. Hierarchical metadata management (tiered metastore) is a strategy that can not only expand infinitely, but also ensure that active data performance does not decline.The recently visited hot metadata, which is called cached layer, is a data that has not been accessed for a long time (also called cold data), and is called persisted storage. In this mode system, the service only caches the current active data, so there will be no problem such as memory bottlenecks. This figure is a metadata management model diagram of a sample system in this mode.
Compared with HDFS Namenode, the metadata management method that loads all metadata to memory and improves fast access capabilities, the memory has become the upper limit of metadata expansion, and Ozone has to read and write all metadata through rocksdb, which has reduced performance has become the biggest problem. Therefore, HDDSFS NN needs to make optimizations and improvements at this point, only cache those active data. For cold data that has not been accessed recently, it is saved in the local rocksdb.
In the hierarchical metadata management strategy, the storage layer of cache active data in memory is called cache store, and the underlying rocksdb layer is called backing store. We also refer to the design of Alluxio and Ozone filesystem.
NN on HDDS——RAFT Based HA
The last big architecture upgrade is the HA upgrade. You only need to have 3 HDFS NameNodes that are composed of RAFT Group that does not rely on external components to implement the HA function. Different from Ozone Manager, the HA implementation requires the client to communicate with the RAFT Group composed of three OMs through the RAFT protocol, that is, communicate with the leader OM. The leader OM records the client's request as the RAFT log, and then synchronizes it to two followers. Our design consists of three NameNodes to form a RAFT Group, but the client uses the following method to find the leader's NameNode to communicate with it. The leader's NameNode will execute the client's request, and then record the write operation to the journal based on the operation log. When the leader of the RAFT Group switches, each NameNode will set its own leader state, and only the leader will accept external requests. When the leader switches, the NameNode that loses the leader state will reset the state and load it into the NameNode, so that the memory state can be achieved, which is consistent with the RAFT log. In the HA design of HDDS NameNode, we are also referring to the HA of Ozone Manager, the HA of SCM, and the Ratis-based HA introduced in Alluxio 2.0. At present, our first two stages have actually been implemented, but our third stage is still in progress, so interested friends can join us to do these great things.
3 Tencent’s contribution
Tencent’s contribution list is a lot, and I won’t introduce them one by one. There are 6 committers, including 2 PMCs and 1 chair. This chair is Sammi (Chen Yi) of our team. It is also used as the very important version of Release Manager for Ozone 1.0.0.4The future of Ozone 1. The future of Ozone
Let’s look at some of the future of Ozone. There are still many things to do in the future of Ozone, such as SCM HA, HA state switching, etc., that need to continue to develop. In the big data ecosystem, Ozone still has many corresponding functions to implement. The Erasure Coding function has been implemented in Hadoop, which is not bad, but it is still in the design stage in Ozone. If you want to replace it, you must perfectly support these APIs, such as append/truncate/hflush, you need to continue to mention them to the community. Datanode's health checks, dynamic configuration modifications, centralized configuration management, and some functions such as balancers in containers still need to be added, so we will not introduce them one by one. In fact, there is still a lot of work to be done.
- Native Object Store https://www.slidestalk.com/TencentCloud/TencentCloudOzone
- toward maturity - Tencent Ozone Thousands of Capacity Breakthrough https://cloud.tencent.com/developer/article/1667033
- Differences in file system and object storage https://ubuntu.com/blog/what-are-the-different-types-of-storage-block-object-and-file
- Ozone Developer Resources https://docs.qq.com/doc/DZUJFSXFuZHFXRGZp
- Goofys Incremental Edition Goofuse https://github.com/opendataio/goofuse
- HCFSFuse https://github.com/opendataio/hcfsfuse
This is some of the articles I have quoted this time. Among them, the Ozone developer resources marked with green is recommended. It contains many resources for developers, such as some Ozone videos mentioned in this article, and how new developers can develop them, etc. It is recommended that you join Apache slack workspace and add ozone channel. There is an invitation connection here. The following are two Fuse projects we developed, HCFSFuse, and GooFuse. As the Fuse implementation of Ozone, you can use Fuse to mount Ozone for the land. If you are interested, you can also pay attention.
. Summary . Finally, we have summarized that we are still ahead of the design, but we iterate in stages. We have also formulated some priorities, first completing the high-priority needs, and producing some results in this stage. We do not do some subcontracting systems like some companies, and this function will be packaged for you, and then you will be responsible for its results yourself. We are not like this. We are in a high degree of collaboration and we all face the problems together. Then, in this aspect of testing, we also attach importance to quality assurance, such as TDD (test-driven development), CI (continuous integration), Nightly build, and code review. We borrowed the design and source code of many open source software such as Alluxio, HDFS, Ozone, RATIS, Ceph, etc., and also referred to Meituan's lock-unlocking solution.
Ozone If you want to become the next generation of HDFS storage system, you need to effectively replace some of the scenario capabilities of HDFS. We look forward to the next decade that Ozone can become a household name and become the underlying storage in the technology stack among major technology companies.
th the end of today's sharing, thank you everyone.
Shared Guest:
Mao Baolong Tencent | Senior engineer comes from Tencent Data Lake team, currently focusing on the development of Ozone, as well as the implementation and application work of Alluxio in Tencent. is a PMC member of the Committer and Alluxio communities of the Ozone open source community.
Shared Guest: Mao Baolong Tencent Senior Engineer
Editing and Organizing: Yang Xueling
Produced Platform: DataFunTalk
