With the development of the big data industry, related technologies in the big data ecosystem are also constantly improving iteratively. The author is fortunate to have personally experienced the development process of the domestic big data industry from zero to one. Through this article, I hope to help everyone quickly build a complete knowledge system of the big data ecosystem.
The core technologies in the current big data ecosystem are summarized as shown in Figure 1. They are divided into the following 9 categories, which are introduced below.
1. Data collection technology framework
Data collection is also called data synchronization. With the rise of technologies such as the Internet, , mobile Internet, , and the Internet of Things, massive amounts of data have been generated. This data is scattered everywhere. We need to fuse this data together and then calculate some valuable content from these massive data. The first step you need to do at this point is to collect the data. Data collection is the basis of big data. Without data collection, there is no big data!
data collection technology framework includes the following:
- Flume, Logstash and FileBeat are often used for real-time monitoring and collection of log data. The detailed differences between them are shown in Table 1:
- Sqoop and Datax are often used for offline data collection of relational databases. The detailed differences between them are shown in Table 2:
- Cannal and Maxwell are often used for real-time data collection of relational databases. The differences between them The detailed differences are shown in Table 3:
The technical selection between Flume, Logstash and FileBeat is shown in Figure 2:
The technical selection between Sqoop and Datax is shown in Figure 3:
The technology selection between Cannal and Maxwell is shown in Figure 4:
2. Data storage technology framework
The rapid growth of data has promoted the development of technology, and a number of excellent storage systems that support distributed storage have emerged.
data storage technology framework includes HDFS, HBase, Kudu, Kafka, etc.
- HDFS can solve the problem of massive data storage, but its biggest disadvantage is that it does not support the modification operation of a single piece of data, because it is not a database after all.
- HBase is a distributed NoSQL database based on HDFS. This means that HBase can take advantage of the massive data storage capabilities of HDFS and support modification operations. But HBase is not a relational database, so it cannot support traditional SQL syntax.
- Kudu is a technical component between HDFS and HBase. It supports both data modification and SQL-based data analysis functions. Currently, Kudu's positioning is rather awkward and it is a compromise solution with limited application in actual work.
- Kafka is often used for temporary buffer storage of massive data to provide high-throughput reading and writing capabilities to the outside world.
3. Distributed resource management framework
In the traditional IT field, enterprise server resources (memory, CPU, etc.) are limited and fixed. However, the application scenarios of servers are flexible and changeable. For example, a system is temporarily online today and requires several servers. After a few days, the system needs to be taken offline and these servers need to be cleaned up.
Before the advent of the big data era, changes in server resources corresponded to the system going online and offline, and these changes were limited. With the advent of the big data era, the demand for temporary tasks has greatly increased, and these tasks often require a large amount of server resources. It is obviously unrealistic to rely on operation and maintenance personnel to manually handle changes to server resources at this time. Therefore, distributed resource management systems emerged as the times require. Common ones include YARN, Kubernetes and Mesos. Their typical application areas are shown in Figure 5.
4. Data calculation technology framework
Data calculation is divided into offline data calculation and real-time data calculation.
1. Offline data calculation
After more than ten years of development, the offline data calculation engine in big data has undergone three major changes so far.
- MapReduce can be called the first generation of offline data computing engine in the big data industry. It is mainly used to solve distributed parallel computing of large-scale data sets. The core idea of the MapReduce computing engine is to abstract computing logic into two stages: Map and Reduce for processing.
- The Tez computing engine has a weak presence in the big data technology ecosystem. In actual work, Tez is rarely used alone to develop computing programs. The biggest feature of
- Spark is memory computing: all intermediate results in the task execution phase are placed in memory, without the need to read and write disks, which greatly improves the computing performance of data. Spark provides a large number of high-order functions (also called operators), which can implement iterative calculations of various complex logics, and is very suitable for application in fast and complex computing requirements of massive data.
2. Real-time data calculation
The most typical real-time data calculation scenario in the industry is the large data screen of Tmall "Double Eleven". Data indicators such as total transaction amount and total order volume displayed on the large data screen are all calculated in real time. After the user purchases the product, the amount of the product will be added to the total transaction amount on the big data screen in real time.
- Storm is mainly used to implement real-time data distributed computing.
- Flink belongs to a new generation of real-time data distributed computing engine, and its computing performance and ecosystem are better than Storm. The SparkStreaming component in
- Spark can also provide real-time data distributed computing functions based on seconds. The differences between
and Storm and Flink are shown in Table 4. The technology selection among
Storm, Spark, and Flink is shown in Figure 6.
Currently, enterprises mainly use Spark for offline computing and Flink for real-time computing.
5. Data analysis technology framework
The data analysis technology framework includes Hive, Impala, Kylin, Clickhouse, Druid, Drois, etc. Their typical application scenarios are shown in Figure 7.
Hive, Impala and Kylin are typical offline OLAP data analysis engines, mainly used in the field of offline data analysis. The differences between them are shown in Table 5.
Table 5
- Hive has average execution efficiency, but extremely high stability;
- Impala can provide excellent execution efficiency based on memory, but has average stability;
- Kylin can provide millisecond response for PB-level data through pre-calculation.
Clickhouse, Druid and Drois are typical real-time OLAP data analysis engines, mainly used in the field of real-time data analysis. The differences between them are shown in Table 6.
- Druid and Doris can support high concurrency, but ClickHouse has limited concurrency capabilities; SQL support in Druid is limited. ClickHouse supports non-standard SQL, and Doris supports standard SQL, and has better SQL support.
- At present, Druid and ClickHouse are relatively mature, and Doris is in a rapid development stage.
6. Task scheduling technology framework
includes Azkaban, Ooize, DolphinScheduler, etc. They are suitable for scheduling routine tasks that are executed at ordinary times, as well as multi-level tasks containing complex dependencies. They support distribution and ensure the performance and stability of the scheduling system . The differences between them are shown in Table 7.
Their previous technology selection is shown in Figure 8.
7. The underlying basic technology framework of big data
The underlying basic technology framework of big data mainly refers to Zookeeper.Zookeeper mainly provides commonly used basic functions (such as namespace, configuration services, etc.). Zookeeper is used to run technical components such as Hadoop (HA), HBase, and Kafka in the big data ecosystem.
8. Data retrieval technology framework
With the gradual accumulation of data in enterprises, the demand for statistical analysis of massive data will become more and more diverse: not only analysis, but also fast and complex queries with multiple conditions. For example, the product search function in e-commerce websites and the information retrieval function in various search engines all fall into the category of fast and complex multi-condition queries.
When choosing full-text search engine tools, you can compare them in terms of ease of use, scalability, stability, cluster operation and maintenance difficulty, project integration level, and community activity. The comparison of Lucene, Solr and Elasticsearch is shown in Table 8.
9. Big data cluster installation and management framework
If an enterprise wants to transform from traditional data processing to big data processing, the first thing to do is to build a stable and reliable big data platform.
A complete big data platform needs to include data collection, data storage, data calculation, data analysis, cluster monitoring and other functions, which means that it needs to include components such as Flume, Kafka, Haodop, Hive, HBase, Spark, Flink, etc. These components need to be deployed on hundreds or even thousands of machines.
If you rely on operation and maintenance personnel to install each component individually, the workload will be relatively large, and the matching issues and various conflicts between versions need to be considered, and the later cluster maintenance work will also put a lot of pressure on the operation and maintenance personnel.
Therefore, some foreign manufacturers have encapsulated the components in big data and provided an integrated big data platform, which can be used to quickly install big data components. At present, the most common ones in the industry include CDH, HDP, CDP, etc.
- HDP: The full name is Hortonworks Data Platform. It is packaged by Hortonworks based on Apache Hadoop, provides interface installation and management with the help of Ambari tool, and integrates common components in big data to provide one-stop cluster management. HDP is an open source free big data platform and does not provide commercial services;
- CDH: the full name is Cloudera Distribution Including Apache Hadoop. It is commercialized by Cloudera based on Apache Hadoop. It provides interface installation and management with the help of Cloudera Manager tool, and integrates common components in big data to provide one-stop cluster management. CDH is a commercial paid big data platform and can be tried for 30 days by default. After that, if you want to continue to use advanced functions and commercial services, you need to pay for a license. If you only use basic functions, you can continue to use it for free;
- CDP: Cloudera acquired Hortonworks in October 2018, and then launched a new generation of big data platform product CDP (Cloudera Data Center). The version number of CDP continues the previous version number of CDH. Starting with version 7.0, CDP supports Private Cloud and Hybrid Cloud. CDP integrates the better components of HDP and CDH and adds some new components. The relationship between
is shown in Figure 9.
The above content comes from the book "Big Data Technology and Architecture Illustration Practical School".
