Assistant Director of the Data Platform Department of the Ministry of Information Technology of China Dadi Insurance Zhang Lei
On September 20, 2019, Mr. Zhang Lei, Assistant Director of the Data Platform Department of the Ministry of Information Technology of China Dadi Insurance, was invited to attend the first financial technology salon hosted by Kyligence, a supplier of big data analysis technology and solution, and gave a speech entitled "The Road to the Evolution of Big Data Application Architecture of China Dadi Insurance", sharing China Dadi Insurance's practice and experience in big data.
China Land Insurance is a national property insurance company. It has been 16 years since its establishment in 2003. At the end of 2018, the official opening of China Dadi Insurance Tibet Branch marked the company's insurance business covering the whole country. In the same year, China Dadi Insurance's premium scale reached 42.6 billion, ranking fifth in the industry.
China Dadi Insurance's four major data application scenarios
With the advent of the big data era, various types of new data are emerging continuously. Whether driven by business or business competition, it is necessary to analyze and explore these data from a new perspective to help enterprises achieve the goal of "big data empowering the company's various business scenarios". This is also the problem and challenge facing every industry in the big data era.
There are four parts of the big data application scenarios of China Dadi Insurance.
●Report analysis. Since the second half of 2017, China Dadi Insurance has started a promotion journey of self-service analysis of reports with the help of lightweight reporting/BI tools. At present, self-service analysis reports have been promoted to all 36 branches.
●Data mining. The content contained in data mining is more complex than report analysis, including behavioral exploration of various users, precise marketing, etc.
●Supervisory reporting. Regulatory reporting is a very important task for all insurance companies, and China Land Insurance attaches great importance to this and invests a lot of manpower every year to carry out this work.
●Risk control. In terms of identifying and predicting risks based on big data, China Dadi Insurance has continued to explore in recent years, such as the anti-fraud system based on SNA (social network analysis), which has been included in typical cases of information construction in China's insurance industry.
China Land Insurance's big data technology evolution process
China Land Insurance used Oracle as early as 2014. In order to improve the performance of Oracle, it introduced the Huawei all-in-one machine in 2014 and rebuilt the data warehouse platform. Whether it is a business system, a data warehouse, or a data mart, it is built on Oracle.
As the data volume becomes larger and larger, traditional data warehouses based on Oracle are becoming increasingly difficult to support business development. China Land Insurance began to explore the construction of the Hadoop big data platform since 2016 - initially built a cluster containing 5 servers. After experiments, some content on Oracle was gradually transferred to the Hadoop platform, and at the same time, Impala was used as the SQL query engine. After starting this way, China Land Insurance gradually became more and more widely used in Hadoop, and the cluster size was also expanding, but at the same time, some problems in Impala were exposed.
In 2018, after several months of communication, exchange, investigation and POC with many manufacturers, China Dadi Insurance built a Hadoop MOLAP self-service analysis platform that uses Kyligence products to build application layer data sets.
However, China Dadi Insurance's big data development path has not been smooth, especially since it encountered many difficulties in the first two stages.
The first stage is the traditional data warehouse stage based on Huawei all-in-one machine + Oracle. The main data application forms of this stage are multi-dimensional analysis and fixed reports. At the same time, it provides business users with an environment and tools that can self-extract data, so that business personnel can obtain some personalized list data without relying entirely on technical personnel. This practice greatly improves the flexibility of business personnel to obtain data, but there are also some problems. For example, the open tool is a professional reporting tool ReportStudio, which has a high threshold for getting started. At this stage, there are four issues that must be mentioned:
●Cube response is slow, and 70% of query responses are at the minute level.
● Poor concurrency performance, China Land Insurance uses Cognos Power Cube to generate files and store them on disk. It has experienced it from native disk to NAS shared storage, from disk arrays to flash arrays, but with the increase in data volume and number of users, the system is becoming increasingly difficult to support high concurrency.
●Insufficient timeliness, on the one hand, this is due to the insufficient computing power and scalability of Oracle. On the other hand, it is due to the slow update of too many and too large Cognos Power Cubes, which leads to users waiting for data frequently and poor user experience.
●Maintenance is difficult, China Dadi Insurance's report volume is close to 1,000. It is understood that the scale of statements of many leading companies in the financial industry is also at this level, and the workload of report maintenance is huge.
The second stage is the Hadoop big data platform with Impala as the SQL engine. The very intuitive problem at this stage of is:
●Imapla's stability problem. The most common thing is that the front-end application suddenly reports a query error when using it.
●Imapla has a large demand for memory and CPU resources. Taking the memory demand as an example, as small as a few hundred GB to as large as a few TB, making it difficult to sustain hardware resources.
●The problem of business use is restricted. Kudu has great restrictions on the number of fields. The self-service analysis that China Dadi Insurance is promoting often requires a large number of dimensions and indicator fields support, but the Kudu + Imapla's solution does have major shortcomings.
●Maintenance is difficult. China Land Insurance has developed hundreds of wide tables (including detailed and summarized wide tables) to meet business needs. The workload of daily updates, maintenance and iterations is huge.
Big data platform architecture for building application layer data with Kyligence
The problems mentioned above were relatively well solved in the third stage.
The third stage is to build a big data platform architecture that uses Kyligence to build application layer data. China Dadi Insurance applied Kyligence to its existing data architecture, but the overall platform architecture has not been adjusted much. It only replaced the market layer originally placed on Hadoop with Kyligence tool to implement it. Since it has not been used for a long time, it has only put some application scenarios with large visits and many complaints from users. Others, including the underlying data, storage, computing engine, and EDW models, have not made any changes. After the
Kyligence product was launched, the big data analysis efficiency of China Dadi Insurance has greatly improved, greatly improving the user experience. Let us take a practical application scenario as an example to illustrate.
China Land Insurance will conduct some assessments, analysis and data mining on salesmen. Analysis and mining refers to grouping salesmen to determine the common characteristics of high-capacity salesmen, such as education, gender, service life, etc. Based on this, the company has a clearer guidance direction in recruiting or training for new employees, and can also allow the company to estimate when the salesmen will be in high-yield period. It can be said that through a series of big data analysis and guidance applications, the salesmen and the company will eventually achieve good development at the same time.
To achieve this goal, China Land Insurance processed a table of 3 billion records, generated a 4TB size Cube, and created 40+ dimensions, including product, organization, channel, and personnel information such as age, gender, education, time of entry, etc., and 30+ indicators.
Before using Kyligence, based on the existing cluster size, the response time of user operations was at the minute level. However, after generating Cube based on Kyligence, the front-end response time was basically controlled within 10 seconds. It not only supports assessment, but also supports data analysis and data mining well, truly realizing the use of big data to guide business departments to improve production capacity and empower business.
What are the big data achievements of China Dadi Insurance?
The most important thing to determine whether a tool is valuable is to see whether it is useful. This project of China Dadi Insurance can explain its value from the perspective of IT personnel and business personnel.
From the perspective of business personnel:
●First of all, the query efficiency is high and the interactive experience is good. Compared with the fact that a report can be opened in a few minutes, the good experience of business personnel is greatly improved.
●Secondly, it truly supports independent business analysis. IT only needs to process dimensions and indicators to provide business personnel. Business personnel can analyze data as they wish, avoiding that any data analysis required by the business department must be implemented through IT, which is very dependent on IT and the IT workload is also very large.
●Thirdly, it supports massive data analysis, and there is no need to split historical data from current data. Whether you want to see historical trends or year-on-year data, it can be done through a Cube.
●Fourth, supports finer-grained analysis. When both concurrency support and permission control capabilities are improved, the platform can support more people to view and analyze data.
From the perspective of IT personnel, let’s look at the benefits of this project:
●First of all, Kyligence’s product deployment is very fast, it can be run by decompression, and the cluster deployment speed is also relatively fast.
●Secondly, Kyligence provides a better configuration interface, especially the graphical interface of many tools, which is convenient for users to use and operate and maintain management.
●Thirdly, the platform can make good isolation for resource occupation. Imapla often needs to compete with other applications for resources, which sometimes drags down the entire cluster. After using Kyligence, Kyligence's products occupy up to one-eighth of the platform's resources, which can ensure that the platform does not collapse due to the influx of users. Of course, we can adjust the resource configuration at any time as the application and users increase.
●Fourth, it is very convenient to connect with BI. We have now connected at least 3 BI tools on Kyligence: Tableau, Yonghong BI, and Cognos.
●Fifth, enjoy professional technical services, and since 2018, I have been exposed to POC and the current formal cooperation. Kyligence's technical engineers are basically on call and have very sufficient support.
China Dadi Insurance's future big data development plan
China Dadi Insurance's big data application will move in the following three directions in the future:
First, establish a unified indicator library. In the past two years, China Dadi Insurance has performed unified processing of statistical indicators as much as possible, but due to technical reasons, there are still some inconsistent indicator results. In the future, we will sort out the indicators in a round, and then use Kyligence to generate some general Cubes as much as possible to solve the inconsistency of indicators.
The second is to build a data service platform. Everyone just mentioned the Data Middle Station, and China Land Insurance is also planning. The company will build a new data service platform. In application-level data, it plans to still use Kyligence products to build a data application caliber, and at the same time unify the data application caliber, thus striving to build a platform that allows users to obtain accurate data quickly and well.
The third is to further realize the popularization of data analysis. This is the key task of China's insurance data application. In order to ensure the interaction effect between users and self-service analysis platforms, Kyligence products must also be used. China Dadi Insurance has been promoting self-service analysis for more than a year. What needs to be done now is to further promote it to ensure that data analysis is truly popularized. It is expected that all branches will better have this capability by the end of this year.
