Background
Jingdong App made a round of weight reduction in 2019. At that time, it carried out all-round weight reduction. However, with the rapid iteration of business and the introduction of new technology frameworks and insufficient management and control, the installation package volume rebounded. Combining the previous treatment without management and the weight-loss experience of some peers in the industry, this article introduces some practical experience and management and control plans accumulated during this weight-loss process. First, the installation package was split and sorted out, and the component data of the installation package was made into an online dashboard. Based on the data, the main factors for the growth of the installation package volume were found. Based on the current situation of JD.com App, a series of comprehensive slimming actions were carried out, and a new set of management and control specifications and management and control platforms were also precipitated. Finally, the installation package volume was reduced by more than 30%.
Installation package component analysis
Slimming has not moved, "data" comes first. The first step of package slimming is to analyze the size data of the components of the App, to provide data support for package slimming goal setting and task dismantling, to achieve the purpose of measurability, online, and unified data standards. How to analyze the package components of Android App? Android developers will basically use the Apk Analyzer tool that comes with Android Studio. You can drag the Apk file into Android Studio to view it. The picture below is the package size data displayed in Android Studio after dragging the JD App into it. It can be clearly seen that the main files that account for the largest proportion of the package volume are lib (storing dynamic libraries and Android plug-ins), r (storing pictures, xml and other resources), assets directory, dex and resources.arsc (resource mapping table). The
apk Analyzer tool is very convenient for viewing preliminary and specific package size data. However, for the package slimming project that needs to be carried out, the data is too general and is not divided according to the module dimension. The responsibility for package slimming cannot be assigned to the corresponding R&D team, which is inconvenient for the specific implementation of subsequent package slimming projects. In order to measure the impact of each module on packet size, we need a refined packet size analysis solution.
First, let’s take a look at the engineering structure of Jingdong App. The simplified model is shown in the figure below. Jingdong App adopts a plug-in architecture as a whole. During the development and iteration process, it is gradually divided into many modules according to functions and business types. Each module has its own project project and warehouse, and is independently developed and maintained on a daily basis, and is finally integrated into the Application project. Functional modules are basically introduced into the project in the form of library dependencies, such as commonly used network libraries, image libraries, buried point libraries, etc.; business modules mostly exist in the form of plug-ins, such as search, business details, shopping carts, etc.; in addition, there is a small amount of historical legacy logic that has not been split into modules in the main project, and it accounts for a very small proportion in daily iterations.
Based on the current modularization of JD App, the component analysis of the installation package can be carried out from three aspects: analyzing the overview data of the package components, analyzing the plug-in data and analyzing the library data.
1
Apk overview data analysis
The Android package product apk file is essentially a zip file. You can use the zipinfo command to output the detailed information log of each file in the compressed package. Usage:
Zipinfo -l --t --hxxxx.apk The log file output by xxxx.txtopens as shown below. Each file has one line of compression information, including file name, original size, compressed size and other indicators.
parses the above log information line by line, and performs classification statistics based on the deobfuscated file name path and file type. It can obtain the overview information of the apk, including the number of various types of files, total size, single file size and other indicators, and establish a file size index.
2
plug-in data analysis
The essence of the plug-in product is also an apk. Currently, there are more than 50 pre-installed plug-ins in the JD App. When the App is packaged, each plug-in will be placed in the lib directory with the .so suffix.The analysis of the plug-in can be directly based on the package product. When parsing, you only need to decompress the apk, separate the plug-in from all so files according to the file name characteristics, and analyze them one by one according to the same method as the zipinfo mentioned above.
3
library data analysis
Each library module is mostly introduced into the project in the form of aar/jar dependency. Currently, there are more than 300 dependencies in all JD App projects, including direct dependencies and indirect dependencies. The analysis of library data mainly focuses on two aspects: analyzing the impact of each dependency on package size and analyzing and maintaining developers.
1
Analyze the impact of dependencies on package size
When the Android project is built, the Gradle build tool will automatically synchronize the dependent libraries to the local cache, compile merge together with the code resources in the project, and finally output the apk. Aar/jar dependency files can be obtained during the build process by writing gradle script tasks. However, considering the independence of the analysis tool, try to avoid coupling with the packaging process and adopt a method of self-parsing dependencies. The overall process is as follows:
a. During the packaging process, execute the gradle command to generate the app runtime dependency tree log file. Usage:
./gradlew:app:dependencies--configuration xxxReleaseRuntimeClasspath The part of the log file generated by dep.txtis as follows:
b. Parse the log file to generate the N fork tree model. Each line of log generates a dependency node, including gr oupId, artifact_name, version information, - in the log indicates a version conflict and takes the higher version, (*) indicates repeated dependencies. The dependencies added in application and common_library are direct dependencies, corresponding to the child nodes of the root node and common_library node, and other nodes are indirect dependencies.
c. Flatten the N-ary tree to remove duplication, parse the dependencies one by one, and download the aar/jar file. During the parsing process, the URL polling warehouse is spliced. In order to speed up the parsing, the source of the dependency can be distinguished according to the dependency groupId. If it is a dependency within jd, the internal private server will be requested first, otherwise the commonly used mirror warehouse will be accessed first. In addition, the LRU caching mechanism can be added, so that locally existing dependent products no longer request the warehouse, further speeding up the parsing speed. A potential problem is that the dependencies of the snapshot version will correspond to multiple products. The latest version taken during parsing may not be the same as the one used when the app was built. Usually, dependencies on the release branch are stipulated not to use the snapshot version, so this problem is indirectly avoided.
d. aar is also a zip file. You can also use the zipinfo command to parse the component size and establish an index of internal jar, so, pictures, xml, asset and other files.
e. By associating each file within the aar according to its name with the file size index established during the overview data analysis, you can know the final size of the so, pictures, xml, asset and other files inside the aar in the apk compressed package. Since the code file jar is eventually compiled, deleted, obfuscated and merged into dex, the source cannot be accurately traced. The size of the code part is calculated using a proportional conversion method from the original size. The conversion ratio is the difference in package size after removing relatively independent dependencies in the experiment. After removing the size of the non-code part, the conversion ratio of the code part can be calculated. Repeat N times to remove different dependencies and take the average.
2
Head of the analysis module
The aforementioned Jingdong App has 300+ dependencies. Due to business adjustments, personnel changes and other reasons, there is no unified library to maintain relationship data.For the direct dependency part, the git blame command is used to output the build.gradle file modification log of the app and common_library project in the early stage, and the corresponding relationship between the dependency and the maintainer is matched regularly; for the indirect dependency part, each indirect dependency node in the aforementioned N-tree is searched upwards for its parent node until the direct dependency node, then the developer of the direct dependency is also the person responsible for the introduction of this indirect dependency. There will be situations where multiple direct dependencies reference the same indirect dependency, that is, a certain indirect dependency library corresponds to multiple introduction responsible persons.
Through the above analysis solution, we can completely analyze the size of each component of the installation package and the corresponding person in charge, providing data guidance for the implementation of weight loss work. The analysis plans are all implemented using python scripts and have been integrated into the company's internal CD system. The effect of the data dashboard is as follows.
The developer team of JD.com App is relatively large. After the online construction of the installation package component data is completed, all developers have unified the data viewing standards and channels. The growth and decline of each App version can also be attributed to the specific module level and the corresponding developer. At the same time, this data analysis dashboard also provides basic data capabilities for weight loss measurement and control. According to data analysis, we found that the number and size of resource files, ReactNative, and plug-ins in the App are relatively large, so the slimming focus is on special governance in these aspects. At the same time, based on the lessons learned from previous slimming rebounds, the analysis found that there was a lack of constraint control in past development iterations, and technology selection and R&D reuse also exposed some problems. Therefore, refined management and control specifications were formulated and implemented in the research and testing process.
Slimming plan
Draws lessons from the industry’s slimming measures and combines it with JD App’s own characteristics. We first carried out regular weight-loss measures, including compressing resources, converting built-in ReactNative to download, converting plug-ins to download, R8 compilation and upgrade, etc. Secondly, we carried out special management of the images in the App and built an image management platform. In the end, the size of the Android installation package was reduced by nearly 30%+. The package volume reduction trend is as shown below.
1
weight loss measures
- plug-in transfer download
Jingdong mPaaS platform provides the ability to download and install Android plug-ins. In early 2022, we worked with the mPaaS team and the operation and maintenance team to optimize it and improve its usability and stability. Set the downloader to unknown The host problem was solved, the httpdns capability was added, and the ability to switch domain names was added. The operation and maintenance team solved the file md5 inconsistency problem caused by some cdn node abnormalities through dial testing. Finally, the average single download network request cost in a real user network environment The efficiency is above 98%. Since there are multiple opportunities to trigger downloads before the user enters the page, the probability of entering the page without successfully loading is less than 4/100,000. This probability is lower than the crash rate of the App. The experience is completely guaranteed, and no abnormal user experience feedback has occurred during practice. Sorted in reverse order through the uv of the business module, and finally selected 10+ plug-ins for download and installation.
- RN built-in package transfer download
At the beginning of optimization, most of JD.com’s business modules developed with React Native used built-in apps, with the peak number reaching 50+. We worked with the mPaaS team for optimization. We initially evaluated using Brotli to compress local built-in packages. It was evaluated that the decompression time was twice that of gzip, and the memory usage was tens to hundreds of times. This solution was eventually abandoned. Unified adopts the solution of downloading and installing built-in packages, reduces the size of JSBundle by splitting basic packages and business packages, and sets up pre-download non-mandatory updates, pre-download mandatory updates, and opening a certain RN business will trigger the download of other undownloaded services. This effectively improves the coverage of new versions in a single week, and provides h5 downgrades, unified all-in-one downgrades, etc., ultimately improving the stability and availability of RN-related capabilities and ensuring user experience. Currently, the number of built-in RN business modules has been reduced to single digits.
- resource compression
For image compression, use the integrated TinyPNG compression tool to compress images above 10K; limited by the Android project minSdkVersion=16, all png images cannot be converted into webp format, and only images without transparent channels can be converted into webp. In the future, if there are stricter requirements on package size, you can consider remoteizing the pictures.
- iconfont
used the slimming script analysis tool to scan the small, square, solid-color pictures in the installation package and convert them into iconfonts. More than 800 pictures were scanned in the package and could be converted into iconfonts. A large number requires long-term management. Although the benefits are not obvious, it is beneficial to the unified management of scattered small icons for each business, and it is also conducive to the unification of the overall visual style of the App.
- R file inline
uses ASM to perform bytecode operations on the class file through a custom Gradle plug-in, replaces the R class resource id referenced in the class file with the corresponding constant value and deletes the R file, thereby reducing the package size. Since JD App adopts a plug-in approach to business development, each business plug-in will reference public resources, and public resource IDs cannot simply be added to the whitelist. Therefore, in combination with the public resource ID fixing capability of JD App plug-in solution (aura), public resource inlining in the plug-in project can be realized. Currently, R file inlining has been enabled in grayscale and will be launched in the future. The overall package volume benefit is more than 5.5MB.
- enables R8 compilation.
is adapted and upgraded with Target31 to upgrade AGP to version 4.1. R8 compilation is enabled simultaneously. Please note that R8 will ignore some Pr. oguard obfuscation rules. In response to the changes in obfuscation rules in R8 compilation, a mapping file detection script tool was written to compare the mapping obfuscation files generated by Proguard and R8 compilation, and detect whether the correct obfuscation rules were added to classes related to network analysis and reflection. The tool was used to ensure the stability of the App after upgrading to R8 compilation. R8 compilation is better than Proguard in optimizing code obfuscation. After turning on R8, the package size is reduced by 1.6MB, and the build time is also reduced by more than 30% (3 minutes).
- Zip
compressed
After completing these conventional slimming program optimizations, the size of the App installation package is still large. Previously, image resource optimization was mainly carried out through image compression, using a set of resolution xhdpi images, converting images to webp, and confusing resource names. These optimization methods all slim down the built-in images. With the continuous iteration of business functions, the benefits of slimming will become less and less. The development method of each business of Jingdong App is in the form of plug-ins. Through the analysis of these business plug-in packages, it is found that the size of the built-in picture resources in these business modules accounts for more than 25% of the total size of the respective plug-in packages. In order to solve the problem of too many built-in picture resources, an optimization plan for the remoteization of picture resources is proposed.
2
Remote image resource
Directly converting image resources to online loading may affect the user experience and strip the built-in image resources from the App , and then upload the image to CDN to obtain the network link of the image. The application displays the image by loading the image link online. Online loading of the image requires the process of downloading the image. Considering that the network environment where the user is located is affected by various factors, the default bottom image is often used to place a placeholder in the image display area, or a full-screen loading animation is used for transition to optimize the user experience.From the perspective of optimizing user experience, an optimization plan of 2-layer network loading + 3-layer downgrade measures is proposed, which can improve the loading success rate of network images, and at the same time give users the same experience of displaying images as loading built-in image resources in the application. The overall solution is as follows:
This optimization solution mainly consists of an image management CMS platform and a client component that obtains image information. The business R&D configures the image information of each module in CMS, and the client obtains the image address and displays the image through the component.
1
CMS image management
- CMS supports image types including png, jgp, webp, dot 9 images, etc., by industry The service module sorts out the built-in image materials;
- CMS creates a business module based on users, uploads images in the module to obtain image CDN links, and provides two resolution settings of 2x and 3x. At the same time, you can set the version range for the image to be effective on the client and the effective platform Android or iOS;
- CMS provides the function of packaging pictures into zip compressed packages. After the user uploads the pictures, it will generate zip packages with different version ranges and 2x and 3x resolutions according to the client version range in which each picture takes effect. It will issue zip package links with different resolutions according to the resolution parameters in the client request interface;
- CMS provides pre-added zip packages The loading strategy is divided into preloading when the App starts, preloading x seconds after the App homepage is loaded successfully, preloading when entering a certain page, etc.;
- After the user completes the configuration according to the previous steps, the configuration information composed of the image name in the business module and the corresponding CDN link is exported by the CMS, and then the configuration information file is built into the client to provide a complete experience following the installation package.
2
client image preloading and caching
- . The client component requests the query interface provided by the CMS background when the App is started, obtains the configuration information corresponding to the current client version, compares it with the local configuration information, and persists the obtained differential information locally; the
- client component also parses the image zip package information issued by the background interface. Download the corresponding image zip package according to the preloading strategy of the zip package. After the download is completed, decompress it to a local folder named after the module name;
- When displaying an image, the client component first queries whether the image already exists in the local zip package decompression directory based on the module ID + image ID. If the image already exists When downloading, the local path path of the image is directly returned, and the client directly uses the local path path to display the image; if the image has not been downloaded yet, or does not exist, the CDN link of the image is obtained from the built-in configuration information file, and the network link of the image is loaded as before; if loading of the network link still fails, the default bottom image is displayed.
In this optimization plan, the first layer of the 2-layer network loading refers to the zip package of the business module, which is downloaded once and reused in multiple versions; the second layer of network loading refers to loading the image network link to display the image through the image CDN link configuration information built in the installation package when the image zip package preloading is not completed or fails. The configuration information is saved in the form of image name + image CDN link key-value pair. The first level of downgrade of the
-layer downgrade measure is to obtain the local cached image preloaded in the image zip package; the second level of downgrade is to obtain the built-in image CDN link to load when the local image cache fails; and the third level of downgrade is to use the default bottom-up image. When the slimming business module turns on the image zip preloading function, CMS supports the business module to set the zip package preloading strategy. The business module chooses different loading strategies according to the actual situation. For example, some business module page entrances are deep, and these business modules can choose to trigger the zip package preloading when the user enters the page;
Some business modules can set the page entry to be shallow and can be preloaded when idle after the App is started. Other secondary and tertiary pages can be set to start downloading x seconds (random time) after the App is launched.By setting these preloading strategies, image zip packages are downloaded in different time periods to reduce the peak network traffic when images are loaded.
The above solution requires the active cooperation of business research and development, and requires the business module to load images by giving priority to the local cache through a fixed module ID + image ID. The use process is cumbersome and lacks flexibility. In order to make it easier for developers to use, provide more choices, and reduce access costs, we have made some improvements in the CMS and client, and proposed the following optimization plan:
- Image CMS change point
CMS background does additional processing for image zip packaging: calculate the MD5 value from the fixed path in the image CDN link, and rename the image name based on the MD5 value, without retaining the image type suffix. For example, link: https://xxx/jfs/xxx/name.png, calculate the MD5 value of the path string between the beginning of jfs and .png in the link: MD5(/jfs/xxx/name) =MD5-Value, use the MD5-Value value as the image name, and remove the image type suffix (.png, etc.). Other images are also renamed according to the same steps, and finally packaged into a compressed image package named after the module name.
- client change point
provides a tool class that takes the module name and image CDN link as input parameters, performs the same processing on the image CDN link, and obtains the fixed path character in the CDN link. MD5 value of the string, and then check whether the local cache of the image already exists based on the module name and MD5 value, that is, whether the image zip package preloading is successful. If the image has been successfully cached, the cache path of the image (file:// protocol) will be returned to the image loading framework. If the image cache does not exist, the image CDN link (http:// protocol) will be returned as is. It can also be implemented in combination with the image loading framework, and a new image loading attribute with the business module name as an alias is added. This alias attribute is the local directory where the business module image is cached. If this attribute is set, the image loading framework will first determine whether the image has been cached successfully. If the image has been cached successfully, the image will be displayed through the local path. If the cache is not successful, the image will be loaded directly online. This solution requires the client image loading framework to support both the file protocol and the http protocol. This solution can also be applied to image preloading in scenes such as big sales.
Check the success rate of zip package preloading by buried data. The picture zip package preloading is downloaded x seconds after the configuration homepage is started. The zip package loading success rate on iOS fluctuates between 98.9% and 99.1%. The zip loading success rate on Android fluctuates between 96.7% and 98.1%. , the remaining 1% to 3% of users load online through the built-in CDN link, and the success rate of displaying images when the two are combined is close to 100%; the efficiency of the client loading images from the local cache is higher than loading images online from the network, eliminating the need to wait for the image download process, reducing the display probability of the default pocket image, and the experience is close to the effect of loading local built-in images.
Control
Jingdong App rebounded by 50% after a period of time after a special slimming down in 2019. On the one hand, it was due to the rapid iterative development of the business and the introduction of some relatively large basic libraries. On the other hand, the reason was that it focused on slimming down but did not do in-depth control and access to prevent deterioration. Therefore, in the new round of weight-loss optimization process, while doing weight-loss management, we are exploring a normalized management and control mechanism, and finally settled a set of management and control specifications and management control platform. The purpose of
control is not to limit demand iteration and increase code. The purpose is to control and allow reasonable code to be put in, unreasonable rejection and obsolete code to improve developers' awareness of weight loss. Management and control have put forward new requirements for the entire industry-research link, which will impose constraints on the previous extensive demand development iteration methods. The product side needs to cooperate with the R&D side in a timely manner to take the abandoned experimental code of ABTest offline. The R&D side needs to do a good job in technical planning and technology selection, streamline and reuse code as much as possible, strengthen collaboration between technical teams, and not place excessively large or redundant resource files, etc.The premise of
control is that the App has been fully componentized and decoupled, and the scale of App developers is relatively large. For example, the current Android version of the JD App is composed of business plug-ins and AAR dependency libraries. The components of the App are represented by a configuration table as a whole. The configuration table describes the configuration version information of all components. The premise of whether a component is allowed to be integrated into the App is that the component's volume growth meets the specifications we have established. Only then can the latest version of the component be integrated into the configuration table. The core mechanism of our control is to control the update and integration permissions of the configuration table to constrain the developers of each component. Through the public control specifications formulated by the App Architect Committee (a virtual organization formed by architects within JD.com), we will judge whether the volume growth of each component complies with regulations. The configuration table will be updated if it complies with the rules. Otherwise, the abnormal application process will be followed.
1
management and control process and plan
The control process is accompanied by the entire process of development, testing, and release. The overall process is as shown in the figure. After the developer completes the development, the developer will release a new component version on JD.com’s internal component construction and publishing platform (mPaaS). After the construction is completed, the developer will package and test based on the temporary configuration table on the packaging platform (Bamboo). During this process, the volume change data of the corresponding component will be calculated based on the package, and the packaging system will synchronize the data in the form of email. To the developer, inform the developer as early as possible whether there is any non-compliance; after the
test is passed, the developer applies to integrate the component version information to the configuration table. If the rules are complied with, the configuration table is updated and the process is completed. If it is not compliant, the developer can optimize the code and then apply for integration, or apply for the exception approval process to the App Architect Committee. Most regular business iterations will not cause violations, but there will also be some abnormal situations. The handling of exceptions often generates more disputes. The volume control of the installation package developed needs to withstand the pressure of products and business. The exception handling process needs to be objective, transparent, and flexible. The exception handling process is shown in the figure.
If a certain component violates the rules and initiates an application, it can be reviewed and approved for third-party libraries, nationally required privacy rectification, Android system upgrade adaptation, etc. Other special circumstances can also be passed, and will be controlled by the App Architect Committee at its discretion. For some normal circumstances where component volume growth exceeds the limit and cannot be optimized in the short term, you can apply to set the slimming back after n versions in the future, or component A negotiates with components B and C to help share the slimming back. The final slimming plan will be judged by the App Architect Committee for reasonableness. Subsequently, within n versions, the platform will automatically calculate the phased progress data of the slimming and inform the relevant personnel who initiated the slimming; if the target of the slimming plan is not achieved after n versions, special offline discussions will be held, the plan will be re-formulated, or some public punishment will be imposed.
Determining whether a component is compliant or not depends on the control specifications. The control specifications are discussed and formulated by the App Architect Committee. Initially, the control specifications we set were relatively rough: the current version of each component should be less than or equal to n relative to the previous version. KB (initially n was set to 100), as the slimming progressed to the later stage, it was discovered that it had many disadvantages. For large modules, it is often the collaboration of multiple developers that often leads to violations. For small modules, there are often fewer participants and less frequent iteration changes, so the specifications do not work for them at all.
Based on this situation, we improved the specification: the components are divided into two categories: A1 and A2. A1 category components = n MB, and A2 category components n MB. For A1 category components, the annual growth rate should be less than or equal to x%, and for A2 category components, the single version increment should be = yKB. A1-type modules generally have high iteration frequency and have many developers, and each version has more room for adjustment if the annual increase is limited. For A2-type modules, the iteration frequency is generally not high, and setting a smaller increment of a single version can also constrain its growth accordingly; whitelist processing is set for some third-party libraries and public libraries; from an overall perspective, management and control specifications are basically fair and reasonable.Combined with the actual situation of the App, by reasonably setting the n, x and y values, the increase can be controlled within the planned range.
Summary
With the development of business and user needs, the size of Apps will maintain an upward growth trend, and all larger Apps in the industry are slimming down. We learned from some excellent weight-loss programs in the industry, combined with the actual scenarios of JD.com App to carry out a comprehensive weight-loss program that combined multiple measures, put forward the idea of both governance and control, precipitated a set of management and control specifications, and implemented them into the R&D and testing process system. With the iterative updates of new technologies, new weight-loss programs and measures will continue to emerge. We will continue to keep track and welcome readers to communicate and make suggestions.
[Reference materials]
. 5 years of sharpening a sword | Complete solution to Youku’s Android package slimming management ideas: https://developer.aliyun.com/article/953463
. Douyin package size optimization-resource optimization: https:// juejin.cn/post/6844904106696376334
. Google reduces application size: https://developer.android.google.cn/studio/build/shrink-code#shrink-resources
. Android App package slimming optimization practice: https://tech.meituan.com/2017/04/07/android-shrink-overall-solution.html
, Baidu App Android package volume optimization practice: https://mp.weixin.qq.com/s?__biz=MzUxMzk2ODI1NQ==&mid=2247485123&idx=1&sn=09e7fb181f8d1e8d0736931c42be0707&chksm=f94 c57d3ce3bdec52c78db8b86e1fdac1f4564ccff94382aec1e5fb060b543ab134de7ac9c3d&scene=21#wechat_redirect
, R8:
https://r8.googlesource.com/r8
Author: App Slimming Team
Source: WeChat public account: JD Retail Technology
Source: https://mp.weixin.qq.com/s/sSpbbq-v04CAB5ifKmC20g
