Data gives you a pair of eyes that can see through the essence. Here is the "Data Analysis Thinking Course".
We have said before that the average cannot represent the overall level, and we have also taught you about the law of large numbers and scatter plots. Next, let’s move on to a common question: How can quickly see the general situation of a set of data ?
For this problem, we don't have to use very complicated scatter plots or text to express it. At this time, it is the standard deviation's turn to appear. Standard deviation, together with data distribution and mean, can easily describe the general situation of a set of data.
standard deviation also has a twin brother called standard error. These two brothers are indeed very similar. We will often hear that "this problem is within the allowable range of the error." It feels like once this sentence is said, it seems like this The stuff seems pretty reliable, but is it really so? Today I will tell you about standard deviation and standard error.
standard deviation
The concept of standard deviation is relatively simple. It represents the degree of spread of a set of values compared to the average. In other words, a large standard deviation means that most of the values are significantly different from the mean, and a small standard deviation means that this set of numbers is closer to the mean. I have put the formula for calculating the standard deviation of
in the appendix. The formula looks a little more complicated, but it mainly calculates the difference between each data and the mean. You often hear that the average salary in a city is We can look at the following example. Suppose the monthly salary of the two groups is roughly as follows, and the units are both "10,000".
first group: [1.72, 1.70, 1.68, 1.71, 1.69]; second group: [1.70, 5.20, 0.60, 0.2, 0.8].
You can do a simple calculation for these two groups of people, and you will find that the average monthly salary is 17,000. But it is obvious that the difference in salary between group 2 people is much greater than that of group 1 people. The first group of people all have a salary of about 17,000, and there is not much difference. Unfortunately, you are in the second group. Your monthly salary is 6,000, and you are surrounded by friends who earn 2,000 and 8,000. But in fact, there are people in your group with a monthly salary of 50,000 that you don’t know, so you will be “raised”. "Yes.
Through formulas or Excel functions (I will teach you how to calculate easily in the last chapter), you can calculate that the standard deviation of the first group is 0.014, and the standard deviation of the second group is 1.818. The difference can be more than a hundred times. If we only give you the average salary of a certain region or department every time, but don't tell you how big the standard deviation is in this region and department, then we will inevitably feel confused. "Don't worry about scarcity but worry about inequality" is still used here. very suitable.
So when looking at salary, you not only need to know an average, but also a standard deviation , so that you can know the overall salary level, your own level and where your future ceiling is.
But this concept is not enough. Assuming that for the salary unit of Group 1, I use "hundred yuan" or even "yuan" as the unit instead of "ten thousand yuan", its standard deviation will be to 1.414 and 141.4. At this time, when compared with the second group of people, it seems that the dispersion of the standard deviation is higher, but the actual data is not like this.
So generally when we are really doing data analysis, we will often use another data to avoid this problem, which is called the discrete coefficient CV (coefficient of variation). Its calculation formula is very simple, which is to divide the standard deviation by the mean ( dispersion coefficient = standard deviation / mean ), thus avoiding these differences in units or other factors. If we look directly at the data of dispersion coefficient, we can know what the degree of dispersion of and is between these sets of data.
Next time you ask about the average salary of the human resources department, you can ask more, "What is the dispersion coefficient of this department?" You will probably know the maximum salary you can get and your future increase. How much salary space will there be?
Specific uses of standard deviation
In addition to measuring the difference between specific values in a group, such as measuring the differences in our salary, height, and weight, what else does standard deviation do?
It will also be used to measure the stability of a person or a team, for example. In your common NBA, we will use average data to measure a player's combat effectiveness, such as points per game, blocks, steals, assists, etc.
At the same time, we will use the standard deviation to measure the stability of a player.
If we only look at the players who averaged 20+ points per game, LeBron-James is the most stable, with a standard deviation of 5.8 points. Looking through all his games this season, he has neither surged more than 40 points, nor has he ever scored more than 40 points. A slump of 13 points below.
Similarly, when we measure the overall sales performance of a team, we use the average. But if we want to look at the income stability and ability of team members over a period of time, we will look at the standard deviation of his recent orders. The same goes for
, which corresponds to management. For example, when I was a management programmer for CTO, I would pay attention to the rhythm of everyone submitting code. Some people just like to submit everything until the last day, and some people like to work evenly and submit in various time periods. If you look at the standard deviation of
, you will find that some people have a very large standard deviation of , and they belong to the assault type player ; some people have a very small standard deviation of , and they belong to the long-term player . For a person with a relatively large standard deviation, his risk is relatively high, because he may complete the task perfectly at the last moment, or he may procrastinate until the end of the task is not completed, and the overall average value is not reached in the end; while for a step-by-step person, Its standard deviation is relatively small, and its advantage is that it is relatively stable, but its breakthrough may not be strong enough. After learning this, you can also try to evaluate your work rhythm. What kind of player are you?
When making investments, standard deviation is also an important risk/return measure. If you look at our savings in the bank, the fluctuations in interest rates will be very small, and accordingly the standard deviation will be very small; the fluctuations in stocks will be larger, and the standard deviation of returns will also be larger; if you look at Bitcoin again, after a while Musk will change his sentence Several times, even if it falls by 30% in a short period of time, the standard deviation of currency speculation income may be tens of thousands of times the standard deviation of bank income, and hundreds of times that of stocks.
So if you put your money in the bank, the standard deviation is small and the income is stable; but if you want to speculate in currencies, the standard deviation is so large, you may make a lot of money or lose everything. Standard deviation actually represents the fluctuations in an industry. Especially when facing an investment product that you don’t understand, you can look at the historical standard deviation of this product and compare it with your commonly used investment products, and you will know in your mind Counted. For a very stable product like gold, a few standard deviations are very large. For example, when gold plummeted on April 16, 2013, Reuters analyst John Kemp lamented that the volatility of gold exceeded 6 standard deviations, which he found incredible.
When something that rarely fluctuates like gold has such a large fluctuation, and reaches a fluctuation of 6 standard deviations (the originally stable standard deviation has changed dramatically), we call this event a " black swan event" ". This event was also called the "Golden Black Swan Event" by later generations, so the next time you see a black swan event, you must know that this term is a concept derived from standard deviation.
standard error
After talking about standard deviation, let’s talk about its twin brother: standard error. We often mention the word error in our life and work, saying "This is acceptable within our error range." So what exactly does the "error range" mentioned in this sentence mean? How does it relate to standard deviation? The two concepts of
are often confused in many places, so that the standard deviation mentioned in many statistical models is actually the standard error.The biggest difference between these two concepts is that the standard deviation is based on the exact known statistical results. It reflects the degree of dispersion between individuals in a statistics. It can also be said that the standard deviation is based on specific instances. Descriptive statistics for .
and standard error represents an inferential estimate . It reflects the degree of dispersion between sample means in multiple samplings, that is, it reflects the representativeness of the sampling sample mean to the overall expected mean. It is mainly used for Extrapolate the overall situation using forecasts and extrapolations. If you still can't tell the difference between the two brothers, you can use the following two formulas to compare and distinguish.
Standard deviation = the degree of dispersion between individual scores in a statistics, which reflects the representativeness of the individual to the overall mean of the sample and is used for descriptive statistics.
standard error (Standard error) = the degree of dispersion between sample means in multiple samplings, which reflects the representativeness of the sample mean to the overall mean and is used for inferential statistics. The specific use of
standard error
standard error is often used to take out a part of the sample to judge the product quality of the overall product line, or to judge whether a thing falls into a common range.
For example, our common Six Sigma (Six Sigma) actually means that all product quality issues need to be controlled within 6 standard errors. When you hear that product quality or operation and maintenance failures are controlled within 3 9s or 5 9s, it also refers to the error range. Five nines means that 99.99966% of products have no quality problems.
is 99.99966%. How to calculate it? This involves the knowledge of normal distribution in our lecture 06. If you can’t remember it clearly, you can go back and review it again.
For example, if we use the following figure for quality control, then these values are the standard error range. For example, we say that within one standard error range, it is approximately 68.3% of the figure; within two standard error ranges, it is 95.4% of the distance from the mean (standard parts); three standard errors is 99.7%; 6 standard errors ( That is, 6-sigma) means that 99.99966% of the products produced are controlled to have no quality problems (only 3.4 of every million products are defective).
So from the standard error point of view, the stability of the system must ensure 5 9s and 6 9s. In other words, the quality control of the code we develop is 6-sigma. This quality is very good. You may not feel it yet, so let me give you an analogy to help you understand.
Handsome men and beautiful women are actually very rare in society. After all, most of us are ordinary people. Let us first assume that the degree of human beauty and handsomeness is randomly distributed (there are not so many people who have plastic surgery). If you see a beautiful woman (handsome guy) every day, then the following formula is established:
- 1 standard errors of beautiful women are about once every 3 days;
- 2 standard error beauties occur once in about 22 days;
- 3 standard error beauties occur approximately once in 370 days;
- 4 standard error beauties occur approximately once in 43 years;
- 5 standard error beauties occur approximately once in 4779 years. The beauty of
- 6 standard errors is about once in 1.39 million years; the probability of
- 7 standard errors is about once in 1 billion years.
Look at it this way, and you will know how strict 6 standard errors are. Next time you meet a particularly beautiful girl and you think she is a once-in-a-million beauty, you can say to her: "Ah, you are a once-in-six-standard beauty!" This is definitely better than Saying "you are so beautiful" is much more profound, and she will definitely admire your knowledge very much (just kidding, if you really say it, she will probably be kicked out). This way you should know what the standard error means.
Summary
OK, let’s review today’s content. Today I mainly talked to you about two concepts: standard deviation and standard error.
The standard deviation is a supplementary standard to the mean for what has happened. The standard error is a description of the degree of sample dispersion in multiple samplings and is used in inference.In the following content, we will also use these two brothers to evaluate and measure the stability of an algorithm and the quality of its implementation results.
To see whether a person, a company, or an investment product is reliable, in addition to the success rate of the person doing things, the average revenue of the company, and the profitability of the product, you also need to look at its standard deviation. It is possible that this so-called "successful person" only succeeded once and made a lot of money, but in fact he failed in everything else. This means that this person's standards are very different. It is possible that he just relied on luck and not very much. Reliable. We Chinese actually prefer the feeling of "moderate". From the perspective of standard deviation, it means that the standard deviation of doing things and being a person is smaller.
Regarding the standard error, I will give you an idiom called " be strict with oneself and be lenient with others". The first half of the sentence means that in our work and life, we should try to make as few mistakes as possible, or even make no mistakes. In this way, not only will our work be beautiful and our leaders will like it, but this concept of constant pursuit of perfection will always push us forward. You can try to apply the ideas of Six Sigma not only in work, but also in life. If you have high standards and strict requirements for yourself for a period of time, I believe you will achieve further growth. The second half of the sentence means that there is no guilt in lying down, but it is justified to struggle. We can use six standard errors to demand ourselves, but others can also use one standard error to demand their own freedom.
If you can sum it up in one sentence, hopes that you will try your best to reduce the standard deviation of your life and work, and improve your standard deviation expectations .
data gives you a pair of eyes that can see through the essence. There is no end to learning data knowledge. Let us continue to learn and encourage together.
Thinking
Have you encountered any black swan events in the past? From your perspective, how many standard errors does it range from? You are welcome to share your thoughts in the comment area and let us improve together.
Appendix: variance and standard deviation formula
Pay attention to practical education, we grow together
