The Theory of Pilot Ability Evaluation from the System Perspective

: The ability evaluation of pilots is the basis of the three basic construction, and the quantified pilot abilities are required as basic data in various aspects such as precision training, qualification management, and safety management. The International Civil Aviation Organization (ICAO) has published 9 core competencies of pilots (manual flight control capabilities, automatic flight control management capabilities, program execution capabilities, the ability to master and apply knowledge, communication skills, problem-solving and decision-making capabilities, situational awareness ability, fatigue management ability and team management ability), but ICAO did not elaborate on how to quantitatively evaluate these nine abilities, and the current industry’s common practice is mainly to adopt manual evaluation methods. Subjective factors are too large to be objectively, fairly, scientifically and comprehensively conclusions.


Concept of flight
1) The generation of flight "The true beginning of human mechanical flight is marked by a maneuverable aircraft." The method and skill of the pilot using the mechanical control system to control the power and lift of the aircraft has become the "all problems" of the pilot in controlling the aircraft. Even today, the concept of "state flight" is still quite common. It is believed that the pilot controls the motion state of the aircraft in the air, and the problem that occurs when the aircraft is in a controllable and trouble-free state is the pilot's problem. Because of the indispensability of the pilot in mechanical flight, the adaptability between the pilot's flying skills and the aircraft has become a key factor in flight.
2) The conception of system Bertalanffy who is the system research pioneer showed that the definition of system is "a complex of interacting multiple elements [1] ". "The system is relatively divided from the network of universally connected objective things. The system is one of the ways of existence of all things. Everything is a system and has a structure. In this structure there is interaction [2] ".
Flight is the result of the interaction between the pilot and the aircraft. Even to this day, the flight of a "maneuverable, heavier-than-air aircraft" still requires the pilot's manipulation and control. The pilot is an indispensable component of flight. According to the system point of view, it can be said that the beginning of human mechanical flight marks the birth of the flight system. Flight is a system composed of pilots and airplanes, which we call it the flight system.
When "flying" refers to the movement of an airplane in the air, what "flying" emphasizes is the state of the airplane's movement. This system can be expressed as:

 
Fly System= Plane Pilot ， 3) The conception of flight from different perspectives a) Flight in the view pilots Flight is "the process of using airborne equipment to control the motion state of the aircraft and satisfy the flight path". The pilot's flying skills emphasize the ability to control the performance of the aircraft, to be able to control and maintain the motion state and trajectory of the aircraft proficiently, accurately and stably. b) Flight in the view systems The flight system consists of the pilot and the aircraft. Flight arises from the interaction between the pilot and the aircraft, and the flight of the system can be defined as "the movement of the system composed of the pilot and the aircraft in the air".
The so-called system is a "complex of interacting multiple elements". The meaning of this definition is: any system is the sum of the feature set and the relation set. "The whole is greater than the sum of its parts" means that the system is a whole with its own structural relationship, not a simple "sum" of components. The flight system is an independent whole relatively divided from the network of universally connected objective things. The formation of the flight system is the existence of the relationship between these elements.

1) Flight is a dynamic process
Dynamic is the attribute of flight, and the meaning of flight is in the dynamic process. Airline flight consists of different flight phase. This flight process is a process of interconnection, mutual influence and interaction between the aircraft, the pilot, and the route, as well as between the system and the environment. This means that the crew always keeps "flying" during the flight to deal with the problems during the flight. If you rush to pay attention to the problem itself, you will lose the "flying" conditions and status. Factors such as mechanical failures and bad weather in flight are handled in dynamic flight. First of all, it is necessary to ensure that the aircraft has the conditions and capabilities to "fly", and then deal with the failures in a safe environment. The correctness and timely handling of the crew will directly affect the accident, especially in the critical phase of the "take-off and landing phase." 2) Flight is changing Change is another attribute of flight. There are no two identical flights in the world. For the crew, they often fly on different routes, airports and encounter different weather conditions. The differences in airports and approach methods are a test of the pilot's skills; even on the same route. It can also be different aircraft and different weather conditions. These different external conditions require different technical skills and procedures for the pilot; the "unsafe state of objects" during the flight is analyzed and analyzed by the crew. The time for evaluation and decision-making is very limited. Maintaining the proficiency, correctness, accuracy and stability of flight technology are the basis of safety. For airlines, the combination of pilots with different technical backgrounds and personalities that the crew members often change may pose a risk to flight safety. Trust or distrust that is not based on the principles of crew working procedures can become the risk of flight safety.
Flight is dynamically changing, and the ability of the flight system is also unstable. This uncertainty comes from both the crew's human factors and the uncertainty of environmental factors. Human abilities contain the most difficult to quantify and estimate factors, such as fatigue, motivation and attention.

Flight system
The flight system is formed in the environment, and operates, evolves and develops in the environment [3] . The evolution of the flight system is a process from simple to complex, from singularity to hierarchy, and from disorder to order.
Aviation technology is the main line of flight development, and the development of flight is also the development of systems. In the course of flight, any change of any element will affect the entire system, which is determined by the nature of the system. Therefore, the research flight needs to start from the nature of the system, based on the flight of the route, analyze and study the individual to find out the characteristics and rules of the flight. On this basis, researches on the identity, difference and level of flight systems at different levels are carried out.
Flight is an artificial thing, and the artificial factor is a complex factor. The man-made factors of the flight system include the technical factors of the pilot, the behavioral factors of individuals and organizations, and the physical and psychological factors of people. It is precisely because of human complexity that aviation safety is a complex system engineering.

The structure of the flight system
The airline flight is the result of the functionalization of state flight. The function of airline flight is transportation, which refers to the process of an aircraft from the starting point to the destination along a prescribed route. The flight system of route transportation can be expressed as: Flight is the process of the relationship between the flight system and the environment. Therefore, airline flight refers to the process of interaction between the pilot and the aircraft and the environment along the prescribed route, that is, the process of the relationship between the flight system and the environment.
The structure of the flight system includes frame structure, operation structure, space structure and time structure.

The ability of the flight system
The ability of the pilot is related to the technical capabilities of the aircraft, the route technology, and environmental factors. The capabilities of crew are related to the management of aircraft, routes and environmental factors and changes, as well as the management of the flight process. The ability of the flight system is not determined by the ability of one components, but is formed by the interaction and mutual restriction of the three. That is to say, the part that the system must adapt to is the concept of intersection.
System capability is the ability of the system to complete certain tasks or certain activities assigned or assigned by the system. The ability of a system usually refers to its comprehensive ability. The comprehensive ability of the system is based on the ability of each component of the system, and is an organic combination of the partial capabilities of the components, such as the "bucket principle".
The system capability of airline flight is the capability calculated by the airline based on the capability of the aircraft, the technical requirements of the route, and the weather conditions.
The route is the route between the starting airport, the destination airport, and the alternate airport. S: represents the system; E: represents the environment, it is the day and night, wind and rain, haze, ice and snow, as well as air pressure, altitude, terrain and other factors, as well as air traffic capacity and separation factors. The technical ability of the pilots can also have the ability to land safely in a complex environment. Weather conditions such as strong wind, rain, haze and low visibility, short and narrow runways and non-standard runway lighting conditions for night flights.
The diversity and hierarchical nature of flight technology requires more time and training for pilots to be fully proficient and accurate in mastering these technologies, which makes it more difficult to maintain a variety of technical capabilities.
b) The management capabilities of crew Airline flight is carried out by the crew in accordance with the flight plan established by the airline. Before the flight, the crew needs to analyze and evaluate the aircraft technology, route technology and environmental conditions, as well as their own qualifications and capabilities, and determine how to execute the flight plan. During the flight, the crew must make correct judgments and timely disposals in a timely manner based on changes in the system and environment, and always maintain the relationship between the system and the environment in an "acceptable state." The crew must also have the ability to divide, collaborate and communicate among crew members based on the principle of division of labor between PF and PNF. The indicators to measure crew management capabilities include: planning, predictability, order, analysis and decision-making, and disposal. c) System capabilities at different levels There are individual differences among pilots, and there are wide differences among crews composed of different individuals. There will be differences in crew capabilities for different combinations of individuals.
The scale of the air transportation system is composed of the number of aircraft, the number of pilots, and the number of routes, while the capacity of the system is determined by the relationship between the system structure and the environmental system in which the system is located.

The pilot evaluation
The International Civil Aviation Organization (ICAO) has published 9 core competencies of pilots (flight manual control ability, flight automatic control management ability, program execution ability, ability to master and apply knowledge, communication ability, problem-solving and decision-making ability [4] , situational awareness ability, fatigue management ability and team management ability). These nine capabilities include both technical and non-technical capabilities. Technical capabilities are biased toward the pilot's perspective, while non-technical capabilities are biased toward the crew's perspective. The two complement each other and merge with each other, but for the convenience of description, we divide the quantitative evaluation of these nine abilities into two dimensions: the pilot dimension and the crew dimension. The pilot dimension includes accuracy, stability, proficiency, and adaptability; the crew dimension includes planning, predictability, and orderliness.

The capability evaluation of pilot dimension
From the perspective of the pilot, flying is a sport, and the task of the pilot is to maneuver and control the aircraft correctly to ensure that the aircraft status continues to meet expectations. The evaluation of this dimension includes four aspects: accuracy, stability, proficiency and adaptability.

1) Accuracy
Although there is no standard flight, there is a standard for flight. The accuracy dimension is the evaluation of the pilot's ability to comply with the standard. For each flight stage of each aircraft type, there are standards for technical points publicly released by the industry or manufacturer. The evaluation method of this dimension is to compare the actual value of the flight with the standard value according to the technical points of different aircraft types. The smaller the gap, the better the accuracy.
In mathematical expression, for a certain technical point, record the actual value as a , record the recommended value as b and set a maximum deviation value c for this technical point. When 0 ba , the score of this technical point is 100, when 0 ba , the score of this technical point is 0, when ba  is between 0 and c , the score is calculated by linear interpolation.
Accuracy is not only for a single evaluation but also for a series of flights. For a series of flights, taking the median of the 95% confidence interval of the actual flight value of this series and substitute into the above formula to participate in the calculation.

2) Stability
Stability is the predictability evaluation of a pilot in a certain technical point, and is completely evaluated separately from accuracy. Stability is mainly aimed at the stability of the actual performance of a certain technical point under a large number of flights. The better the stability of a certain technical point, the more predictable the pilot's performance on this technical point. The predictability dimension has extremely important reference value for flight safety management.
In mathematical expression, for a certain technical point, the entropy value is calculated according to the actual value of a series of flights, and entropy is used to represent stability. When the entropy is 1, it is 100 point, and when the entropy is 0, it is zero point. The score is calculated by linear interpolation between 0 and 1.
Since entropy can only express the degree of dispersion of the data, it cannot describe the extent of the deviation of the data. In order to make up for this defect, we have added a large deviation correction to the stability dimension. That is, the ratio of c that exceeds the definition of the accuracy dimension is used to make weighted corrections to the stability score.

3) Proficiency
During the flight, due to various internal and external factors, the state of the aircraft will inevitably deviate from the acceptable level. At this time, the pilot needs to correct these deviations, the proficiency dimension is the evaluation of the pilot's ability to correct deviations. The proficiency dimension is divided into two small dimensions, one is the accuracy of the correction action, and the other is the effectiveness of the correction operation. For the accuracy of the deviation correction action, it mainly depends on whether the deviation correction action meets the requirements of the procedure, such as the correction of excessive airspeed caused by up-windshear during the descent phase. The effectiveness of the corrective action depends on whether and to what extent the aircraft's condition has been improved after the corrective action is applied.
In terms of mathematical expression, the actual action is compared with the prescribed action for the dimension of whether the corrective action is accurate. If the action is consistent, the score is full, and if it is inconsistent, the score is zero. For the effective negative of the correction action, it is expressed by the number of corrections from the start of the deviation to the completion of the correction. The smaller the number, the better the effect of the correction.

4) Adaptability
Flight is a complex system engineering. Affected by various external factors, there are no two flights with exactly the same external conditions in the world. An excellent pilot should have the ability to successfully complete aircraft driving tasks under different conditions, which is adaptability. The adaptability dimension considers the breadth of the pilot's technology, which is a relatively comprehensive indicator. The breadth is not only affected by technology, but also by psychological quality.
In mathematical expression, first, list some special scenarios as needed, such as crosswind landing, night landing, non-precision approach, turbulent landing, tailwind landing, tailwind takeoff, high altitude airport operation, etc., then the corresponding flights under these scenarios are scored separately, and these scores are subtracted from the full flight scores regardless of special scenarios. The smaller the difference and the standard deviation of the difference, the better the adaptability, that is, the broader the technology.

The capability evaluation of crew dimension
As mentioned above, flight from the system perspective is a system engineering. The flight crew in this perspective is not just an airplane pilot, but a temporary team. In addition to the driving skills required to perform the functions of a pilot, the flight crew also needs possess many abilities in non-technical dimensions, and this part of the abilities is more reflected in the management level. After refining, we use the following three dimensions to measure, namely planning, predictability and order.

1) Planning
Airline flight is carried out by the crew in accordance with the flight plan, and the release of a flight is made by the dispatch and the crew [5] . Therefore, the crew is responsible for making a scientific and reasonable flight plan based on the actual situation and the dispatch before the flight. The so-called planning is to evaluate the rationality of the flight plan that the crew participated in before the flight.
In mathematical expression, we use the deviation of actual flight conditions and flight plan to measure. Specifically, it includes trajectory profile (height and horizontal trajectory), oil volume profile, velocity profile, time profile and morphological profile. Then find the cosine similarity between the plan and the actual profile. The larger the value, the better the planning.

2) Predictability
As mentioned above, flight is a dynamic process that is constantly changing, and an excellent pilot should have the ability to make timely amendments to the plan based on actual conditions to make up for the deficiencies of the initial plan. The foreseeability dimension measures the crew's ability to make small-scale revisions to the plan. From this point of view, foreseeability is similar to planning, and both plan future actions based on known information, but the scale of planning is more predictable. Larger, predictability is more inclined to measure short-term, small-scale corrections.
In terms of mathematical expression, the dimensions have not yet been put into practice. We plan to use the result direction of the crew operation to measure, that is, when there is no major deviation in the state of the aircraft, if the crew has an operation action input, the actual result of the operation will be carried out. Measure, if the actual state of the aircraft becomes better after the operation action, take 1 and take -1 if the direction is bad. Finally, the cumulative value represents the predictability score of the current flight.

3) Orderliness
Flying is a rigorous and meticulous work, and the crew needs to arrange work in the cockpit in an orderly and organized manner. There are two aspects here. One is organized, which mainly refers to the order of the actions of the unit. For example, the operation sequence of each button in the direct preparation phase should be in a fixed order; the placement of various manuals should be Fixed, the shouts when making the checklist should also be fixed. The second is organization, which mainly refers to CRM. Organization refers to clear division of labor and clear tasks.
In terms of mathematical expression, this dimension has not yet been implemented. We plan to consider two aspects. One is to collect the actual operation sequence in each scenario from the QAR data and compare it with the standard operation sequence. Similarity to quantify. The second is to quickly access the cabin sound through voice recognition, collect the text of the crew's call data, and compare it with the standard call.

Conclusion
The skill evaluation of pilots is different from user portraits in the internet industry. Flying skills and flying business have a very strong logical correlation, so they cannot be achieved through the form of labeling commonly used in the internet industry. We derive the above evaluation method based on the system flight theory, and there may be a lot of space for correction. We are willing to provide a feasible solution for the industry in the quantitative evaluation of flight skills, aiming to provide flight training, flight safety management, etc. The field provides objective, fair, scientific and comprehensive decision-making guidance.