How do you fit the "elephant" of machine learning into the "fridge" of MCU?

Many people regard artificial intelligence (AI) as something out of a science fiction film that’s far removed from ordinary life. This notion is being rapidly revised, since within the next five to 10 years, AI will progress at an unimaginable speed and become an integral part of our daily lives. At this point, you may be wondering – how? Read on and all will be explained.

The basic model of AI Internet of Things

The majority of people think of AI as a science fiction fantasy because, in the past, it was a prohibitively extravagant concept. This was due to the fact that the machine learning (ML) on which AI relies involves a process of training and reasoning that requires extremely high computing power. To overcome this challenge, data processing utilizing centralized computing power on the cloud became the standard approach to realizing machine learning.

However, with the advent of the era of IoT, the feasibility of this model was questioned as centralized cloud computing consumes large bandwidth and storage resources as well as massive amounts of power for real-time data transmission. It also causes long data transmission delays between terminals and the cloud, and poses high security risks in the processes of data transmission and cloud centralized storage. When these drawbacks emerged, it soon became apparent that simple end-to-end computing led to a dead end.

Consequently, the focus of developers has shifted to edge computing, which is complementary to cloud computing. In edge computing, most computing tasks are directly processed on the edge device, and only a portion of the pre-processed data is transmitted to the cloud for "finishing" when necessary. This improves the real-time response and level of intelligence of the edge, and also reduces the load for network transmission channels and cloud data centers. Hence this hybrid computing model is able to perfectly resolve the issues present in traditional cloud computing.

This shift in computing architecture has also had an impact on the machine learning model, transforming it from a computing-centric model to a data-centric model. In these two models, the former entails completing the training and inference of machine learning in the cloud data center, while the latter entails the completion of model training in the cloud while inference is completed on the edge device, thus shaping the basic form of the AI Internet of Things (AIoT) architecture.

Expanding the boundaries of machine learning to MCU

As you can see, edge computing has greatly expanded the boundaries of machine learning, enabling it to transform from a product of the data center server room to a more edge network intelligence with diverse possibilities. However, it still falls far short of the requirements of IoT applications. Because inference on edge devices still requires relatively powerful computing power, which in turn usually requires more complex heterogeneous microprocessors including ML auxiliary processors to achieve acceleration, this type of configuration is already considered "high-end" in the embedded field. This fact alone will eliminate many applications that are sensitive to power consumption, cost, and immediacy of the benefits of machine learning.

Therefore, if machine learning is to keep breaking new ground, it requires the development of microcontrollers (MCUs) with simpler resources and more limited computing power capable of running and wielding the powers of machine learning. Based on research data released by IC Insights, 28.1 billion MCUs were shipped worldwide in 2018, and this number will grow to 38.2 billion units by 2023, at which time the global MCU inventory will be in the hundreds of billions. Whoever succeeds in applying equipment of this magnitude to machine learning will be richly rewarded, in more ways than one.

But, as with any dream, reality often seems to "fall short." This is because deploying machine learning to MCUs is like trying to stuff an elephant into a fridge. This analogy is no joke, but a real technology issue that must be considered from two different perspectives.

Slimming machine learning models

The first challenge is to “slim down” the “elephant-sized” ML model, which means that corresponding technology must be developed so that "miniaturized" machine learning inference models can be deployed and run on the microcontroller. Even after it has been slimmed down, the model must meet the following criteria:

• The power consumption of the terminal running the model is generally at the mW level, or even lower;
• The memory occupied is generally below a few hundred kB;
• Inference time is ms level and generally needs to be completed within 1 second.

Towards this end, TinyML technology was devised. As its name suggests, this technology can make ML models "tinier." Like the basic AIoT machine learning model mentioned above, TinyML also needs to collect data and train in the cloud, but the difference lies in the optimization and deployment of the trained model. In order to cope with the limited computing resources of the MCU, TinyML must perform "deep compression" and can only be deployed to edge terminals after the model has undergone a multi-step process including distillation, quantization, encoding and compilation.

Figure 1: Schematic diagram of TinyML deployed in embedded devices (Image source: Internet)

Key steps within this multi-step process include:

• Distillation: This refers to the technical means of culling and distilling knowledge after the model has undergone training in order to create a more compact form of representation.
• Quantization: After distillation, quantization is implemented to approximate the 32-bit floating-point data with a smaller number of data types, thus reducing model size and memory consumption and accelerating the inference speed of the model within an acceptable range of accuracy loss.
• Encoding: This is the application of more effective coding methods (such as Huffman coding) to store data in order to further reduce the scale of the model.
• Compilation: After undergoing the above steps, the model is compressed and compiled into C or C++ code, which can be used by most MCUs, and run through the featherweight network compiler on the device (such as TF Lite and TF Lite Micro).

While TinyML technology takes off, many manufacturers have stepped up their investment in this field over the last two years. According to Silent Intelligence's forecast, TinyML will create more than US$70 billion in economic value over the next five years and maintain a compound annual growth rate of over 27.3%.

Create a “new species” of machine learning MCU

In order to “put the elephant into the fridge,” in addition to slimming down the "elephant" (that is, the ML model), the "fridger" must also be transformed. In other words, we must optimize and transform the MCU we are familiar with so that it can accommodate the requirement for running ML.

For example, to meet the needs of implementing complex machine learning functions in IoT edge devices, Maxim Integrated has launched an ML microcontroller MAX78000 designed for low power consumption applications. The component features a built-in Arm Cortex-M4F processor (100MHz) and a 32-bit RISC-V auxiliary processor (60MHz), as well as a convolutional neural network accelerator that supports a 64-layer deep network, which can perform AI inference in battery-powered applications, consuming only a few microjoules of energy. In contrast with traditional software solutions, this hardware acceleration-based solution reduces the energy consumption of complex AI inference to one percent of the former while inference speed is 100 times faster.

Therefore it is fair to say that “new species” with characteristics similar to ML will, in the not too distant future, become integral to the product blueprints of major MCU manufacturers.

Figure 2: Maxim Integrated's low-power ML microcontroller MAX78000 (Image source: Maxim)

In conclusion

In conclusion, compared with embedded computing architectures such as microprocessors or x86, MCUs are characterized by low power consumption, low cost, short development cycle, fast time to market, high immediacy, and large market volume. If these features can be combined with high-performance machine learning, the resulting product will have endless potential.

In the course of combining the two, if developers can access a "new species" of MCU that supports machine learning functions, along with a complete development tool chain to facilitate the optimization and deployment of ML models, then putting the "elephant" of machine learning into the "fridge" of the MCU will indeed be possible.

Since this trend is in its infant stages, the early bird will not only catch the worm but find itself soaring towards exciting new horizons.