Need to find the best AI model for your problem? Try neural architecture search
NAS algorithms are efficient problem solvers
Search strategy
Even basic search spaces usually require plenty of trials and error to find the optimal deep learning architecture. Therefore, a neural architecture search algorithm also needs a “search strategy.” The search strategy determines how the NAS algorithm experiments with different neural networks.
The most basic strategy is “random search,” in which the NAS algorithm randomly selects a neural network from the search space, trains and validates it, registers the results, and moves on to the next. Random search is extremely expensive because the NAS algorithm is basically brute-forcing its way through the search space, wasting expensive resources on testing solutions that can be eliminated with easier methods. Depending on the complexity of the search space, random search can take days’ or weeks’ worth of GPU time to verify every possible neural network architecture.
There are other techniques that speed up the search process. An example isBayesian optimization, which starts with random choices and gradually tunes its search direction as it gathers information about the performance of different architectures.
Another strategy is to frame neural architecture search as areinforcement learning problem. In this case, the RL agent’s environment is the search space, the actions are the different configurations of the neural network, and the reward is the performance of the network. The reinforcement learning agent starts with random modifications, but over time, it learns to choose configurations that produce better improvements to the neural network’s performance.
Other search strategies includeevolutionary algorithmsandMonte Carlo tree search. Each search strategy has its strengths and weaknesses, and engineers must find the right balance between “exploration and exploitation,” which basically means testing totally new architectures or tweaking the ones that have so far proven promising.
Performance estimation strategy
As the NAS algorithm goes through the search space, it must train and validate deep learning models to compare their performance and choose the optimal neural network. Obviously, doing full training on each neural network takes a long time and requires very large computational resources.
To reduce the costs of evaluating deep learning models, engineers of NAS algorithms use “proxy metrics” that can be measured without requiring full training of the neural network.
For example, they can train their models for fewer epochs, on a smaller dataset, or on lower resolution data. While the resulting deep learning model will not reach its full potential, these lower fidelity training regimes provide a baseline to compare different models at a lower cost. Once the set of architectures has been culled to a few promising neural networks, the NAS algorithm can do more thorough training and testing of the models.
Another way to reduce the costs of performance estimation is to initialize new models on the weights of previously trained models. Known as transfer learning, this practice results in a much faster convergence, which means the deep learning model will need fewer training epochs. Transfer learning is applicable when the source and destination model have compatible architectures.
A work in progress
Neural architecture search still has challenges to overcome, such as providing explanations of why some architectures are better than others and addressing complicated applications that go beyond simple image classification.
NAS is nonetheless a very useful and attractive field for the deep learning community and can have great applications both foracademic research and applied machine learning.
Sometimes, technologies like NAS are depicted as artificial intelligence that creates its own AI, making humans redundant and taking us towardAI singularity. But in reality, NAS is a perfect example of how humans andcontemporary AI systemscan work together to solve complicated problems. Humans use their intuition, knowledge, and common sense to find interesting problem spaces and define their boundaries and intended result. NAS algorithms, on the other hand, are very efficient problem solvers that can search the solution space and find the best neural network architecture for the intended application.
This article was originally published by Ben Dickson onTechTalks, a publication that examines trends in technology, how they affect the way we live and do business, and the problems they solve. But we also discuss the evil side of technology, the darker implications of new tech, and what we need to look out for. You can read the original articlehere.
Story byBen Dickson
Ben Dickson is the founder of TechTalks. He writes regularly about business, technology and politics. Follow him on Twitter and Facebook(show all)Ben Dickson is the founder ofTechTalks. He writes regularly about business, technology and politics. Follow him onTwitterandFacebook
Get the TNW newsletter
Get the most important tech news in your inbox each week.
