DOI QR코드

DOI QR Code

Path-Based Computation Encoder for Neural Architecture Search

  • Yang, Ying (Dept. of Computer Science and Technology, Chongqing University of Posts and Telecommunications) ;
  • Zhang, Xu (Dept. of Computer Science and Technology, Chongqing University of Posts and Telecommunications) ;
  • Pan, Hu (Dept. of Computer Science and Technology, Chongqing University of Posts and Telecommunications)
  • Received : 2022.01.21
  • Accepted : 2022.03.08
  • Published : 2022.04.30

Abstract

Recently, neural architecture search (NAS) has received increasing attention as it can replace human experts in designing the architecture of neural networks for different tasks and has achieved remarkable results in many challenging tasks. In this study, a path-based computation neural architecture encoder (PCE) was proposed. Our PCE first encodes the computation of information on each path in a neural network, and then aggregates the encodings on all paths together through an attention mechanism, simulating the process of information computation along paths in a neural network and encoding the computation on the neural network instead of the structure of the graph, which is more consistent with the computational properties of neural networks. We performed an extensive comparison with eight encoding methods on two commonly used NAS search spaces (NAS-Bench-101 and NAS-Bench-201), which included a comparison of the predictive capabilities of performance predictors and search capabilities based on two search strategies (reinforcement learning-based and Bayesian optimization-based) when equipped with different encoders. Experimental evaluation shows that PCE is an efficient encoding method that effectively ranks and predicts neural architecture performance, thereby improving the search efficiency of neural architectures.

Keywords

Acknowledgement

This work was supported by the Natural Science Foundation of Chongqing, China under Grant cstc2019jscx-mbdxX0021, and in part by the Major Industrial Technology Research and Development Project of Chongqing High-tech Industry (No. D2018-82).

References

  1. C. White, W. Neiswanger, and Y. Savani, "Bananas: Bayesian optimization with neural architectures for neural architecture search," in Proceedings of the 35th AAAI Conference on Artificial Intelligence, Virtual Event, 2021, pp. 10293-10301.
  2. M. Chatzianastasis, G. Dasoulas, G. Siolas, and M. Vazirgiannis, "Operation embeddings for neural architecture search," 2021 [Online]. Available: https://arxiv.org/abs/2105.04885.
  3. C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, and F. Hutter, "Nas-bench-101: towards reproducible neural architecture search," in Proceedings of the 36th International Conference on Machine Learning, vol. 97, pp. 7105-7114, 2019.
  4. X. Dong and Y. Yang, "Nas-bench-201: Extending the scope of reproducible neural architecture search," 2020 [Online]. Available: https://arxiv.org/abs/2001.00326.
  5. X. Zhang, Y. Zhang, Z. Zhang, and J. Liu, "Discriminative learning of imaginary data for few-shot classification," Neurocomputing, vol. 467, pp. 406-417, 2022. https://doi.org/10.1016/j.neucom.2021.09.070
  6. W. Wen, H. Liu, Y. Chen, H. Li, G. Bender, and P. J. Kindermans, "Neural predictor for neural architecture search," in Computer Vision - ECCV 2020. Cham, Switzerland: Springer, 2020, pp. 660-676.
  7. C. White, W. Neiswanger, S. Nolen, and Y. Savani, "A study on encodings for neural architecture search," Advances in Neural Information Processing Systems, vol. 33, pp. 20309-20319, 2020.
  8. H. Shi, P. Pi, H. Xu, Z. Li, J. Kwok, and T. Zhang, "Bridging the gap between sample-based and one-shot neural architecture search with BONAS," Advances in Neural Information Processing Systems, vol. 33, pp. 1808-1819, 2020.
  9. X. Ning, Y. Zheng, T. Zhao, Y. Wang, and H. Yang, "A generic graph-based neural architecture encoding scheme for predictor-based NAS," in Computer Vision - ECCV 2020. Cham, Switzerland: Springer, 2020, pp. 189-204.
  10. M. Zhang, S. Jiang, Z. Cui, R. Garnett, and Y. Chen, "D-VAE: a variational autoencoder for directed acyclic graphs," Advances in Neural Information Processing Systems, vol. 32, pp. 1586-1598, 2019.