通道剪枝(Channel Pruning)
通道剪枝是一种有效的神经网络压缩/加速方法,在业界已广泛使用剪枝方法包括三个阶段:
训练大型的超参数化网络,修剪不太重要的权重或通道,微调或重新训练修剪的网络。
其中,第二阶段是关键,实现逐层的迭代修剪和快速微调或权重重构,以保持精度【Jose M Alvarez and Mathieu Salzmann. Learning the number of neurons in deep networks. In Advances in Neural Information Processing Systems, pages 2270-2278, 2016】。
在权重修剪时, Han等修剪单个权重以压缩模型尺寸【Song Han, Huizi Mao, and William J Dally. Deep compression:
Compressing deep neural networks with pruning】。
但是,权重修剪会导致非结构化的稀疏滤波器,通用硬件几乎无法加速这种结构。
近年来,Hu和Li等聚焦于CNN中的通道修剪,而不是单个权重,该方法删除了整个权重过滤器【Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang. Network trimming:
A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:
1607.03250, 2016】。
传统的通道剪枝方法主要依靠数据驱动的稀疏度约束【Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 304-320, 2018】或人为设计的策略【Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1389-1397, 2017】。
在大多数传统的通道剪枝中,需要人为设置每一层的压缩率,比较耗时,且容易陷入次优解。
AutoML(Automated Machine Learning)
AutoML方法基于反馈循环【Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint arXiv:
1812.08928, 2018】或强化学习【Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc:
Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference on Computer Vision (ECCV), pages 784-800, 2018】,以迭代模式自动修剪通道。
与传统的通道剪枝方法相比,AutoML方法节省人力,有助于减轻通道剪枝中的超参数所需的手动工作,并且可以直接优化如硬件latency(推理时延)之类的指标。
元学习(Meta Learning)
元学习是指通过观察不同的机器学习方法如何执行各种学习任务来学习。
权重预测是指神经网络的权重是由另一个神经网络预测的,而非直接学习【David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:
1609.09106, 2016】。
元学习可用于零样本学习(few/zero-shot learning)【Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016】和转移学习【Yu-Xiong Wang and Martial Hebert. Learning to learn:
Model regression networks for easy small sample learning. In European Conference on Computer Vision, pages 616-634. Springer, 2016】。
神经网络架构搜索(Neural Architecture Search)
神经网络架构搜索通过强化学习【Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:
1611.01578,2016】,遗传算法【Lingxi Xie and Alan Yuille. Genetic cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 1379-1388, 2017.】或基于梯度【Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, YimingWu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet:
Hardware-aware efficient convnet design via differentiable neural architecture search. arXiv preprint arXiv:
1812.03443, 2018.】的方法找到最佳的网络结构和超参数。
作者将训练PruningNet通过权重预测来做连续的通道修剪。