机器之心专栏
作者：陈惠婵
从 RCNN 到 Faster RCNN，再到最近的 FPN 和获得 ICCV Best Paper 的 Mask RCNN，深度学习在物体检测中以绝对优势从众多机器学习算法中脱引而出。大家对 Facebook 的计算机视觉研究项目的开源期盼已久，经过 1 年多的漫长等待，今天 Facebook 终于开源了 Detectron，Detectron 开源项目使用 caffe2 和 python 接口。实现了 10 多篇计算机视觉最新的成果。下面我们简单介绍一下 Detectron 所实现的论文。并且对 Detectron 进行初次测试，我们会在随后的博客中更新我们自己测试得到的 Detectron 训练模型和速度标准。
Fast RCNN、Faster RCNN、RFCN、FPN、RetinaNet
Detectron 实现了物体检测的标准模型，并且添加了 Feature Pyramid Network 和 RetinaNet 等 state-of-the-art 的物体检测模型。FPN 是 two-stage 检测的 state-of-the-art，RetinaNet 是 one-stage 的 best-performing 模型，并且也是 ICCV 的 best student paper。
ResNet，ResNeXt
Detectron 实现了 Residual Network 和 ResNeXt 等基础的神经网络结构。ResNext 使用 depthwise convolution 的技术大大降低了参数，并且保证了分类结果。
Human-object Interaction Detection
物体检测可以得到 bounding box 如图（a），Human-object interaction 通过预测不同的 bounding box 之间的概率密度可以学习不同 bounding box 之间的关系。如图（c），人和刀之间的关系是切（cut）。
Mask RCNN
Mask RCNN 通过改进 Faster RCNN 可以实现 7 FPS 的 instance segmentation 和关键点检测，并且超过当时的所有方法。Mask RCNN 在 COCO 和 CITYSCAPES 数据集上面取得了好的结果。Mask RCNN 的示意图如下。
Training Imagenet in one hour
本篇论文发现了 large batch 可以大大的提升分类网络的收敛速度，通过把 batch size 从 256 提升到 8192，将训练时间从几周降低到了 1 个小时，大大提升了神将网络的训练速度。
CVPR 2018 投稿论文：Learning to segment everything
收集 mask rcnn 的标注十分昂贵，在 cityscapes 上面一张图的标注需要 1 个小时。这篇论文提出了 weight transfer 的办法来分割所有的物体，免去了收集分割数据的巨大时间和金钱耗费。本篇论文使用 bounding box detection branch 的权重来预测 mask branch 的权重来实现此目的。
Non Local Neural Convolution
Convolution Neural Network 只能够传递邻域的信息，本篇论文参照 non local means 和 self attention 的方法设计出了 non local convolution，从而能够捕捉到非邻域的信息。如下图，中心点可以捕捉到非邻域的重要信息。
Detectron 框架初探
要使用 Detectron 框架需要安装 caffe2，caffe2 的安装请参照 caffe2 官网。然后参照 INSTALL.md 安装 Detectron，Detectron 提供了方便的测试和添加 op 功能。添加 op 具体参照 test_zero_even_op.py。
Detectron 框架包含 config，demo，lib，tests 和 tools 等文件夹。Config 包含着各个模型的训练和测试参数，lib 是 detectron 的核心文件夹，例如 data loader，model builder，operator definition 和 utils（学习率等非核心函数）。
Detectron 安装
Caffe2 安装，参照网址 https://caffe2.ai/docs/getting-started.html?platform=ubuntu&configuration=compile
核心命令：
git clone --recursive https://github.com/caffe2/caffe2.git && cd caffe2
make && cd build && sudo make install
python -c 'from caffe2.python import core' 2>/dev/null && echo "Success" || echo "Failure"
Detectron 安装，参照 https://github.com/facebookresearch/Detectron/blob/master/INSTALL.md
Detectron 测试
使用 Mask RCNN FPN ResNet 50 进行测试，命令如下：
CUDA_VISIBLE_DEVICES=3 python tools/train_net.py --cfg configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml OUTPUT_DIR /tmp/detectron-output
在 Titan X 上面测试速度：
INFO infer_simple.py: 111: Processing demo/16004479832_a748d55f21_k.jpg -> /tmp/detectron-visualizations/16004479832_a748d55f21_k.jpg.pdf
INFO infer_simple.py: 119: Inference time: 1.402s
INFO infer_simple.py: 121: | im_detect_bbox: 1.329s
INFO infer_simple.py: 121: | misc_mask: 0.034s
INFO infer_simple.py: 121: | im_detect_mask: 0.034s
INFO infer_simple.py: 121: | misc_bbox: 0.005s
INFO infer_simple.py: 124: \ Note: inference on the first image will be slower than the rest (caches and auto-tuning need to warm up)
INFO infer_simple.py: 111: Processing demo/18124840932_e42b3e377c_k.jpg -> /tmp/detectron-visualizations/18124840932_e42b3e377c_k.jpg.pdf
INFO infer_simple.py: 119: Inference time: 0.411s
INFO infer_simple.py: 121: | im_detect_bbox: 0.305s
INFO infer_simple.py: 121: | misc_mask: 0.058s
INFO infer_simple.py: 121: | im_detect_mask: 0.044s
INFO infer_simple.py: 121: | misc_bbox: 0.004s
INFO infer_simple.py: 111: Processing demo/24274813513_0cfd2ce6d0_k.jpg -> /tmp/detectron-visualizations/24274813513_0cfd2ce6d0_k.jpg.pdf
INFO infer_simple.py: 119: Inference time: 0.321s
INFO infer_simple.py: 121: | im_detect_bbox: 0.264s
INFO infer_simple.py: 121: | misc_mask: 0.034s
INFO infer_simple.py: 121: | im_detect_mask: 0.018s
INFO infer_simple.py: 121: | misc_bbox: 0.005s
INFO infer_simple.py: 111: Processing demo/33823288584_1d21cf0a26_k.jpg -> /tmp/detectron-visualizations/33823288584_1d21cf0a26_k.jpg.pdf
INFO infer_simple.py: 119: Inference time: 0.722s
INFO infer_simple.py: 121: | im_detect_bbox: 0.515s
INFO infer_simple.py: 121: | misc_mask: 0.127s
INFO infer_simple.py: 121: | im_detect_mask: 0.072s
INFO infer_simple.py: 121: | misc_bbox: 0.007s
INFO infer_simple.py: 111: Processing demo/17790319373_bd19b24cfc_k.jpg -> /tmp/detectron-visualizations/17790319373_bd19b24cfc_k.jpg.pdf
INFO infer_simple.py: 119: Inference time: 0.403s
INFO infer_simple.py: 121: | im_detect_bbox: 0.292s
INFO infer_simple.py: 121: | misc_mask: 0.067s
INFO infer_simple.py: 121: | im_detect_mask: 0.038s
INFO infer_simple.py: 121: | misc_bbox: 0.006s
Detectron 框架训练
在 COCO 数据集上面使用 FPN ResNet50 进行 Faster RCNN 训练
使用命令 ：
CUDA_VISIBLE_DEVICES=3 python tools/train_net.py --cfg configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml OUTPUT_DIR /tmp/detectron-output
输出如下：
Namespace(cfg_file='configs/getting_started/tutorial_1gpu_e2e_faster_rcnn_R-50-FPN.yaml', multi_gpu_testing=False, opts=['OUTPUT_DIR', '/tmp/detectron-output'], skip_test=False)
INFO train_net.py: 188: Training with config:
INFO train_net.py: 189: {'BBOX_XFORM_CLIP': 4.1351665567423561,
'CLUSTER': {'ON_CLUSTER': False},
'DATA_LOADER': {'NUM_THREADS': 4},
'DEDUP_BOXES': 0.0625,
'DOWNLOAD_CACHE': '/tmp/detectron-download-cache',
'EPS': 1e-14,
'EXPECTED_RESULTS': [],
'EXPECTED_RESULTS_ATOL': 0.005,
'EXPECTED_RESULTS_EMAIL': '',
'EXPECTED_RESULTS_RTOL': 0.1,
'FAST_RCNN': {'MLP_HEAD_DIM': 1024,
'ROI_BOX_HEAD': 'fast_rcnn_heads.add_roi_2mlp_head',
'ROI_XFORM_METHOD': 'RoIAlign',
'ROI_XFORM_RESOLUTION': 7,
'ROI_XFORM_SAMPLING_RATIO': 2},
'FPN': {'COARSEST_STRIDE': 32,
'DIM': 256,
'EXTRA_CONV_LEVELS': False,
'FPN_ON': True,
'MULTILEVEL_ROIS': True,
'MULTILEVEL_RPN': True,
'ROI_CANONICAL_LEVEL': 4,
'ROI_CANONICAL_SCALE': 224,
'ROI_MAX_LEVEL': 5,
'ROI_MIN_LEVEL': 2,
'RPN_ANCHOR_START_SIZE': 32,
'RPN_ASPECT_RATIOS': (0.5, 1, 2),
'RPN_MAX_LEVEL': 6,
'RPN_MIN_LEVEL': 2,
'ZERO_INIT_LATERAL': False},
'MATLAB': 'matlab',
'MEMONGER': True,
'MEMONGER_SHARE_ACTIVATIONS': False,
'MODEL': {'BBOX_REG_WEIGHTS': (10.0, 10.0, 5.0, 5.0),
'CLS_AGNOSTIC_BBOX_REG': False,
'CONV_BODY': 'FPN.add_fpn_ResNet50_conv5_body',
'EXECUTION_TYPE': 'dag',
'FASTER_RCNN': True,
'KEYPOINTS_ON': False,
'MASK_ON': False,
'NUM_CLASSES': 81,
'RPN_ONLY': False,
'TYPE': 'generalized_rcnn'},
'MRCNN': {'CLS_SPECIFIC_MASK': True,
'CONV_INIT': 'GaussianFill',
'DILATION': 2,
'DIM_REDUCED': 256,
'RESOLUTION': 14,
'ROI_MASK_HEAD': '',
'ROI_XFORM_METHOD': 'RoIAlign',
'ROI_XFORM_RESOLUTION': 7,
'ROI_XFORM_SAMPLING_RATIO': 0,
'THRESH_BINARIZE': 0.5,
'UPSAMPLE_RATIO': 1,
'USE_FC_OUTPUT': False,
'WEIGHT_LOSS_MASK': 1.0},
'NUM_GPUS': 1,
'OUTPUT_DIR': '/tmp/detectron-output',
'PIXEL_MEANS': array([[[ 102.9801, 115.9465, 122.7717]]]),
'RESNETS': {'NUM_GROUPS': 1,
'RES5_DILATION': 1,
'STRIDE_1X1': True,
'TRANS_FUNC': 'bottleneck_transformation',
'WIDTH_PER_GROUP': 64},
'RETINANET': {'ANCHOR_SCALE': 4,
'ASPECT_RATIOS': (0.5, 1.0, 2.0),
'BBOX_REG_BETA': 0.11,
'BBOX_REG_WEIGHT': 1.0,
'CLASS_SPECIFIC_BBOX': False,
'INFERENCE_TH': 0.05,
'LOSS_ALPHA': 0.25,
'LOSS_GAMMA': 2.0,
'NEGATIVE_OVERLAP': 0.4,
'NUM_CONVS': 4,
'POSITIVE_OVERLAP': 0.5,
'PRE_NMS_TOP_N': 1000,
'PRIOR_PROB': 0.01,
'RETINANET_ON': False,
'SCALES_PER_OCTAVE': 3,
'SHARE_CLS_BBOX_TOWER': False,
'SOFTMAX': False},
'RFCN': {'PS_GRID_SIZE': 3},
'RNG_SEED': 3,
'ROOT_DIR': '/home/huichan/caffe2/detectron',
'RPN': {'ASPECT_RATIOS': (0.5, 1, 2),
'RPN_ON': True,
'SIZES': (64, 128, 256, 512),
'STRIDE': 16},
'SOLVER': {'BASE_LR': 0.0025,
'GAMMA': 0.1,
'LOG_LR_CHANGE_THRESHOLD': 1.1,
'LRS': [],
'LR_POLICY': 'steps_with_decay',
'MAX_ITER': 60000,
'MOMENTUM': 0.9,
'SCALE_MOMENTUM': True,
'SCALE_MOMENTUM_THRESHOLD': 1.1,
'STEPS': [0, 30000, 40000],
'STEP_SIZE': 30000,
'WARM_UP_FACTOR': 0.3333333333333333,
'WARM_UP_ITERS': 500,
'WARM_UP_METHOD': u'linear',
'WEIGHT_DECAY': 0.0001},
'TRAIN': {'ASPECT_GROUPING': True,
'AUTO_RESUME': True,
'BATCH_SIZE_PER_IM': 256,
'BBOX_THRESH': 0.5,
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0.0,
'CROWD_FILTER_THRESH': 0.7,
'DATASETS': ('coco_2014_train',),
'FG_FRACTION': 0.25,
'FG_THRESH': 0.5,
'FREEZE_CONV_BODY': False,
'GT_MIN_AREA': -1,
'IMS_PER_BATCH': 2,
'MAX_SIZE': 833,
'PROPOSAL_FILES': (),
'RPN_BATCH_SIZE_PER_IM': 256,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 0,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POST_NMS_TOP_N': 2000,
'RPN_PRE_NMS_TOP_N': 2000,
'RPN_STRADDLE_THRESH': 0,
'SCALES': (500,),
'SNAPSHOT_ITERS': 20000,
'USE_FLIPPED': True,
'WEIGHTS': u'/tmp/detectron-download-cache/ImageNetPretrained/MSRA/R-50.pkl'},
'USE_NCCL': False,
'VIS': False,
'VIS_TH': 0.9}
I0123 13:14:38.367794 36482 context_gpu.cu:325] Total: 311 MB
INFO train_net.py: 330: Loading dataset: ('coco_2014_train',)
loading annotations into memory...
Done (t=15.17s)
creating index...
index created!
INFO roidb.py: 49: Appending horizontally-flipped training examples...
INFO roidb.py: 51: Loaded dataset: coco_2014_train
INFO roidb.py: 135: Filtered 1404 roidb entries: 165566 -> 164162
INFO roidb.py: 67: Computing bounding-box regression targets...
INFO roidb.py: 69: done
INFO roidb.py: 191: Ground-truth class histogram:
INFO roidb.py: 195: 0__background__: 0
INFO roidb.py: 195: 1 person: 363358
INFO roidb.py: 195: 2 bicycle: 9824
INFO roidb.py: 195: 3 car: 61106
INFO roidb.py: 195: 4 motorcycle: 11944
INFO roidb.py: 195: 5 airplane: 7656
INFO roidb.py: 195: 6 bus: 8642
INFO roidb.py: 195: 7 train: 6316
INFO roidb.py: 195: 8 truck: 14094
INFO roidb.py: 195: 9 boat: 14912
INFO roidb.py: 195: 10 traffic light: 18248
INFO roidb.py: 195: 11 fire hydrant: 2632
INFO roidb.py: 195: 12 stop sign: 2744
INFO roidb.py: 195: 13 parking meter: 1666
INFO roidb.py: 195: 14 bench: 13482
INFO roidb.py: 195: 15 bird: 14226
INFO roidb.py: 195: 16 cat: 6598
INFO roidb.py: 195: 17 dog: 7534
INFO roidb.py: 195: 18 horse: 9304
INFO roidb.py: 195: 19 sheep: 12916
INFO roidb.py: 195: 20 cow: 11196
INFO roidb.py: 195: 21 elephant: 7760
INFO roidb.py: 195: 22 bear: 1806
INFO roidb.py: 195: 23 zebra: 7316
INFO roidb.py: 195: 24 giraffe: 7186
INFO roidb.py: 199: total: 1195680
INFO train_net.py: 334: 164162 roidb entries
INFO net.py: 54: Loading from: /tmp/detectron-download-cache/ImageNetPretrained/MSRA/R-50.pkl
I0123 13:16:41.699045 36482 net_dag_utils.cc:118] Operator graph pruning prior to chain compute took: 0.000500389 secs
I0123 13:16:41.699774 36482 net_dag.cc:61] Number of parallel execution chains 340 Number of operators = 632
INFO loader.py: 232: [62/64]
INFO detector.py: 434: Changing learning rate 0.000000 -> 0.000833 at iter 0
json_stats: {"accuracy_cls": 0.000000, "eta": "2 days, 6:05:39", "iter": 0, "loss": 5.814330, "loss_bbox": 0.008809, "loss_cls": 4.863443, "loss_rpn_bbox_fpn2": 0.000000, "loss_rpn_bbox_fpn3": 0.000000, "loss_rpn_bbox_fpn4": 0.002576, "loss_rpn_bbox_fpn5": 0.264878, "loss_rpn_bbox_fpn6": 0.000000, "loss_rpn_cls_fpn2": 0.455301, "loss_rpn_cls_fpn3": 0.091068, "loss_rpn_cls_fpn4": 0.022299, "loss_rpn_cls_fpn5": 0.105955, "loss_rpn_cls_fpn6": 0.000000, "lr": 0.000833, "mb_qsize": 64, "mem": 3253, "time": 3.245656}
json_stats: {"accuracy_cls": 0.940430, "eta": "8:29:26", "iter": 20, "loss": 1.839182, "loss_bbox": 0.071032, "loss_cls": 0.897934, "loss_rpn_bbox_fpn2": 0.077837, "loss_rpn_bbox_fpn3": 0.005068, "loss_rpn_bbox_fpn4": 0.014110, "loss_rpn_bbox_fpn5": 0.013995, "loss_rpn_bbox_fpn6": 0.000000, "loss_rpn_cls_fpn2": 0.425642, "loss_rpn_cls_fpn3": 0.099356, "loss_rpn_cls_fpn4": 0.034078, "loss_rpn_cls_fpn5": 0.019162, "loss_rpn_cls_fpn6": 0.000000, "lr": 0.000900, "mb_qsize": 64, "mem": 3267, "time": 0.509616}
Memory 占用如下：
总结
Detecrton 框架给予 caffe2 和 python 接口，caffe2 对 Multi-GPU 和分布式训练提供了很好的支持，GPU 现存的利用率也大大提升，并且对很多 state-of-the-art 的方法提供了很好的 baseline 实现。相信 Detectron 框架会在未来的 computer vision 领域大放异彩。
安装小提示 1：
>>> import caffe2
>>> from caffe2.python import core
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "caffe2/python/core.py", line 24, in <module>
from past.builtins import basestring
ImportError: No module named past.builtins
>>> quit()
方法： sudo pip install future
安装小提示 2：
caffe2 安装之后需要将 caffe2 添加到 PYTHONPATH 和 LD_LIBRARY_PATH 路径
nano ~/.bashrc
输入：
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
export PYTHONPATH=$PYTHONPATH:/home/huichan/caffe2/caffe2/build
source ~/.bashrc