Posted on   |   In ,   |  

测试DFANet语义分割网络，基于论文DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation，主要特点在于它的实时性：

测试使用cityscapes数据集，可以在这里下载。

服务器及数据准备

假设有已有一个远程docker服务器root@0.0.0.0 -p 9999。

Dependence

pytorch==1.0.0
python==3.6
numpy
torchvision
matplotlib
opencv-python
tensorflow
tensorboardX
apt install -y libsm6 libxext6
pip3 install opencv-python
pip3 install pyyaml
pip3 install tensorboardX

检查pytorch版本：

import torch
print(torch.__version__)
# Linux, pip, Python 3.6, CUDA 9
pip3 install --upgrade pip
pip3 install --upgrade torch torchvision
使用scp指令将本地程序和数据集上传到服务器：
scp -P 9999 local_file root@0.0.0.0:remote_directory
解压缩zip文件
apt-get update
apt-get install zip -y
unzip local_file
下载DFANet
git clone https://github.com/huaifeng1993/DFANet.git
cd DFANet
Pretrained model

打开utils/preprocess_data.py，修改dataset位置：

cityscapes_data_path = "/home/luohanjie/Documents/SLAM/data/cityscapes"
cityscapes_meta_path = "/home/luohanjie/Documents/SLAM/data/cityscapes/gtFine"

运行脚本，生成labels：

python3 utils/preprocess_data.py
main.py

打开main.py，修改dataset位置：

train_dataset = DatasetTrain(cityscapes_data_path="/home/luohanjie/Documents/SLAM/data/cityscapes",
                             cityscapes_meta_path="/home/luohanjie/Documents/SLAM/data/cityscapes/gtFine/")
                             
val_dataset = DatasetVal(cityscapes_data_path="/home/luohanjie/Documents/SLAM/data/cityscapes",
                         cityscapes_meta_path="/home/luohanjie/Documents/SLAM/data/cityscapes/gtFine/")

2019.4.24 An function has been writed to load the pretrained model which was trained on imagenet-1k.The project of training the backbone can be Downloaded from here -https://github.com/huaifeng1993/ILSVRC2012. Limited to my computing resources(only have one RTX2080),I trained the backbone on ILSVRC2012 with only 22 epochs.But it have a great impact on the results.

由于我们没有ILSVRC2012的pretrained model，所以需要关掉标志位：

net = dfanet(pretrained=False, num_classes=20)
ERROR: TypeError: init() got an unexpected keyword argument 'log_dir'

打开train.py，修改为：

writer = SummaryWriter(logdir=self.log_dir)
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)

出现这个错误的情况是，在服务器上的docker中运行训练代码时，batch size设置得过大，shared memory不够（因为docker限制了shm）.解决方法是，将Dataloader的num_workers设置为0[1]。

打开main.py，修改：

train_loader = DataLoader(dataset=train_dataset,
                          batch_size=10, shuffle=True,
                          num_workers=0)
val_loader = DataLoader(dataset=val_dataset,
                        batch_size=10, shuffle=False, 
                        num_workers=0)

Train

python3 main.py

  1. https://blog.csdn.net/hyk_1996/article/details/80824747 ↩

Posted on   |   In ,   |  

下载opencv_contrib：

git clone https://github.com/opencv/opencv_contrib

cd opencv_contrib

git checkout 3.4.5

官网下载sources。

sudo apt-get install build-essential git make git yams libgtk2.0-dev pig-config libavcodec-dev libavformat-dev python-dev python-bumpy python-tk libtbb2 libtbb-dev libjpeg-dev libpng12-dev libtiff5-dev libjasper-dev libdc1394-22-dev libswscale-dev libopenexr-dev libeigen2-dev libeigen3-dev libfaac-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev libx264-dev libqt4-dev libqt4-opengl-dev sphinx-common texlive-latex-extra libv4l-dev

mkdir build
cd build

cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_TBB=ON -D BUILD_EXAMPLES=OFF -D BUILD_DOCS=OFF -D BUILD_PERF_TESTS=OFF -D BUILD_TESTS=OFF -D WITH_GTK_2_X=ON -D WITH_QT=ON -D WITH_OPENGL=ON -D WITH_CUDA=ON -D CMAKE_CXX_FLAGS="-std=c++11" -D CUDA_NVCC_FLAGS="-std=c++11 --expt-relaxed-constexpr" -D OPENCV_EXTRA_MODULES_PATH=/home/luohanjie/Documents/software/opencv-3.4.5/opencv_contrib/modules ..


make -j4
sudo make install

sudo /bin/bash -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/opencv.conf'
sudo ldconfig

Posted on   |   In ,   |  

给定图片$I$上的一个特征点$\mathbf{x}$和对应的搜索向量$\mathbf{n}$，求在另一张图$J$中找到匹配的点$\mathbf{x'}$。假定在图$J$上的搜索起始位置为$\mathbf{x} _{init}'$，搜索窗口为$W$，迭代次数$K$（在实际使用中，我们往往会将$\mathbf{x} _{init}'=\mathbf{x}$）[1][2][3]。

计算步骤

(1) 分别对图像$I$和图像$J$建立金字塔${I^L} _{L=0,…,L _m}$和${J^L} _{L=0,…,L _m}$，$L _m$为给定的金字塔层数，一般为3（图像金字塔化一般包括两个步骤：首先对图像进行一次低通滤波进行平滑，然后对图像的横纵两个方向1/2抽样，从而得到一系列尺度缩小的图像。当L＝0时，为原图，当向金字塔的上层移动时，尺寸和分辨率降低，伴随的细节就越少。我们从顶层开始对目标点进行跟踪，先获得一个粗糙的结果，然后将结果作为下一层的初始点再进行跟踪，不断迭代直到到达第0层。这是一种由粗到细分析策略）。

jinzita

(2) 初始化顶层金字塔的搜索偏移位置：$\mathbf{g}^{L _m}=[g _{x}^{L _m}\ g _{y}^{L _m }]^T =[0\ 0]^T$

(3) 从第$L=L _m$层(顶层)金字塔图像开始不断往下，对每一层图像作作以下操作：

(3.1) 计算特征点$\mathbf{x}$在金字塔第$L$层图$I^L$上的位置：$\mathbf{x}^L=[p _x\ p _y]^T=\mathbf{x}/2^L$。

(3.2) 计算搜索起始位置$\mathbf{x} _{init}'$在金字塔第$L$层图$J^L$上的位置：$\mathbf{x}'^L=[p' _x\ p' _y]^T=\mathbf{x} _{init}'/2^L$。

(3.3) 计算最速下降矩阵$\mathbf{S}(u,v)=[I _x (p _x+u,p _y+v)\ I _y (p _x+u,p _y+v)]\mathbf{n}$，$(u,v)⊆W$，$\mathbf{S}$矩阵跟窗口$W$大小一致。其中，$I _x (x,y)$，$I _y (x,y)$为图$I^L$在$(x,y)$位置$X$和$Y$两个方向的梯度。

(3.4) 计算在特征点$\mathbf{x}$在第$L$层的空间梯度值$\textstyle m^L = \sum _{(u,v)⊆W} \mathbf{S}(u,v)^2$。$m^L$体现的是，图$I^L$中，位于$\mathbf{x}^L$的窗口$W$内，图像在$\mathbf{n}$方向的梯度变化。

(3.5) 初始化位置迭代参数$\mathbf{\Gamma}^0 = [{\Gamma} _{x}^0 \ {\Gamma} _{y}^0]^T = [0\ 0]^T$，参数记录了偏移位置，用于寻找偏移了的特征点。

(3.6) 变量$k$从1到$K$（$K$为控制变量，用于控制(3.6.1)至(3.6.3)的迭代次数），迭代以下操作：

(3.6.1) 此时，特征点在图$J^L$的跟踪位置在$(p' _x+g _x^L+{\Gamma} _x^{k-1},p' _y+g _y^L+{\Gamma} _y^{k-1} )$，计算图像偏差值$b^k = \sum _{(u,v)⊆W}[\mathbf{S}(u,v)[I^L (p _x+u,p _y+v)-J^L (p' _x+g _x^L+{\Gamma} _x^{k-1},p' _y+g _y^L+{\Gamma} _y^{k-1} )]$。

(3.6.2) 更新位置迭代参数$\mathbf{\Gamma}^k=\mathbf{\Gamma}^{k-1}-(b^k/m^L)\mathbf{n}$。

(3.6.3) $k=k+1$，回到(3.6.1)继续迭代。

(3.7) 在$L$层金字塔最终跟踪偏移：$\mathbf{d}^L=\mathbf{\Gamma}^k$。

(3.8) 初始化下一层金字塔的跟踪偏移位置：$g^{L-1}=2(g^L+d^L)$。

(3.9) $L=L-1$，回到(3.1)继续迭代。

(4) 特征点$x$在图$J$中的匹配的点位置$\mathbf{x}'=\mathbf{x} _{init}'+g^0+d^0$。

1d_klt

  1. Baker, Simon, and Iain Matthews. "Lucas-kanade 20 years on: A unifying framework." International journal of computer vision 56.3 (2004): 221-255. ↩

  2. Bouguet, Jean-Yves. "Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm." Intel Corporation 5.1-10 (2001): 4. ↩

  3. Bouguet, Jean-Yves. "Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm." Intel Corporation 5.1-10 (2001): 4. ↩

Posted on   |   In ,   |  

Introduction

在使用Semi-direct Method[1]跑Euroc Dataset的v103数据时，发现效果很不好。导致错误的主要的原因有：图片太暗，对比度太低；图片亮度变化很大（不限于帧间，左右目有时候也会出现亮度不一致的情况）。于是，需要对输入图像进行预处理，提高图片的对比度，并且使得进行跟踪的两张图片亮度一致。

对于提高图片的对比度，最简单的方法是使用直方图均衡化。不过直方图均衡化有一些很明显的缺点，如变换后细节消失；不自然的过分增强。对于SLAM系统，往往会在过份增强的纹理上提取出一些关键点，而这些关键点我们认为是不稳定的（如下图的窗帘）。

所以，我们需要一种更加先进的图像增强算法用于SLAM的图像预处理。

image_enhancement1

Related Work

根据BIMEF[2]算法提供的对比程序，我们测试了几个图像增强算法的结果。根据对比，我们认为LIME[3]算法无论在增强效果还有速度上都有较好的表现。

image_enhancement2

image_enhancement3

Image Enhancement

为了进一步提高算法性能，我们集合了LIME[3]和FGS[4]滤波算法，提出了一种新的图像增强算法。下图展现了增强算法的增强结果： image_enhancement4

为了满足SLAM系统的需求，我们对于增强图再进行了一次去噪还有对比度增强处理：

image_enhancement5 image_enhancement6

我们对参考帧和当前帧图像进行图像增强处理。经过处理后，两者的亮度差异已经变很小了。然后，我们再对当前帧进行线性变换，使得当前帧的平均灰度值和均方差与参考帧一致，从而达到亮度一致的目的。

Experiment

根据下图可以看出，我们的算法能够很好地恢复出图像暗处的纹理，并且对于噪音有比较好的抑止。

enhancement7

为验证图像增强算法对于Semi-direct Method的影响，我们设计了一个对比实验，分别对数据图片进行图像增强和直方图均衡化操作，并且输入到Semi-direct Method，观察输出的pose与ground truth pose的差异。

实验视频右下角结果窗口中，蓝点为根据Semi-direct Method结果进行的关键点重投影，而绿点是根据ground truth pose进行的重投影，当蓝点变红时，表示此时Semi-direct Method无解或者residual过大。

enhancement8

实验结果表明，我们的图像增强算法能够使得Semi-direct Method在Euroc v103 dataset中正常运行。并且相对于直方图均衡化，我们的图像增强法能够使得Semi-direct Method的结果精度有所提升。

  1. Forster, Christian, et al. "Svo: Semidirect visual odometry for monocular and multicamera systems." IEEE Transactions on Robotics 33.2 (2017): 249-265. ↩

  2. https://github.com/baidut/BIMEF ↩

  3. Guo, Xiaojie, Yu Li, and Haibin Ling. "LIME: Low-light image enhancement via illumination map estimation." IEEE Transactions on Image Processing 26.2 (2017): 982-993. ↩ ↩

  4. Min, Dongbo, et al. "Fast global image smoothing based on weighted least squares." IEEE Transactions on Image Processing 23.12 (2014): 5638-5653. ↩

Posted on   |   In ,   |  

环境

  • 系统: Ubuntu 16.04.4 LTS
  • 内核: 4.13.0-36-generic
  • CUDA: 9.0.176
  • 显卡: 940mx
  • 显卡驱动: 384.13
  • GCC: 5.4.0
  • python: 2.7.12

Dependences

sudo apt-get install python-pip python-dev python3-pip python3-dev cuds-command-line-tools

sudo pip install testresources enum34 mock
sudo edit ~/.bashrc

添加:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/extras/CUPTI/lib64

刷新:

source ~/.bashrc

安装CUDA 9.0

官网下载CUDA cuda-repo-ubuntu1604-9-0-local_9.0.176-1_amd64.deb。

注意，使用.run文件安装cuda的话，在安装TensorRT时会发生错误！

sudo dpkg -i cuda-repo-ubuntu1604-9-0-local_9.0.176-1_amd64.deb
sudo apt-key add /var/cuda-repo-9-0-local/7fa2af80.pub
sudo apt-get update
sudo apt-get install cuda

重启，配置环境：

sudo edit ~/.bashrc

添加：

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

刷新：

source ~/.bashrc

重启：

nvcc -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuba compilation tools, release 9.0, V9.0.176
dpkg-query -W | grep cuda-cubla

cuda-cublas-9-0	9.0.176-1
cuda-cublas-dev-9-0	9.0.176-1
Building Samples (optional)
cd /NVIDIA_CUDA-9.1_Samples/1_Utilities/deviceQuery
make
./deviceQuery 

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce 940MX"
  CUDA Driver Version / Runtime Version          9.0 / 9.0
  CUDA Capability Major/Minor version number:    5.0
  Total amount of global memory:                 2003 Mbytes (2100232192 bytes)
  ( 3) Multiprocessors, (128) CUDA Cores/MP:     384 CUDA Cores
  GPU Max Clock rate:                            1242 MHz (1.24 GHz)
  Memory Clock rate:                             1001 Mhz
  Memory Bus Width:                              64-bit
  L2 Cache Size:                                 1048576 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Supports Cooperative Kernel Launch:            No
  Supports MultiDevice Co-op Kernel Launch:      No
  Device PCI Domain ID / Bus ID / location ID:   0 / 2 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 9.0, CUDA Runtime Version = 9.0, NumDevs = 1
Result = PASS

cuDNN

下载cuDNN v7.0.5 (Dec 5, 2017), for CUDA 9.0:

  • cuDNN v7.0.5 Runtime Library for Ubuntu16.04 (Deb)
  • cuDNN v7.0.5 Developer Library for Ubuntu16.04 (Deb)
  • cuDNN v7.0.5 Code Samples and User Guide for Ubuntu16.04 (Deb)

Navigate to your directory containing cuDNN Deb file:

sudo dpkg -i libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb 
sudo dpkg -i libcudnn7-dev_7.0.5.15-1+cuda9.0_amd64.deb
sudo dpkg -i libcudnn7-doc_7.1.4.18-1+cuda9.0_amd64.deb
Verifying
cp -r /usr/src/cudnn_samples_v7/ $HOME
cd $HOME/cudnn_samples_v7/mnistCUDNN
make clean && make
./mnistCUDNN
cudnnGetVersion() : 7005 , CUDNN_VERSION from cudnn.h : 7005 (7.0.5)
Host compiler version : GCC 5.4.0
There are 1 CUDA capable devices on your machine :
device 0 : sms  3  Capabilities 5.0, SmClock 1241.5 Mhz, MemSize (Mb) 2002, MemClock 1001.0 Mhz, Ecc=0, boardGroupID=0
Using device 0

...

Test passed!

NVIDIA TensorRT 3.0.4(optional)

官网下载nv-tensorrt-repo-ubuntu1604-ga-cuda9.0-trt3.0.4-20180208_1-1_amd64.deb[1]。

sudo dpkg -i nv-tensorrt-repo-ubuntu1604-ga-cuda9.0-trt3.0.4-20180208_1-1_amd64.deb
sudo apt-get update
sudo apt-get install tensorrt

sudo apt-get install python-libnvinfer-doc python-libnvinfer python-libnvinfer-dev swig3.0  # python 2.7

sudo apt-get install python3-libnvinfer-doc # python 3.5
Verifying
dpkg -l | grep TensorRT

ii  libnvinfer-dev                                              4.0.4-1+cuda9.0                                             amd64        TensorRT development libraries and headers
ii  libnvinfer-samples                                          4.0.4-1+cuda9.0                                             amd64        TensorRT samples and documentation
ii  libnvinfer4                                                 4.0.4-1+cuda9.0                                             amd64        TensorRT runtime libraries
ii  python-libnvinfer                                           4.0.4-1+cuda9.0                                             amd64        Python bindings for TensorRT
ii  python-libnvinfer-dev                                       4.0.4-1+cuda9.0                                             amd64        Python development package for TensorRT
ii  python-libnvinfer-doc                                       4.0.4-1+cuda9.0                                             amd64        Documention and samples of python bindings for TensorRT
ii  python3-libnvinfer                                          4.0.4-1+cuda9.0                                             amd64        Python 3 bindings for TensorRT
ii  python3-libnvinfer-dev                                      4.0.4-1+cuda9.0                                             amd64        Python 3 development package for TensorRT
ii  python3-libnvinfer-doc                                      4.0.4-1+cuda9.0                                             amd64        Documention and samples of python bindings for TensorRT
ii  tensor

安装TensorFlow 1.8 [2]

For python 2.7

sudo apt-get install python-pip python-dev

pip install pip==9.0 # Don't use pip > 9.0 !!!!
sudo pip  install --upgrade https://download.tensorflow.google.cn/linux/gpu/tensorflow_gpu-1.8.0-cp27-none-linux_x86_64.whl
sudo pip install --upgrade pip

For python 3.5

sudo apt-get install python3-pip python3-dev

sudo pip3 install --upgrade https://download.tensorflow.google.cn/linux/gpu/tensorflow_gpu-1.8.0-cp35-cp35m-linux_x86_64.whl

卸载指令: sudo pip uninstall tensorflow or sudo pip3 uninstall tensor flow

Verifying
python

写入:

# Python
import tensorflow as tf
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'

config = tf.ConfigProto(allow_soft_placement=True)

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.7)

config.gpu_options.allow_growth = True
sess = tf.Session(config=config)

hello = tf.constant('Hello, TensorFlow!')
print(sess.run(hello))

输出:

Hello, TensorFlow!

  1. https://docs.nvidia.com/deeplearning/sdk/tensorrt-install-guide/index.html ↩

  2. https://www.tensorflow.org/install/install_linux?hl=zh-cn#InstallingNativePip ↩