Posted on   |   In ,   |  

环境

  • 系统: Ubuntu 16.04.4 LTS
  • 内核: 4.15.0-50-generic
  • CUDA: 9.0.176
  • 显卡: 940mx
  • 显卡驱动: 384.13
  • GCC: 5.4.0
  • python: 3.5.2

protobuf

sudo apt-get install autoconf automake libtool curl make g++ unzip

git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
git checkout v3.6.0
git submodule update --init --recursive
./autogen.sh

./configure
make -j4
make check -j4
sudo make install  -j4
sudo ldconfig # refresh shared library cache.
protoc --version
libprotoc 3.6.0
sudo pip3 uninstall protobuf
cd protobuf/python
python3 setup.py build
python3 setup.py test
sudo python3 setup.py install

Bazel

下载bazel-0.18.1-installer-linux-x86_64.sh：

chmod +x bazel-0.18.1-installer-linux-x86_64.sh
sudo ./bazel-0.18.1-installer-linux-x86_64.sh --user

The --user flag installs Bazel to the $HOME/bin directory on your system and sets the .bazelrc path to $HOME/.bazelrc. Use the --help command to see additional installation options.

Check CuDNN Version

cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2

or

cat /usr/include/x86_64-linux-gnu/cudnn.h | grep CUDNN_MAJOR -A 2

TensorFlow

git clone https://github.com/tensorflow/tensorflow.git
git checkout v1.12.2
./configure


WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.protobuf.UnsafeUtil (file:/home/luohanjie/.cache/bazel/_bazel_luohanjie/install/cdf71f2489ca9ccb60f7831c47fd37f1/_embedded_binaries/A-server.jar) to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of com.google.protobuf.UnsafeUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
You have bazel 0.18.1 installed.
Please specify the location of python. [Default is /usr/bin/python]: /usr/bin/python3


Found possible Python library paths:
  /home/luohanjie/Documents/software/caffe/python
  /usr/lib/python3.5/dist-packages
  /usr/local/lib/python3.5/dist-packages
  /usr/lib/python3/dist-packages
Please input the desired Python library path to use.  Default is [/home/luohanjie/Documents/software/caffe/python]
/usr/lib/python3/dist-packages
Do you wish to build TensorFlow with Apache Ignite support? [Y/n]: n
No Apache Ignite support will be enabled for TensorFlow.

Do you wish to build TensorFlow with XLA JIT support? [Y/n]: n
No XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with OpenCL SYCL support? [y/N]: n
No OpenCL SYCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with ROCm support? [y/N]: n
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 9.0]: 


Please specify the location where CUDA 9.0 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: 


Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7]: 7.4.2


Please specify the location where cuDNN 7 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]: /usr/include/x86_64-linux-gnu


Do you wish to build TensorFlow with TensorRT support? [y/N]: y
TensorRT support will be enabled for TensorFlow.

Please specify the location where TensorRT is installed. [Default is /usr/lib/x86_64-linux-gnu]:/usr/src/tensorrt


Please specify the NCCL version you want to use. If NCCL 2.2 is not installed, then you can use version 1.3 that can be fetched automatically but it may have worse performance with multiple GPUs. [Default is 2.2]: 1.3


Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size. [Default is: 3.5,7.0]: 5.0


Do you want to use clang as CUDA compiler? [y/N]: n
nvcc will be used as CUDA compiler.

Please specify which gcc should be used by nvcc as the host compiler. [Default is /usr/bin/gcc]: 


Do you wish to build TensorFlow with MPI support? [y/N]: n
No MPI support will be enabled for TensorFlow.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]: 


Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: n
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See tools/bazel.rc for more details.
	--config=mkl         	# Build with MKL support.
	--config=monolithic  	# Config for mostly static monolithic build.
	--config=gdr         	# Build with GDR support.
	--config=verbs       	# Build with libverbs support.
	--config=ngraph      	# Build with Intel nGraph support.
Configuration finished
bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2 --config=nonccl //tensorflow:libtensorflow_cc.so

bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2 --config=nonccl //tensorflow:libtensorflow_framework.so

sudo mkdir -p /usr/local/include/tf/tensorflow
sudo ln -s abs_path_to_tensorflow/bazel-genfiles/ /usr/local/include/tf
sudo ln -s abs_path_to_tensorflow/tensorflow/cc /usr/local/include/tf/tensorflow
sudo ln -s abs_path_to_tensorflow/tensorflow/core /usr/local/include/tf/tensorflow
sudo ln -s abs_path_to_tensorflow/third_party /usr/local/include/tf
sudo ln -s abs_path_to_tensorflow/bazel-bin/tensorflow/libtensorflow_cc.so /usr/local/lib
sudo ln -s abs_path_to_tensorflow/bazel-bin/tensorflow/libtensorflow_framework.so /usr/local/lib
sudo ln -s tensorflow/contrib/makefile/downloads/absl/absl /usr/local/include/tf/third_party

如果想要卸载请运行如下命令[1]：

sudo rm -r /usr/local/include/tf
sudo rm /usr/local/lib/libtensorflow_*.so

TensorFlow C++ Demo

Demo.cpp

#include <tensorflow/core/platform/env.h>
#include <tensorflow/core/public/session.h>
#include <iostream>

using namespace std;
using namespace tensorflow;

int main()
{    
    Session* session;
    Status status = NewSession(SessionOptions(), &session);
    if (!status.ok()) {
        cout << status.ToString() << "\n";
        return 1;
    }
    cout << "Session successfully created.\n";
    return 0;
}

CMakeLists.txt

cmake_minimum_required(VERSION 3.5)
 
set(CMAKE_CXX_STANDARD 11)
 
find_package(OpenCV REQUIRED)
find_package(Eigen REQUIRED )

add_definitions(${EIGEN_DEFINITIONS})
 
set(TENSORFLOW_INCLUDES
        /usr/local/include/tf/
        /usr/local/include/tf/bazel-genfiles
        /usr/local/include/tf/tensorflow/
        /usr/local/include/tf/third-party
)
 
set(TENSORFLOW_LIBS
        /usr/local/lib/libtensorflow_cc.so
        /usr/local/lib//libtensorflow_framework.so)
 
 
include_directories(
    ${TENSORFLOW_INCLUDES}
    ${OpenCV_INCLUDE_DIRS}
    ${EIGEN_INCLUDE_DIR}
)
 
add_executable(demo demo.cpp)
target_link_libraries(demo ${TENSORFLOW_LIBS} ${OpenCV_LIBS})

  1. http://www.liuxiao.org/2018/08/ubuntu-tensorflow-c-%E4%BB%8E%E8%AE%AD%E7%BB%83%E5%88%B0%E9%A2%84%E6%B5%8B1%EF%BC%9A%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BA/ ↩

Posted on   |   In ,   |  

测试DFANet语义分割网络，基于论文DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation，主要特点在于它的实时性：

测试使用cityscapes数据集，可以在这里下载。

服务器及数据准备

假设有已有一个远程docker服务器root@0.0.0.0 -p 9999。

Dependence

pytorch==1.0.0
python==3.6
numpy
torchvision
matplotlib
opencv-python
tensorflow
tensorboardX
apt install -y libsm6 libxext6
pip3 install opencv-python
pip3 install pyyaml
pip3 install tensorboardX

检查pytorch版本：

import torch
print(torch.__version__)
# Linux, pip, Python 3.6, CUDA 9
pip3 install --upgrade pip
pip3 install --upgrade torch torchvision
使用scp指令将本地程序和数据集上传到服务器：
scp -P 9999 local_file root@0.0.0.0:remote_directory
解压缩zip文件
apt-get update
apt-get install zip -y
unzip local_file
下载DFANet
git clone https://github.com/huaifeng1993/DFANet.git
cd DFANet
Pretrained model

打开utils/preprocess_data.py，修改dataset位置：

cityscapes_data_path = "/home/luohanjie/Documents/SLAM/data/cityscapes"
cityscapes_meta_path = "/home/luohanjie/Documents/SLAM/data/cityscapes/gtFine"

运行脚本，生成labels：

python3 utils/preprocess_data.py
main.py

打开main.py，修改dataset位置：

train_dataset = DatasetTrain(cityscapes_data_path="/home/luohanjie/Documents/SLAM/data/cityscapes",
                             cityscapes_meta_path="/home/luohanjie/Documents/SLAM/data/cityscapes/gtFine/")
                             
val_dataset = DatasetVal(cityscapes_data_path="/home/luohanjie/Documents/SLAM/data/cityscapes",
                         cityscapes_meta_path="/home/luohanjie/Documents/SLAM/data/cityscapes/gtFine/")

2019.4.24 An function has been writed to load the pretrained model which was trained on imagenet-1k.The project of training the backbone can be Downloaded from here -https://github.com/huaifeng1993/ILSVRC2012. Limited to my computing resources(only have one RTX2080),I trained the backbone on ILSVRC2012 with only 22 epochs.But it have a great impact on the results.

由于我们没有ILSVRC2012的pretrained model，所以需要关掉标志位：

net = dfanet(pretrained=False, num_classes=20)
ERROR: TypeError: init() got an unexpected keyword argument 'log_dir'

打开train.py，修改为：

writer = SummaryWriter(logdir=self.log_dir)
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)

出现这个错误的情况是，在服务器上的docker中运行训练代码时，batch size设置得过大，shared memory不够（因为docker限制了shm）.解决方法是，将Dataloader的num_workers设置为0[1]。

打开main.py，修改：

train_loader = DataLoader(dataset=train_dataset,
                          batch_size=10, shuffle=True,
                          num_workers=0)
val_loader = DataLoader(dataset=val_dataset,
                        batch_size=10, shuffle=False, 
                        num_workers=0)

Train

python3 main.py

  1. https://blog.csdn.net/hyk_1996/article/details/80824747 ↩

Posted on   |   In ,   |  

下载opencv_contrib：

git clone https://github.com/opencv/opencv_contrib

cd opencv_contrib

git checkout 3.4.5

官网下载sources。

sudo apt-get install build-essential git make git yams libgtk2.0-dev pig-config libavcodec-dev libavformat-dev python-dev python-bumpy python-tk libtbb2 libtbb-dev libjpeg-dev libpng12-dev libtiff5-dev libjasper-dev libdc1394-22-dev libswscale-dev libopenexr-dev libeigen2-dev libeigen3-dev libfaac-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev libx264-dev libqt4-dev libqt4-opengl-dev sphinx-common texlive-latex-extra libv4l-dev

mkdir build
cd build

cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_TBB=ON -D BUILD_EXAMPLES=OFF -D BUILD_DOCS=OFF -D BUILD_PERF_TESTS=OFF -D BUILD_TESTS=OFF -D WITH_GTK_2_X=ON -D WITH_QT=ON -D WITH_OPENGL=ON -D WITH_CUDA=ON -D CMAKE_CXX_FLAGS="-std=c++11" -D CUDA_NVCC_FLAGS="-std=c++11 --expt-relaxed-constexpr" -D OPENCV_EXTRA_MODULES_PATH=/home/luohanjie/Documents/software/opencv-3.4.5/opencv_contrib/modules ..


make -j4
sudo make install

sudo /bin/bash -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/opencv.conf'
sudo ldconfig

Posted on   |   In ,   |  

给定图片$I$上的一个特征点$\mathbf{x}$和对应的搜索向量$\mathbf{n}$，求在另一张图$J$中找到匹配的点$\mathbf{x'}$。假定在图$J$上的搜索起始位置为$\mathbf{x} _{init}'$，搜索窗口为$W$，迭代次数$K$（在实际使用中，我们往往会将$\mathbf{x} _{init}'=\mathbf{x}$）[1][2][3]。

计算步骤

(1) 分别对图像$I$和图像$J$建立金字塔${I^L} _{L=0,…,L _m}$和${J^L} _{L=0,…,L _m}$，$L _m$为给定的金字塔层数，一般为3（图像金字塔化一般包括两个步骤：首先对图像进行一次低通滤波进行平滑，然后对图像的横纵两个方向1/2抽样，从而得到一系列尺度缩小的图像。当L＝0时，为原图，当向金字塔的上层移动时，尺寸和分辨率降低，伴随的细节就越少。我们从顶层开始对目标点进行跟踪，先获得一个粗糙的结果，然后将结果作为下一层的初始点再进行跟踪，不断迭代直到到达第0层。这是一种由粗到细分析策略）。

jinzita

(2) 初始化顶层金字塔的搜索偏移位置：$\mathbf{g}^{L _m}=[g _{x}^{L _m}\ g _{y}^{L _m }]^T =[0\ 0]^T$

(3) 从第$L=L _m$层(顶层)金字塔图像开始不断往下，对每一层图像作作以下操作：

(3.1) 计算特征点$\mathbf{x}$在金字塔第$L$层图$I^L$上的位置：$\mathbf{x}^L=[p _x\ p _y]^T=\mathbf{x}/2^L$。

(3.2) 计算搜索起始位置$\mathbf{x} _{init}'$在金字塔第$L$层图$J^L$上的位置：$\mathbf{x}'^L=[p' _x\ p' _y]^T=\mathbf{x} _{init}'/2^L$。

(3.3) 计算最速下降矩阵$\mathbf{S}(u,v)=[I _x (p _x+u,p _y+v)\ I _y (p _x+u,p _y+v)]\mathbf{n}$，$(u,v)⊆W$，$\mathbf{S}$矩阵跟窗口$W$大小一致。其中，$I _x (x,y)$，$I _y (x,y)$为图$I^L$在$(x,y)$位置$X$和$Y$两个方向的梯度。

(3.4) 计算在特征点$\mathbf{x}$在第$L$层的空间梯度值$\textstyle m^L = \sum _{(u,v)⊆W} \mathbf{S}(u,v)^2$。$m^L$体现的是，图$I^L$中，位于$\mathbf{x}^L$的窗口$W$内，图像在$\mathbf{n}$方向的梯度变化。

(3.5) 初始化位置迭代参数$\mathbf{\Gamma}^0 = [{\Gamma} _{x}^0 \ {\Gamma} _{y}^0]^T = [0\ 0]^T$，参数记录了偏移位置，用于寻找偏移了的特征点。

(3.6) 变量$k$从1到$K$（$K$为控制变量，用于控制(3.6.1)至(3.6.3)的迭代次数），迭代以下操作：

(3.6.1) 此时，特征点在图$J^L$的跟踪位置在$(p' _x+g _x^L+{\Gamma} _x^{k-1},p' _y+g _y^L+{\Gamma} _y^{k-1} )$，计算图像偏差值$b^k = \sum _{(u,v)⊆W}[\mathbf{S}(u,v)[I^L (p _x+u,p _y+v)-J^L (p' _x+g _x^L+{\Gamma} _x^{k-1},p' _y+g _y^L+{\Gamma} _y^{k-1} )]$。

(3.6.2) 更新位置迭代参数$\mathbf{\Gamma}^k=\mathbf{\Gamma}^{k-1}-(b^k/m^L)\mathbf{n}$。

(3.6.3) $k=k+1$，回到(3.6.1)继续迭代。

(3.7) 在$L$层金字塔最终跟踪偏移：$\mathbf{d}^L=\mathbf{\Gamma}^k$。

(3.8) 初始化下一层金字塔的跟踪偏移位置：$g^{L-1}=2(g^L+d^L)$。

(3.9) $L=L-1$，回到(3.1)继续迭代。

(4) 特征点$x$在图$J$中的匹配的点位置$\mathbf{x}'=\mathbf{x} _{init}'+g^0+d^0$。

1d_klt

  1. Baker, Simon, and Iain Matthews. "Lucas-kanade 20 years on: A unifying framework." International journal of computer vision 56.3 (2004): 221-255. ↩

  2. Bouguet, Jean-Yves. "Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm." Intel Corporation 5.1-10 (2001): 4. ↩

  3. Bouguet, Jean-Yves. "Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm." Intel Corporation 5.1-10 (2001): 4. ↩

Posted on   |   In ,   |  

Introduction

在使用Semi-direct Method[1]跑Euroc Dataset的v103数据时，发现效果很不好。导致错误的主要的原因有：图片太暗，对比度太低；图片亮度变化很大（不限于帧间，左右目有时候也会出现亮度不一致的情况）。于是，需要对输入图像进行预处理，提高图片的对比度，并且使得进行跟踪的两张图片亮度一致。

对于提高图片的对比度，最简单的方法是使用直方图均衡化。不过直方图均衡化有一些很明显的缺点，如变换后细节消失；不自然的过分增强。对于SLAM系统，往往会在过份增强的纹理上提取出一些关键点，而这些关键点我们认为是不稳定的（如下图的窗帘）。

所以，我们需要一种更加先进的图像增强算法用于SLAM的图像预处理。

image_enhancement1

Related Work

根据BIMEF[2]算法提供的对比程序，我们测试了几个图像增强算法的结果。根据对比，我们认为LIME[3]算法无论在增强效果还有速度上都有较好的表现。

image_enhancement2

image_enhancement3

Image Enhancement

为了进一步提高算法性能，我们集合了LIME[3]和FGS[4]滤波算法，提出了一种新的图像增强算法。下图展现了增强算法的增强结果： image_enhancement4

为了满足SLAM系统的需求，我们对于增强图再进行了一次去噪还有对比度增强处理：

image_enhancement5 image_enhancement6

我们对参考帧和当前帧图像进行图像增强处理。经过处理后，两者的亮度差异已经变很小了。然后，我们再对当前帧进行线性变换，使得当前帧的平均灰度值和均方差与参考帧一致，从而达到亮度一致的目的。

Experiment

根据下图可以看出，我们的算法能够很好地恢复出图像暗处的纹理，并且对于噪音有比较好的抑止。

enhancement7

为验证图像增强算法对于Semi-direct Method的影响，我们设计了一个对比实验，分别对数据图片进行图像增强和直方图均衡化操作，并且输入到Semi-direct Method，观察输出的pose与ground truth pose的差异。

实验视频右下角结果窗口中，蓝点为根据Semi-direct Method结果进行的关键点重投影，而绿点是根据ground truth pose进行的重投影，当蓝点变红时，表示此时Semi-direct Method无解或者residual过大。

enhancement8

实验结果表明，我们的图像增强算法能够使得Semi-direct Method在Euroc v103 dataset中正常运行。并且相对于直方图均衡化，我们的图像增强法能够使得Semi-direct Method的结果精度有所提升。

  1. Forster, Christian, et al. "Svo: Semidirect visual odometry for monocular and multicamera systems." IEEE Transactions on Robotics 33.2 (2017): 249-265. ↩

  2. https://github.com/baidut/BIMEF ↩

  3. Guo, Xiaojie, Yu Li, and Haibin Ling. "LIME: Low-light image enhancement via illumination map estimation." IEEE Transactions on Image Processing 26.2 (2017): 982-993. ↩ ↩

  4. Min, Dongbo, et al. "Fast global image smoothing based on weighted least squares." IEEE Transactions on Image Processing 23.12 (2014): 5638-5653. ↩