How to configure an environment for URPD research

June 13, 2022

Due to code consistency reasons, the URPD code relies on a cpp module compiled by OpenCV 3, and the python scripts by OpenCV4. This makes the environment configuration of URPD code much more complex. This blog will show you how to configure such an environment for URPD research purpose. The experience can also benefit other configuration tasks relies on OpenCV.

Platform: WSL + VSCode + Docker Install libraries: OpenCV 3 & contrib, Boost, Miniconda/Anaconda , Python OpenCV 4 & contrib, etc.

• Platform Setup
  • WSL
    • Windows OS Version Pre-check
    • How to install
    • Memory Issue
  • VSCode
    • VSCode Extension
  • Docker
    • Docker Installation
• Configure the Environment
  • Pull the Docker image of Python
  • Start the container with a Volume
  • OpenCV3 Installation (Not Python-OpenCV)
  • Boost Installation
  • Conda Installation (Python Side)
  • Create a Virtual Environment (Optional)
  • Python Libraries Installation
  • Pytorch Installation (Optional)

Platform Setup

WSL

Windows OS Version Pre-check

Before installing WSL, make sure you have Windows 10 version 2004 and higher (Build 19041 and higher) or Windows 11. Otherwise, the installation will be much more complicated and unstable due to windows updates. You can follow the manually installation tutorial here: Manual installation steps for older versions of WSL | Microsoft Docs Updating windows is recommended.

How to install

You can install everything you need to run Windows Subsystem for Linux (WSL) by entering this command in an administrator PowerShell or Windows Command Prompt and then restarting your machine. wsl --install

By default, the installed Linux distribution will be Ubuntu.

Memory Issue

WSL can easily consume all your RAM without control. Here is a blog reported such an issue and provided the solution how to release the memory: Taking Back Memory From Vmmem/WSL | BlogLogBlog

We can configure limits on the memory, CPU, and swap size allocated to WSL 2 in a .wslconfig file. Advanced settings configuration in WSL | Microsoft Docs Remark:

• If the corresponding .wslconfig file doesn’t existed, we can create one.

VSCode

VSCode is a lightweight and powerful editor which integrated with WSL, Git and Docker nicely.

You can download and install VSCode through this link: Download Visual Studio Code - Mac, Linux, Windows You can find VSCode official document here: Documentation for Visual Studio Code

VSCode Extension

Some recommended extensions which will be used in this article: CMake, Docker, Python, Python Environment Manager, Remote-WSL

Docker

With the help of Docker, we can pull an image from the Hub with pre-installed desired Python version and no worries about the dependencies. Moreover, after we configure such a complex environment in a container, we can build an image based on this container. This makes other collaborators much easier to contribute to the code, instead of spending time on re-configuring environment on their local machines.

Docker Installation

Since we have installed WSL on windows system, we can either install Docker Engine in WSL Ubuntu system, or install Docker Desktop in Windows with an interactive surface.

• Install Docker-Engine in WSL Ubuntu system: Install Docker Engine on Ubuntu
• Install Docker-desktop in Windows (WSL is required): Install Docker Desktop on Windows

Some Useful Docker Commands

• Build an image from a Dockerfile: docker build [OPTIONS] PATH | URL | -
• List the docker images: docker images [OPTIONS] [REPOSITORY[:TAG]]
• List the docker containers: docker ps [OPTIONS]
• Run a command in a new container: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
  • Usually run a docker in backend, with interactive: -it
• Run a command in a running container: docker exec [OPTIONS] CONTAINER COMMAND [ARG...]
• Create a new image from a container’s changes: docker commit [OPTIONS] CONTAINER [REPOSITORY[:TAG]]
• Save one or more images to a tar archive (streamed to STDOUT by default): docker save [OPTIONS] IMAGE [IMAGE...] Example docker save busybox > busybox.tar
• See more about docker commands: Reference documentation

Configure the Environment

1. Pull the Docker image of Python

   We start the environment configuration with a prebuild Python image. Hub link: Python - Official Image | Docker Hub Pull command (latest tag): docker pull python After pulling this image, we can create a container based on this image.

2. Start the container with a Volume

   It is worthy to notice that a docker container shall be kept small to avoid the image growing incredibly large. However, in research, we are not just configuring an environment, but also run multiple experiments and generates a lots of result files. If these files are generated inside the container, it will make the image too large to be easily share among contributors or to the other users.

   To resolve this problem, we can use Volume, which is a preferred mechanism for persisting data generated by and used by Docker containers. It works similarly to bind mounts on OS, but has the advantage that fully controlled by Docker. By creating a Volume, we can easily bind a local external directory to a container, and it can detach the container after stopped.

   In this article, we create a container called RP-Test from the Python image, and bind the local directory E: to container directory /mnt/e

   The Docker-Desktop makes the creation of containers very simple, and can check the status of each container intuitively. For example, by clicking the corresponding running container, we can check whether the volumes are mounted correctly.

3. OpenCV3 Installation (Not Python-OpenCV)

   Here we first install the OpenCV 3 for cpp module usage. The installation of OpenCV 3.4.11 is mostly follow this official doc:OpenCV: Installation in Linux

   Some remarks:

   • After cloning the opencv repo, do not forget to switch branch to the corresponding release version (i.e. 3.4.11) by: git checkout tags/3.4.11. Otherwise, OpenCV 4 version will be installed.
   • To compile OpenCV with contrib packages, the cmake commend shall be modified like this: cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local OPENCV_EXTRA_MODULES_PATH=/home/tools/opencv_contrib/modules .. Here OPENCV_EXTRA_MODULES_PATH shall be modified to your opencv contrib folder directory.
   • After build by make -jn, do not forget to install libraries by running make install in the build directory. Otherwise, CMake cannot find OpenCV libraries automatically.

4. Boost Installation

   The installation of boost can follow this official doc: Boost Getting Started on Unix Variants - 1.79.0 If you need to install a specific version of boost, please select the corresponding version documents.

5. Conda Installation (Python Side)

   OpenCV 3 and Boost are served for the cpp module compiling. By finishing the above steps, we can start installing the Python side packages & libraries.

   We choose to use Miniconda to control the Python packages & virtual environments. The installation of Miniconda can follow this official doc: Installing on Linux — conda 4.13.0.post16+23ba63de documentation

6. Create a Virtual Environment (Optional)

   To better arrange the Python packages for different code purpose, creating a virtual environment is a good choice. A virtual environment in conda can be easily created by such a commend: conda create --name myenv python=specified_version

   In VSCode, by the Python Environment Manager extension, we can easily choose which environment to use, and see which packages are installed.

7. Python Libraries Installation

   Here we listed the installation commend for some python libraries used in our project. pip is recommended.

   • Python-OpenCV (cpu-only) full package (contains both main modules and contrib/extra modules): pip install opencv-contrib-python
   • IPython Kernel for Jupyter: pip install ipykernel
   • …

8. Pytorch Installation (Optional)

   • Install Pytorch with CPU only version by conda: conda install pytorch torchvision torchaudio cpuonly -c pytorch