.. _perception-setup:

Setup Camera Perception
=======================

.. warning::

    **Dependency on local mapping**

    For performance optimisation, the way the *camera perception* and *local mapping* work during sensorfusion makes
    them depend on each others' configuration. This means that, even if sensorfusion is not active, the *local mapping*
    configuration needs to be loaded into the appropriate namespace for the *camera perception* to function.

Prerequisites
-------------

Ensure the ``camera_perception`` package is built and the setup file is sourced.

.. code-block:: bash

    catkin build camera_perception
    source devel/setup.bash

YOLOv8-seg Model with TensorRT
------------------------------

How to export a trained model to a TensorRT engine:

1. **Export the ONNX model** from a ``.pt`` file using the script ``export-seg.py``
2. **Convert the ONNX model** to a TensorRT engine using ``trtexec``

Export ONNX Model
^^^^^^^^^^^^^^^^^

All trained models can be found in `this folder <https://drive.google.com/drive/u/0/folders/1B5CR_I1qjSARNe_Qc-mxOpvzZmweSUm3>`_.

1. **Download the model** into an accessible directory inside the Docker container, for example:
``/workspace/as_ros/src/camera_perception/engine``

2. **Export the ONNX model** from your ``.pt`` file:

.. code-block:: bash

    cd /workspace/as_ros/src/camera_perception/scripts
    python3 export-seg.py --weights ../engine/YOUR_MODEL.pt --opset 14 --input-shape 1 image_channel(3) model_input_height(608) model_input_width(1280) --device cuda:0

Example:

.. code-block:: bash

    python3 export-seg.py --weights ../engine/uni_4_V100_seg_n_34.pt --opset 14 --input-shape 1 3 608 1280 --device cuda:0

You will get an ONNX model whose prefix is the same as the input weights. This ONNX model does not contain postprocessing.

- ``--weights`` : The PyTorch model you trained.
- ``--opset`` : ONNX opset version, default is 11.
- ``--sim`` : (Optional) Whether to simplify your ONNX model.
- ``--input-shape`` : Input shape for your model, should be (batch, channel, height, width).
- ``--device`` : The CUDA device you use to export the engine.

Export Engine by trtexec
^^^^^^^^^^^^^^^^^^^^^^^^

Use NVIDIA's ``trtexec`` tool to convert the ONNX model to a TensorRT engine:

.. code-block:: bash

    /usr/src/tensorrt/bin/trtexec \
    --onnx=YOUR_ONNX_MODEL.onnx \
    --saveEngine=YOUR_ENGINE.engine \
    --fp16

- ``--onnx`` : The ONNX model you exported.
- ``--saveEngine`` : The name of the engine you want to save.
- ``--fp16`` : (Optional) Whether to use FP16 (converting from FP32 to FP16).

Example:

.. code-block:: bash

    cd /workspace/as_ros/src/camera_perception/engine
    /usr/src/tensorrt/bin/trtexec \
    --onnx=uni_4_V100_seg_n_34.onnx \
    --saveEngine=uni_4_V100_seg_n_34.engine \
    --fp16 \
    --workspace=16384

Running the Node
----------------

.. code-block:: bash

    roslaunch camera_perception camera_perception.launch

You can set the following argument:

- ``send_debug_informations`` (yes/no): If "yes," the node outputs debug information to ``/rosout``. Default: "no."

Adjust other parameters in ``perception.yaml`` in the ``config`` folder. When changing the camera resolution in the ``perception.yaml``, set the proper directory for the intrinsic matrix as well.

Setting up the camera
---------------------

The current camera configuration is set to a resolution of 720p and a framerate of 60 fps. A fixed offset of 35 ms is subtracted from the timestamp of the image to correct driver delay (based on the `ZED SDK documentation <https://support.stereolabs.com/hc/en-us/articles/206918319-What-is-the-latency-of-the-ZED-camera>`_).

Reference
---------

The inference is partly based on: `YOLOv8-TensorRT <https://github.com/triple-Mu/YOLOv8-TensorRT>`_
and `YOLACT <https://openaccess.thecvf.com/content_ICCV_2019/papers/Bolya_YOLACT_Real-Time_Instance_Segmentation_ICCV_2019_paper.pdf>`_.