.. _camera-calibration:

How to calibrate the camera
===========================

The current system relies on identifying cones on camera images. As camera we use the stereo camera zed2i.
In the next step the local mapping module calculates where the identified cone is in the real world.

The local mapping module needs a intrinsic and extrinsic matrix to calculate this info.

The intrinsic camera matrix is calculated by the ZED camera itself and pubished as ROS message.
The extrinsic camera matrix has to be calculated by an alternative method. We chose [coming soon].

While this process was an elaborate manual task in the past. Since October 2022 we have a graphical user
interface for the camera calibration.

In the future there should be a new graphical user interface for a new, more precise calibration method.


.. click:: calibrate_extrinsic_matrix:init
   :prog: calibrate_extrinsic_matrix
   :nested: full

Prerequisites
-------------

#. Connect a display to the ACU, or

#. :ref:`ethernet-connection` and :ref:`vnc-connection`

#. .. tab:: Calibration board

      #. Lay the calibration board in front of the car in such a way that the middle crosses align with the x axis of the vehicle.

         a. the weird corner of the blue polygon must be on the upper right.

      #. Make sure, that all crosses can be viewed on the camera image:

         a. Open foxglove on your local machine or on the ACU itself (display or vnc)

         b. Open a new data source: Open connection to live robot: ROS 1: ``localhost`` or ``<ACU_IP>`` (based on the decision of previous step)

         c. Open a image panel and check both cameras.

         d. You may have to get some items to block some sun light.

      #. Measure the projected distance between the camera and the lowest cross. 

   .. tab:: Ground Truth

      #. Mark an big (e.g. 15x15m or 20x20m) perfect square with big orange cones

      #. Place yellow, blue and small orange cones inside this square so that they are equally distributed.

         Yellow and blue cones will be used for calibrating the camera (use registration for those), orange cones for controlling the calibration (don't use registration for those).
      
      #. Place the vehicle approximately in the middle of one side of the square. Make a picture with a drone.

      #. Create a track from this picture using the :ref:`track-creator`. Specify a useful test day and calibration as track layout. No manual map needed.


Instruction
-----------

#. Follow the basic instructions in :ref:`gui-in-container`

#. Record a rosbag with some images from both cameras:

   .. code-block:: bash

      roslaunch local_mapping camera_calibration.launch

   Cancel the recording after 3-5 seconds by pressing ``Ctrl + C``

#. Rename the new rosbag under ``./rosbags/`` to ``calibration<number>.bag``.

#. Calibrate the camera via the graphical user interface

   If you are using a VNC session, don't forget to specify which display the gui should use: ``export DISPLAY=1``

   .. tab:: Calibration board

      #. Start the GUI (see above for options):
      
         .. code-block:: bash

            calibrate_extrinsic_matrix /workspace/as_ros/rosbags/calibration<number>.bag -ds board

      #. If the terminal outputs, that it did not find any image, there is something wrong. You have to think for yourself :o

      #. If the shown image is black, you can restart the graphical user interface with the option ``-s <delay in seconds>``

      #. Click on each cross from left to right, from the top to the bottom. You can delete a point by clicking <backspace>

      #. After clicking on nine crosses, the blue control polygon is overlayed. The terminal will ask you whether this looks good or you want to try it again.

      #. You have to do the previous two steps for other camera aswell.

      #. If you are finished, please commit the matrices at some point in the near futures

   .. tab:: Ground Truth

      #. Start the GUI (see above for options):
      
         .. code-block:: bash

            calibrate_extrinsic_matrix /workspace/as_ros/rosbags/calibration<number>.bag -ds ground_truth -tl calibration -td <test_day>

      #. enter the projected distance between the rear axle an camera in x direction in mm (emma: 590mm)

      #. If the terminal outputs, that it did not find any image, there is something wrong. You have to think for yourself :o

      #. If the shown image is black, you can restart the graphical user interface with the option ``-s <delay in seconds>``

      #. Click on the tip of every yellow and blue cone in the same order as when creating the track.

      #. After you clicked on each cone, the small orange cones should be visualized as orange crosses.

      #. You have to do the previous two steps for other camera aswell.

      #. If you are finished, please commit the matrices at some point in the near futures