Detailed Description
A class for grabing "range images", intensity images and other information from an Xbox Kinect sensor.
Configuration and usage:
Data is returned as observations of type
mrpt::slam::CObservation3DRangeScan (and
mrpt::slam::CObservationIMU for accelerometers data). See those classes for documentation on their fields.
As with any other CGenericSensor class, the normal sequence of methods to be called is:
Calibration parameters
For an accurate transformation of depth images to 3D points, you'll have to calibrate your Kinect, and supply the following
threee pieces of information (default calibration data will be used otherwise, but they'll be not optimal for all sensors!):
Coordinates convention
The origin of coordinates is the focal point of the depth camera, with the axes oriented as in the diagram shown in
mrpt::slam::CObservation3DRangeScan. Notice in that picture that the RGB camera is assumed to have axes as usual in computer vision, which differ from those for the depth camera.
The X,Y,Z axes used to report the data from accelerometers coincide with those of the depth camera (e.g. the camera standing on a table would have an ACC_Z=-9.8m/s2).
Some general comments
- Depth is grabbed in 10bit depth, and a range N it's converted to meters as: range(m) = 0.1236 * tan(N/2842.5 + 1.1863)
- This sensor can be also used from within rawlog-grabber to grab datasets within a robot with more sensors.
- There is no built-in threading support, so if you use this class manually (not with-in rawlog-grabber), the ideal would be to create a thread and continuously request data from that thread (see mrpt::system::createThread ).
- There is a built-in support for an optional preview of the data on a window, so you don't need to even worry on creating a window to show them.
- This class relies on an embedded version of libfreenect (you don't have to install it in your system). Thanks guys for the great job!
Converting to 3D point cloud
You can convert the 3D observation into a 3D point cloud with this piece of code:
Then the point cloud mrpt::slam::CColouredPointsMap can be converted into an OpenGL object for rendering with mrpt::slam::CMetricMap::getAs3DObject() or alternatively with:
Raw depth to range conversion
At construction, this class builds an internal array for converting raw 10 or 11bit depths into ranges in meters. Users can read that array or modify it (if you have a better calibration, for example) by calling
CKinect::getRawDepth2RangeConversion(). If you replace it, remember to set the first and last entries (index 0 and KINECT_RANGES_TABLE_LEN-1) to zero, to indicate that those are invalid ranges.
R(d) = k3 * tan(d/k2 + k1);
k1 = 1.1863, k2 = 2842.5, k3 = 0.1236
| |
Platform-specific comments
For more details, refer to
libfreenect documentation:
- Linux: You'll need root privileges to access Kinect. Or, install
MRPT/scripts/51-kinect.rules
in /etc/udev/rules.d/
to allow access to all users.
- Windows: Install CL NUI SDK: http://codelaboratories.com/nui
- MacOS: (write me!)
Format of parameters for loading from a .ini file
PARAMETERS IN THE ".INI"-LIKE CONFIGURATION STRINGS:
-------------------------------------------------------
[supplied_section_name]
sensorLabel = KINECT
preview_window = false
device_number = 0
grab_image = true
grab_depth = true
grab_3D_points = true
grab_IMU = true
rgb_cx = 328.9427
rgb_cy = 267.4807
rgb_fx = 529.2151
rgb_fy = 525.5639
d_cx = 339.3078
d_cy = 242.7391
d_fx = 594.2143
d_fy = 591.0405
relativePoseIntensityWRTDepth = [0 -0.02 0 -90 0 -90]
pose_x=0
pose_y=0
pose_z=0
pose_yaw=0
pose_pitch=0
pose_roll=0
More references to read:
Definition at line 194 of file CKinect.h.
#include <mrpt/hwdrivers/CKinect.h>
List of all members.
Public Types |
typedef float | TDepth2RangeArray [KINECT_RANGES_TABLE_LEN] |
| A typedef for an array that converts raw depth to ranges in meters.
|
Public Member Functions |
| CKinect () |
| Default ctor.
|
| ~CKinect () |
| Default ctor.
|
virtual void | initialize () |
| Initializes the 3D camera - should be invoked after calling loadConfig() or setting the different parameters with the set*() methods.
|
virtual void | doProcess () |
| To be called at a high rate (>XX Hz), this method populates the internal buffer of received observations.
|
void | getNextObservation (mrpt::slam::CObservation3DRangeScan &out_obs, bool &there_is_obs, bool &hardware_error) |
| The main data retrieving function, to be called after calling loadConfig() and initialize().
|
void | getNextObservation (mrpt::slam::CObservation3DRangeScan &out_obs, mrpt::slam::CObservationIMU &out_obs_imu, bool &there_is_obs, bool &hardware_error) |
| This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.
|
virtual void | setPathForExternalImages (const std::string &directory) |
| Set the path where to save off-rawlog image files (this class DOES take into account this path).
|
|
|
void | open () |
| Try to open the camera (set all the parameters first!) - users may also call initialize(), which in turn calls this.
|
bool | isOpen () const |
| Whether there is a working connection to the sensor.
|
void | close () |
| Close the conection to the sensor (not need to call it manually unless desired for some reason, since it's called at destructor).
|
void | setDeviceIndexToOpen (int index) |
| Set the sensor index to open (if there're several sensors attached to the computer); default=0 -> the first one.
|
int | getDeviceIndexToOpen () const |
void | setTiltAngleDegrees (double angle) |
| Change tilt angle.
|
double | getTiltAngleDegrees () |
void | enablePreviewRGB (bool enable=true) |
| Default: disabled.
|
void | disablePreviewRGB () |
bool | isPreviewRGBEnabled () const |
void | setPreviewDecimation (size_t decimation_factor) |
| If preview is enabled, show only one image out of N (default: 1=show all).
|
size_t | getPreviewDecimation () const |
double | getMaxRange () const |
| Get the maximum range (meters) that can be read in the observation field "rangeImage".
|
size_t | getRowCount () const |
| Get the row count in the camera images, loaded automatically upon camera open().
|
size_t | getColCount () const |
| Get the col count in the camera images, loaded automatically upon camera open().
|
const mrpt::utils::TCamera & | getCameraParamsIntensity () const |
| Get a const reference to the depth camera calibration parameters.
|
void | setCameraParamsIntensity (const mrpt::utils::TCamera &p) |
const mrpt::utils::TCamera & | getCameraParamsDepth () const |
| Get a const reference to the depth camera calibration parameters.
|
void | setCameraParamsDepth (const mrpt::utils::TCamera &p) |
void | setRelativePoseIntensityWrtDepth (const mrpt::poses::CPose3D &p) |
| Set the pose of the intensity camera wrt the depth camera.
|
const mrpt::poses::CPose3D & | getRelativePoseIntensityWrtDepth () const |
TDepth2RangeArray & | getRawDepth2RangeConversion () |
| Get a reference to the array that convert raw depth values (10 or 11 bit) into ranges in meters, so it can be read or replaced by the user.
|
const TDepth2RangeArray & | getRawDepth2RangeConversion () const |
void | enableGrabRGB (bool enable=true) |
| Enable/disable the grabbing of the RGB channel.
|
bool | isGrabRGBEnabled () const |
void | enableGrabDepth (bool enable=true) |
| Enable/disable the grabbing of the depth channel.
|
bool | isGrabDepthEnabled () const |
void | enableGrabAccelerometers (bool enable=true) |
| Enable/disable the grabbing of the inertial data.
|
bool | isGrabAccelerometersEnabled () const |
void | enableGrab3DPoints (bool enable=true) |
| Enable/disable the grabbing of the 3D point clouds.
|
bool | isGrab3DPointsEnabled () const |
Protected Member Functions |
virtual void | loadConfig_sensorSpecific (const mrpt::utils::CConfigFileBase &configSource, const std::string §ion) |
| Loads specific configuration for the device from a given source of configuration parameters, for example, an ".ini" file, loading from the section "[iniSection]" (see utils::CConfigFileBase and derived classes)
- Exceptions:
-
| This | method must throw an exception with a descriptive message if some critical parameter is missing or has an invalid value. |
|
Protected Attributes |
mrpt::poses::CPose3D | m_sensorPoseOnRobot |
bool | m_preview_window |
| Show preview window while grabbing.
|
size_t | m_preview_window_decimation |
| If preview is enabled, only show 1 out of N images.
|
size_t | m_preview_decim_counter_range |
size_t | m_preview_decim_counter_rgb |
mrpt::gui::CDisplayWindowPtr | m_win_range |
mrpt::gui::CDisplayWindowPtr | m_win_int |
mrpt::utils::TCamera | m_cameraParamsRGB |
| Params for the RGB camera.
|
mrpt::utils::TCamera | m_cameraParamsDepth |
| Params for the Depth camera.
|
mrpt::poses::CPose3D | m_relativePoseIntensityWRTDepth |
| See mrpt::slam::CObservation3DRangeScan for a diagram of this pose.
|
double | m_maxRange |
| Sensor max range (meters).
|
int | m_user_device_number |
| Number of device to open (0:first,...).
|
bool | m_grab_image |
bool | m_grab_depth |
bool | m_grab_3D_points |
bool | m_grab_IMU |
| Default: all true.
|
Private Member Functions |
void | calculate_range2meters () |
| Compute m_range2meters at construction.
|
Private Attributes |
std::vector< uint8_t > | m_buf_depth |
std::vector< uint8_t > | m_buf_rgb |
| Temporary buffers for image grabbing.
|
TDepth2RangeArray | m_range2meters |
| The table raw depth -> range in meters.
|
Member Typedef Documentation
A typedef for an array that converts raw depth to ranges in meters.
Definition at line 199 of file CKinect.h.
Constructor & Destructor Documentation
mrpt::hwdrivers::CKinect::CKinect |
( |
|
) |
|
mrpt::hwdrivers::CKinect::~CKinect |
( |
|
) |
|
Member Function Documentation
void mrpt::hwdrivers::CKinect::calculate_range2meters |
( |
|
) |
[private] |
Compute m_range2meters at construction.
void mrpt::hwdrivers::CKinect::close |
( |
|
) |
|
Close the conection to the sensor (not need to call it manually unless desired for some reason, since it's called at destructor).
void mrpt::hwdrivers::CKinect::disablePreviewRGB |
( |
|
) |
[inline] |
virtual void mrpt::hwdrivers::CKinect::doProcess |
( |
|
) |
[virtual] |
To be called at a high rate (>XX Hz), this method populates the internal buffer of received observations.
This method is mainly intended for usage within rawlog-grabber or similar programs. For an alternative, see getNextObservation()
- Exceptions:
-
| This | method must throw an exception with a descriptive message if some critical error is found. |
- See also:
- getNextObservation
Implements mrpt::hwdrivers::CGenericSensor.
void mrpt::hwdrivers::CKinect::enableGrab3DPoints |
( |
bool |
enable = true |
) |
[inline] |
Enable/disable the grabbing of the 3D point clouds.
Definition at line 317 of file CKinect.h.
void mrpt::hwdrivers::CKinect::enableGrabAccelerometers |
( |
bool |
enable = true |
) |
[inline] |
Enable/disable the grabbing of the inertial data.
Definition at line 313 of file CKinect.h.
void mrpt::hwdrivers::CKinect::enableGrabDepth |
( |
bool |
enable = true |
) |
[inline] |
Enable/disable the grabbing of the depth channel.
Definition at line 309 of file CKinect.h.
void mrpt::hwdrivers::CKinect::enableGrabRGB |
( |
bool |
enable = true |
) |
[inline] |
Enable/disable the grabbing of the RGB channel.
Definition at line 305 of file CKinect.h.
void mrpt::hwdrivers::CKinect::enablePreviewRGB |
( |
bool |
enable = true |
) |
[inline] |
Default: disabled.
Definition at line 270 of file CKinect.h.
Get a const reference to the depth camera calibration parameters.
Definition at line 291 of file CKinect.h.
Get a const reference to the depth camera calibration parameters.
Definition at line 287 of file CKinect.h.
size_t mrpt::hwdrivers::CKinect::getColCount |
( |
|
) |
const [inline] |
Get the col count in the camera images, loaded automatically upon camera open().
Definition at line 284 of file CKinect.h.
int mrpt::hwdrivers::CKinect::getDeviceIndexToOpen |
( |
|
) |
const [inline] |
double mrpt::hwdrivers::CKinect::getMaxRange |
( |
|
) |
const [inline] |
Get the maximum range (meters) that can be read in the observation field "rangeImage".
Definition at line 279 of file CKinect.h.
This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.
- Note:
- This method also grabs data from the accelerometers, returning them in out_obs_imu
The main data retrieving function, to be called after calling loadConfig() and initialize().
- Parameters:
-
| out_obs | The output retrieved observation (only if there_is_obs=true). |
| there_is_obs | If set to false, there was no new observation. |
| hardware_error | True on hardware/comms error. |
- See also:
- doProcess
size_t mrpt::hwdrivers::CKinect::getPreviewDecimation |
( |
|
) |
const [inline] |
TDepth2RangeArray& mrpt::hwdrivers::CKinect::getRawDepth2RangeConversion |
( |
|
) |
[inline] |
Get a reference to the array that convert raw depth values (10 or 11 bit) into ranges in meters, so it can be read or replaced by the user.
If you replace it, remember to set the first and last entries (index 0 and KINECT_RANGES_TABLE_LEN-1) to zero, to indicate that those are invalid ranges.
Definition at line 301 of file CKinect.h.
const TDepth2RangeArray& mrpt::hwdrivers::CKinect::getRawDepth2RangeConversion |
( |
|
) |
const [inline] |
const mrpt::poses::CPose3D& mrpt::hwdrivers::CKinect::getRelativePoseIntensityWrtDepth |
( |
|
) |
const [inline] |
size_t mrpt::hwdrivers::CKinect::getRowCount |
( |
|
) |
const [inline] |
Get the row count in the camera images, loaded automatically upon camera open().
Definition at line 282 of file CKinect.h.
double mrpt::hwdrivers::CKinect::getTiltAngleDegrees |
( |
|
) |
|
virtual void mrpt::hwdrivers::CKinect::initialize |
( |
|
) |
[virtual] |
Initializes the 3D camera - should be invoked after calling loadConfig() or setting the different parameters with the set*() methods.
- Exceptions:
-
| This | method must throw an exception with a descriptive message if some critical error is found. |
Reimplemented from mrpt::hwdrivers::CGenericSensor.
bool mrpt::hwdrivers::CKinect::isGrab3DPointsEnabled |
( |
|
) |
const [inline] |
bool mrpt::hwdrivers::CKinect::isGrabAccelerometersEnabled |
( |
|
) |
const [inline] |
bool mrpt::hwdrivers::CKinect::isGrabDepthEnabled |
( |
|
) |
const [inline] |
bool mrpt::hwdrivers::CKinect::isGrabRGBEnabled |
( |
|
) |
const [inline] |
bool mrpt::hwdrivers::CKinect::isOpen |
( |
|
) |
const |
Whether there is a working connection to the sensor.
bool mrpt::hwdrivers::CKinect::isPreviewRGBEnabled |
( |
|
) |
const [inline] |
virtual void mrpt::hwdrivers::CKinect::loadConfig_sensorSpecific |
( |
const mrpt::utils::CConfigFileBase & |
configSource, |
|
|
const std::string & |
section | |
|
) |
| | [protected, virtual] |
Loads specific configuration for the device from a given source of configuration parameters, for example, an ".ini" file, loading from the section "[iniSection]" (see utils::CConfigFileBase and derived classes)
- Exceptions:
-
| This | method must throw an exception with a descriptive message if some critical parameter is missing or has an invalid value. |
Implements mrpt::hwdrivers::CGenericSensor.
void mrpt::hwdrivers::CKinect::open |
( |
|
) |
|
Try to open the camera (set all the parameters first!) - users may also call initialize(), which in turn calls this.
Raises an exception upon error.
- Exceptions:
-
| std::exception | A textual description of the error. |
void mrpt::hwdrivers::CKinect::setDeviceIndexToOpen |
( |
int |
index |
) |
[inline] |
Set the sensor index to open (if there're several sensors attached to the computer); default=0 -> the first one.
Definition at line 262 of file CKinect.h.
virtual void mrpt::hwdrivers::CKinect::setPathForExternalImages |
( |
const std::string & |
directory |
) |
[virtual] |
Set the path where to save off-rawlog image files (this class DOES take into account this path).
An empty string (the default value at construction) means to save images embedded in the rawlog, instead of on separate files.
- Exceptions:
-
| std::exception | If the directory doesn't exists and cannot be created. |
Reimplemented from mrpt::hwdrivers::CGenericSensor.
void mrpt::hwdrivers::CKinect::setPreviewDecimation |
( |
size_t |
decimation_factor |
) |
[inline] |
If preview is enabled, show only one image out of N (default: 1=show all).
Definition at line 275 of file CKinect.h.
void mrpt::hwdrivers::CKinect::setRelativePoseIntensityWrtDepth |
( |
const mrpt::poses::CPose3D & |
p |
) |
[inline] |
void mrpt::hwdrivers::CKinect::setTiltAngleDegrees |
( |
double |
angle |
) |
|
Change tilt angle.
- Note:
- Sensor must be open first.
Member Data Documentation
Temporary buffers for image grabbing.
Definition at line 371 of file CKinect.h.
Params for the Depth camera.
Definition at line 361 of file CKinect.h.
Params for the RGB camera.
Definition at line 360 of file CKinect.h.
Default: all true.
Definition at line 368 of file CKinect.h.
Sensor max range (meters).
Definition at line 364 of file CKinect.h.
Show preview window while grabbing.
Definition at line 341 of file CKinect.h.
If preview is enabled, only show 1 out of N images.
Definition at line 342 of file CKinect.h.
The table raw depth -> range in meters.
Definition at line 372 of file CKinect.h.
Number of device to open (0:first,...).
Definition at line 366 of file CKinect.h.