{"id":27,"date":"2020-05-02T18:31:25","date_gmt":"2020-05-02T18:31:25","guid":{"rendered":"https:\/\/jasoninerlangen.myqnapcloud.com:8081\/WordPress\/?p=27"},"modified":"2020-05-28T11:41:48","modified_gmt":"2020-05-28T10:41:48","slug":"project-computer-vision-3d-box-detection","status":"publish","type":"post","link":"https:\/\/jasoninerlangen.myqnapcloud.com\/WordPress\/project-computer-vision-3d-box-detection\/","title":{"rendered":"Project Computer Vision – 3D Box Detection"},"content":{"rendered":"
Motivation<\/h6>\n

In this exercise, we acquire a brightness image (424×512), a depth image (424×512) and a 3D point cloud matrix (424x512x3) from Microsoft kinect camera.  From the datasets stated above, we are asked to estimate the dimension of the object that lies in the center of the scene.<\/p>\n

Visualization of the raw data <\/h6>\n\n\n
\"\"
Brightness (after Histogram Equalization)<\/em><\/figcaption><\/figure>\n\n\n\n
\"\"
Depth (after Histogram Equalization)<\/figcaption><\/figure>\n\n\n\n
\"\"
View point: azimuth = -45\/ elevation = -135<\/figcaption><\/figure>\n\n\n
RANSAC floor plane fitting<\/h6>\n\n\n
\"\"
Floor plane segmentation (RANSAC fitting)<\/figcaption><\/figure>\n\n\n
Relation between camera and point cloud<\/h6>\n\n\n
\"\"<\/figure>\n\n\n
Fitting top plane and refining of potential  region<\/h6>\n

Excluding the invalid set of 3D point cloud and floor plane segmentation from above, there is large, well connected region in center of the image. However,  if we apply RANSAC algorithm again to fit the top plane of the box, some regions which are near to the image boundary are often falsely regarded as “top plane”, so that we apply flood-fill algorithm to mark the only region in the middle.<\/p>\n\n\n

\"\"<\/figure>\n\n\n\n
def floodfill_topplane(topPlaneImg):\n    \"\"\"\n    apply floodfill algorithm to define more concise top plane\n    Input:\n    topPlaneImg(array(bool)): top plane image after RANSAC plane fitting\n    Return:\n    returnImg(array(bool))\n    \"\"\"\n    (height,width) = topPlaneImg.shape\n    returnImg = np.zeros(topPlaneImg.shape,dtype=np.int64)\n    maskIndices = np.arange(height*width).reshape(height,width)\n    validSet = maskIndices[topPlaneImg.astype(bool)]\n    # retrieve the median \"pixel\"\n    idx = np.floor(validSet.size\/2).astype(int)\n    x = validSet[idx]%width\n    y = np.floor(validSet[idx]\/width).astype(int)\n    mask = skimage.morphology.flood(np.int64(topPlaneImg),(y,x),connectivity=2)\n    returnImg[mask] = 1\n    return returnImg.astype(bool)<\/pre><\/div>\n\n\n\n
\"\"
Top plane segmentation (RANSAC fitting + Floodfill)<\/figcaption><\/figure>\n\n\n
Sorting the result of corner detector<\/h6>\n
The output of corner detector is sorted w.r.t. “cornerness<\/strong>” which is used to specify potential corner candidates in the corner detector, however, if we draw the edges of the box directly with the order of the output of the corner detector, it is not guaranteed that we can generate a convex polygon. As a consequence, we compute the azimuth of 4 corners w.r.t. the geometric center of these and sort it  in an ascending order which denotes that the range of angle lies within -\u03c0 to \u03c0 and the direction of rotation is counter clockwise.<\/p>\n\n\n
\"\"<\/figure>\n\n\n\n
def sort_corners(corners):\n    \"\"\"\n    sort corners' azimuth in counter clockwise order (3rd->4th->1st->2nd quadrant) \n    w.r.t geometric center(mean value of 4 vertices) of the box\n    Input:\n    corners(array(int)): output(ndarray) of corner detection\n    Return:\n    returnMatx(array(int))\n    \"\"\"\n    returnMatx = np.zeros(corners.shape,dtype=np.int64)\n    ### check Quadrant\n    centroid = np.mean(corners,axis=0)\n    vector = corners - centroid\n    angle = np.arctan2(vector[:,0],vector[:,1])\n    ind = np.argsort(angle)\n    for i in range(ind.size):\n        returnMatx[i,:] = corners[ind[i],:]\n    return returnMatx<\/pre><\/div>\n\n\n\n
\"\"
3D Box Detection<\/figcaption><\/figure><\/div>\n\n\n\n
\"\"
3D Point Cloud (blue: floor plane\/orange: top plane)<\/figcaption><\/figure>\n\n\n
Box dimension estimation<\/h2>\n\n\n\n\n\t\n\n\t\n\t\n\t\n\t
<\/td>Length(m)<\/th>Width(m)<\/th>Height(m)<\/th>\n<\/tr>\n<\/thead>\n
cloud1<\/td>0.4402<\/td>0.3319<\/td>0.1837<\/td>\n<\/tr>\n
cloud2<\/td> 0.4386<\/td>0.3381<\/td> 0.1904<\/td>\n<\/tr>\n
cloud3<\/td> 0.4327<\/td>0.3416<\/td> 0.1932<\/td>\n<\/tr>\n
cloud4<\/td>0.4824<\/td>0.3096<\/td>0.1828<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
Hardware<\/h2>\n\n\n\n\n\t\n\n\t\n\t\n\t
CPU<\/th>Intel 8700K @4.4GHz all cores<\/th>\n<\/tr>\n<\/thead>\n
RAM<\/td>16GB @3000MHz<\/td>\n<\/tr>\n
NVMe SSD<\/td>512GB<\/td>\n<\/tr>\n
GPU<\/td>GTX 1070Ti 8GB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n
So far as we know, we can find out that the object is a DHL packet box size L
https:\/\/www.dhl.de\/en\/privatkunden\/pakete-versenden\/pakete-abgeben\/verpacken.html<\/a><\/p>\n
The execution time for both top plane and floor plane RANSAC plane fitting is about 7 seconds.<\/p>\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Motivation In this exercise, we acquire a brightness image (424×512), a depth image (424×512) and a 3D point cloud matrix […]<\/p>\n","protected":false},"author":1,"featured_media":5,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[2],"tags":[7],"class_list":["post-27","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-coding","tag-computer-vision"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/jasoninerlangen.myqnapcloud.com\/WordPress\/wp-json\/wp\/v2\/posts\/27","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jasoninerlangen.myqnapcloud.com\/WordPress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jasoninerlangen.myqnapcloud.com\/WordPress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jasoninerlangen.myqnapcloud.com\/WordPress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/jasoninerlangen.myqnapcloud.com\/WordPress\/wp-json\/wp\/v2\/comments?post=27"}],"version-history":[{"count":53,"href":"https:\/\/jasoninerlangen.myqnapcloud.com\/WordPress\/wp-json\/wp\/v2\/posts\/27\/revisions"}],"predecessor-version":[{"id":200,"href":"https:\/\/jasoninerlangen.myqnapcloud.com\/WordPress\/wp-json\/wp\/v2\/posts\/27\/revisions\/200"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/jasoninerlangen.myqnapcloud.com\/WordPress\/wp-json\/wp\/v2\/media\/5"}],"wp:attachment":[{"href":"https:\/\/jasoninerlangen.myqnapcloud.com\/WordPress\/wp-json\/wp\/v2\/media?parent=27"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jasoninerlangen.myqnapcloud.com\/WordPress\/wp-json\/wp\/v2\/categories?post=27"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jasoninerlangen.myqnapcloud.com\/WordPress\/wp-json\/wp\/v2\/tags?post=27"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}