Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

From binary masks with holes to COCO JSON format #574

Closed
2 tasks done
MihaiDavid05 opened this issue Nov 4, 2023 · 3 comments
Closed
2 tasks done

From binary masks with holes to COCO JSON format #574

MihaiDavid05 opened this issue Nov 4, 2023 · 3 comments
Labels
duplicate This issue or pull request already exists enhancement New feature or request

Comments

@MihaiDavid05
Copy link

MihaiDavid05 commented Nov 4, 2023

Search before asking

  • I have searched the Supervision issues and found no similar feature requests.

Description

The code does not seem to support masks with holes, feature that is quite important for instance segmentation (especially for food images datasets for example). By using the RLE format from COCO this would be attainable.

Use case

It will help improve the binary masks exportation to COCO JSON format, by relying on RLE encoding when needed, otherwise keeping the polygon format.

For example, if we take the binary mask of a specific class/category, from an image mask, we can find all possible contours in the class-mask (using RETR_TREE in openCV). If there is only 1 contour in the mask that is for sure 1 full polygon, therefore, we keep the polygon encoding. Otherwise, if it means that either there are multiple instances in the mask (multiple contours), either there are some holes in some of the instances (we can check the contour hierarchy for contours that are parents to others). In this case we can go from binary to uncompressed RLE, which will encode the possible holes in some instances. (for finding the area and bbox for this cases, it is needed to go from uncompressed RLE to compressed RLE first).

Additional

I built such an exporter here based on other sources, and I could contribute with a PR, if needed.

Are you willing to submit a PR?

  • Yes I'd like to help by submitting a PR!
@MihaiDavid05 MihaiDavid05 added the enhancement New feature or request label Nov 4, 2023
Copy link
Contributor

github-actions bot commented Nov 4, 2023

Hello there, thank you for opening an Issue ! 🙏🏻 The team was notified and they will get back to you asap.

@SkalskiP SkalskiP added the duplicate This issue or pull request already exists label Nov 15, 2023
@SkalskiP
Copy link
Collaborator

It looks like this issue is a duplicate of #373. For this reason, I am closing this issue. If I misunderstood something, feel free to reopen the issue.

@ryouchinsa
Copy link

ryouchinsa commented Nov 21, 2023

Using the script general_json2yolo.py, you can convert the RLE mask with holes to the YOLO segmentation format.

The RLE mask is converted to a parent polygon and a child polygon using cv2.findContours().
The parent polygon points are sorted in clockwise order.
The child polygon points are sorted in counterclockwise order.
Detect the nearest point in the parent polygon and in the child polygon.
Connect those 2 points with narrow 2 lines.
So that the polygon with a hole is saved in the YOLO segmentation format.

def is_clockwise(contour):
    value = 0
    num = len(contour)
    for i, point in enumerate(contour):
        p1 = contour[i]
        if i < num - 1:
            p2 = contour[i + 1]
        else:
            p2 = contour[0]
        value += (p2[0][0] - p1[0][0]) * (p2[0][1] + p1[0][1]);
    return value < 0

def get_merge_point_idx(contour1, contour2):
    idx1 = 0
    idx2 = 0
    distance_min = -1
    for i, p1 in enumerate(contour1):
        for j, p2 in enumerate(contour2):
            distance = pow(p2[0][0] - p1[0][0], 2) + pow(p2[0][1] - p1[0][1], 2);
            if distance_min < 0:
                distance_min = distance
                idx1 = i
                idx2 = j
            elif distance < distance_min:
                distance_min = distance
                idx1 = i
                idx2 = j
    return idx1, idx2

def merge_contours(contour1, contour2, idx1, idx2):
    contour = []
    for i in list(range(0, idx1 + 1)):
        contour.append(contour1[i])
    for i in list(range(idx2, len(contour2))):
        contour.append(contour2[i])
    for i in list(range(0, idx2 + 1)):
        contour.append(contour2[i])
    for i in list(range(idx1, len(contour1))):
        contour.append(contour1[i])
    contour = np.array(contour)
    return contour

def merge_with_parent(contour_parent, contour):
    if not is_clockwise(contour_parent):
        contour_parent = contour_parent[::-1]
    if is_clockwise(contour):
        contour = contour[::-1]
    idx1, idx2 = get_merge_point_idx(contour_parent, contour)
    return merge_contours(contour_parent, contour, idx1, idx2)

def mask2polygon(image):
    contours, hierarchies = cv2.findContours(image, cv2.RETR_CCOMP, cv2.CHAIN_APPROX_TC89_KCOS)
    contours_approx = []
    polygons = []
    for contour in contours:
        epsilon = 0.001 * cv2.arcLength(contour, True)
        contour_approx = cv2.approxPolyDP(contour, epsilon, True)
        contours_approx.append(contour_approx)

    contours_parent = []
    for i, contour in enumerate(contours_approx):
        parent_idx = hierarchies[0][i][3]
        if parent_idx < 0 and len(contour) >= 3:
            contours_parent.append(contour)
        else:
            contours_parent.append([])

    for i, contour in enumerate(contours_approx):
        parent_idx = hierarchies[0][i][3]
        if parent_idx >= 0 and len(contour) >= 3:
            contour_parent = contours_parent[parent_idx]
            if len(contour_parent) == 0:
                continue
            contours_parent[parent_idx] = merge_with_parent(contour_parent, contour)

    contours_parent_tmp = []
    for contour in contours_parent:
        if len(contour) == 0:
            continue
        contours_parent_tmp.append(contour)

    polygons = []
    for contour in contours_parent_tmp:
        polygon = contour.flatten().tolist()
        polygons.append(polygon)
    return polygons 

def rle2polygon(segmentation):
    if isinstance(segmentation["counts"], list):
        segmentation = mask.frPyObjects(segmentation, *segmentation["size"])
    m = mask.decode(segmentation) 
    m[m > 0] = 255
    polygons = mask2polygon(m)
    return polygons

The RLE mask.

スクリーンショット 2023-11-22 1 57 52

The converted YOLO segmentation format.

スクリーンショット 2023-11-22 2 11 14

To run the script, put the COCO JSON file coco_train.json into datasets/coco/annotations.
Run the script. python general_json2yolo.py
The converted YOLO txt files are saved in new_dir/labels/coco_train.

スクリーンショット 2023-11-23 16 39 21

Edit use_segments and use_keypoints in the script.

if __name__ == '__main__':
    source = 'COCO'

    if source == 'COCO':
        convert_coco_json('../datasets/coco/annotations',  # directory with *.json
                          use_segments=True,
                          use_keypoints=False,
                          cls91to80=False)

To convert the COCO bbox format to YOLO bbox format.

use_segments=False,
use_keypoints=False,

To convert the COCO segmentation format to YOLO segmentation format.

use_segments=True,
use_keypoints=False,

To convert the COCO keypoints format to YOLO keypoints format.

use_segments=False,
use_keypoints=True,

This script originates from Ultralytics JSON2YOLO repository.
We hope this script would help your business.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate This issue or pull request already exists enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants