[CV] Segmentation and Feature Extraction of Fingernail Plate and Lunula Based on Deep Learning

Fan, Y., You, M., Ge, J., & Zhai, G. (2024, July 26). Segmentation and feature extraction of fingernail plate and lunula based on deep learning [Preprint]. bioRxiv. https://doi.org/10.1101/2024.07.26.605289

 

dataset

a total of 6250 nail photos, each with a photo size of 300*300

manually labeled using labelme

DeepLabv3+

NailNet을 이해하기 위한 DeepLabv3+ 모델 논문 Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017, June 17). Rethinking Atrous Convolution for Semantic Image Segmentation (arXiv:1706.05587) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.1706.05587

Encoder

• ResNet101 backbone
  • standard convolution layers → low-level segmentation map from Stage2 and the high-level segmentation map from Stage4 생성
• ASPP(Atrous Spatial Pyramid Pooling)astrous convolution은 pooling 없이 receptive field 확보, 공간 해상도 손실 없이 성능 향상 가능
  • input: high-level feature map
  • multi-scale feature map(3x3 conv. layers and 1 global average pooling layer) 추출을 위해 5 parallel heads 사용
• 같은 연산량으로 더 넓은 영역을 보기 위함
• Depthwise separable convolution

  

Decoder

• concatenates unsample ASPP output and low-level feature

NailNet

• nail segmentation : DL based network for nail plate and lunula segmentation

→ nail segmentation & finger classification 동시에 수행

 

DeepLabv3+와의 차이점

• Encoder = ResNet50 backbone (ResNet101 대신)
• Nail image는 3개 클래스만 포함(nail, lunula, background)
• depthwise separable convolution 사용 → 모델 경량화
• high-level feature(ASPP module) & low-level feature(ResNet in decoder) 결합. feature gap 줄이기 위해 feature pyramid 사용
• classification head 추가 - parallel w/ the original segmentation layer

 

Phenotype Classifier

• regional color: nail plate 중앙 색
• global color: 손톱 전체의 위에서 아래까지 각 픽셀 행(row)별 색상 평균값

free edge, lunula, and obvious black or white spots on the fingernail plate는 제거

고려해야할 사항: nail plate shape / lunula color / lunula ratio

1. NailNet 사용 (nail classifier 없이) → 300x300 pixels이 충분함을 확인
2. 라벨링 (nail, lunula, background) → NailNet으로 정제된 segmentation, classification 결과 도출

segmentation 성능 평가: mIoU(mean Intersection over Union)

K = # of pixel classes (현 연구에서 3개; background, nail, lunula)

pij

i: # of pixels class

j: predicted # of pixels class

classification 성능 평가: accuracy

optimizer: Nesterov momentum optimizer

그냥 momentum보다 μv_{t−1} 만큼 앞의 기울기를 계산하여 정확한 방향, overshooting 감소

하이퍼파라미터 튜닝은 empirical testing

data augmentation: random horizontal flipping, rotation, color jittering, scaling

early stopping

 

한계 

nail edge: dirt, damage 등의 요소로 평균에서 벗어나는 경우가 많았음 → the midpoint of the nail as the center and averaged the colors in a 20x20 pixel grid 만 뽑아옴

 

추가로 해볼 만한 내용

• surface texture phenotype of nails 연구