The objective of this study is to suggest a method for classifying archeological fragments into groups. For this task, the method suggested begins with conversion of images from their original RGB color to a Hue, Saturation and Value (HSV) color. From that point forward, a 2D median filtering algorithm is implemented to remove any resultant noise. Next, each image is separated into six sub-block of equivalent size. In order to extract the feature for each sub-block, it is represented as a vector intersection of colors between each part of the image and the corresponding parts of the five remaining images. At this stage, we obtain a vector that consists of the six values for each image. For the last stage, a Self-Organization Map (SOM) Neural Network classifies the fragments into groups relying upon their HSV color feature. The algorithm was tested on several images of pottery fragments and the results achieved demonstrate this approach is promising and is able to cluster fragments into groups with high precision.