Abstract Most clustering algorithms are proposed by optimizing an objective function which is based on a within-cluster scatter matrix. In this paper, we define the fuzzy scatter matrix and propose a novel fuzzy clustering algorithm, called the fuzzy compactness & separation (FCS), based on a fuzzy scatter matrix which the FCS algorithm is derived by the minimization of the compactness measure and simultaneously the maximization of the separation measure. The compactness is measured by a fuzzy within-cluster variation and the separation is measured by a fuzzy between-cluster variation. The proposed FCS objective function is a modification of the FS validity index proposed by Fukuyama and Sugeno and also a generalization of the fuzzy c-means (FCM) clustering model. The FCS algorithm assigns a crisp boundary (cluster kernel) for each cluster such that hard memberships and fuzzy memberships could be co-existed in the clustering results. Thus, FCS can be seen as a clustering algorithm with a novel sense between hard c-means and fuzzy c-means. Some numerical examples are demonstrated to show its robust properties and effectiveness.