Purpose: The study attempted to construct and validate a systemized evaluation for critical thinking performance in the Taiwanese medical cultural context. Methods: Literature related to critical thinking and rubric development was reviewed and discussed in expert panels in order to set up a theoretical framework for the proposed Analytic Critical Thinking Scoring Rubric for Healthcare Students and Providers (ACTSR‒HSP). Three experts used the index of Item Objective Congruence (IOC) for the critical thinking and scoring rubric to evaluate the congruence between the qualitative scoring items and the objectives. Percent exact agreement and percent adjacent agreement were applied to examine the inter-rater consensus estimates. Pearson correlation coefficients for inter-rater and intra-rater reliabilities were also calculated for consistency estimates. Results: The IOC results showed that among 18 qualitative criteria statements, 14 statements fell in the acceptable IOC range of 0.5 to 1.0; 4 items not meeting the criteria were revised and reviewed to confirm the content validity. The inter-rater consensus estimates were all above 70%, between 90% and 100%. The inter-rater consistency estimates were between 0.83 and 0.89; the intra-rater constancy estimates were between 0.95 and 0.98. Conclusion: Results of content validity, IOC, inter-rater reliabilities, and intra-reliabilities proved that the ACTSR‒HSP could be a valid and systemized evaluation tool for critical thinking performance in the Taiwanese medical cultural context.