Purpose: Instructors' perspectives on students' evidence-based practice (EBP) competencies may not reflect their true performance. Using an objective assessment tool, this pilot study explored the effectiveness of EBP teaching for undergraduate year (UGY) medical students at the beginning of UGY5-7 levels, and offered important suggestions for developing an EBP curriculum. Methods: An assessment programme was conducted for novice (UGY5) and advanced beginner-1 (UGY6) and -2 (UGY7) students with the Knowledge of Research Evidence Competencies (K-REC) instrument. Students' performance on EBP knowledge and skills were scored and analysed. Results: A low proportion of novice and advanced beginner-1 students were discovered to be competent at understanding biostatistics and Boolean search strategies. Critical appraisal skills were found to be undeveloped in > 50% of novice level students, while advanced beginner-1 students obtained significantly higher scores than novice level students. Novice level students in 2013 had significantly higher scores than novice level students in 2012, specifically in the categories of problems, interventions, comparisons and outcomes formation, study design knowledge, search strategy, critical appraisal, and levels of evidence processes. A significant improvement in EBP competencies was observed in the cohort over the duration of their novice to advanced beginner-2 levels. However, there was no significant enhancement in their medical subject headings search strategy and biostatistics. No change in total scores was recorded for two advanced beginner-2 cohorts between 2014 and 2016. Conclusions: The K-REC assessment results of the achievements and insufficiencies of EBP learning at novice and advanced beginner stages are helpful for curriculum reform. Teaching of research biostatistics, search strategies, and critical appraisal skills should be incorporated at the UGY4 level or sooner. EBP competencies after curriculum reform and resource coordination are being investigated.