提升多作品多評委評選品質之研究：以專題作品為例

A STUDY ON IMPROVING THE RANKING QUALITY UNDER MULTI-WORK AND MULTI-REVIEWER: AN EXAMPLE OF PROJECT WORKS RANKING

黃允成(Yun-Cheng Huang) ； 潘彥如(Yen-Ru Pan)

《品質學報》 25卷2期 (2018/04) Pp. 69-88

https://doi.org/10.6220/joq.201804_25(2).0001

摘要

本研究主要係針對組織評選機制進行研究，組織內部經常對專案成果、工作績效或作品進行評選，但常有不公或護短現象發生。為能公正地進行評比，本研究提出多專案（或多作品）多評委多階段之評選機制設計，據以找出優秀之作品。以區間尺度為評選基準，設計出兩種評分方式，分別為「循序兩階段」與「平行兩階段」評分法。然因內部評審委員也參與相關作品之規劃與指導，因此評分法皆採取利益迴避原則進行評審（即只評審他人之作品，不評審與自己相關之作品）。考量每一位評審委員（不論內部委員或外部委員）之評審標準寬嚴不一，易產生評選方式不當而致排序 逆轉之現象。本文首先證明排序逆轉現象之存在，再提出標準化與線性轉換之方法，加以改善。最後，經由範例分析證實不同的排序方法會產生不同的排序結果，因此，選擇排序方法時應考慮其可行性、公平性與經濟性。

關鍵字

評選機制 ； 排序逆轉 ； 標準化 ； 線性轉換

並列摘要

In this paper, we proposed a multi-work, multi-reviewer and multi-stage ranking mechanism to evaluate works, projects or alternatives. We created Parallel and Sequential scoring methods, respectively, that based on institution requirement. At first stage of Sequential scoring method, each internal reviewer evaluated all 20 project works. Second, we delivered the top 10 project works to external experts and got the final results as well. On the other hand, at first stage of Parallel scoring method, we gathered the internal and external reviewers' rating scores based on project documents. Second we obtained the scores through by project presentations and Q&A. However, if the project was guided by internal reviewer, then the reviewer will avoid reviewing the project. Due to the rating standard of reviewers are not the same each other, so we need to design a transformation mechanism to normalize the scores such that the final ranking results are more fair and objective. At first, we proved the existence of ranking reverse, and then proposed the method to standardize the raw scores and the linear transformation method was also taken to public for easy understand the transformed scores. Finally, we concluded that different rating methods will cause different ranking results; therefore, to design a rating mechanism needs to consider its feasibility fairness and economic evaluation.

並列關鍵字

ranking mechanism ； ranking reverse ； standardize ； linear transformation

參考文獻

Bian, Y. and Li, S., 2011, Ranking decision making units with large set of correlated performance indicators: a method based on Gram-Schmidt process, Expert Systems with Applications, 38(8), 10518-10523.

Cai, L., Qu, S., Yuan, Y., and Yao, X., 2015, A clustering-ranking method for many-objective optimization, Applied Soft Computing, 35, 681-694.

Chen, Y., Kilgour, D. M., and Hipel, K. W., 2011, An extreme-distance approach to multiple criteria ranking, Mathematical and Computer Modelling, 53(5-6), 646-658.

Fernandez, E. and Olmedo, R., 2005, An agent model based on ideas of concordance and discordance for group ranking problems, Decision Support Systems, 39(3), 429-443.

Foroughi, A. A. and Tamiz, M., 2005, An effective total ranking model for a ranked voting system, Omega, 33(6), 491-496.