中文概要
众包近来已经成为了许多领域解决大规模人力需求的有效范式。但是任务发布者通常预算有限，因此有必要使用一种明智的预算分配策略以获得更好的质量。在这篇论文中，我们在 HodgeRank 框架中研究了用于主动采样策略的信息最大化原理；其中HodgeRank 这种方法基于多个众包工人（worker）的配对排名数据的霍奇分解（Hodge Decomposition）。
该原理给出了两种主动采样情况：费希尔信息最大化（Fisher information maximization）和贝叶斯信息最大化（Bayesian information maximization）。其中费希尔信息最大化可以在无需考虑标签的情况下基于图的代数连接性（graph algebraic connectivity）的序列最大化而实现无监督式采样；贝叶斯信息最大化则可以选择从先验到后验的过程有最大信息增益的样本，这能实现利用所收集标签的监督式采样。实验表明，相比于传统的采样方案，我们提出的方法能提高采样效率，因此对实际的众包实验而言是有价值的。
英文概要
Recently, crowdsourcing has emerged as an effective paradigm for human-powered large scale problem solving in various domains. However, task requester usually has a limited amount of budget, thus it is desirable to have a policy to wisely allocate the budget to achieve better quality. In this paper, we study the principle of information maximization for active sampling strategies in the framework of HodgeRank, an approach based on Hodge Decomposition of pairwise ranking data with multiple workers.
The principle exhibits two scenarios of active sampling: Fisher information maximization that leads to unsupervised sampling based on a sequential maximization of graph algebraic connectivity without consideringlabels; and Bayesian information maximization that selects samples with the largest information gain from prior to posterior, which gives a supervised sampling involving the labels collected. Experiments show that the proposed methods boost the sampling efficiency as compared to traditional sampling schemes and are thus valuable to practical crowdsourcing experiments.
英文演讲PPT
In this paper, we present a principle of active sampling based on information maximization in the framework of HodgeRank.
Our contributions in this work are three fold:
1. A new version of Hodge decomposition of pairwise comparison data with multiple voters is presented. Within this framework, two schemes of information maximization, Fisher and Bayesian that lead to unsupervised and supervised sampling respectively, are systematically investigated.
2. Closed form update and a fast online algorithm are derived for supervised samplingwith Bayesian information maximization for HodgeRank, which is shown faster and more accurate than the state-of-the-art method Crowd-BT (Chen et al.2013).
3. These schemes exhibit better sampling efficiency than random sampling as well as a better loop-free control in clique complex of paired comparisons, thus reduce the possibility of causing voting chaos by harmonic ranking (Saari 2001) (i.e., the phenomenon that the inconsistency of preference data may lead to totally different aggregate orders using different methods).
