While Decision Support Systems (DSS) have a long history, their usefulness for non-experts outside specific organisations has not lived up to the promise. A key reason for this is the high cost associated with populating the underlying knowledge bases. In this article, we describe how DSSs can leverage crowds and their wisdom in constructing knowledge bases to overcome this challenge. We also demonstrate how to construct DSSs on-the-fly using the collected data. Our user-driven laboratory studies focus on user perceptions of the concept itself, and motives for contributing and using such a DSS. To the best of our knowledge, our approach and implementation are the first to demonstrate such use of crowdsourcing in building DSSs.