I think the work of the growth economist Garrett Jones on the externalities of human capital needs to be mentioned and understood to see the most plausible economic case against complete labor market liberalization. He finds very large externalities in macro regressions, and there are plausible micro-foundations: higher-IQ individuals cooperate more in Prisoner's Dilemmas, save and invest more, and have policy views that more closely resemble those of economists. Similar results have been found by other economists, such as Hanushek and Rindermann, using PISA and other international assessments of academic performance.
Citizens of the poorest developing countries where surveys show the largest desire to migrate tend to have very low levels of human capital using multiple measures including educational attainment, PISA scores, and IQ compared to developed countries (China has strong test performance, but only a smaller portion of the population expresses desire to migrate, the population of migratory age is a small portion of the world's potential migrants, and it is already rapidly developing). So this is not just a theoretical issue: full labor mobility would drastically lower the average level of education and human capital in immigrant-receiving countries.
Clemens' work is an excellent contribution to the debate, but it assumes that differences in productivity between countries are exogenous to the population inhabiting those countries. If the human capital, political views, and other characteristics of migrants affect the level of productivity in countries through externalities, such as on institutions, then this assumption breaks down. The analysis would then need to weigh the gains from moving people to better institutions against any deleterious (or positive) effects on the quality of those institutions.
This could still easily yield a huge net gain, but the case is not as simple as Clemens' presentation.