In the thriving field of network studies, there has been a recently emerging practice of comparing the optimal modularity scores across different social networks to evaluate the variation of network-module-related substantive concepts, such as the levels of consensus, polarization, or community boundary rigidity. Although the rationale for this practice is comprehensible, we caution that it suffers from various conceptual and empirical issues. Conceptually, modularity scores may misalign with the substantive concepts in which social scientists are interested. Empirically, the estimated optimal modularity scores across social networks are highly sensitive to the algorithms employed and various network characteristics that are irrelevant to those substantive concepts, hence biasing the comparison results. We illustrate these conceptual and analytical problems with toy examples and systematic simulations. We demonstrate the practical significance of these lessons by replicating an empirical study that examines the temporal trend of modularity scores for job mobility networks to evaluate the evolution of mobility rigidity in the US labor market. We conclude with a summary of these lessons and offer practical suggestions for future applications of modularity comparison.