Three of the main criticisms of Open Access have already been addressed in other tabs.
Author fees are a significant burden that most faculty cannot or will not afford without support from another source. The two most likely sources are research grants and institutional funding to cover the faculty's author fees.
Tenure and promotion policies and practices need to be such that that publication in Open Access journals is valued equally with conventional journals, so that faculty will choose where to publish based on academic and scholarly communication factors rather than complying with an arbitrary requirement for the sake of their career.
Not all scholarly journals are equally rigorous, quality controlled or significant in the scholarly conversation. This is true of both conventional and Open Access journals. There has been a proliferation of low quality and fraudulent Open Access journals, which has tarnished its reputation, but it is important to consider that at the same time, there was also an explosion in the number of low quality and fraudulent conventional journals, some of them from extremely reputable publishers. This highlights the need to examine the actual selection and peer review processes of individual journals, and even to look at the merits of individual articles, rather than to accept an aggregate reputation or appearance of scholarliness.
The hardest challenge to Open Access is the question of whether it is a viable business model. There are three possible answers to this question:
The short version is that most scholars agree that Open Access is viable and sustainable at least as something that operates alongside conventional publishing, improving access and providing competition to keep prices down.
There is also the question of whether Open Access is the cheapest model for institutions of higher education to support. While it certainly has its advantages, it is not without costs. In fact, no one expects much savings from Open Access. Any money redirected from overpriced library subscriptions will probably be moved over to paying for the costs of creating and evaluating knowledge in the form of Open Access publications.
While this topic is much discussed, it is (forgive me) a largely academic discussion, since institutions of higher education treat economic advantage as only one consideration among several, including academic freedom, equity of access, justice and sustainability, and the progress of human knowledge.