@Greg What part of the 14th amendment's equal protection and due process clauses do you not understand?
All persons born or naturalized in the United States, and subject to the jurisdiction thereof, are citizens of the United States and of the State wherein they reside. No State shall make or enforce any law which shall abridge the privileges or immunities of citizens of the United States; nor shall any State deprive any person of life, liberty, or property, without due process of law; nor deny to any person within its jurisdiction the equal protection of the laws.
Edit: forgot to nention the due process clause
As Berkeley is a state institution, it falls under the purview of the 14th amendment and, therefore, the first amendment as well.
@Kate_Monster If the group withdrew their invitation, that's one thing, although if the rationale was due to being concerned about the security of their guest... that is certainly a frightening concept, but more one due to individuals blocking free speech than to a government institution blocking free speech.