Mark says that at the foot of the cross was “Mary the mother of the younger James and of Joses (Joseph), and Salome” This was obviously not Mary the mother of Jesus, so there is another Mary with sons called James and Joseph.
Matthew similarly says of the women at the foot of the cross “Among them were Mary Magdalene and Mary the mother of James and Joseph” (Mt 26:56)
Luke says that at the tomb were “Mary Magdalene, Joanna, and Mary the mother of James” (Lk 24:10)
John says “Standing by the cross of Jesus were his mother and his mother’s sister, Mary the wife of Clopas, and Mary of Magdala.” (Jn 19:25). Now this could mean that Jesus’ mother’s sister was there (whatever is meant by “sister”) and Mary the wife of Clopas or they were the same person, but either way there were at least three Mary’s at the cross – Mary the mother of Jesus, Mary the wife of Clopas and Mary of Magdalene. Now Mary the mother of James and Joseph could have been a fourth or she could have been Mary wife of Clopas. Either way Mary the mother of Jesus was not the mother of James and Joseph mentioned as Jesus’ brothers. And since they were listed first, neither was Simon and Judas, since if the were they would hardly have been listed after non-brothers.
The Church historian Eusebius quoting from Hegesippus (110-180 AD) writes
"After the martyrdom of James and the conquest of Jerusalem which immediately followed, it is said that those of the apostles and disciples of the Lord that were still living came together from all directions with those that were related to the Lord according to the flesh (for the majority of them also were still alive) to take counsel as to who was worthy to succeed James. They all with one consent pronounced Symeon, the son of Clopas, of whom the Gospel also makes mention; to be worthy of the episcopal throne of that parish. He was a cousin, as they say, of the Saviour. For Hegesippus records that Clopas was a brother of Joseph."
So Symeon (Simeon, Simon) was the cousin of Jesus, and Mary Clopas was therefore the sister-in-law of Mary the mother of Jesus. Again note the loose use of relationships. Mary Clopas is referred to as Mary’s “sister” in Jn 19:25 when she is actually her sister-in-law.
In the book of Jude he says “Jude, a slave of Jesus Christ and brother of James” (Jude 1:1) So Jude (or Judas) is probably the brother of James the son of Clopas.
Then also Luke when listing the apostles says James, son of Alpheus. But the Aramaic Alpheus can be rendered in Greek as either Alpheus or Clopas. So again James, the “brother” of the Lord is probably the son of Clopas.
Then there is this text:
"So his brothers said to him, “Leave here and go to Judea, so that your disciples also may see the works you are doing. No one works in secret if he wants to be known publicly. If you do these things, manifest yourself to the world.” (Jn 7:3-4) If Mary had other children these “brothers” would be younger that Jesus. But in that culture to speak to an older brother in this way would be extremely rude. So it would seem likely they were older than Jesus and therefore could not have been Mary’s children.
Finally there is the Protoevangelium of James. This is an early document which describes the early life of Mary. It is not canonical but seems to have been accepted by several early fathers as historical. In this Joseph, the husband of Mary, is described as an older man with children by a previous marriage.
This is from Catholic Answers:
"Backing up the testimony of Scripture regarding Mary’s perpetual virginity is the testimony of the early Christian Church. Consider the controversy between Jerome and Helvidius, writing around 380. Helvidius first brought up the notion that the "brothers of the Lord" were children born to Mary and Joseph after Jesus’ birth. The great Scripture scholar Jerome at first declined to comment on Helvidius’ remarks because they were a "novel, wicked, and a daring affront to the faith of the whole world." At length, though, Jerome’s friends convinced him to write a reply, which turned out to be his treatise called On the Perpetual Virginity of the Blessed Mary. He used not only the scriptural arguments given above, but cited earlier Christian writers, such as Ignatius, Polycarp, Irenaeus, and Justin Martyr. Helvidius was unable to come up with a reply, and his theory remained in disrepute and was unheard of until more recent times."
Papias (d. 163), fragment X The Exposition Of The Oracles Of The Lord.
"(1) Mary the mother of the Lord; (2) Mary the wife of Cleophas or Alphaeus, who was the mother of James the bishop and apostle, and of Simon and Thaddeus, and of one Joseph; (3) Mary Salome, wife of Zebedee, mother of John the evangelist and James; (4) Mary Magdalene. These four are found in the Gospel. James and Judas and Joseph were sons of an aunt (2) of the Lord's. James also and John were sons of another aunt (3) of the Lord's. Mary (2), mother of James the Less and Joseph, wife of Alphaeus was the sister of Mary the mother of the Lord, whom John names of Cleophas, either from her father or from the family of the clan, or for some other reason. Mary Salome (3) is called Salome either from her husband or her village. Some affirm that she is the same as Mary of Cleophas, because she had two husbands."