The census and other records give varying and contradictory information.
She was the daughter of John Christopher Hiffernan, Hefferman or Heffernan and his wife Katherine or Catherine (née Hannan).
The 1851 census for London, Middlesex, records the family as John Hefferman (or Hifferman - the 'e' or 'i' is unclear) aged 32, school master, and Catherine, his wife, aged 36, both born in Ireland. Their children are all recorded as born in London: John, aged 6 and Catherine, aged 8, both born in Marylebone; Ann, aged 10, born in Chelsea; Bridget, aged 9, born in St George, and Ellen, aged 15 born in Maylebone.
This suggests that Anne, who is assumed to be Joanna, was born in 1841. However, there is conflicting evidence on her birth0date and place.
The 1861 Greenwich census records 'Anna Whistler', aged 19, and her status as 'Wife' of 'James Whistler'. This would mean she was born in 1842.
Jo's mother was almost certainly the 'Katherine Heffernan' who died in London in 1862, her age give as 44 (although that conflicts with the 1851 census information).
Jo's father, John Christopher Hiffernan, was a schoolmaster. He was described by the Pennells as 'a sort of Captain Costigan' (a drunken Irishman in Thackeray's Pendennis). The Pennells also described him as 'a teacher of polite chirography [calligraphy]' who used to speak of Whistler as 'me son-in-law'.
Joanna's sister Ellen was married in 1866 at the age of 21 to Frederick Walter Stokes at the Parish church St Luke Chelsea. She gave her name as Mary Ellen Hiffernan, her father as John Christopher Hiffernan.
The census for 1881 records Jo' visiting her sister Bridget Agnes Hiffernan. They were staying in Thistle Grove Lane, the house of Charles J. Singleton, accountant, aged 37. 'Bridget Hiffernan' was listed as aged 36 and born in Ireland, 'Johanna Hiffernan' as aged 38, also born in Ireland. This would suggest Jo' was born in 1843, rather than the 1842 or 1841 previously recorded - or that she wanted to appear younger, or that she simply didn't know her precise birth date. Whistler's illegitimate son, Charles James Whistler Hanson, was also staying in the house.
Agnes was Charles Singleton's mistress for many years, until the death of his wife, when they were married. The marriage took place in January 1901, when 'Bridget Agnes Hifferman' was this time recorded as born in 1841 in Limerick, Ireland, her parents being 'Christopher Hefferman' and 'Catherine Hannan'. The various spellings of the names confuses this family tree considerably.
Walter Greaves claimed that Joanna Hiffernan had a son called Harry but there is no record of him. A 'Family Tree' now (2016) on Ancestry.com includes unconfirmed data, including the existence of another son, John Henry Hifferman
It has also been suggested that Jo' was married at some point after 1881, perhaps abroad, to a man named Abbot, because she is said to have used the name 'Mrs Abbott' in business transactions done on Whistler's behalf, but she could simply have adopted Whistler's middle name for convenience.
The name Joanna or Johanna Hiffernan (and variations) are surprisingly common, and dates of marriage or death have not been located.
'Johanna Hefferman' died in Holborn, London, in the autumn of 1886.
Whistler first met Hiffernan in 1860 while she was at a studio in Rathbone Place, according to Ionides, and she went on to have a six year liaison with him. She modelled for some of Whistler's most famous paintings during this period. She was in France with Whistler during the summer of 1861, and in Paris during the winter of 1861-62 sitting for Symphony in White, No. I: The White Girl y038 at a studio in Boulevard des Batignolles. It is possible that this is where she met Courbet for whom she later modelled. However, as well as being a model, she painted and drew a little herself.
Hiffernan was not fully accepted within Whistler's family. They did not consider her to be respectable. When Whistler's mother visited from America in 1864, alternative accommodation had to be found for her away from 7 Lindsey Row. She also seems to have been the cause of Whistler's quarrel with Legros in 1863.
Hiffernan attended séances with Whistler at D. G. Rossetti's house in Chelsea in 1863.
She spent the summer and autumn of 1865 in Trouville with Whistler, and posed for Courbet in Portrait de Jo, la Belle Irlandaise: there are four versions of this portrait (New York: Metropolitan Museum of Art, Zurich: private collection, Kansas: Nelson-Atkins Museum of Art, New York: private collection). Hiffernan also posed for an oil sketch Portrait de Jo, 1865 (New York: private collection). Courbet reminisced years later:
'... rappelez-vous Trouville et Jo qui faisait le clown pour nous egayer. le soir elle chantait si bien les chants Irlandais car elle avait elle avait l'esprit et la distinction de l'art. ...
J'ai encore le portrait de Jo que je ne vendrai jamais il fait l'admiration de tout le monde' (Courbet to Whistler, 14 February 1877, GUW #00695).
Translation: 'do you remember Trouville and Jo who played the clown to amuse us. the evening she sang Irish songs so well she had the spirit and the distinction of art..... I still have the portrait of Jo which I will never sell everyone admires it'.
In January 1866, Whistler gave 'Joanna Hiffernan of 7 Lindsey Row' power of attorney over his affairs whilst he was in Valparaiso for seven months - and letters were signed by her with that name. Whistler made provision for household expenses and gave authority to Hiffernan to act as an agent in the sale of his works.
Hiffernan travelled to Paris and posed for Courbet in The Sleepers or Le Sommeil (1866, Paris: Musée du Petit Palais). It has been suggested that they had an affair at this time, but there is absolutely no proof of that.
On 22 January 1866, even as Whistler was giving Jo control of his affairs, his mother suggested that he used a legacy from his Aunt Alicia McNeill 'to bestow on your Model ... you promised me to promote a return to virtue in her. I never forget to pray for her.' (GUW #06527). This may have influenced Whistler, for it appears that they parted amicably after his return from Valparaiso.
However, Jo helped to raise Whistler and Louisa Fanny Hanson's son Charles who was born in 1870 (the Pennells say she adopted him, but there is no record of this). On 11 January 1877 Whistler's friend Alan S. Cole recorded calling with Whistler in Thistle Grove 'upon the original of his White Girl now a buxom short woman of say 40.'
Hiffernan continued to look after Charlie as late as 1880 when Whistler was away in Venice with Maud Franklin, his current mistress (at that time, Whistler referred to her as 'Auntie Jo' in a letter to his young son). They are recorded as living with her sister Agnes Bridget Hiffernan at 5 Thistle Grove for some time.
Du Maurier, Daphne, (ed.), The Young George du Maurier: A Selection of his Letters, 1860-67, London, 1951 ; Ionides, Luke, Memories, Paris, 1925 ; Pennell, Elizabeth Robins, and Joseph Pennell, The Life of James McNeill Whistler, 2 vols, London and Philadelphia, 1908 ; Pennell, Elizabeth Robins, and Joseph Pennell, The Whistler Journal, Philadelphia, 1921 ; MacDonald, Margaret F., Susan Galassi, Aileen Ribeiro and Patricia de Montfort, Whistler, Women and Fashion, New Haven and London, 2003 .