He was the son of George Washington Whistler and his first wife Mary Roberdeau Swift. He married twice: 1. Mary Ann Ducatel, by whom he had one child, George Worthen; 2. Julia De Kay Winans, the sister of Thomas De Kay Winans, by whom he had five children: Julia de Kay b. 1855); Thomas Delano (b. 1857); Ross Winans (b. 1858); Neva (1860-1907); Joseph Swift (1865-1905).
He was a self-taught engineer, and was connected with various railroads in the USA. He was Superintendent of the Erie, and New York and New Haven Railroads. In 1856 he went to Russia to work at Alexandroffsky, in the firm of Harrison, Winans and Eastwick.
Pennell, Elizabeth Robins, and Joseph Pennell, The Life of James McNeill Whistler, 2 vols, London and Philadelphia, 1908 .