Collier was from a talented and successful family. His grandfather, John Collier, was a Quaker merchant who became a member of parliament. His father, Robert, (who was a member of parliament, Attorney General and, for many years, a full-time judge of the Privy Council) was created the first Lord Monkswell. He was also a member of the Royal Society of British Artists. John Collier's elder brother, the second Lord Monkswell, was Under-Secretary of State for War and Chairman of the London County Council.

In due course, Collier became an integral part of the family of Thomas Henry Huxley PC, President of the Royal Society from 1883 to 1885. Collier married two of Huxley's daughters and was "on terms of intimate friendship" with his son, the writer Leonard Huxley. Collier's first wife, in 1879, was Marian (Mady) Huxley. She was a painter who studied, like her husband, at the Slade and exhibited at the Royal Academy and elsewhere. After the birth of their only child—a daughter, Joyce—she suffered severe post-natal depression and was taken to Paris for treatment where, however, she contracted pneumonia and died in 1887. Joyce became a portrait miniaturist and was a member of the Royal Society of Miniature Painters.

In 1889 Collier married Mady's younger sister Ethel Huxley.  Until the Deceased Wife's Sister's Marriage Act 1907 such a marriage was not possible in England, so the ceremony took place in Norway. By his second wife he had a daughter and a son, Sir Laurence Collier, who was the British Ambassador to Norway 1941–1951.