The son of a sawmill worker, and the grandson of a member of the Industrial Workers of the World, Brown grew up in the Pacific Northwest in a family of Democrats with 100-year-old roots in the area. He graduated from Olympia High School in Olympia, Washington, in 1979, and from the University of Washington. He holds a bachelor’s degree in economics. He was appointed to, but chose to leave, the United States Military Academy at West Point.

Brown credits meeting Ronald Reagan at a Masonic Temple in 1976 for sparking his interest in politics when he was 15 years old. He is married to NYT Bestselling author Mary Beth Brown, author of The Faith of Ronald Reagan, with whom he has three children.

In 1992, Brown was quoted in the Washington Times:

I have a sense of what connects with people like me. We’re not culturally Republicans. We’re not libertarians. We’re not neo-conservatives or former liberals. We’re just old-fashioned, blue-collar social conservatives. These are people who couldn’t care less about politics, want to be left alone by government, but if their country calls for them to fight abroad, will. You win elections by cultivating people like me.