Thirteen-year-old Rudy and his friends routinely ride their bikes through their rundown neighborhood, shouting insults at their neighbor, Jacob, an elderly Jewish man out tending his garden. Then Rudy discovers that his mother has arranged for him to help Jacob tear down his fence that summer. When a sullen Rudy shows up at Jacob's door, it's hard to know which of them is most wary of the other. Yet when Rudy sees the beautiful gardens Jacob and his neighbors, Frederick and Yoshito, have created in their backyards, he can't help but be impressed.

During the hot summer days that follow, fatherless Rudy, who wants to "belong to something," toils in the sun with a shovel and an attitude, reluctant to accept even a glass of Frederick's iced tea. Gradually he learns that the older men—Jacob, from ; Frederick, an African American from the pre–civil rights movement South; and Yoshito, a Japanese American who spent three years in an internment camp during World War II—have become like brothers, bonded through tragedy and the drive to transform barren dirt into something beautiful.

Frederick and Yoshito have made peace with their pasts and removed the fence between their yards, but Jacob is still haunted by what happened to his family at Auschwitz, memories retriggered by Rudy and his friends. As they work alongside each other, Jacob and Rudy do more than tear down a fence in this story of healing and hope that changes Rudy's life in ways he never imagined.