Sid's parents, Jacob and Rebecca Siegel, came to the United States from Romania at the turn of the century. In New York they operated a succession of bakeries and restaurants in which they and their children worked. Sid, born on January 4, 1916, was the youngest of their children; they were a talented and accomplished group of youngsters in whom the mother instilled strong drives for achievement. At the time Sid was born, when his parents were in their forties, his oldest brother, Max, was already at NYU and headed toward a brilliant career in dentistry .A second brother, Morris, was also an accomplished student; he became a Columbia-trained anthropologist. The son nearest to Sid in age, Sam, was early identified as a gifted child and was nurtured to become a prodigy as a violinist and chess player. An older sister, Kitty, was a center of family concern because of serious illness.

He was not identified as talented by his family; the signs that might seem unmistakable to an outsider were overlooked in a family habituated to achievement. Before entering school, Sid learned arithmetic through making change in the restaurant; he learned to read through attending silent films at movie houses and overhearing people reading the subtitles aloud to illiterate companions. Although he was a reasonably apt pupil in the grade-school years, he was in no sense a model schoolboy, and he came to think of himself as having no special talent for school. The fact that he read widely and enthusiastically attracted no attention, especially because often the books were spirited away from a library and then just as stealthily returned. His parents were not religious, and his religious training was confined to one or two sessions at shul before the rabbi expelled him for bad behavior .

Jacob Siegel, Sid's father, was a loved figure in the neighborhood, a storyteller and a wit. He accommodated to American life by remaining somewhat aloof from it. There can be no doubt of his special warmth for Sid, and perhaps there was a secret delight that this was one son who was not submissively internalizing his mother's Puritanism and drive for achievement in American culture. When Jacob Siegel would be called to school to hear a principal recount Sid's misdeeds in the boy's presence, he would express alarm and outrage, and his promises to punish his son would be so vehement as to move the principal to plead for moderation. Then the two would leave school together, the father would laugh and give Sid a nickel for ice cream, and the incident would be closed. No doubt it was from his father that Sid learned his lifelong belief that the central job of a parent is to stand by the child, to be his unfailing ally. And certainly it was from his father that Sid acquired his abilities as a raconteur, a spellbinding storyteller.

Sid's interest in schoolwork, never very strong, fell off as he entered his teens. He was attracted by other interests outside of school. And perhaps his efforts at school seemed foredoomed in comparison to the marked successes of his older brothers. There was also the fact that he became markedly nearsighted in adolescence. Since this was not diagnosed, neither Sid nor anyone else realized how much he was missing of the things presented visually to him. In addition, at some point in these years he contracted tuberculosis, then a common disease among city slum dwellers. It was not diagnosed until years later, after healing had occurred, but it may well have accounted for his lassitude and disinterest. Finally, his teachers were beginning to label him as a delinquent, holding out little hope for his future. In any event, he dropped out of high school during his freshman year, after having flunked algebra, and obtained working papers at age fifteen.

In the culture of the streets in which Sid lived, petty pilferage and disrespect for the law were routine. He became a pool shark, playing for the house in a neighborhood pool hall, and this brought his life in touch with the fringes of organized crime. Although some of his friends went into careers of crime and eventually ended up in prison, Sid was never a juvenile delinquent in the contemporary sense: wanton destructiveness and meaningless violence were totally alien to him. His delinquencies were strictly a way to get by in the life of the city. Some were routine cadging a quarter to "watch" a parked car under the implied threat that its tires would be slashed otherwise. Others were more inventive standing in a subway station and reaching through an open window of a subway car, just as its doors were closing and it was gathering speed to leave the station, and lifting a straw hat from the head of its surprised and helpless owner seated at the window. (The hats were later sold to a fence for a quarter.) Throughout his life; Sid was strongly sympathetic with "underprivileged" and "delinquent" kids, but he felt no kinship with the hoodlums. His own delinquencies had been strictly. a way of getting by in the life of a city which offered no jobs and few channels to success to its slum children. He was eventually alarmed by the direction in which this path might lead him, however, and he decided to learn a respectable trade.

While continuing to help in the restaurant, he studied radio at a trade school sponsored by the YMCA, becoming a skilled radio repairman. At about this time he became interested in politics and became a strong anti-Fascist, active in work in this country to oppose Franco in Spain and to provide aid to refugees from Hitler's Germany. When his father died in the late thirties and his mother's health required a move to a warmer climate, Sid and his new wife, Frances, went with her to California, where he found work in a Los Angeles radio shop.