The poor prognosis of oral cavity squamous cell carcinoma (OCSCC) patients is associated with residual tumor after surgery. Raman spectroscopy has the potential to provide an objective intra-operative evaluation of the surgical margins. Our aim was to understand the discriminatory basis of Raman spectroscopy at a histological level. In total, 127 pseudo-color Raman images were generated from unstained thin tissue sections of 25 samples (11 OCSCC and 14 healthy) of 10 patients. These images were clearly linked to the histopathological evaluation of the same sections after hematoxylin and eosin-staining. In this way, Raman spectra were annotated as OCSCC or as a surrounding healthy tissue structure (i.e., squamous epithelium, connective tissue (CT), adipose tissue, muscle, gland, or nerve). These annotated spectra were used as input for linear discriminant analysis (LDA) models to discriminate between OCSCC spectra and healthy tissue spectra. A database was acquired with 88 spectra of OCSCC and 632 spectra of healthy tissue. The LDA models could distinguish OCSCC spectra from the spectra of adipose tissue, nerve, muscle, gland, CT, and squamous epithelium in 100%, 100%, 97%, 94%, 93%, and 75% of the cases, respectively. More specifically, the structures that were most often confused with OCSCC were dysplastic epithelium, basal layers of epithelium, inflammation- and capillary-rich CT, and connective and glandular tissue close to OCSCC. Our study shows how well Raman spectroscopy enables discrimination between OCSCC and surrounding healthy tissue structures. This knowledge supports the development of robust and reliable classification algorithms for future implementation of Raman spectroscopy in clinical practice.