Food Analysis Lab · W11

Spectral preprocessing and similarity matching (qualitative identification)

Preprocessing & Spectral Similarity — Qualitative Identification

Food Analysis Lab · 4-hour session
Raman/NIR preprocessing · building a spectral library · similarity metrics · NNLS mixture preview

ALS baseline (λ, p)SNV/MSCSG smoothing & derivativesPearson r / cosine / SAMLibrary matchingNNLS mixture fitting

Before we start, look at this packet of powder

A packet of white powder says “sugar substitute” —
is it aspartame? Sucralose? Erythritol? Or a blend of several?

To the eye, all four are fine white powders that look identical. But measure one spectrum and the shape never lies.

But don’t rush to match: measure the same powder again on another instrument, or packed a different way, and the raw spectrum can look completely different — a fluorescence background (baseline) buries the signal, different particle sizes cause scatter that tilts the baseline, and the intensity units of different instruments do not line up. Today we learn to remove this “noise” first, then use similarity metrics to say scientifically “which entry in the spectral library this powder resembles most”.

Today’s task: use the pure-compound spectra the whole class measured in W10 to build a small spectral library, then identify the TA’s blind samples and commercial sugar substitutes — and finally compare whose matching scores are the most convincing.

W11 / COURSE TOUR

Remove measurement effects, then identify spectra

Full narration and captions; simulated spectra, not measured fingerprints. Player and downloads · Interactive lab

W11 / PREDICT · OBSERVE · EXPLAIN

Six-act lab: choose preprocessing with evidence

Open the full interactive lab

Today’s learning goals

After W11 you should be able to…

Explain why raw spectra cannot be compared directly: three problems — fluorescence baseline, particle-size scatter, intensity differences
Apply cosmic-ray spike removal and ALS baseline removal to a Raman spectrum, and know how λ (smoothness) and p (asymmetry) affect the result
Apply SNV / MSC to remove scatter from an NIR spectrum, and use SG smoothing and 1st/2nd derivatives to sharpen detail

Compare spectra with four similarity metrics — Pearson r, cosine, SAM, Euclidean — and know whether each is sensitive to “intensity”
Build the class spectral library, identify the TA’s blind samples and commercial sugar substitutes, and report the top-3 rather than only the first place
Use NNLS for a first quantitative preview of an unknown mixture (Σcᵢ·referenceᵢ), leading into W16
In Orange OR11, read the distance matrix and dendrogram and find the library sample most similar to the unknown

Preprocessing tool: this lesson uses SpectraView (GitHub) ↗ (user guide ↗) for SG smoothing, baseline, SNV/MSC, derivatives, peak picking and .speclib library matching; as an easy start you can also use the Orange workflow OR11 ↗: File → Preprocess Spectra (ALS baseline subtraction → vector normalization) → Distances (Pearson) → Distance Matrix + Hierarchical Clustering, which only does qualitative identification (“which one is it most similar to”) — NNLS quantification of mixtures still needs Colab NB09.

What is a spectrum, really?

A spectrum = a string of numbers

Each wavelength/wavenumber is just one “column” and each intensity just one “cell” — the illustrative spectra in this lesson have 71 data points (one every 20 cm⁻¹), so one spectrum = 71 numbers in a row. Preprocessing (spike removal, baseline, SNV…) simply means “apply the same operation to every row”.

Raman shift (cm⁻¹)	400	420	440	460	480
Intensity (a.u.)	0.431	0.448	0.457	1.355	0.466

In Excel: one row = one spectrum, 71 columns = 71 wavenumbers; applying the same formula (e.g. SNV) across a whole row is “preprocessing”; 71 samples × 71 columns is the data table X used from W12 on.

Principle

Why can’t raw spectra be compared directly?

① Fluorescence baseline: impurities excited by the laser add a broad background, making signal peaks less obvious
② Particle-size scatter: particle size/packing density changes the light path, so the same component shows different intercepts and slopes
③ Intensity differences: different instruments/integration times put the values on different scales. All three have nothing to do with “which compound it is”, yet they distort the match

Preprocessing ①

Cosmic-ray spike removal (despiking)

First use a “median filter” to estimate what each point should look like nearby, then see how far each point deviates from that estimate. The deviation is measured with a “robust standard deviation”, MAD (median absolute deviation), converted into a “modified z-score”; a deviation of more than 6 robust SDs is judged a spike and replaced with the median-filter value.

Position	1	2	3	4	5	6	7
Raw value	0.30	0.31	0.29	2.10	0.32	0.30	0.29
7-pt median filter	0.30	0.30	0.30	0.30	0.30	0.29	0.29
Modified z-score	0.0	0.7	-0.7	121.4	1.3	0.7	0.0
Corrected	0.30	0.31	0.29	0.30	0.32	0.30	0.29

Point 4 is 2.10, z ≈ 121 (far above 6) → judged a cosmic-ray spike → replaced with the median-filter estimate 0.30. A window that is too wide may flag even real narrow peaks as spikes and delete them; too narrow may miss spikes — NB09/OR11 both use a 7-point window, z>6.

Preprocessing ②

ALS baseline: two parameters, λ and p

The defaults in NB09 / Orange OR11 are both λ=10⁶, p=0.01 — the common standard for this course. Orange path: Preprocess Spectra → ALS.

λ (smoothness): the baseline is a line drawn along the “bottom” of the spectrum — the larger λ is, the more it acts like a stiff ruler that ignores peak ups and downs; if λ is too small it acts like a soft rope that sinks into real peaks
p (asymmetry): p = 0.01 means points currently above the baseline get only 1% weight, forcing the baseline to be fitted using points below the peaks
airPLS is a variant of ALS that adjusts weights automatically from the residuals, so only λ needs tuning; all calculations in this course use ALS with p given explicitly

Data · Raman

Raman preprocessing: spike removal + ALS baseline removal

Illustrative spectrum (not measured by this class), but the preprocessed curve is the result of actually running despike() + ALS(λ=1e6, p=0.01) on the raw curve at left, using the same formulas as NB09 clean_spectrum(). Before preprocessing there are 2 cosmic-ray spikes and a broad fluorescence baseline; if λ is too small, real peaks are eaten as baseline, and p controls how much “only points below the baseline are trusted”. The x-axis is evenly spaced Raman shift.

Preprocessing ③

SNV step by step

Formula: y′=(y−ȳ)/s, where ȳ = the mean of this spectrum itself and s = the standard deviation of this spectrum itself (population SD, dividing by n, not n−1).

	Pt 1	Pt 2	Pt 3	Pt 4	Pt 5
y	0.2	0.4	0.9	0.5	0.3
y′ (after SNV)	-1.08	-0.25	1.82	0.17	-0.66

ȳ=0.46, s=0.242 (population SD). Replace each y with “how many standard deviations from the mean” and you get y′. Try replacing y with 2y+0.1 (intensity doubled, then shifted) and run SNV again: the result is exactly the same — that is why SNV is insensitive to both shift and scaling.
Excel: =(B2-AVERAGE($B2:$F2))/STDEV.P($B2:$F2) (note: use STDEV.P, not STDEV.S). MSC is conceptually the same; the difference is that it regresses against a reference spectrum (usually the mean spectrum), y=a+b·y_ref, then corrects with (y−a)/b.

Data · NIR

NIR preprocessing: SNV removes particle-size scatter

Before preprocessing: 3 spectra of the same component with different intercepts and slopes because of particle size/packing (illustrative data).

After SNV: each spectrum is re-standardized with its own mean and standard deviation, and the 3 spectra nearly coincide.

Intensity differences can also be handled by vector normalization (divide by vector length) or area normalization (divide by the area under the spectrum) — used in the next section.

Preprocessing ④

Savitzky–Golay smoothing and derivatives

Within a sliding window (NB09/OR11 commonly use 15 points), fit a quadratic polynomial to the points, keep only the fitted value at the window center as the smoothed result, then move one step right and repeat
With the window center as origin, let the fitted quadratic be a₀+a₁u+a₂u²: the center’s linear coefficient a₁ is the “smoothed 1st derivative”, and the center’s second derivative is 2a₂ (not a₂ itself) — no separate differencing is needed; if u is a point index, you must also divide by the square of the wavelength spacing to get the true derivative value
The wider the window, the smoother — but adjacent narrow peaks get blurred together (lower resolution); too low a polynomial order flattens peak shape, while too high almost removes the smoothing effect

Derivative intuition

Derivative = slope (think of differences)

1st derivative = the difference between neighboring points; 2nd derivative = the difference of differences.

Spectrum y	1st derivative (difference)	2nd derivative (difference again)
[1, 2, 4, 7]	[1, 2, 3]	[1, 1]

Add a constant to the whole spectrum (a uniform baseline shift, e.g. [6, 7, 9, 12]) and the 1st derivative is completely unchanged — differencing cancels the constant; a linear slope (the whole spectrum rising steadily in one direction) is also removed in the 2nd derivative. The cost: noise (random point-to-point jitter) changes quickly, and differencing amplifies it, so derivatives are often used together with smoothing (e.g. SG).

Data · NIR

NIR preprocessing: 1st/2nd derivative — peaks become negative minima

Illustrative data. The 2nd derivative turns into negative minima at the two overlapping peak positions (1450, 1500 nm), resolving the overlapping broad peaks more clearly; the cost is that point-to-point jitter (noise) is clearly amplified, so window width and polynomial order must balance “resolution” against “noise resistance”.

Preprocessing ⑤

Vector / area normalization — changes the intensity scale first

Vector normalization: y/‖y‖, where ‖y‖=√Σy² is the “length” of the spectrum treated as a vector; after normalization every spectrum has vector length 1
Area normalization: y/Σ|y|, using the “area under the spectrum” in place of vector length; the idea is similar
Today it is used mainly for qualitative “find the compound with the most similar shape”; it can also be used in empirical quantitative models such as PLS, but you must validate separately whether it really helps — don’t assume it always does
Never apply vector normalization before NNLS quantification of a mixture (W16 stresses this again): once normalization changes the spectral scale, the coefficients solved by NNLS can no longer be read directly as original mass fractions

Mini-game ①8 preprocessing methods — which problem does each solve?0 / 8

Sort the 8 preprocessing methods by the main problem each solves: scatter / baseline / noise / intensity.

Similarity metrics

Four metric formulas for matching spectra

Metric	Formula	Sensitive to intensity magnitude?
Pearson r	How strongly the “shapes” of two spectra correlate (covered in W04 ↗)	Not sensitive (unaffected by shift or scaling)
cosine	Treat spectra as vectors; cosθ=(x·y)/(‖x‖‖y‖)	Insensitive to scaling, but sensitive to shift
SAM (spectral angle)	SAM = arccos(cosine) × 180/π (converted to “degrees”)	Same as cosine — the smaller the angle, the more alike
Euclidean	‖x−y‖, a direct numerical difference	Very sensitive — always normalize before matching

False-positive warning: matching a mixture often only hits the main component and looks like a single compound; and on raw NIR, two completely different compounds can still give r > 0.9 because they share broad O–H/C–H combination bands — in that case switch to derivative spectra or look at the Raman fingerprint region, and always report the top-3 scores rather than only the first place.

Worked example: ranking reversal

Without normalization, Euclidean picks the wrong answer

Unknown U=[0.10,0.35,0.90,0.40,0.15], candidate C=[0.12,0.32,0.95,0.38,0.22] (shape slightly different from U, but the absolute values are very close to U).

Metric	U vs C
cosine	0.9968
SAM	4.59°
Pearson r	0.9911
Euclidean (raw, not normalized)	0.095
Euclidean (after vector normalization)	0.080

Compare with the “truly correct” candidate: raw Euclidean d(U, truly correct)=0.510 is larger than the 0.095 of C above — raw distance wrongly judges C to be closer! After normalization, d(U, truly correct)=0.048 is smaller than C’s 0.080, so the judgment is correct. Pearson r, cosine and SAM are insensitive to intensity by design, so they never made this mistake. (Which candidate is truly correct? Work it out yourself in Calculation Challenge ⑥ later.)

Similarity metrics

Four metrics, four kinds of sensitivity

📈

Pearson correlation coefficient r

Looks only at the relative changes in “shape”; insensitive to both offset and scale — the most common way to check whether waveforms look alike

📐

Cosine similarity

The cosine of the angle between spectra treated as vectors; insensitive to scale but sensitive to overall offset

🧭

SAM spectral angle (radians/degrees)

The angle version of cosine — the smaller the angle, the closer the directions; SAM=0 means the two vectors point in exactly the same direction

📏

Euclidean distance

A direct numerical difference, very sensitive to intensity — always normalize before matching, or intensity differences will lead you astray

Data · spectral library

Library matching example: TA blind sample #1 (Raman, illustrative data)

Candidate compound ⇅	Pearson r ⇅	cosine ⇅	SAM (°) ⇅	Rank ⇅

Click a column header to sort. SAM is always converted from cosine (arccos(cos)×180/π), so the two columns agree. Preprocessing (spike removal + ALS + vector normalization) has been applied identically to the library and the unknown spectrum. There is a clear score gap between 1st place aspartame and 2nd place sucrose, so the call is safe; but if the top two scores are close (e.g. r differs by <0.02), always go back and confirm with a visual overlay and fingerprint-region peak positions — never rely on the numbers alone.

Mini-game ②Order the spectral identification workflow— / 6

Use ▲▼ to put the 6 steps of “identifying an unknown sample with a spectral library” in the correct order.

Mini-game ③For this identification task, Raman or NIR?0 / 6

Sort the 6 scenarios into “suits Raman” or “suits NIR”. Isomers = compounds with the same molecular formula but different atom arrangement (structure).

Orange starter path

OR11: distance matrix and dendrogram

File loads the library + unknown samples → Preprocess Spectra: ALS baseline subtraction (λ=10⁶, p=0.01) → vector normalization — use exactly the same settings for the library and the unknowns
Distances: choose Pearson; Orange converts the correlation coefficient to a distance = (1−r)/2 — the closer r is to 1 (more alike), the closer the distance is to 0
Distance Matrix: find the row of the unknown sample; the library sample at the cell with the smallest number is the most similar answer
Hierarchical Clustering: a dendrogram of the same set of spectra — the shorter the branch joining two samples, the more similar they are

Orange has no dedicated “Similarity / Library / Mixture” widgets — the OR11 identification is built from these four general-purpose widgets; NNLS mixture quantification and peak assignment can’t be done in Orange and need Colab NB09.

Data · mixture

Mixture preview: NNLS non-negative least-squares fitting

Illustrative data. Unknown spectrum ≈ 0.62 × sucralose + 0.34 × maltodextrin + intercept (proportions taken from NB09’s mix_demo); NNLS fits c₁=0.61, c₂=0.34, intercept=0.018, R²=0.995. “Least squares” = find the coefficients that minimize the “sum of squared residuals”; R²=1−Σresidual²/Σ(y−ȳ)² means the same as the “proportion of reference-value variance explained by the model” taught in W07 ↗. NNLS forbids negative coefficients, matching the physical fact that “a component’s content cannot be negative”; the reference spectra here are not vector-normalized, otherwise the linear “coefficient = proportion” relationship would be broken — a warm-up for W16 quantification.

Mini-game ④Preprocessing concepts: live quiz0 / 5

Peak assignment practice

Back to library matching: are these peak assignments correct?

The aspartame benzene-ring breathing mode falls near about 1003 cm⁻¹; the sucralose C–Cl stretch falls in the range about 600–850 cm⁻¹. These numbers are approximate literature values — always confirm where the peaks really fall using this class’s own W10 spectra, then use the same positions to assign today’s blind samples and commercial sugar substitutes.

Mini-game ⑤Decision challenge: how to interpret a match result?0 / 5

4-hour lab schedule

Lecture → preprocessing practice → build library → match → report

Time	Contents
0:00–0:50	Lecture: the three “can’t compare directly” problems, spike removal, ALS baseline, SNV/MSC/SG, four similarity metrics
0:50–1:40	SpectraView preprocessing practice: try different parameters (λ, p, SG window/order) on the same spectrum and record your reasons for each choice
1:40–2:35	Build the class spectral library from the W10 pure-compound spectra (or use Orange OR11) and match the TA’s blind samples
2:35–3:30	Measure commercial sugar substitutes (Raman 785 nm + each group’s NIR) and match against the library
3:30–4:00	Each group reports its top-3 results and similarity scores

🔦

Laser safety

With the Raman 785 nm/532 nm laser, never look into the beam path and never aim at anyone’s eyes; before adjusting, make sure the shield cover is closed

😷

Powder handling

Weigh sugar-substitute powder in a draft-free spot and avoid inhaling dust; clean the balance and bench after use

Mini-game ⑥Calculation challenge: SAM spectral angle (degrees)Unanswered

Two 5-point illustrative spectra: U = [0.10, 0.35, 0.90, 0.40, 0.15], C1 = [0.20, 0.55, 1.30, 0.60, 0.25]. First compute cosine = (U·C1)/(‖U‖‖C1‖), then SAM = arccos(cosine), and finally convert to degrees (°).

degrees

Homework

Colab NB09

📚

Library matching table

Include: the unknown sample, top-3 candidates, each similarity score (r/cosine/SAM), and the preprocessing methods used

🖼️

Overlay figure

Attach a screenshot overlaying the unknown spectrum and the top-1 candidate on one plot

🔍

One false-positive / ambiguous-match discussion

Pick a case where the top-1 and top-2 scores are close or the call was clearly wrong; explain the likely cause and how to confirm

🧮

NNLS practice

Pick any one mixture sample, run one NNLS fit, and record the coefficients and R²

Colab NB09: Open NB09 in Colab ↗ · Easy start (no coding): Orange workflow OR11 ↗. The report must include: a library matching table (top-3), the overlay, a false-positive/ambiguous-match discussion, and your reasons for the preprocessing parameters.

Self-check

By the end of today you should be able to…

✓State the three reasons raw spectra can’t be compared directly: fluorescence baseline, particle-size scatter, intensity differences
✓Do spike removal, ALS baseline removal, SNV/MSC and SG smoothing/derivatives, and state the trade-offs of λ, p, window and order
✓State whether r, cosine, SAM and Euclidean are sensitive to “intensity”, and why we normalize
✓Identify an unknown with the class spectral library using exactly the same preprocessing, and report the top-3 rather than only the first place
✓Use NNLS for a first quantitative preview of a mixture and understand the physical meaning of the non-negativity constraint
✓Read the distance matrix and dendrogram in Orange OR11 and identify the most similar sample
✓Follow laser safety and powder-handling rules

Tap to tick.