Pulse shaping and matched filters

The previous lesson mapped bits to constellation points and treated each symbol as a number. A radio cannot transmit a number. It transmits a waveform, and the shape of the pulse that carries each symbol decides two things: how much spectrum the signal occupies and whether neighbouring symbols interfere with each other at the receiver. This lesson is for readers who can run numpy and want to see both effects measured. We build a root-raised-cosine filter from its formula, compare spectra, add a matched filter at the receiver, and read the eye diagram that every modem engineer uses as a first check.

The problem with rectangular pulses

The obvious way to send a symbol is to hold its value for one symbol period T. That is a rectangular pulse. Its spectrum is a sinc function. The main lobe is 2/T wide, and the sidelobes fall off slowly: the first sidelobe is only about 13 dB below the peak, and the envelope drops by 6 dB per octave after that. A channel allocation is a fixed slice of spectrum, so energy in those sidelobes lands on your neighbours. Regulators and standards set spectral masks to stop exactly this.

The opposite extreme is a sinc pulse in time. Its spectrum is a perfect rectangle of width 1/T, the narrowest a signal at symbol rate 1/T can be without losing information. No signal can carry independent symbols at rate 1/T without interference in less bandwidth than this. The catch is that a sinc pulse lasts forever and its tails decay as 1/t. Any timing error at the receiver picks up contributions from many neighbouring symbols, and the sum can grow large.

What we want sits between the two: a pulse whose spectrum is compact, whose tails decay quickly, and which still has zero crossings at every other symbol instant. This condition is the Nyquist criterion for zero intersymbol interference, named after Harry Nyquist, whose 1928 paper on telegraph transmission is listed at the end. It is a different idea from the Nyquist rate for sampling, though the name is shared. If the overall pulse p(t) is zero at t = kT for every k other than 0, then sampling at the correct instant sees only the current symbol.

Raised cosine and root raised cosine

The raised-cosine pulse meets that criterion with a tunable trade-off. Its spectrum is flat up to (1 − β)/(2T), then rolls off as half a cosine period to zero at (1 + β)/(2T). The roll-off factor β runs from 0 to 1. With β = 0 the pulse is a sinc. With β = 1 the spectrum is twice as wide as the minimum and the time-domain tails die away fast. The total occupied bandwidth of a complex baseband signal is (1 + β)/T, so a 1 Mbaud signal with β = 0.35 needs 1.35 MHz.

In a real link the raised-cosine response is split in two. The transmitter applies a root-raised-cosine (RRC) filter, whose frequency response is the square root of the raised cosine. The receiver applies the same RRC filter. Their product is a raised cosine, so the cascade has zero ISI. The reason for the split comes in the matched-filter section below.

The block builds an RRC impulse response from its closed-form expression, with 8 samples per symbol and a span of 12 symbols. It checks the ISI at the symbol instants for one RRC and for two in cascade. Then it generates 16 384 random BPSK symbols, shapes them with a rectangular pulse and with the RRC pulse, and estimates both spectra.

import numpy as np

def rrc_taps(beta: float, sps: int, span: int) -> np.ndarray:
    """Root-raised-cosine impulse response, span symbols long, unit energy."""
    t = np.arange(-span * sps // 2, span * sps // 2 + 1) / sps   # time in symbols
    h = np.zeros_like(t)
    for i, ti in enumerate(t):
        if np.isclose(ti, 0.0):
            h[i] = 1.0 - beta + 4 * beta / np.pi
        elif beta > 0 and np.isclose(abs(ti), 1 / (4 * beta)):
            h[i] = (beta / np.sqrt(2)) * ((1 + 2 / np.pi) * np.sin(np.pi / (4 * beta))
                                          + (1 - 2 / np.pi) * np.cos(np.pi / (4 * beta)))
        else:
            num = np.sin(np.pi * ti * (1 - beta)) + 4 * beta * ti * np.cos(np.pi * ti * (1 + beta))
            h[i] = num / (np.pi * ti * (1 - (4 * beta * ti) ** 2))
    return h / np.sqrt(np.sum(h**2))

sps, span, beta = 8, 12, 0.35          # 8 samples per symbol, 12-symbol filter, roll-off 0.35
h_rrc = rrc_taps(beta, sps, span)
h_rc = np.convolve(h_rrc, h_rrc)       # RRC at the transmitter, RRC at the receiver
centre = len(h_rc) // 2

def isi_at_symbol_instants(p: np.ndarray, c: int) -> float:
    """Largest |p(kT)| for k != 0, relative to p(0)."""
    k = np.arange(-(c // sps), c // sps + 1) * sps + c
    k = k[(k >= 0) & (k < len(p)) & (k != c)]
    return float(np.max(np.abs(p[k])) / abs(p[c]))

print(f"RRC filter: {len(h_rrc)} taps, peak tap {h_rrc.max():.4f}")
print(f"RRC alone, worst ISI sample: {isi_at_symbol_instants(h_rrc, len(h_rrc) // 2):.4f}")
print(f"RRC * RRC (raised cosine), worst ISI sample: {isi_at_symbol_instants(h_rc, centre):.5f}")

# Power spectral density of a random BPSK stream with a rectangular pulse and with the RRC pulse.
rng = np.random.default_rng(6)
n_sym = 16384
symbols = rng.choice([-1.0, 1.0], n_sym)
up = np.zeros(n_sym * sps)
up[::sps] = symbols
x_rect = np.convolve(up, np.ones(sps) / np.sqrt(sps))[: n_sym * sps]
x_rrc = np.convolve(up, h_rrc)[: n_sym * sps]

def psd_db(x: np.ndarray, nfft: int = 256) -> tuple[np.ndarray, np.ndarray]:
    segs = x[: len(x) // nfft * nfft].reshape(-1, nfft) * np.hanning(nfft)
    p = np.mean(np.abs(np.fft.fftshift(np.fft.fft(segs, axis=1), axes=1)) ** 2, axis=0)
    f = np.fft.fftshift(np.fft.fftfreq(nfft, d=1 / sps))   # frequency in units of the symbol rate
    return f, 10 * np.log10(p / p.max())

f, p_rect = psd_db(x_rect)
_, p_rrc = psd_db(x_rrc)

def occupied_bw(x: np.ndarray, frac: float = 0.99) -> float:
    X = np.abs(np.fft.fftshift(np.fft.fft(x))) ** 2
    fr = np.fft.fftshift(np.fft.fftfreq(len(x), d=1 / sps))
    keep = fr >= 0                                        # real signal: one side is enough
    cum = np.cumsum(X[keep]) / np.sum(X[keep])
    return 2 * float(fr[keep][np.searchsorted(cum, frac)])

print(f"99 % power bandwidth, rectangular: {occupied_bw(x_rect):.2f} x symbol rate")
print(f"99 % power bandwidth, RRC beta={beta}: {occupied_bw(x_rrc):.2f} x symbol rate (edge-to-edge width 1+beta = {1 + beta:.2f})")
print("f/Rs, rect dB, RRC dB")
for i in range(128, 194, 2):            # 0 to 2 times the symbol rate
    print(f"{f[i]:.3f}, {p_rect[i]:.1f}, {p_rrc[i]:.1f}")

RRC filter: 97 taps, peak tap 0.3874
RRC alone, worst ISI sample: 0.0773
RRC * RRC (raised cosine), worst ISI sample: 0.00120
99 % power bandwidth, rectangular: 6.88 x symbol rate
99 % power bandwidth, RRC beta=0.35: 1.16 x symbol rate (edge-to-edge width 1+beta = 1.35)
f/Rs, rect dB, RRC dB
0.000, -0.0, -0.2
0.062, -0.2, -0.5
0.125, -0.1, -0.3
0.188, -0.4, 0.0
0.250, -1.0, -0.4
0.312, -1.3, -0.3
0.375, -2.4, -0.3
0.438, -2.8, -1.2
0.500, -4.0, -3.3
0.562, -4.9, -6.2
0.625, -6.7, -12.9
0.688, -8.1, -28.5
0.750, -10.4, -42.1
0.812, -12.9, -45.3
0.875, -16.7, -48.8
0.938, -23.1, -52.7
1.000, -34.2, -54.8
1.062, -24.3, -54.4
1.125, -18.8, -54.3
1.188, -16.1, -56.3
1.250, -14.7, -59.4
1.312, -13.4, -58.7
1.375, -13.3, -57.6
1.438, -12.7, -58.6
1.500, -13.1, -61.8
1.562, -13.3, -61.4
1.625, -14.5, -59.8
1.688, -15.4, -60.5
1.750, -17.2, -63.1
1.812, -19.3, -63.5
1.875, -22.7, -61.6
1.938, -28.8, -61.9
2.000, -39.6, -63.7

A single RRC pulse is not a Nyquist pulse. Its worst sample at a neighbouring symbol instant is 7.7 % of the peak. After the second RRC the worst residual is 0.12 %, which comes from cutting the filter off at 12 symbols. A longer span pushes it lower at the cost of more taps.

The spectra tell the bandwidth story. With β = 0.35, 99 % of the power sits within 1.16 times the symbol rate, inside the ideal edge-to-edge width of 1.35. The rectangular pulse needs at least 6.88 times the symbol rate for the same 99 %, and that figure is limited by our simulation, which only spans 8 times the symbol rate. At 0.75 times the symbol rate from the centre the RRC spectrum is 42.1 dB down, while the rectangular one is only 10.4 dB down. Its sidelobe near 1.4 times the symbol rate sits at −13.3 dB, as expected for a sinc.

The RRC curve flattens between −55 dB and −65 dB instead of falling forever. Truncating the impulse response to 12 symbols puts a floor on the stopband, and the window in the spectral estimate adds a little leakage. If your spectral mask needs more rejection, lengthen the filter.

Intersymbol interference

Intersymbol interference (ISI) is energy from other symbols present at the instant you sample the current one. It has three common sources. The pulse itself may not meet the Nyquist criterion, as with a single RRC. The receiver may sample at the wrong time, which turns the tails of neighbouring pulses into interference. And the channel may smear the pulse, which lesson 7 covers as multipath.

ISI differs from noise in one important way. Noise power falls as you increase transmit power relative to it. ISI scales with the signal, so raising power does not remove it. A link limited by ISI shows an error floor: past some SNR, the bit error rate stops improving.

The matched filter

The receiver sees the transmitted pulse plus white noise. Before deciding on each symbol it filters the signal and samples once per symbol. The filter that maximises the SNR at the sampling instant is the matched filter: its impulse response is the time-reversed complex conjugate of the transmitted pulse. For a pulse with energy E and white noise of spectral density N0/2, the best achievable SNR at the sample is 2E/N0, and only the matched filter reaches it.

The RRC pulse is symmetric and real, so its matched filter is the same RRC. This is why the raised cosine is split. One RRC at each end gives the best SNR and, because the two multiply to a raised cosine, zero ISI as well. You get both properties from one filter design.

The second block checks this with 400 000 BPSK bits at an Eb/N0 of 6 dB. Receiver A applies the matched filter and samples once per symbol. Receiver B skips the filter and samples the noisy waveform at the pulse peaks. The same block then removes the noise and measures the eye opening against timing offset for three roll-off factors.

import numpy as np
from scipy.special import erfc

def rrc_taps(beta: float, sps: int, span: int) -> np.ndarray:
    """Root-raised-cosine impulse response, span symbols long, unit energy."""
    t = np.arange(-span * sps // 2, span * sps // 2 + 1) / sps
    h = np.zeros_like(t)
    for i, ti in enumerate(t):
        if np.isclose(ti, 0.0):
            h[i] = 1.0 - beta + 4 * beta / np.pi
        elif beta > 0 and np.isclose(abs(ti), 1 / (4 * beta)):
            h[i] = (beta / np.sqrt(2)) * ((1 + 2 / np.pi) * np.sin(np.pi / (4 * beta))
                                          + (1 - 2 / np.pi) * np.cos(np.pi / (4 * beta)))
        else:
            num = np.sin(np.pi * ti * (1 - beta)) + 4 * beta * ti * np.cos(np.pi * ti * (1 + beta))
            h[i] = num / (np.pi * ti * (1 - (4 * beta * ti) ** 2))
    return h / np.sqrt(np.sum(h**2))

rng = np.random.default_rng(7)
sps, span, beta = 8, 12, 0.35
h = rrc_taps(beta, sps, span)
delay = len(h) - 1                      # total delay of two filters, in samples (a multiple of sps)

# BPSK, real baseband, sample rate = 8 x symbol rate. Each pulse has unit energy, so Eb = 1.
n_sym = 400_000
bits = rng.integers(0, 2, n_sym)
sym = 2.0 * bits - 1.0
up = np.zeros(n_sym * sps)
up[::sps] = sym
tx = np.convolve(up, h)

ebn0_db = 6.0
n0 = 1.0 / 10 ** (ebn0_db / 10)
rx = tx + rng.normal(0.0, np.sqrt(n0 / 2), tx.size)   # real white noise, two-sided PSD N0/2 per sample

# Receiver A: matched filter (the same RRC), then sample once per symbol at the peak.
mf = np.convolve(rx, h)[delay : delay + n_sym * sps : sps]
ber_mf = np.mean((mf > 0) != bits)

# Receiver B: no filter, sample the noisy waveform directly at the pulse peaks.
raw = rx[delay // 2 : delay // 2 + n_sym * sps : sps]
ber_raw = np.mean((raw > 0) != bits)

ber_theory = 0.5 * erfc(np.sqrt(10 ** (ebn0_db / 10)))
loss_db = -20 * np.log10(h.max())       # SNR lost by skipping the filter
print(f"Eb/N0 = {ebn0_db} dB, {n_sym} bits")
print(f"BER with matched filter:  {ber_mf:.2e}  (theory {ber_theory:.2e})")
print(f"BER without any filter:   {ber_raw:.2e}  (SNR penalty {loss_db:.1f} dB)")

# Eye opening against sampling offset, noise-free, for three roll-offs.
# Opening = (smallest |sample| over all symbols) / (ideal amplitude). Zero or below means the eye is closed.
print("offset (T), eye opening for beta = 0.2, 0.35, 0.5")
sps_e = 16                              # finer time grid for the eye
sym_short = sym[:4000]
offsets = range(-sps_e // 2, sps_e // 2 + 1)
eye = {}
for b in (0.2, 0.35, 0.5):
    hb = rrc_taps(b, sps_e, span)
    up_s = np.zeros(sym_short.size * sps_e)
    up_s[::sps_e] = sym_short
    y = np.convolve(np.convolve(up_s, hb), hb)
    d = len(hb) - 1
    eye[b] = []
    for off in offsets:                 # skip the first and last span symbols (filter transients)
        s = y[d + off + span * sps_e : d + off + (sym_short.size - span) * sps_e : sps_e]
        eye[b].append(np.min(s * sym_short[span : span + s.size]))
for k, off in enumerate(offsets):
    print(f"{off / sps_e:+.4f}, {eye[0.2][k]:.3f}, {eye[0.35][k]:.3f}, {eye[0.5][k]:.3f}")

Eb/N0 = 6.0 dB, 400000 bits
BER with matched filter:  2.30e-03  (theory 2.39e-03)
BER without any filter:   1.40e-01  (SNR penalty 8.2 dB)
offset (T), eye opening for beta = 0.2, 0.35, 0.5
-0.5000, -0.793, -0.485, -0.300
-0.4375, -0.621, -0.316, -0.128
-0.3750, -0.424, -0.133, 0.048
-0.3125, -0.207, 0.058, 0.226
-0.2500, 0.026, 0.253, 0.400
-0.1875, 0.268, 0.448, 0.567
-0.1250, 0.515, 0.640, 0.725
-0.0625, 0.760, 0.825, 0.870
+0.0000, 0.980, 0.995, 0.998
+0.0625, 0.757, 0.826, 0.869
+0.1250, 0.512, 0.643, 0.725
+0.1875, 0.267, 0.452, 0.567
+0.2500, 0.025, 0.257, 0.400
+0.3125, -0.207, 0.062, 0.226
+0.3750, -0.424, -0.130, 0.049
+0.4375, -0.620, -0.313, -0.128
+0.5000, -0.792, -0.484, -0.299

With the matched filter the measured bit error rate is 2.30 × 10⁻³, against 2.39 × 10⁻³ from the BPSK formula. The gap is statistical: 400 000 bits give about 920 errors. Without the filter the BER is 1.40 × 10⁻¹. The unfiltered sample collects noise from the full 8-times-oversampled bandwidth, while the signal sample is only the peak tap of the pulse. The printed penalty, 8.2 dB, is the ratio between the energy of the whole pulse and the square of its peak tap. Any receiver that leaves out the matched filter, or uses the wrong one, pays some of that penalty.

Reading an eye diagram

An eye diagram overlays many two-symbol-long slices of the matched-filter output, all aligned to the symbol clock. For BPSK the traces cross between +1 and −1 and leave an open region in the middle: the eye. Its height at the sampling instant is the noise margin. Its width is the timing margin: how far the sampling instant can drift before the decision goes wrong even without noise.

The second half of the block measures the worst-case eye opening at each sampling offset. It takes the smallest value of the sample multiplied by the correct symbol over 4 000 symbols, so 1.0 means a perfect sample and zero or below means at least one symbol is already wrong with no noise at all.

At the correct instant all three eyes are almost fully open: 0.980 for β = 0.2, 0.995 for β = 0.35 and 0.998 for β = 0.5. The β = 0.2 eye loses a little height because its tails decay slowly and the 12-symbol filter cuts more of them off. A quarter of a symbol off, the differences are large. The β = 0.2 eye has shrunk to about 0.025, nearly closed. The β = 0.35 eye keeps 0.253 and the β = 0.5 eye keeps 0.400. A larger roll-off buys timing tolerance with bandwidth. That is the main trade you make when you pick β, along with filter length.

On a real receiver the eye also tells you about the channel and the hardware. A vertically squashed eye points to noise or ISI. Traces that thicken at the crossings show timing jitter. An eye that slowly rotates or closes over time suggests a frequency offset that the receiver has not corrected. Timing recovery loops work by steering the sampling instant towards the widest point of the eye.

The complete chain

A few practical points apply to every implementation. The filter span (12 symbols here) sets the stopband floor and the delay: each filter adds half its length, so the cascade delays the signal by 12 symbols. The upsampling ratio must leave room for the (1 + β) bandwidth, so 2 samples per symbol is the practical minimum and 4 to 8 is common in software. On hardware, the same filters are usually polyphase FIR structures, and the FPGA Guide DSP hub covers how they are built. Many single-carrier standards specify an RRC filter and its roll-off. Multicarrier systems such as LTE and 5G NR deal with ISI differently, using the cyclic prefix you will build in lesson 8.

Exercise

1. In the first block, change the span from 12 to 6 symbols. Report the new worst ISI sample for the cascade and the level of the RRC spectrum at 1.0 times the symbol rate. Explain both changes.
2. In the second block, give receiver A a sampling offset of 2 samples (a quarter of a symbol) and rerun for β = 0.2 and β = 0.5. Compare the BER with the eye openings in the chart.
3. Replace the receive RRC with a rectangular filter of 8 taps (an integrate-and-dump receiver) while keeping the RRC transmitter. Measure the BER at 6 dB and explain why it lies between receivers A and B.

Summary

• Rectangular pulses have sinc spectra with sidelobes only about 13 dB down. Raised-cosine pulses confine the spectrum to (1 + β) times the symbol rate.
• A pulse has zero ISI if it crosses zero at every other symbol instant. The raised cosine does. A single root raised cosine does not, but two in cascade do.
• The matched filter maximises SNR at the sampling instant. For an RRC transmitter it is the same RRC, which is why the raised cosine is split between transmitter and receiver.
• Skipping the matched filter cost 8.2 dB of SNR in our BPSK test at 8 samples per symbol.
• A larger roll-off widens the eye and tolerates more timing error, at the cost of bandwidth.

Further reading

• H. Nyquist, “Certain topics in telegraph transmission theory”, Transactions of the AIEE, vol. 47, no. 2, 1928: the paper the Nyquist criterion is named after.
• J. G. Proakis and M. Salehi, Digital Communications, 5th edition, McGraw-Hill, 2008: see the chapters “Optimum Receivers for AWGN Channels” and “Digital Communication Through Band-Limited Channels”.
• A. Goldsmith, Wireless Communications, Cambridge University Press, 2005: see the chapter “Digital Modulation and Detection”.