Calling just about any function once-per-pixel is going to be slow as molasses in debug mode because functions are never inlined in debug mode.  Luckily you can bypass all of these expensive function calls by writing directly to memory:

std::vector<sf::Uint8> SfmlFilm::ToRgba(const std::vector<Spectrum>& image)
{
	std::vector<sf::Uint8> rgba(GetWidth() * GetHeight() * 4);
  
	auto p = &rgba[0];
    
	for (auto spectrum : image)
	{
		const auto max = 255;
		*p++ = static_cast<sf::Uint8>(spectrum.r * max);
		*p++ = static_cast<sf::Uint8>(spectrum.g * max);
		*p++ = static_cast<sf::Uint8>(spectrum.b * max);
		*p++ = max;
	}
	return rgba;
}

Thanks for the help, everybody!

For the ones interested, splitting up the steps shows that it really is in the push(/emplace)_back() calls:

Using a light breeze's suggestion, FPS went up from 1 to 95.