To simplify what I need to do is to create a RGB surface from an AVFrame. SDL has a function just for this:
SDL_Surface *SDL_CreateRGBSurfaceFrom(void *pixels,
int width, int height, int depth, int pitch,
Uint32 Rmask, Uint32 Gmask, Uint32 Bmask, Uint32 Amask);
The function is straightforward, I just don't know how to get from an AVFrame to raw pixel data readable into this function. I've never been good with this. The Rmask, Gmask, Bmask stuff I believe has to do with the byte order. For another similar function which is SDL_CreateRGBSurface, SDL has an example which includes the following:
/* Create a 32-bit surface with the bytes of each pixel in R,G,B,A order,
as expected by OpenGL for textures */
SDL_Surface *surface;
Uint32 rmask, gmask, bmask, amask;
/* SDL interprets each pixel as a 32-bit number, so our masks must depend
on the endianness (byte order) of the machine */
#if SDL_BYTEORDER == SDL_BIG_ENDIAN
rmask = 0xff000000;
gmask = 0x00ff0000;
bmask = 0x0000ff00;
amask = 0x000000ff;
#else
rmask = 0x000000ff;
gmask = 0x0000ff00;
bmask = 0x00ff0000;
amask = 0xff000000;
#endif
. . .
. . .
. . .
You can use [font=courier new,courier,monospace]sws_scale[/font] to convert the [font=courier new,courier,monospace]AVFrame[/font] to an RGB24/RGBA32 frame ([font=courier new,courier,monospace]PIX_FMT_RGB24[/font] or [font=courier new,courier,monospace]PIX_FMT_RGBA[/font]), similar to how you're already doing it. I don't have time right now to post some sample code, but I'll try and do that in the morning. The RGB pixel data will be stored in the [font=courier new,courier,monospace]AVFrame[/font]'s [font=courier new,courier,monospace]data[0][/font] member, in the format: [r, g, b, r, g, b, r, g, b, ...] or [r, g, b, a, r, g, b, a, r, g, b, a, ...] (regardless of endianness). The [font=courier new,courier,monospace]AVFrame[/font]'s [font=courier new,courier,monospace]linesize[0][/font] member will tell you how many bytes are in each row of pixel data in the frame (it'll be at least 3 * the frame's width (or 4 * the frame's with, if RGBA), but it can be slightly more (for padding... they do this so they can use certain SIMD instructions for optimizations)). You should be able to create a byte buffer the size of width * height * 3 (or * 4 if RGBA), and copy each row of pixels in the [font=courier new,courier,monospace]AVFrame[/font] to this buffer that you can pass to [font=courier new,courier,monospace]SDL_CreateRGBSurfaceFrom()[/font]. I'll post a sample tomorrow. Here's some small psuedo-code until I can write some proper stuff tomorrow (it's C++, sorry, but the concept is there):
char *buffer = new char[frameWidth * frameHeight];
AVFrame *frame = yourRgb24Frame;
for (int i = 0; i < frameHeight; ++i)
{
// The primary purpose I don't just copy the frame->data[0] straight into the buffer is because frame->linesize[0] may not equal
// the frame's width
std::copy(frame->data[0] + i * frame->linesize[0], frame->data[0] + i * frame->linesize[0] + 3 * frameWidth, buffer + i * frameWidth);
}
SDL_CreateRGBSurfaceFrom(buffer, ...);
That code is for RGB24 pixel data, but I don't know if [font=courier new,courier,monospace]SDL_CreateRGBSurfaceFrom()[/font] expects a fourth alpha channel to be in the buffer. If so, the code will need to be modified (the simplest would probably be to use [font=courier new,courier,monospace]sws_scale[/font] to make a RGBA frame and replace the 3 from above to a 4).
Alright, here's some code. It's not quite fully functional, but it has the basics of each step. You'll probably need to separate out each chunk of put it where it belongs. I'm not sure about how to use SDL, so you're kind of on your own there (unless someone else can comment on that). There could be some errors in this code sample, but it should help illustrate the idea. You could also safely make some alterations to this, but I'll leave that up to you. I went for simplicity in this small sample.
// I'm assuming you've already opened the video...
SwsContext* imageConvertContext = NULL;
int frameFinished = 0;
int videoStreamIndex = 0; // I'm making this up, of course
char* buffer;
AVPacket packet;
AVFrame* decodingFrame;
AVFrame* rgbFrame;
// Use PIX_FMT_RGBA instead if you need rgba...
if (avpicture_alloc((AVPicture*)rgbFrame, PIX_FMT_RGB24, codecContext->width, codecContext->height) != 0)
{
// error
}
imageConvertContext = sws_getCachedContext(NULL, // current swscontext
codecContext->width, // source width
codecContext->height, // source height
codecContext->pix_fmt, // source format
codecContext->width, // dest width
codecContext->height, // dest height
PIX_FMT_RGB24, // dest format
SWS_FAST_BILINEAR, // flags
NULL, // source filter
NULL, // dest filter
NULL); // params
if (m_imageConvertContext == NULL)
{
// error...
}
// This is where the decoding and converting part go. I'll assume you've got a decoding loop already.
// This isn't quite a proper loop, but it has the important part of using sws_scale()
av_free_packet(&packet);
while (av_read_frame(formatContext, &packet) == 0)
{
if (packet.stream_index == videoStreamIndex)
{
avcodec_decode_video2(codecContext, decodingFrame, &frameFinished, &packet);
if (frameFinished)
{
sws_scale(imageConvertContext,
decodingFrame->data,
decodingFrame->linesize,
0,
decodingFrame->height,
rgbFrame->data,
rgbFrame->linesize);
break;
}
}
av_free_packet(&packet);
}
// Now copy the rgbFrame into the buffer
for (int i = 0; i < codecContext->height; ++i)
{
memcpy(buffer + i * codecContext->width, rgbFrame->data[0] + i * rgbFrame->linesize[0], codecContext->width * 3); // change the 3 to a 4 if doing rgba
}
SDL_CreateRGBSurfaceFrom(buffer, ...);
// Clean up when you're done... probably at the end of your program
sws_freeContext(imageConvertContext);
avpicture_free((AVPicture*)rgbFrame);
av_free(rgbFrame);
av_free(decodingFrame);
Alright, here's some code. It's not quite fully functional, but it has the basics of each step. You'll probably need to separate out each chunk of put it where it belongs. I'm not sure about how to use SDL, so you're kind of on your own there (unless someone else can comment on that). There could be some errors in this code sample, but it should help illustrate the idea. You could also safely make some alterations to this, but I'll leave that up to you. I went for simplicity in this small sample.
// I'm assuming you've already opened the video...
SwsContext* imageConvertContext = NULL;
int frameFinished = 0;
int videoStreamIndex = 0; // I'm making this up, of course
char* buffer;
AVPacket packet;
AVFrame* decodingFrame;
AVFrame* rgbFrame;
// Use PIX_FMT_RGBA instead if you need rgba...
if (avpicture_alloc((AVPicture*)rgbFrame, PIX_FMT_RGB24, codecContext->width, codecContext->height) != 0)
{
// error
}
imageConvertContext = sws_getCachedContext(NULL, // current swscontext
codecContext->width, // source width
codecContext->height, // source height
codecContext->pix_fmt, // source format
codecContext->width, // dest width
codecContext->height, // dest height
PIX_FMT_RGB24, // dest format
SWS_FAST_BILINEAR, // flags
NULL, // source filter
NULL, // dest filter
NULL); // params
if (m_imageConvertContext == NULL)
{
// error...
}
// This is where the decoding and converting part go. I'll assume you've got a decoding loop already.
// This isn't quite a proper loop, but it has the important part of using sws_scale()
av_free_packet(&packet);
while (av_read_frame(formatContext, &packet) == 0)
{
if (packet.stream_index == videoStreamIndex)
{
avcodec_decode_video2(codecContext, decodingFrame, &frameFinished, &packet);
if (frameFinished)
{
sws_scale(imageConvertContext,
decodingFrame->data,
decodingFrame->linesize,
0,
decodingFrame->height,
rgbFrame->data,
rgbFrame->linesize);
break;
}
}
av_free_packet(&packet);
}
// Now copy the rgbFrame into the buffer
for (int i = 0; i < codecContext->height; ++i)
{
memcpy(buffer + i * codecContext->width, rgbFrame->data[0] + i * rgbFrame->linesize[0], codecContext->width * 3); // change the 3 to a 4 if doing rgba
}
SDL_CreateRGBSurfaceFrom(buffer, ...);
// Clean up when you're done... probably at the end of your program
sws_freeContext(imageConvertContext);
avpicture_free((AVPicture*)rgbFrame);
av_free(rgbFrame);
av_free(decodingFrame);
Man, it's amazing to work with so much code, Just last year I couldn't possibly conceive of doing something like this... Ah well enough of that, I'm getting another segmentation fault when attempting to blit it to the screen.
I modified my struct with the appropriate variables I'll need to manipulate:
Going back to my main function I created an SDL_Surface* bmp as well as the Uint32s for rmask, gmask, and etc. that were defined in my previous example. So here's where the conversion and stuff is actually taking place:
I assume I'm doing something wrong when I'm creating the RGB surface. If it would be any easier to you since you aren't familiar with SDL, I would be just fine with an OpenGL texture. The actual game I plan on doing will be in SDL_OpenGL. (basically OpenGL that instead of using GLUT, it uses SDL) I just need to play these two intro videos, and one of them with audio and both are at 720p. I just would've really liked to make something for SDL, but I guess I'm just not ready.
