Artifacts after seekign to first frame of the file

I am trying to decode h264 video from a file using avcodec, I pull all the
frames into a circular buffer and get a frame from the buffer, submit to the
decoder. Everything seems to be fine except when I reach the end of the file, I
do a seek to the beginning, weird artifacts starts hapenning.Here is what I have
to seek to beginning:  

if(av_read_frame(pFormatCtx, packet) < 0)
{

   av_seek_frame(pFormatCtx, ffStreamIdx, 0, AVSEEK_FLAG_FRAME);
}
else
{
   stuff frame to buffer..
}

Is there something wrong in the way I am seeking to the beginning of file? If
not, is there something else I have to do to, like reset some element of
avformat or avcontext?

Thanks