Home
Reading
Searching
Subscribe
Sponsors
Statistics
Posting
Contact
Spam
Lists
Links
About
Hosting
Filtering
Features Download
Marketing
Archives
FAQ
Blog
 
Gmane
From: =?UTF-8?Q?Dominique_Pell=C3=A9?= <dominique.pelle <at> gmail.com>
Subject: How to use afl-fuzz to find bugs in Vim
Newsgroups: gmane.editors.vim.devel
Date: Monday 11th January 2016 22:12:15 UTC (about 1 year ago)
By popular demand, here is a tutorial describing how to fuzz Vim
with afl-fuzz (American Fuzzy Lop) on Linux.

For details about afl-fuzz, make sure you read:

  http://lcamtuf.coredump.cx/afl/
  http://lcamtuf.coredump.cx/afl/README.txt

I also recommend reading this fascinating blog entry:

  http://lcamtuf.blogspot.nl/2014/11/pulling-jpegs-out-of-thin-air.html

afl-fuzz has successfully found several bugs in Vim:

  http://permalink.gmane.org/gmane.editors.vim.devel/54481
(Jan 10, 2016)
  http://permalink.gmane.org/gmane.editors.vim.devel/54447
(Jan 10, 2016)
  http://permalink.gmane.org/gmane.editors.vim.devel/54279
(Jan  5, 2016)
  http://permalink.gmane.org/gmane.editors.vim.devel/54261
(Jan  5, 2016)
  http://permalink.gmane.org/gmane.editors.vim.devel/54227
(Jan  4, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/52921
(Nov  7, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/52920
(Nov  6, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/52863
(Nov  4, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/52861
(Nov  4, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/52306
(Sep 27, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/51655
(Aug 13, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/51643
(Aug 12, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/51637
(Aug 12, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/51619
(Aug 11, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/51587
(Aug  9, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/51570
(Aug  8, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/51568
(Aug  7, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/51478
(Aug  1, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/51473
(Aug  1, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/51425
(Jul 28, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/51400
(Jul 26, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/51394
(Jul 26, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/50722
(May 23, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/50487
(Apr 28, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/50406
(Apr 22, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/50381
(Apr 20, 2015)
  http://permalink.gmane.org/gmane.editors.vim.devel/50120
(Mar 30, 2015)

Steps to use afl-fuzz with Vim:

Step 1) Download and install afl-fuzz

  $ wget http://lcamtuf.coredump.cx/afl/releases/afl-latest.tgz
  $ tar zxvf afl-latest.tgz
  $ cd afl-1.96b    # the version might be different.
  $ make -j4
  $ sudo make install

Step 2) Build vim with afl-gcc to instrument vim

  $ cd vim
  $ make clean
  $ rm src/auto/config.cache
  $ CC=/usr/local/bin/afl-gcc ./configure \
      --with-features=huge \
      --enable-gui=none
  $ make -j4

Instrumentation with afl-gcc guides the fuzzer to find
new code paths in Vim.

Step 3) Setup input and output directories of afl-fuzz

  $ cd vim/src
  $ mkdir in/
  $ mkdir out/

The out/ directory should be left empty, it is used by afl-fuzz
to store its state, the discovered test cases that cause crashes,
hangs and a corpus of test cases that exercise many paths in vim.

You need to put at least one input file in the in/ directory.
You also need to chose the options to start vim.

afl-fuzz will first run vim with the provided input example(s)
and then mutate the input to create other interesting test cases
that trigger new code paths in vim.

What follows are two examples: the first example takes an
input regexp to fuzz vim regexp engine:

Example #1:

  $ cat in/regexp
  a*b\+\|[0-9]\|\d{1,9}

  $ export AFL_USE_ASAN=1
  $ afl-fuzz -M fuzzer01 -m 99999999 -t 4000 -i in/ -o out/ \
      ./vim -u NONE -X -Z -e -s \
      -c 'call search(getline("."))'' -c ':qa!' @@

The second example takes a VimL source file as input to stress
the VimL parser.

Example #2:

  Copy a viml file into in/ the directory and start afl-fuzz with:

  $ export AFL_USE_ASAN=1
  $ afl-fuzz -M fuzzer01 -m 99999999 -t 4000 -i in/ -o out/ \
      ./vim -u NONE -X -Z -e -s -S @@ -c ':qa!'

The @@ part is automatically replaced by the test case file
being tested and its mutations.

After a few seconds of running afl-fuzz, you should expect to see
the value 'path' which keeps incrementing.  If it stays at 1, it
means that something is wrong (afl-fuzz is not discovering paths).

Make sure that you keep the input in the in/ directory not too
large as large inputs will slow down fuzzing.  In particular,
do not store vim backup files there  i.e. you should not put two
almost identical files like in/ex.vim and in/ex.viml~ as they would
contain mostly useless duplication. Remove comments from VimL
input file, as fuzzing comments is most likely a waste of time.

You then need to let afl-fuzz run for a long time (days, or weeks)
but sometimes bugs can be found in a few seconds. Afl-fuzz displays
many stats. The most important stat is the number of discovered
crashes. For more information about afl-fuzz status screen, see:

  http://lcamtuf.coredump.cx/afl/status_screen.txt

Test cases that cause a crash are stored in fuzzer01/crashes/.
You can re-run them later for debugging the crash. When a crash is
found, it's good to simplify it as much as possible while still
reproducing the crash, either manually or by using the afl-tmin tool.

Different inputs and different command line options will
fuzz different parts of Vim and will thus find different
bugs. This is where you should be creative by selecting the
input and command line options to start vim.

The above examples use options -u NONE and -X to speed up vim startup.
The faster Vim starts up, the faster afl-fuzz will be able to find
bugs. Options -e -s are used to make vim silent to avoid the 'MORE'
prompt which could block vim. The option -Z disables external commands
which makes fuzzing safer.

WARNING: fuzzing could in theory come up with running shell commands
that deletes your files, open and modify existing files, etc. To be
safer, it is best to fuzz as a user created only for the purpose
of fuzzing.

Besides the fuzzer01/crashes/ directory, afl-fuzz will also create a
directory fuzzer01/hangs/ for test cases that cause vim to  hang.
There can be false positives there, depending on which time-out
value (-t option of afl-fuzz) was used. afl-fuzz also creates a
corpus of test cases which exercise many code paths in Vim in
out/fuzzer01/queue/.  The corpus can be used later to test vim with
slower checkers such as valgrind, ubsan, msan, etc.  Running
valgrind for example is useful to find use of uninitialized
memory or memory leaks which are not detected when running afl-fuzz
with asan.

If you want to fuzz in parallel on several cores, make sure that
you read:

  https://github.com/mirrorer/afl/blob/master/docs/parallel_fuzzing.txt

For parallel fuzzing, I run the following command in a 2nd terminal
for example:

  $ afl-fuzz -S fuzzer02 -m 99999999 -t 4000 -i in/ -o out/ \
      ./vim -u NONE -Z -X -e -s -S @@ -c ':qa!'

Then in a 3rd terminal:

  $ afl-fuzz -S fuzzer03 -m 99999999 -t 4000 -i in/ -o out/ \
      ./vim -u NONE -Z -X -e -s -S @@ -c ':qa!'

Etc.

Make sure that the first fuzzer is started with the -M option
(for master) and the subsequent parallel fuzzers are run with
the -S option (for slave) with different IDs (fuzzer01, fuzzer02,
fuzzer03, etc.)

That's it. There is certainly more to it. In particular, I have not
yet explored the dictionary feature of afl-fuzz, which could be a
powerful feature to fuzz Vim. See:

  http://lcamtuf.blogspot.nl/2015/01/afl-fuzz-making-up-grammar-with.html

Good lock finding and sharing interesting bugs in Vim found
with afl-fuzz.

Regards
Dominique

-- 
-- 
You received this message from the "vim_dev" maillist.
Do not top-post! Type your reply below the text you are replying to.
For more information, visit http://www.vim.org/maillist.php

--- 
You received this message because you are subscribed to the Google Groups
"vim_dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [email protected]
For more options, visit https://groups.google.com/d/optout.
 
CD: 3ms