Create your own syntax highlighter/code beautifier with python and pygments

Now a days I have seen that there are lots and lots of people who want to share their code to this world and help people who are in need of that code(or for your personal satisfaction 😜 ). Even though sharing code is easy just by using a <pre> tag or a <code> tag, but sometimes many people including me feel comfortable when the code is neatly highlighted. 

I will show you a comparison between a neatly highlighted code and un-highlighted code.

Not highlighted code

Highlighted Code

Okay don't worry if you don't have a code highlighter, you can create your own code beautifier/syntax highlighter and present your code in a neat and tidy format. To do this you just need to know basics of python and I will teach you the rest in this post.

Okay let's get started.

 To do syntax highlighting with python we will use a library called pygments. We are using this library because:

• It supports more than 300 languages/markups.
• It can output the code in HTML, LaTeX, RTF, ASCII sequences.
• It is very fast.
• You can run it from command line also.
•  You can add your own programming language if you have one.
• You can add your own style and reuse it any number of times.
• Last but not the least, it is open source 😃

 We will start with installation of pygments:

Installing with pip

$ pip install pygments

Installing with easy_install

$ easy_install pygments

Installing manually

First download the package from pypi and then cd to the directory. Now when you are sure that you are in the pygments directory then you can type the following command:

$ python setup.py install

About Pygments

Pygments has four important components which work together to high light given code. They are as follows:

•  Lexers: We all know that every programming language has its own reserved keywords. A lexer is the one which will identify these keywords in your program and then split the code depending on the programming language you have given.
• Filter: After the lexer has split the keywords and other stuff then the code is sent to filter where it is filtered and processed in the background which is really not necessary for us to know at this point of time. 
• Formatter: A formatter will take the tokens that are supplied by the filter and then it will convert it to HTML/LaTeX/RTF/ASCII sequence depending on your requirement.
• Style: After the code is converted to required sequence, we also need style or CSS for HTML to make it look beautiful. That part of the work is taken care here.

Creating first code Beautifier

As you have now understood the basics of what is happening behind the scenes of pygments you can now create your own syntax highlighter within no time.

from pygments.lexers import get_lexer_by_name
from pygments.formatters import HtmlFormatter
from pygments import highlight

programming_language = get_lexer_by_name('python')

raw_code = 'print("Programming is fun!")'

highlighted_code = highlight(raw_code, programming_language, HtmlFormatter())

print highlighted_code

I have saved above program as tutorial.py. If you will run above program either on command line or on IDLE you will get a result something like this:

pygments from command line

Okay lets first understand what happened before looking at the output.
In the first line I have imported get_lexer_by_name, a function which, when called with a programming language as a parameter will act like a lexer for the programming language. You can also directly import python lexer just by typing from pygments.lexers import PythonLexer which is equivalent to get_lexer_by_name('python'). If you want to import some other lexer then you can visit this page and find out your lexer: Available Lexers - Pygments
Next from pygments.formatters we have imported HtmlFormatter a formatter, which will format the given code and output it to a HTML Format.
Next we have imported the highlight function.
We have created a variable programming_language for the lexer.
We have saved our code that we need to highlight in a variable named raw_code.
In the next line we have highlighted our code calling the highlight function with raw_code, lexer and formatter as parameters.
In the last line we have printed the HTML Code we have generated.    
As we have used HtmlFormatter we have got the code ouput in HTML Format. If you were to get code in some other format other than HTML then you can see Pygments Formatters,  and import the required formatter.
Coming back, the code generated by running tutorial.py looks as follows:

<div class="highlight">
 <pre>
  <span></span>
  <span class="k">print</span>
  <span class="p">(</span>
  <span class="s2">&quot;Programming is fun!&quot;</span>
  <span class="p">)</span>
 </pre>
</div>

If you will save above code as .html file, open it in a browser then you will see something like this:

Screen Shot of pygments without CSS

 By now you should have understood on why the code is still black and white and not in a colorful way. 

Yes you have guessed it right!

The reason is because even though we have generated our HTML code we have not yet added the CSS(Style) information. To get the css information you can do as follows:

print HtmlFormatter().get_style_defs('.highlight')

 This will generate CSS styling information for the .highlight class.

Digging Deep

Till now we have seen the basics of Pygments, but we will dig a little deep and finish of this post.

Now see the following code and try to understand it:

from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter
from pygments import highlight

programming_language = PythonLexer()

raw_code = 'print("Programming is fun!")'

formatter = HtmlFormatter(cssclass='radiusofcircle',linenos=True)

highlighted_code = highlight(raw_code,programming_language,formatter)

print(highlighted_code)

Lets first understand the above program before we will see the output:
In the first three lines we have imported the lexer, formatter, highlightner.
In the next line we have assigned our PythonLexer to variable programming_language.
As in the previous code in the next line we have written code and saved it to variable raw_code. 
 Next we have called HtmlFormatter with parameters cssclass = radiusofcircle which means that the output HTML should have class=radiusofcircle instead of default .highlight, the one we have got in the previous code. linenos=True which means that the code should also have line numbers.
All the remaining code is same as the previous code.
 Now run the above program and you will see code generated like this:

  <table class="radiusofcircletable">
    <tr>
      <td class="linenos">
        <div class="linenodiv">
          <pre>
1
</pre>
        </div>
      </td>

      <td class="code">
        <div class="radiusofcircle">
          <pre>
<span class="k">print</span><span class="p">(</span><span class=
"s2">"Programming is fun!"</span><span class="p">)</span>
</pre>
        </div>
      </td>
    </tr>
  </table>

I am assuming that you have observed the difference.

Summary

We have come to the end of the post and I think I have covered all the topics which are required for you to highlight your code. If you want to delve deep then you can visit the official documentation from here: Introduction and QuickStart - Pygments
  
As always I have tried to write this post in such a way that it is easy even for the beginners. If you have not understood anything or have any doubt then please do comment in the comment box below and I will be glad to help you. You can also contact me from here: Contact me.

Also please do comment on how I can improve this post so that it be easy for everyone to understand. Please don't hesitate to comment/contact me if I have made any typo or mistake in this post. Please do comment if you want me to add any topic to this post.

Thank you. Have a nice day😄