The next thing to do was to get Jupyter notebooks to display the plots using pure Swift. Currently this was being done using the EnableIPythonDisplay.swift library bundled with swift-jupyter. The image was being passed to a display function after encoding it to the base64 format. The display function then used IPython’s display function along with the image format(i.e. PNG) to generate a display message, which was then sent to the Jupyter Kernel. And voila! The image was displayed!

For those who don’t know what a Jupyter Notebook is, quoting the Jupyter Notebook website:

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

The way Jupyter displays thing is using messages. A message contains all the data that Jupyter requires to display anything.

The general format of a Jupyter message(as mentioned here) is :

{
  # The message header contains a pair of unique identifiers for the
  # originating session and the actual message id, in addition to the
  # username for the process that generated the message.  This is useful in
  # collaborative settings where multiple users may be interacting with the
  # same kernel simultaneously, so that frontends can label the various
  # messages in a meaningful way.
  'header' : {
                'msg_id' : str, # typically UUID, must be unique per message
                'username' : str,
                'session' : str, # typically UUID, should be unique per session
                # ISO 8601 timestamp for when the message is created
                'date': str,
                # All recognized message type strings are listed below.
                'msg_type' : str,
                # the message protocol version
                'version' : '5.0',
     },

  # In a chain of messages, the header from the parent is copied so that
  # clients can track where messages come from.
  'parent_header' : dict,

  # Any metadata associated with the message.
  'metadata' : dict,

  # The actual content of the message must be a dict, whose structure
  # depends on the message type.
  'content' : dict,

  # optional: buffers is a list of binary data buffers for implementations
  # that support binary extensions to the protocol.
  'buffers': list,
}

To display images/graphics the message type(msg_type tag in the header) is display_data. 

So what I had to do was, generate such display messages using Swift and send them over to the Jupyter Kernel.

Initially I found it pretty difficult to understand how messaging worked in Jupyter, so I had next to no idea about what I had to do. I decided to give it a few days and i finally started to understand what it all was. I made a naive implementation of the message generating code but only to fail. I got some error which seemed like gibberish to me. Even Googling didn’t help me here. Then Marc came to rescue. In swift-jupyter we send the message to the kernel in a serialized form(a utf8 CString). Marc found that while converting to this form, an extra null terminator was added to it, which created an error with Jupyter. Also in my implementation the message wasn’t an optional. So if a message was empty, it was still getting sent and this caused some errors. So, Marc removed the null terminators and made the message an optional. 

But just sending messages wasn’t enough. Jupyter also needed message signing. A unique signature was generated for each message which was verified by Jupyter and if successful, only then we could see the result appear in the notebook. All this time we wore working by disabling message signing in Jupyter like this:

jupyter notebook –Session.key=‘b"“’

The signature is the HMAC hex digest of the concatenation of:

  • A shared key (typically the key field of a connection file)
  • The serialized header
  • The serialized parent header
  • The serialized metadata
  • The serialized content

The hashing function to be used was sha256.

The problem that I faced here was, how do I get the HMAC hex digest? Implementing it on my own was out of question, because there was no way to guarantee its security. There are no cryptographic functions in Swift available on Linux. On Mac we could use CommonCrypto. It is an Obj-C library that one could import using bridging headers between Obj-C and Swift. The obstacle here was that using CommmonCrypto wouldn’t make it cross-platform, and also I had no idea of how to use bridging headers with swift-jupyter. 

Brad suggested that we could use IBM’s BlueCryptor library. This was a cross-platform Swift cryptography library. On the Mac platform it used the CommonCrypto module and on Linux it used the libssl-dev package.

So to use the pure swift library the user had to install BlueCryptor before, or disable message signing.

Using BlueCryptor was pretty straightforward. The exact way to get an HMAC hex digest was given in the documentation. I had a few hiccups doing this initially, because I did not know that the key and data for hashing were to be provided as hex strings. I was using simple strings before and therefore wasn’t getting any results. Also the Swift toolchain I was using had some problems due to which it couldn’t compile BlueCryptor. The only Swift toolchains compatible with swift-jupyter were the S4TF nightly builds, which for some reason didn’t have python3.6 support on that specific day, which was a must for swift-jupyter. So I had to build it myself. This took a lot more time than I expected. 

After all these hassles, finally I got it working. Yipee!!

You can find the implementation here: EnableJupyterDisplay.swift

Then I submitted a PR, which was reviewed many times. The comments were mostly about code formatting and style. I still hadn’t gotten used to the Swift style. I did what was asked and the PR was merged. There is still work to be done on this, like supporting more file formats, maybe include audio playback, etc. But all of that can come later. Right now the focus is just on making the data visualization library.

When I had taken a break from thinking about messaging in Jupyter I got to wotk on fixing the plot dimensions issue with the AGG Renderer.

How the AGG Renderer works is that, on initializing an instance of the renderer creates a buffer that holds all the pixels of the image. One could easily say that to have a custom size, one could pass in the image dimensions in the initializer/constructor of the respective class and allocate a buffer of the required size. But on doing this I got a segmentation fault. On investigating the issue using gdb, I found that the error had to do something with the row_accessor not being able to access a pixel. Essentially we were trying to access a wrong memory location somewhere. But even after going through the code many a times I couldn’t figure out where i was going wrong. The previous implementation had the buffer declared globally. It worked that way, but not when the buffer was part of the class.

I decided to dig a bit deeper and tried to find out answers online. There seemed to be no such answers anywhere but, I found an AGG mailing list, that had been inactive for more than two years. I did not have high hopes, but there was no harm in giving it a try. I’ll add the link to the mailing list at the end if anyone would need it in the future.

class Plot{

 public:
   agg::rasterizer_scanline_aa<> m_ras;
   agg::scanline_p8              m_sl_p8;
   agg::line_cap_e roundCap = agg::round_cap;
   renderer_aa ren_aa;
   int pngBufferSize = 0;

   unsigned char* buffer;


   agg::int8u*           m_pattern;
   agg::rendering_buffer m_pattern_rbuf;
   renderer_base_pre rb_pre;

   Plot(float width, float height, float subW, float subH){
     frame_width = width;
     frame_height = height;
     sub_width = subW;
     sub_height = subH;
     buffer = new unsigned char[frame_width*frame_height*3];
     agg::rendering_buffer rbuf = agg::rendering_buffer(buffer,
frame_width, frame_height, -frame_width*3);
     pixfmt pixf = pixfmt(rbuf);
     renderer_base rb = renderer_base(pixf);
     ren_aa = renderer_aa(rb);
     pixfmt_pre pixf_pre(rbuf);
     rb_pre = renderer_base_pre(pixf_pre);
   }

This was the part of the code I was using to initialize the renderer. I sent this snippet to the mailing list. Fortunately someone called Stephan Abmus responded to my mail and pointed out that all the objects required for rendering needed to stay in memory while rendering. I won’t bore you with excessive details but here is a brief summary. I was declaring the pixf and rendering buffer in the constructor and as soon as it went out of scope those declarations went out of memory too and thus we got an error. I am still not sure why making the buffer global worked. Even declaring the buffer and pixf outside the constructor didn’t help. Actually pixf is a typedef for pix_fmt, which holds the format of the pixel to be used in the image, for example RGBA. In fact I couldn’t declare pixf outside the constructor because it didn’t have a default constructor. I had to pass in the rendering buffer while declarinf the pixf object. This meant I couldn’t declare the rendering_buffer outside of the constructor. The only option left was to declare the rendering buffer object and the pixf object inside every function that renderer something onto the image. This method worked!I had finally fixed the bug I was racking my brains on from the first day!

The next task to accomplish was to provide a better way to give data input to the plots, than the Point type. I was still trying to think of a way to do that. I had tried out generics but at the time it didn’t seem like a viable method. This topic is still under development and once something is finalized I’ll write about it.

The goals for my first milestone were done. 

On June 28, the results of the 1st evaluation were declared. I had finally passed the first evaluation. But I did know beforehand that I’d be passing the evaluation as I had already spoken about it with Brad and Marc. Both of them had given me positive feedback regarding it.

Nonetheless, I’ll describe very briefly what the evaluation was. Both the stuent and the mentors were to fill out an evaluation form. The mentor had to give an evaluation of how well the student had performed, and if he/she should pass or not, and the student had to answer a few questions about their GSoC experience and how well they were able to communicate with the mentors.

Phase 1 went pretty smoothly. There were a few ups and downs, but I’ve the got possibly the best mentors ever who guided me through the whole process. I got to know about Swift, it’s coding format and style. I learnt a lot about building frameworks through my mentors’ answers to  my questions. I observes=d how they think about the features to implement and what kind of implementation would be best for both the developer and the end user of the product.

image

This was the feedback I received from my mentors. It was a really great motivator for me and I want to give my best now more than ever. 

I hope the rest of the summers stays on to be the great experience I’ve had till now.

I’ve worked on implementing Histogram since then, and I also seem to have made some progress on the Data Input front. I’ll talk about any further developments in the next post.

See you next time!