Almost 4 years ago, I thought than after the good adoption rate of edLeak from my colleagues, they would love to have a python API to write their own script. Unfortunately, the success has been... very disappointing.
The fact is that developers are lazy! This is probably one of the main reason why we ended up doing software development : We do not want to do the same thing twice, so we code one task and then it is repeated an infinite number of times if necessary while we can relax and drink some beer (or more probably go for another task the second after your last commit reached the CI system). For edLeak the outcome was simple : People liked to use edLeak even tough the web app was very basic but they were not interested in writing their own scripts. In other words, the web app did its job well so they did not try to spend more time learning APIs for little benefits.
So I had to face reality : This really basic web app that was poorly written within a few hours was much more successful than a - hopefully - well defined python API that could allow to do much more things. Time had come to re-write this web-app and hide the python stuff. Before rewriting the edLeak app, I had two things to solve:
- Be clear on the future design of edkit.
- Choose the good web technologies to write the app.
The first point was raised when I wrote the python bindings and started to play with automated leak detection: It was clear that some parts cannot run in a browser but must run on a development system, written in another language. I ended up with the following design that I started to implement at the same time that re-writing the UI part:
The main change is the addition of an edKit server that implements all the needed logic. Also since this server can access resources on the development system it can use debug/unstripped symbols information to improve backtrace and symbols information resolution. The core of this server is written in python3. This will allow to run machine learning frameworks such as scikit-learn and tensorflow (but this will be a story for another post...). The other components, including the UI, talk to this server via the wamp.js protocol. This protocol is the best choice for edKit:
- It supports RPC and pub/sub messaging patterns.
- It allows bi-directional communication.
- It has client implementations in many languages, included javascript (that can be browserified) and C++.
For now, the edLeak agent does not have a wampjs controller, but this will come soon. A benefit of having the agent connected to the server instead of the other way is that it is easier to have several agents connected to server, and do distributed debugging. With this design, it is also simple to replace the UI with some scripts to automate memory leak detection (No I did not give-up on this task).
So now let's see how the new web application is designed.
The Javascript Framework Hell
The most difficult task in web programming is probably to choose what technologies to use. The good news is that many things have - positively - changed since I wrote the edLeak web app: At that time, there was no solution to write a web app, but just libraries. So edLeak was just a web page with jquery and a chart library. A few years later, enyojs was available and I used it for the edVent app. Enyojs is now a dead project but with the advent of mobile web apps there are now many options, too many options, and still few \"all in one\" solutions.
Anyway, some parts were already clear for this re-write. It had to be based on ReactiveX that I started to use on Android and that I liked a lot. The UI would be based on ReactJS, because everybody uses it these days. Also, the usual suspect came quickly to develop all this with the help of nodejs: wepback to package the application and babel to write code in ES2015.
However, journeys never go the way you expect. In this case ReactJS was the disappointment: After playing a little bit with it, I could not feel confortable with its paradigm. This probably comes from a lack of experience, but I do not see the point of defining immutable components with const properties, and then doing workarounds to change the component state. So I quickly gave up with ReactJS and replaced it with semantic-ui that was much more natural to use. Moreover, while playing with rxjs I ended up implementing all the app with cycle.js that also allows to use shadow DOM.
So finally, the new edLeak app is based on the following frameworks: rxjs, cycle.js, semantic-UI, wamp.js, webpack, babel. The whole thing is packaged as a web app and served by the edkit server via crossbar and autobahn.
UI evolutions : Even less is even more
The goal in re-writing the edLeak app was not only to clean the code, but also to change some UI design choices so that the evolutions I have in mind will be easier to add. Here is the new look of edLeak:
There are some visual changes, but the overall structure is the same: The toolbar on the top is the place where one can start and configure new measures. The main change in the toolbar is a real capture area with a real start/stop button as well as a progress bar indicating the current progress of the measure.
Bellow the toolbar is the biggest area : The scatterplot visualization of the current measure. This part was almost kept as is. For now, it is still based on highcharts but I will migrate the plot part to d3js later. This will allow to refresh the plot in real-time as the measure goes on. The biggest change is visually subtle : At the end of the measure, allocation points that leak are drawn in red instead of green. This is done by querying a dedicated service on the server, and is based on machine learning.
A new space was added on the right side of the screen. Today, it only contains the callstack state of the selection allocation point. In the future, there will be much more things such as the callstack and a time-based view of the current allocation point. Things to track reference-counting should also appear in this area once available.
Next steps
More than the UI, all these evolutions should be a solid foundation for all future evolutions that I have in mind. You can use it directly from pypi :
pip3 install edkit-server
or from github: https://github.com/edkit/edkit-server
The next big challenge is to improve call stack retrieval on stripped binaries, without debug symbols. Several small things can also be added easily such as additional views to help visual analysis. Documentation is also an area where something has to be done!
Romain.
Interested in reactive programming ? Here is my book on
Reactive Programming with Python. I wrote this book to explain how to develop RxPY applications step by step.