Why NeuralMarker?
In the last 5 years, I have seen dozens of Deep Learning projects which made into production. Every time, we were able to push the models in production, we always followed more or less similar process. It had more to do with training data than the algorithm. It was always painful and involved us getting our hands dirty in the dataset. NeuralMarker is our attempt to bake all those learnings and best practices in a platform so that AI teams developing computer vision solutions can deliver production quality model more reliably and rapidly. In this post I would go through some of the key features of the platform.
The problems:
1. Developing Production-Grade AI Model is an iterative Process and we still don’t have the right workflow tools:
Deep Learning models are still a black box. The typical AI development workflow looks like this: You collect bunch of training data and train your model on it. You then run the model to see what the model has learned( so what was actually there in the training data) and you realize that there are some gaps (blindspots) in your training data so we need to collect more training data for certain particular types to fill those blindspots. For most projects this happens in multiple iterations.
Currently, our annotation tools don’t allow for building these workflows naturally. Resulting in a lot of communication happening on the phone and emails and data scientists wasting a lot of time on non Machine Learning work.
2. Adding informative training data to a model gets harder in each iteration:
Developing a model that’s 70% accurate is democratised now. Any decent deep learning engineer should be able to deliver that to you with decent amount of training data using transfer learning. However, the problem increases as you go to the next round of training. Now, if you collect more training data, only 30% data would add value to the model and this would keep increasing the cost of further learning in each subsequent round of training. So, without using something like active learning it’s extremely difficult to deliver production grade models within budget.
3. Data Scientists waste a lot of time in identification of right training data to use despite having a data annotation team:
So, a more practical strategy(most of the teams already do this) is to only collect further training data for the failure scenarios. This normally means the data-scientist would run the examples on various test sets and share the output with the annotation manager or the annotation team who would then go through the data and find the scenarios. This process is mostly done on email iteratively with sharing links of Google Drive and Dropbox folders. This also means that the time of the Data Scientists is wasted on running the model on different datasets and sharing output that can be consumed by the annotation team. Data Scientists don’t want to do this grind but they have to as they want to move the models to production with limited budget.
The Solution: NeuralMarker.ai
1. Active Learning for all subsequent Training iterations is a must do:
In order to make the training of the subsequent rounds more effective, Data Scientists use active learning. The idea of active learning is to only label examples which are informative to our trained model. This involves running the trained model from the first iteration on a large unlabeled dataset and then using some metric for informativeness to only select the data which is conductive to learning for the trained model i.e. contribute further to improving the current model. The examples that the current model already understands well won’t contribute to any further learning for the current model and only increase the time and cost of the project.
So, in short: Don’t spend time and energy in labelling what the model has already learnt.
2. AI Pre-Labeling and AI Assisted Labeling result in huge speedup:
AI Pre-labeling refers to using a pre-trained model to generate initial set of annotations. The annotators can:
a). Accept the annotations as it is if they are correct
b). Edit them if there is some small error in the annotation
c). Completely delete the annotation and make a new annotation.
If the model that’s used for pre-labeling has 75% accuracy, a lot of annotation effort is reduced. AI Assisted labeling also helps in generating complex annotations with small human effort. For example for image segmentation, just by marking 4 points on the image, AI can generate a pretty good segmentation mask which can be fixed by the human annotator.
3. Our AI Model Inference sits inside the Annotation platform:
NeuralMarker platform has many public popular models like YOLO, Centernet, DeepLab etc. integrated within the platform. This means you can run inference on these models with a click of a button. So, if you are training any of the available models, you can just drag and drop your weights and classes file and your inference engine is sitting inside your annotation tool. After this, one can run inference with the click of a button. Anyone in the team can run test the model on any dataset without any help from the Data scientist. These in-built AI models mean team can use these models for pre-labeling or active learning to find the relevant examples for labeling further.
Most of the annotation platforms leave this heavy lifting to be done by the AI engineer and this is never achieved resulting in the dependency on the manual work by the AI engineers to even generate outputs.
4. Empower the Annotation Data Manager:
Most of the AI organisations now have a role that works with the team of annotators to explain them the task of labeling, answer any queries or just being a single point of contact between Data Scientist and the annotation team. This role is often called Annotation Data Manager.
Once your annotation manager has the capacity to run the model on a dataset and see the results, you have converted him/her into an AI Strategist. He would be able see the blind spots of the current model and find more examples where the model makes mistakes. This frees the AI Engineer to do things that they should be doing, i.e. finding ways to improve accuracy, using model averaging, hyper parameter search or just reading latest research.
These things have helped us deliver models quicker, better quality and reliably for many production use-cases while staying within budgets. If these features excite you as well, please feel free to signup for a quick demo or just head over to app.neuralmarker.ai to try things on your own.
Refs: 1. https://www.mdpi.com/2220-9964/7/2/65/htm