Introducing Video Compatibility to Support Broader Use Cases in Computer Vision

Why Video Data in Computer Vision?

Videos are just as prevalent within visual data in numerous industries. Even more so, video data has a significant impact on the different steps of a computer vision pipeline. At the data collection level, videos are a common way of collecting visual data, such as streaming data from a live camera. Additionally, videos are a simple way to provide large amounts of individual images as each of the frames in the video can be taken as an image. However, at the data annotation stage, video annotation can become even more tedious than typical tasks, given that the number of frames per second can mean a large multiplier in terms of the quantity of images despite the video being only a certain number of images. On the deployment end of the pipeline, common applications of computer vision models are performing live predictions on streaming video. Therefore, accommodating video formats provides access to a large portion of applications that are reliant on video.

What Are Important Factors for Video Compatibility on an MLOps Platform?

There are a few key considerations that are needed in order for a user to feel at ease with managing and manipulating video data throughout each stage of the MLOps pipeline. Below are points at each relevant stage of the pipeline that users should note to track in order to have an expected and fulfilling experience.

Video Upload

The assurance of data integrity is important for any platform in which users provide their own data. One common pitfall that is used for ease of development is the splitting of videos into images immediately on upload, which removes the user’s ability to view the video frame images in their original form of video. It additionally becomes difficult to track the data originating from the video. Therefore, users should look for platforms that allow for videos to be uploaded and accessed in their original format.

Video Annotation

Data annotation for images is already quite a tedious process. Video annotation increases the annotation task size multiplicatively. Therefore, video annotation requires just as much, if not more support, to make the onerous task of annotating smoother and easy to navigate. Users should look out for annotators with smooth video rendering, useful annotation tools that ease the tedium, and video navigation and visibility tools such as sections which visually display where and what class of annotations are being made in the video.

Model Training

The addition of a new data type like videos should not have an effect on the training portion of the computer vision pipeline. Given that computer vision models handle images independently, videos should be processed by the platform to be fed in as inputs per frame like regular images, thus preserving the same workflow that a user is used to.

How Does Nexus Support Video Assets?

Nexus now supports the video asset ingestion and annotation with our extensive suite of annotation tools. Importantly, the processes that users on our platform are used to remain the same for the new asset format.

How to Upload Videos on Datature Nexus?

Video upload is the same process as image upload on Nexus, where one simply goes to the Assets page and either navigates to the folder containing the assets in the pop-up or drags and drops the file into the designated space at the top of the page.

Additionally, we support external data storage syncing with Nexus through our S3 Bucket Connectivity feature. This is particularly relevant for videos, where video file size can be prohibitively large, and external data storage is preferred. S3 Bucket Connectivity is enabled through a process made simple on the Nexus platform. To better understand how this can facilitate your use case, you can learn more here.

For detailed information on how videos are handled and processed on our platform, please take a look at our documentation.

How to Annotate Videos on Datature Nexus?

Video annotation will be the same at the frame level with the same extensive suite of annotation tools we provide for images. Tools that could be particularly useful for reducing the time needed IntelliBrush for manually creating annotations faster, AI Edge Refinement for automatically improving mask quality, and Model Assisted Labelling to leverage the knowledge of previously trained models to automatically label your frames.

The key additions to ease the user experience primarily deal with navigating frames in the video, and our additional utilities that help users track where they’ve been annotating.

We provide a video player that smoothly renders your video but also allows you to annotate whenever you would like. At the bottom right corner, the settings menu controls video settings to help with video navigation. At the bottom of the annotator, we provide a sliding bar to facilitate video scrubbing, with bars below to indicate where annotations of a certain class have been made.

‍

‍

After you’ve annotated the assets in your project, you can start making your workflow and training a model in a few clicks, in the very same process as what you would do with a project with only images! We want to preserve the same no-code, smooth experience for workflow generation and model training that we’ve done with our image pipeline, and have maintained that despite the additional complexity of video input.

Next Steps

Once your video assets have been successfully onboarded and annotated for your Nexus project, you can analyze the annotations and your data with our Aggregation Statistics that we provide on the project home page to determine whether your dataset is prepared for a successful model training.

Given that videos provide such a large concentration of data in certain settings, it is important to maintain data variety to improve model robustness and increase performance. Our aggregation statistics can provide deeper insights to help you further understand your dataset and improve your overall data quality and adjust your pipeline to optimize your models’ efficacy.

Our Developer’s Roadmap

Including videos as a compatible format on Nexus is just the beginning of our commitment to expanding and improving our tools. For video annotations, we are well aware that the annotation process can be a bottleneck in the process of setting up an MLOps pipeline. In the future, we will be providing tools to assist in speeding up the process to further improve the user annotation experience, such as advanced polygon interpolation and AI assisted object labelling across multiple frames for annotation.

Want to Get Started?

If you have questions, feel free to join our Community Slack to post your questions or contact us about how video onboarding fits in with your usage.

For more detailed information about the video onboarding customization options, or answers to any common questions you might have, read more on our Developer Portal.