Skip to content →

Cloud based image processing for digitized media monitoring documents

Media monitoring solutions for printed media (newspapers, magazines, etc), audiovisual media (tv, radio, internet streams, etc) and social media vary in features and implementation choices. The pipeline of printed media monitoring includes the digitization of printed material and the selective segmentation of that material’s content, a process called clipping. As it’s name indicates, clipping is the act of selecting part of the material using rectangles and polygons and generating content out of that clipped area.

Background

The process of clipping, is performed by most media monitoring companies using desktop software solutions that are either of general image editing purposes, or specialized with features tailored to the media monitoring field. The software solution has to be installed to the desktops of the employees that handle the material clipping process. A couple of problems occur when this approach is used that hurt productivity, maintenance costs and resources allocation. In detail:

  • Having an image processing software installed per employee desktop workstation requires capable hardware of handling the image processing, meaning the allocation of extra resources that force the cost of the workstations to go up.
  • Updating the software requires either manual labor, or the use of tools that automate the update and maintenance procedure (eg chef and puppet) in both cases, meaning extra maintenance costs.
  • The usage of elastic workload services (such as AWS) is prohibited with this architecture since the processing happens on the client (aka the employee workstations). Every client must have enough resources to perform the image processing regardless if it needs them 24/7.
  • To perform remote work (eg work from home), employees have to install the same software on their home computers, let aside make sure that those home computers are powerful enough to run the image processing computations in reasonable time.

Alternatives

Alternative solutions to the problems posed above, focus on either the solution or mitigation of a subset of the problems as the problem lies more on the architectural design of the clipping pipeline and not the implementation.

Automatic updates and maintainance mitigates the extra cost of maintenance (still requires extra devops work-hours).

Usage of remote desktop software can solve the problem of “work-from-home” but still hurt productivity as the interface of the application loses its’s smoothness.

Elasticity cannot be achieved but the usage of a good allocation of work to employee scheme can minimize the hardware neeeded, still not achieving true elasticity.

Proposed Solution

I had to work on a project where the customer faced the problems mentioned above. The most obvious solution to the problems, would be to move the image processing to the cloud and develop a web client for employees to perform the work remotelly.

The solution proposed and implemented had the following key aspects:

  • Microservice oriented architecture
  • Image processing done on the cloud
  • Employee actions forwarded to the cloud through a client
  • Preview and results pushed forward from the cloud to the clients

Technologies utilized

The application was split into frontend (client) that would run in the browser and backend (image processing microservices) that would run on the cloud.  The frontend utilized Google’s Web Toolkit and HTML5 Canvas technology and communicated with the backend through HTTP APIs. The backend was built using Spring Boot.  The frontend was responsible of handling the user input and actions and forwarding them to the backend. It was also in charge of drawing image data and metadata from the backend on the HTML5 Canvas.

Frontend choices

Picking HTML5 Canvas was a one way for drawing shapes and image data on the frontend. The cross-browser support, it’s performance and the smooth integration with Javascript and Google Web Toolkit were the main reasons.

Picking Google’s Web Toolkit had to do with rapid development using an already established and rich framework but it mainly had to do with the fact that GWT is able to compile Java code into Javascript. The frontend had to be able to perform complex mathematical operations to draw/scale/rotate image data and polygons on the canvas. Instead of writting the mathematical functions, mathematical libraries from Java’s rich ecosystem was used and was fused into the code with GWT’s compilation ability. Those libraries were not available in javascript meaning one would have to develop them from scratch causing the cost of the project to increase. While this sounds optimal, one has to mention that compilation was not sucessful by default as GWT did not support advanced java features such as reflection, meaning the code had to be cleansed of them before compilation was attempted.

Backend choices

Spring boot was chosen as the framework to develop the image processing microservices. It’s easy integration with cloud services, the built-in support for microservice architecture and it’s simplicity played a key role at selecting it. For image processing, the backend would forward incoming calls to imagemagik which handled all of the image processing tasks of the project including scaling/rotation/color adjustment and cropping of image data.

Published in Software Development

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *