Wed, 27 Sep 2006

Fading Memories

Investigating Image Libraries

There's a tremendous effort being expended in the free software world to create great raster image applications. And that means that there are also a lot of free or open source libraries that could be used as the basis for an image editor application. Still, most applications use their own core. Krita has pigment for color management and kritaimage for pixel manipulation.

Similarly, digikam has its own image class, Gimp has its own core, although that has been slated to be replaced by Gegl for some time, and so on.

There are only three common ways to store image data: tiles, scanlines and big blogs. Tiles are used by Krita, Gimp, Java's JAI library and Photoshop, among others. Krita's tiles are 64x64 pixels. Scanlines are used by the new Cinepaint and Mosfet's MosfetPaint. Blobs are used by digikam (as far as I know) and, of course, Qt itself: QImage is a small image library in itself, together with QPainter. Likewise, there are three ways of getting at the pixel data: pointer to tiles, scanlines or blob, pointer to a copy of the image data, or, when using a modern programming language, iterators.

From time to time, people come up with the suggestion to use an existing imaging library with Krita, to replace our own core. That used to be a pressing question, but by now, Krita has grown to about 100.000 lines of code and our image handling core, while perhaps not the most optimized, is very nice and easy to use. The library closest to Krita is perhaps Vigra, because both Krita and Vigra don't expose tiles or scanlines, but iterators. In the use of templates, Adobe's Generic Image Libary is close.

What follows is a non-exhaustive list of image libraries (or apps, if they have their own library) with my assessment of them. This list was originally prepared in 2003, but I updated it for the occasion.

Krita

Krita contains its own 2D image library, consisting of the tile manager, the layer classes and KisPainter. Starting with KOffice 2.0, the colorspace library has been separated into pigment, and the image library has been separated completely from the user interface. It would be easy to separate both libraries from KOffice itself, although that's not a priority.

Advantages:
- Already works
- Optimized for interactive work
- Allows different color models

Disadvantages:
- Not optimized (yet)
- Large images are a large problem
- No indexed images

Gimp (http://www.gimp.org)

The Gimp contains a complex core that allows interactive painting of images with a channel depth of 8 bits.

Advantages:
- Well tested, very complete
- Optimized for interactive usage

Disadvantages:
- Written in C.
- Not readily available as a library.
- Depends on glib and gtk
- 8-bit only
- No color models -- hard-coded gray8, rgb8 and sometimes indexed
- Has problems handling really large images

Vigra (http://kogs-www.informatik.uni-hamburg.de/~koethe/vigra/)

Vigra is a C++ imaging library written by Ullrich Koetthe to demonstrate his Phd thesis on flexible image algorithms. The thesis is a cool thing to read and influenced Krita noticeably, although we are probably wagging the dog, instead of tail, template-wise.

Advantages:
- Supports very large images
- Supports color models
- Supports different bit depths through templating
- modern C++
- Relatively small
- well-documented
- interesting algorithms

Disadvantages:
- Not optimized for interactive use
- Really meant for computer vision research

Vips (http://www.vips.ecs.soton.ac.uk/)

Vips is a C library with a C++ layer. It has been designed for handling very large images, mainly in the context of research into paintings in museums like the National Gallery. It comes with a gtk2 gui, nip2, which is a kind of spreadsheet for images.

Advantages:
- Handles very large images
- Handles color models
- Handles different bit depths
- C++ interface

Disadvantages:
- Not optimized for paintbox type apps (even though it is possible).
- Very large.

VXL (http://vxl.sourceforge.net/

VXL is a collection of small libraries that are used for computation vision and imaging purposes in an academic setting.

Advantages:
- Handles very large images
- C++

Disadvantages: - Not recently updated
- Comes with its own core libraries that replace standard C++
- Optimized for simple rgb images.
- No license at all
- Badly documented

CImg (http://cimg.sourceforge.net/)

CImg is a very cool, very small library that offers some extremely innovative image effects, like inpainting of damaged or noise images. We use it in our greycstoration plugin in Krita.

Advantages:
- Small
- GPL
- Cool stuff

Disadvantages: - Everything, including GUI stuff, in one header file.
- badly documented.

Gegl (http://www.gegl.org/, http://pippin.gimp.org/gegl/)

Gegl was intended to become the Gimp 2.0 core, but development had stalled so much that the move to Gegl didn't happen before Gimp 2.0. It is, more or less, an attempt to write a templated library in C with the help of a custom language to describe image operations. Recently, Øyvind Kolås (alias pippin) has picked up development. Progress is interesting, although I'm not convinced that it will be easy or even possible to just port the Gimp to Gegl.

Advantages:
- Small
- Together with babl, support for all kinds of colorspaces
- Optimized for interactive use
- Pippin is cool

Disadvantages:
- Not finished yet
- C
- Complex hack around the fact that C is a low-level language

libart_lgpl (http://www.levien.com/libart/)

Libart isn't really an image library, but rather a canvas that can be used to paint images on. It is optimized for vector graphics, and is used by Karbon 1.x to render shapes before display. (Karbon 2.x will use Arthur.

Advantages:
- Raph Levien is really good at this stuff, so libart is quality

Disadvantages: - C
- It isn't an image library, really
- Hard to use
- Obsoleted by Cairo or Arthur

java2D (http://java.sun.com/j2se/1.4.2/docs/guide/2d/index.html)

Java2D is more or less complete library to write a paint app around. It offers image types, color spaces, kernel convolutions and text. It's in Java, of course, and the free re-implementation is not done yet, and besides, is based around Cairo. Once I had the source for an open source Java image editor based on Java2D, but I lost the source and have forgotten the name.

Advantages:
- Neat OO design
- (Over-) complete

Disadvantages:
- Java
- Not free
- Has some legacy cruft.

ImageMagick/GraphicsMagick (http://imagemagick.org/, http://www.graphicsmagick.org/)

GraphicsMagick is ImageMagick with a different license, and a focus on API stability. GM and IM now also differ slightly in terms of features offered. Krita used to be based around IM (which circa 2004 could still be seen in many places in the code). The IM core was dropped in favor of the present core because it turned out that IM was not re-entrant, making it hard to use in an interactive application. Recently, we've made most of the import/export filters IM-independent, too, and have switched the remaining code to GraphicsMagick, for the API stability.

Advantages:
- Mature
- C++ interface
- Full-featured
- RGB and CMYK (but not more)
- License compatible with Krita
- Under active development

Disadvantages:
- Bit-depth a compile-time option
- Not re-entrant: not optimized for interactive use.

The Visualization Toolkit (VTK, http://public.kitware.com/VTK/)

From the makers of CMAKE comes VTK, a very big C++ library for 2d and 3d image processing and visualisation. It's too big to easily evaluate for me...

Advantages
- It is used in other Qt applications, like Julius
- Probably very good

Disadvantages
- The book is very expensive
- Uses its own make equivalent, CMake -- hey! that's an advantage nowadays :-)
- Very large

GIL

Similarly named to Gegl, GIL is a collection of headers that depend on Boost and provide common image manipulation algorithms. Neat is that GIL can handle both banded and planar images, that is, images where each channel has its own block of memory and images where each pixel is consecutive in memory. Gil is used internally in Adobe, and in Mission Photo>.

Advantages
- Adobe knows what it is doing
- Very modern C++

Disadvantages
- Quite difficult to grasp
- Not quite a complete image library - as someone said, " it's library of generic accessors and converters for a variety of image types that, in effect, allows you to treat them the same regardless of whether the image has a pixel-interleaved representation"

Others

Image libraries and roll-their own applications -- Perico, Digikam, Cinepaint (and Glasgow), PIL, Java Advanced Imaging API -- there are many more ... I have just mentioned the ones that I investigated myself when starting on Krita and looking for a library to do the heavy lifting. I think rolling our own was the right choice: we learned a lot from it and we have some unique capabilities of our own. The decision to use iterators has cost us some performance, but has made development of actual features a lot easier. Right now, we're replacing the straight inheritance design of the colorspace library with templates, making it even easier to code up new colorspaces. No longer will we need two whole days to code support for L*a*b!

/software | permanent link | 18 comments |



Re: Investigating Image Libraries

Lee wrote on Wed, 27 Sep 2006 11:27

I like the ideas and features (so far!) in Krita, but it is *terribly* slow to interact with. OpenGL doesn't seem to help much (which seems to confirm that the bottleneck is code rather than GPU performance). However, I find the idea of using OpenGL worrying for users of older machines or just FOSS drivers anyway.

Have the actual bottlenecks been profiled? What kind of speedup factor is expected once KritaImage/Pigment are optimised, I wonder? Is there a refactoring required, compiler optimisation required? Or is it really just a matter of optimising things once they're finalised?

Reply



Re: Investigating Image Libraries

shamaz wrote on Wed, 27 Sep 2006 11:58

Hello !

You stated that the free implementation of java2D is not ready yet. That's wrong :) You'll find it in the latest gnu classpath version : https://savannah.gnu.org/forum/forum.php?forum_id=4573

Anyway, sun is about to open source java.

Reply



Update

Boudewijn wrote on Wed, 27 Sep 2006 17:21

Update: Gegl doesn't expose tiles or scanlines either, VXL has had plenty of updates this year and the documentation is better, too. And I forgot about Intel's OpenVision.

Reply



Re: Investigating Image Libraries

Pete wrote on Thu, 28 Sep 2006 00:10

Your rating of C and C++ is unfortunate. I'd prefer if you rated by the implementation quality of the library instead of the language itself.

I clean C library is a joy to use. A messy C++ library is impossible. Fortunately it is easy to read your article and manually grepping out the pro/con lines with language reference.

Reply



Re: Investigating Image Libraries

steve wrote on Thu, 28 Sep 2006 18:44

GIL really is an amazing library in how generic and efficient it is. It is an open source work in progress and welcomes contributions to make it more complete.

For a good video tutorial see: http://opensource.adobe.com/gil/presentation/index.htm

For a comparison with other libraries like the one you wrote see: http://opensource.adobe.com/gil/html/gildesignguide.html

Reply



Re: Investigating Image Libraries

Bernt wrote on Thu, 28 Sep 2006 22:36

Have you looked at the Anti-Grain Geometry Project?

www.antigrain.com

I am not a programmer, but it seems to be very intersting!

Reply



Another update

Boudewijn wrote on Fri, 29 Sep 2006 09:30

Mosfet writes to tell me his mosfetpaint core did indeed use tiles, but that the pixel data was exposed through pseudo-scanlines. The scanlines were all I remembered from our discussion about it some time ago.

Reply



Re: Investigating Image Libraries

John Cupitt wrote on Fri, 29 Sep 2006 12:17

Interesting overview, thanks for mentioning VIPS (my project).

I noticed you don't mention SMP on your pros/cons for these libraries. Image processing (often) threads well and with SMP machines becoming very common I think this is a factor to consider.

Another factor I look for is chaining: if you use several operations in a sequence on an image, do they run together or separately? Chaining is an important feature because it lets you construct new operations from existing operations without worrying about memory behaviour.

Finally, VIPS isn't that large. The library is just over a megabyte on my machine. Sorry for the long comment.

Reply


Your Comment


Name:
URL/Email: [http://... or mailto:you@wherever] (optional)
Title: (optional)
Comment:
Save my Name and URL/Email for next time
Captcha: To prevent comment spam, please retype the characters in this image:

Enter the text here: