jkt's blog

Blößmrt

Entries tagged "rant".

QMake Static Libraries, Unit Tests and Much Headache, or the Tale of How Trojita Changed the Build System from CMake to QMake
2010-05-18

I've spent more than 10 hours in total changing the build system of Trojitá from CMake to QMake upon a request from KWest. The process was very painful for me, so I think it's worthwhile to include some comments about what were the major obstacles.

The first step, "getting the beast to build", was actually pretty easy -- I (KWest actually) simply created a list of all files in the tree, added them into a single .pro file, removed the #includes for MOC, commented out the unit tests and rebuilt the project. That was the easy part; qmake simply built bunch of .o files and linked them together.

Problem with unit tests is that they all define the main() function. Therefore, as far as I know, one has to create a separate .pro file for each unit test, use the template = subdirs and put each test into a subdirectory. That's slightly annoying when compared to CTest (see the Trojita's git repository for how the CMakeLists.txt looked before we switched), but doable and actually pretty straightforward. Now, a much bigger problem is persuading QMake to create static libraries and using them properly and in a cross-platform way. I care about the Unix platform, some users want to play with Trojita on Windows, and there's that secret device KWest is building.

Trojita currently consists of several parts; we have the core IMAP stuff (which itself consists of three or four components), the GUI layer, some third-party modules even (like Witold Wysota's qwwsmtpclient, the Qt's iconloader, a unifying layer for QProcess and QSslSocket etc). The unit tests for the IMAP protocol clearly should not care about icons at all and shall ignore GUI classes, too, but they do need to link against the IMAP protocol implementation. A reasonable way to express that in the build system is to create a static library for the IMAP stuff and link it to the rest of the GUI when building the application and to each of the unit tests when building tests. That's where problems with QMake start to hurt.

Unlike CMake, under which the static libraries are extremely easy to write, QMake's support and documentation for static libraries leaves much to be desired. A reasonable request is, for example, expecting the build system to isolate me from stuff like library placement -- I just want to tell it that I need to link against bunch of .arelease and debug when caring about Windows builds.

Another expectation is that QMake should relink each binary if any static library it depends on gets rebuilt. Too bad, using QMake, you have to include the library name in three places for this to happen: at first, you have to use LIBS += -Lpath/to/your/lib/directory, then you got to actually link to it by LIBS += -lnameOfTheLibrary, and finally you have to take care of the rebuilding by PRE_TARGETDEPS += full/path/and/the/libIdentifier.a. Oh, and please do not forget about CONFIG += ordered, or the PRE_TARGETDEPS won't affect much stuff anyway. Yuck!

There's the CONFIG += create_prl and CONFIG += link_prl, but they did not help me. I guess they are used for specifying dynamic libraries on which the currently processed static library depends. They certainly did not fix my problems when I played with them, though.

Anyway, this is what I ended up with:


Project file for the GUI stuff, src/Gui/Gui.pro:
(...)

trojita_libs = Imap/Model Imap/Parser Imap/Network MSA Streams iconloader qwwsmtpclient

myprefix = ../
include(../linking.pri)

Each of tests/test_*.pro:
TARGET = test_Imap_LowLevelParser
include(../tests.pri)

A helper file for unit tests, tests/tests.pri:
QT += core network
CONFIG += qtestlib no_lflags_merge
DEFINES -= QT3_SUPPORT
DEPENDPATH += ../../src/
INCLUDEPATH += ../../src/ ../
TEMPLATE = app
HEADERS += ../qtest_kde.h


trojita_libs = Imap/Parser Imap/Model Imap/Parser Streams
myprefix = ../../src/
include(../src/linking.pri)

SOURCES += $$join(TARGET,,,.cpp)
HEADERS += $$join(TARGET,,,.h)

And finally, the file, src/linking.pri:
for(what, trojita_libs) {
    mylib = $$replace(what,/,)
    unix {
        mypath = $$join(what,,$${myprefix},)
    }
    win32 {
        CONFIG( debug, debug|release ) {
            mypath = $$join(what,,$${myprefix},/debug)
        } else {
            mypath = $$join(what,,$${myprefix},/release)
        }
    }
    LIBS += $$join(mypath,,-L,)
    LIBS += $$join(mylib,,-l,)
    PRE_TARGETDEPS += $$join(mypath,,,$$join(mylib,,/lib,.a))
}

Compare the above to the elegance of CMake, depicted below out of sentiment:
set(libImap_SRCS
    ${CMAKE_CURRENT_SOURCE_DIR}/Imap/Parser/Parser.cpp
    ${CMAKE_CURRENT_SOURCE_DIR}/Imap/Parser/Command.cpp
...
)
...
add_library(Imap ${libImap_SRCS})
...
target_link_libraries(Imap Streams ${QT_QTGUI_LIBRARY} ${QT_QTCORE_LIBRARY})
...
target_link_libraries(trojita Imap MessageView ModelTest MSA QwwSmtpClient QtIconLoader)

That's what I call ease of use. Anyway, the QMake change just had to be done (when the customer asked if I could migrate to QMake, I said I had no preference) and it's been done now. This blog post is just a rant, and hopefully might eventually make its way to Google's results for "qmake static library".

There's always the possibility that I'm just too dumb to miss a completely obvious way to work with static libraries in QMake. If that's indeed the case, I sincerely apologise to QMake designers. Also, I offer one beer as a sign of appreciation to the first person who shows me that I'm indeed missing something from the big picture. In the meanwhile, have fun -- QMake has a lot of nice features and I'm no longer afraid to use it, now that the static libraries are used properly. The make test is still missing, but I guess I can live without that for a while.

Tags: gentoo, kde, qt, rant, trojita.
On Innovation, NIH, Trojita and KDE PIM
2013-05-07

Jos wrote a blog post yesterday commenting on the complexity of the PIM problem. He raises an interesting concern about whether we would be all better if there was no Trojitá and I just improved KMail instead. As usual, the matter is more complicated than it might seem on a first sight.

Executive Summary: I tried working with KDEPIM. The KDEPIM IMAP stack required a total rewrite in order to be useful. At the time I started, Akonadi did not exist. The rewrite has been done, and Trojitá is the result. It is up to the Akonadi developers to use Trojitá's IMAP implementation if they are interested; it is modular enough.

People might wonder why Trojitá exists at all. I started working on it because I wasn't happy with how the mail clients performed back in 2006. The supported features were severely limited, the speed was horrible. After studying the IMAP protocol, it became obvious that the reason for this slowness is the rather stupid way in which the contemporary clients treated the remote mail store. Yes, it's really a very dumb idea to load tens of thousands of messages when opening a mailbox for the first time. Nope, it does not make sense to block the GUI until you fetch that 15MB mail over a slow and capped cell phone connection. Yes, you can do better with IMAP, and the possibility has been there for years. The problem is that the clients were not using the IMAP protocol in an efficient manner.

It is not easy to retrofit a decent IMAP support into an existing client. There could be numerous code paths which just assume that everything happens synchronously and block the GUI when the data are stuck on the wire for some reason. Doing this properly, fetching just the required data and doing all that in an asynchronous manner is not easy -- but it's doable nonetheless. It requires huge changes to the overall architecture of the legacy applications, however.

Give Trojitá a try now and see how fast it is. I'm serious here -- Trojitá opens a mailbox with tens of thousands of messages in a fraction of second. Try to open a big e-mail with vacation pictures from your relatives over a slow link -- you will see the important textual part pop up immediately with the images being loaded in the background, not disturbing your work. Now try to do the same in your favorite e-mail client -- if it's as fast as Trojitá, congratulations. If not, perhaps you should switch.

Right now, the IMAP support in Trojitá is way more advanced than what is shipped in Geary or KDE PIM -- and it is this solid foundation which leads to Trojitá's performance. What needs work now is polishing the GUI and making it play well with the rest of a users' system. I don't care whether this polishing means improving Trojitá's GUI iteratively or whether its IMAP support gets used as a library in, say, KMail -- both would be very succesfull outcomes. It would be terrific to somehow combine the nice, polished UI of the more established e-mail clients with the IMAP engine from Trojitá. There is a GSoC proposal for integrating Trojitá into KDE's Kontact -- but for it to succeed, people from other projects must get involved as well. I have put seven years of my time into making the IMAP support rock; I would not be able to achieve the same if I was improving KMail instead. I don't need a fast KMail, I need a great e-mail client. Trojitá works well enough for me.

Oh, and there's also a currently running fundraiser for better address book integration in Trojitá. We are not asking for $ 100k, we are asking for $ 199. Let's see how many people are willing to put the money where their mouth is and actually do something to help the PIM on a free desktop. Patches and donations are both equally welcome. Actually, not really -- great patches are much more appreciated. Because Jos is right -- it takes a lot of work to produce great software, and things get better when there are more poeple working towards their common goal together.

Update: it looks like my choice of kickstarter platform was rather poor, catincan apparently doesn't accept PayPal :(. There's the possiblity of direct donations over SourceForge/PayPal -- please keep in mind that these will be charged even if less donors pledge to the idea.

Tags: gentoo, kde, qt, rant, trojita.