\author{\htmladdnormallink{Reinhold Kainhofer}{mailto:reinhold@kainhofer.com}, with help of
\htmladdnormallink{A. de Groot}{mailto:groot@kde.org} and \htmladdnormallink{Dan Pilone}{mailto:pilone@slac.com}}
\date{\today}
%\W \newcommand{\texttrademark}{}
\newcommand{\code}[1]{{\small\texttt{#1}}}
\newcommand{\file}[1]{{\small\texttt{#1}}}
\newcommand{\class}[1]{{\small\em\texttt{#1}}}
\newcommand{\codesize}[0]{\scriptsize}
\begin{document}
\maketitle
%\htmlmenu{3}
\tableofcontents
\section{Introduction and General}
\subsection{Introduction}
One of the greatest assets of the handheld computers is their ability to
interconnect with other applications. KPilot supports this capability
through conduits. A conduit is a small separate program that talks to
KPilot during the hot sync. The conduit translates between the Palm
Pilot and the application you're syncing with.
For KPilot to be a really usable synchronization application, it depends
on third-party developers to enlarge the conduit gallery to other applications. In particular, I want to encourage developers of Palm applications to provide not only a Windows\texttrademark conduit, but at the same time put some effort in providing also a conduit for Linux and other platforms.
This is also the reason why I'm writing this tutorial:
To encourage third-party developers to write conduits for KPilot.
I will show the process at several examples: \\
First, the general framework (library, factory etc.) for a conduit is presented
in section \ref{FactorySection}, then in section \ref{SetupSection}.
I will describe how you can write a configuration dialog for your conduit. These
two sections will use the malconduit (AvantGo conduit) as an example.
The synchronization part (the conduit class) will be described in the next few
sections. Section \ref{SectionSimpleConduit} will show a very simple
synchronization at the example of the AvantGo conduit, where we will use the
functions of an external library which will do the synchronization for us.
In section \ref{SectionDatabasesConduit} I will show a synchronization process
where one file on disk corresponds to one database on the palm, and where no conflict
resolution and no record comparison needs to be done, because we have to copy the
whole database either from or to the handheld. The particular example there will be the docconduit which
synchronizes text files on the harddisk with PalmDOC documents for AportisDoc, TealDoc, MobiPocket Reader, Gutenpalm etc.
on the Palm.
Finally, I wanted to show an example of a record-based conduit, but then decided it
would be too extensive to replicate all the complex sync and conflict resolution code.
Instead I refer to the addressbook conduit, which you should be able to understand
quite well after studying the other conduits explained in the previous chapters of this How-To.
Using KDE's KitchenSync general synchronization framework for syncing will be the topic of section
\ref{SectionKitchenSync}, where I give some argument why we do not yet switch to kitchensync yet.
\subsection{Further information and links}
There are loads of information about the PalmOS\texttrademark Database format
out there:
\begin{itemize}
\item Palm's official Developer Knowledge Base document: \htmladdnormallink{http://oasis.palm.com/dev/kb/faq/FileFormat/PDB+PRCFormat.cfm}{http://oasis.palm.com/dev/kb/faq/FileFormat/PDB+PRCFormat.cfm}
\item EBNF representation of the database format: \htmladdnormallink{http://obermuhlner.com/public/Projects/Palm/PDBC/Documentation/pdbc\_file.html}{http://obermuhlner.com/public/Projects/Palm/PDBC/Documentation/pdbc\_file.html}
\item A list of several Palm databse formats: \htmladdnormallink{http://myfileformats.com/search.php?name=Palm}{http://myfileformats.com/search.php?name=Palm}
\end{itemize}
\subsection{Legal}
You can skip this chapter if you are familiar with HOWTOs,
or just hate to read all this assembly-unrelated crap.
\subsubsection{Legal Blurb}
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU \htmladdnormallink{Free Documentation
License}{http://www.gnu.org/copyleft/fdl.html} Version 1.1;
with no Invariant Sections, with no Front-Cover Texts, and no Back-Cover texts.
A copy of the license is included in the \htmladdnormallink{Free Documentation
the user change them in the dialog. If the user exists the dialog by pressing OK, the new values are
read from the dialog controls and written back to the configuration file.
To provide a consistent experience to KPilot users, there already
exists a class \class{ConduitConfig} which is a subclass of \class{KDialog}.
This dialog does most of the basic work for you.
\subsection{The dialog template, using QT Designer}
Of course, first we need to have a dialog template in the form of a QT
Designer file (which has an extension .ui). Start up \file{designer} and
create a new widget (no dialogbox, i.e. no OK or cancel buttons, these will be added automatically). The dialogbox should contain a QTabWidget, even if you only need one tab. A second tab "About" will be added more or less automatically by the conduit listing the copyright and the authors of your conduit. A typical example of the coknduit setup widget dialog is shown in the following screenshot:
\includegraphics[width=14cm]{pictures/ProxyTab}
\subsection{Using the dialog template for your conduit}
Now that we have the dialog template, we can include it in the configuration dialog box.
First, create a subclass of \class{ConduitConfig}, where we only have to implement the con- and
destructor as well as the methods
\begin{itemize}
\item\code{virtual void readSettings()}
\item\code{virtual void commitChanges()}
\end{itemize}
This class \class{ConduitConfig} will do most of the work for us, all
that is left to us is to read the config settings from the config file
and set the values of the controls in the dialog (in the
\code{readSettings()} method). Also, when \code{commitChanges()} is
called, this means that the dialog was existed with the OK button, so
we need to read the new values from the controls and store them to the
In this example, we don't store to the configuration file if a custom proxy port should be used. Instead, we just store a port number, and if the port number is 0 this means to use the default port. In this case, the custom port CheckBox needs to stay unchecked, and the port NumEntry control will stay disabled as it is in the dialog template. In all other cases, however, the custom port CheckBox should be checked, and the port NumEntry control will be enabled and filled with the correct custom port.
The KPilot user can then change all the settings in the dialogbox without any intervention from KPilot, so we don't need to write any code for that. Only the \code{commitChanges()} method remains to be done, which does the opposite of the readSettings() method. It reads the values of the controls and stores them to the configuration file. The \class{TDEConfig} class (the \code{fConfig} variable, resp.) has only one method \code{TDEConfig::writeEntry("entryname", valueOfWhateverType)} to write a value to the configuration file. However, this method has several overloaded implementations so that you can write numeric, string, boolean, date and many more variable types with the same syntax. First, we need to set the correct configuration group again, and then we just read each of the settings and write it out immediately using the \code{writeEntry} method:
This was all that is needed to implement the configuration for your conduit. Simple, isn't it?
\section{How the conduits work}
First a little background on pilot databases would be useful.
Each database on the pilot is divided into multiple records. Each
record is a specific entry into the database. For example, one
address in the Address Book corresponds to one record in AddressDB.
Each record has a unique id assigned by the pilot when the record is
created, and an index number in the database. The index numbers get
reused, for example if you delete the second record from the database
then create a new one it will fill in the empty slot and become index
number two. However, it will have a unique ID number. (The moral of
this story is only use index numbers when you need to iterate through
the whole Database, otherwise use the unique record id.)
When a new record is created by the user on the local side, it
is inserted into the backed up copy of the databases but given a
unique ID of 0. When the syncing occurs this record is installed on
the pilot and the pilot will return with the new unique id for it.
This same process happens with a conduit. If two records have the
same unique id they will overwrite each other.
Lastly, the conduit is requested to
hot-sync whenever KPilot is hot-syncing with the pilot. During this
time the conduit asks KPilot if there are any modified records on the
pilot and should back them up into the program the conduit syncs
with. Then, the conduit checks to see if there are any locally
modified records and asks KPilot to put those on the pilot. Once
finished, it exits gracefully by a
{\small\begin{verbatim}
emit syncDone();
\end{verbatim}
}
and KPilot will continue with the next conduit.
In the next few sections I will present the most important classes KPilot provides to the conduit programmers, and after that I will show several examples of different conduits.
\subsection{The \class{SyncAction} base class for conduits}
All conduits are subclasses of \class{SyncAction}, which provides the following methods. The \code{exec} method is a pure virtual function and needs to be implemented by each conduit:
\begin{itemize}
\item
\code{virtual bool exec()=0} ... This function starts the actual processing done
by the conduit. It should return false if the
processing cannot be initiated, f.ex. because
some parameters were not set or a needed library
is missing. This will be reported to the user.
It should return true if processing is started
normally. If processing starts normally, it is
the {\bf conduit's} responsibility to eventually
\code{emit syncDone();} if processing does not start
normally (ie. \code{exec()} returns false) then the
\code{void addSyncLogEntry(const TQString \&e,bool suppress=false)} ... Write a log entry to the pilot. Causes signal \code{logEntry(const char *)} to be emitted.
\code{int pilotSocket()}\qquad ... returns the pilot socket (needed if you use your own functions to talk to the handheld. This is not recommended, but necessary in some cases)
\item
\code{int openConduit()}\qquad ... Notify the handheld which conduit is running now.
\end{itemize}
In addition to these functions the class also has some signals you can emit to display log messages and to indicate that the conduit has finished and KPilot should go on with the next conduit:\
\begin{itemize}
\item\code{void syncDone(SyncAction *)} ... tell KPilot that the conduit has finished. Every conduit that returns true in its \code{exec()} method needs to emit this signal, otherwise KPilot will wait forever for the conduit to return.
If your conduit will have to ask the user some question, the handheld needs to be tickled to prevent a timeout while the message box or dialog window is shown. The \class{InteractiveAction} class (which is a subclass of \class{SyncAction}) can be used as a base class for your conduit in this case. Call \code{startTickle()} some time before showing a dialog to the user (we're assuming a local event loop here) so that while the dialog is up and the user is thinking, the pilot stays awake. Afterwards, call \code{stopTickle()}. The parameter to \code{startTickle} indicates the timeout, in seconds, before signal timeout is emitted. You can connect to that signal, to take down the user interface part if the user isn't reacting.
In addition to \class{SyncAction}'s methods it adds the following methods:
\begin{itemize}
\item\code{void tickle()} ... Tickles the handheld to reset the timeout counter.
\item\code{void startTickle(unsigned count=0)} ... start a timer to tickle
\section{A very simple conduit: malconduit}\label{SectionSimpleConduit}
To have a working conduit, we still have to implement the synchronization
itself. For the AvantGo conduit there already exists a library named
"libmal" which does all the synchronization for us. In the section on
autoconf and automake I already described the configure commands to check
for the library and to link to the library on runtime (the \code{-lmal}
flag stored in the variable \code{\$(MAL\_LIB)} ).
When using the libmal library, all we have to do is to make some proxy settings, and then call the function \texttt{malsync( pilotSocket(), pInfo);}, which will do the actual sync for us:
if (!fLastSync.isValid() || !now.isValid()) return false;
switch (eSyncTime) {
case eEveryHour:
if ( (fLastSync.secsTo(now)<=3600) && (fLastSync.time().hour()==now.time().hour()) ) return true;
else return false;
case eEveryDay:
if ( fLastSync.date() == now.date() ) return true;
else return false;
case eEveryWeek:
if ( (fLastSync.daysTo(now)<=7) && ( fLastSync.date().dayOfWeek()<=now.date().dayOfWeek()) ) return true;
else return false;
case eEveryMonth:
if ( (fLastSync.daysTo(now)<=31) && (fLastSync.date().month()==now.date().month()) ) return true;
else return false;
case eEverySync:
default:
return false;
}
return false;
}
/* virtual */ bool MALConduit::exec() {
FUNCTIONSETUP;
if (!fConfig) {
kdWarning() << k_funcinfo << ": No config file was set!" << endl;
return false;
}
readConfig();
if (skip()) {
emit logMessage(i18n("Skipping MAL sync, because last synchronization was not long enough ago."));
emit syncDone(this);
return true;
}
// Set all proxy settings
switch (eProxyType) {
case eProxyHTTP:
if (fProxyServer.isEmpty()) break;
setHttpProxy(fProxyServer.latin1());
if (fProxyPort>0 && fProxyPort<65536) setHttpProxyPort( fProxyPort );
else setHttpProxyPort(80);
if (!fProxyUser.isEmpty()) {
setProxyUsername( fProxyUser.latin1() );
if (!fProxyPassword.isEmpty()) setProxyPassword( fProxyPassword.latin1() );
}
break;
case eProxySOCKS:
setSocksProxy( fProxyServer.latin1() );
if (fProxyPort>0 && fProxyPort<65536) setSocksProxyPort( fProxyPort );
else setSocksProxyPort(1080);
break;
default:
break;
}
// Now initiate the sync.
PalmSyncInfo* pInfo=syncInfoNew();
if (!pInfo) {
kdWarning() << k_funcinfo << ": Could not allocate SyncInfo!" << endl;
emit logError(i18n("MAL synchronization failed (no SyncInfo)."));
return false;
}
malsync( pilotSocket(), pInfo);
syncInfoFree(pInfo);
saveConfig();
emit syncDone(this);
return true;
}
\end{verbatim}
}
When you use an external library to do the sync, the external functions need a reference to the current connection to the handheld. In the pilot-link library, which is the base for all of KPilot's communication with the handheld, this is implemented via an integer identifier, which can be obtained by the function \texttt{pilotSocket()} of the SyncAction class.
The libmal also needs some internal data structed "PalmSyncInfo", which is obtained by its own syncInfoNew() function, but this part is libmal-specific.
Another issue is how to propagate log messages from the external library to KPilot's log window. SyncAction provides slots logError, logMessage and logProgress to put messages into KPilot's sync log. All you have to do is to call
\begin{verbatim}
emit logMessage(i18n("My own log message"));
\end{verbatim}
The problem with these slots is that they are Qt-specific, while most libraries are written in C, and expect a hook function that will be called whenever a message needs to be written out. Unfortunately you cannot pass a member of your SyncAction-derived class, either, so the way out is to store a pointer to the current conduit instance (only one will be active at any time, anyway) in a static variable, and call the member method from this pointer:
{\footnotesize
\begin{verbatim}
// static pointer to the current conduit instance
static MALConduit *conduitInstance=0L;
// The hook function which will be called by the library
int malconduit_logf(const char *format, ...) {
FUNCTIONSETUP;
va_list val;
int rval;
va_start(val, format);
#define WRITE_MAX_BUF 4096
char msg[WRITE_MAX_BUF];
msg[0]='\0';
rval=vsnprintf(&msg[0], sizeof(msg), format, val);
The PalmDoc conduit of KPilot takes a directory of text files and synchronized them with PalmDOC databases on the handheld. These PalmDOC documents can be read with AportisDoc, TealReader, and modified with applications like QED. Optionally, the conduit can also keep local copies of the pdb handheld databases in a local directory.
The conduit just needs to find out if a document has changed either on the handheld or on the pc (or on both sides), and then copy the text either to or from the handheld. The docconduit has a class \class{DOCConverter} which does the actual conversion. You only have to set the local path to the text file, and give a pointer to an opened \class{PilotDatabase} (either \class{PilotLocalDatabase} or \class{PilotSerialDatabase}), and then call
\code{docconverter.convertPDBtoDOC();} or \code{docconverter.convertDOCtoPDB();}. I will not explain this class here, but rather the algorithm to determine the sync direction and the actual calls of the DOCConverter.
The conduit has to find out
\begin{itemize}
\item which PalmDoc databases on the handheld have been changed or edited (using the modified flag of the records inside the database)
\item which text files on disk have changed (using an md5 checksum on the text)
\item if a local copy is kept, if the local copy of a database has been changed or added (again using the modified flat of the records inside the database).
\end{itemize}
To assure a responsive user interface, we will once again use \texttt{QTimer::singleShot(this, 0, SLOT(whatever()));} for each of these steps.
The \code{DOCConduit::exec()} function is just the entry point and calls syncNextDB, which will go through all PalmDOC databases on the handheld and determine if any of them has been changed:
{\footnotesize
\begin{verbatim}
/* virtual */ bool DOCConduit::exec() {
FUNCTIONSETUP;
readConfig();
dbnr=0;
QTimer::singleShot(0, this, SLOT(syncNextDB()));
return true;
}
\end{verbatim}
}
syncNextDB then walks through all PalmDoc databases on the handheld and decides if they are supposed to be synced to the PC. The function needsSync (which we will describe later), checks which files have actually changed or were added or deleted and so determines the sync direction. The docSyncInfo is just an internal structure to store all information about the text:
To go through all .txt files on disk, we use a QStringList::Iterator, again set the fields of the docSyncInfo for each text, and call \code{needsSync} to do the actual comparison of the local and handheld text to the versions of the previous sync. If a local copy of the pdb files should be kept, we proceed similar using the slot \code{checkPDBFiles}:
{\footnotesize
\begin{verbatim}
void DOCConduit::syncNextDOC() {
FUNCTIONSETUP;
if (eSyncDirection==eSyncPDAToPC ) {
// We don't sync from PC to PDB, so start the conflict resolution and then the actual sync process
After all databases have been identified, we possibly need to do some conflict resolution in the slot \code{resolve()}. The conflict resolution dialog just displays the list of databases and lets the user choose the sync direction for each database. When the user presses Ok, the direction field of each docSyncInfo object is set to the chosen value.
{\footnotesize
\begin{verbatim}
void DOCConduit::resolve() {
FUNCTIONSETUP;
for (fSyncInfoListIterator=fSyncInfoList.begin(); fSyncInfoListIterator!=fSyncInfoList.end(); fSyncInfoListIterator++) {
// Walk through each database and apply the conflictResolution option.
// the remaining conflicts will be resolved in the resolution dialog
if ((*fSyncInfoListIterator).direction==eSyncConflict){
DEBUGCONDUIT<<"We have a conflict for "<<(*fSyncInfoListIterator).handheldDB<<", default="<<eConflictResolution<<endl;
switch (eConflictResolution)
{
case eSyncPDAToPC:
DEBUGCONDUIT<<"PDA overrides for database "<<(*fSyncInfoListIterator).handheldDB<<endl;
Finally, the actual sync of the databases is done again with \code{QTimer::singleShot}s in the slot \code{syncDatabases()}. Each entry in the list is processed in one pass of \code{syncDatabases}, and then \code{syncDatabases} is again called using a \code{QTimer::singleShot}, until all databases have been synced.
The actual sync is done by the function \code{doSync(docSyncInfo\&)}, which first checks for deletion of the database as a special case. Otherwise, it uses the \class{DOCConverter} class to copy the text file to or from the handheld, and then recalculates the md5 checksum of the text file on disk and stores it in KPilot's config.
{\footnotesize
\begin{verbatim}
bool DOCConduit::doSync(docSyncInfo &sinfo) {
bool res=false;
if (sinfo.direction==eSyncDelete) {
if (!sinfo.docfilename.isEmpty()) {
if (!QFile::remove(sinfo.docfilename)) {
kdWarning()<<i18n("Unable to delete the text file \"%1\" on the PC").arg(sinfo.docfilename)<<endl;
The worst part about the conduit is to find out which side has been changed (and how), and what needs to be done about this. The function \code{needsSync} does exactly this. If the database was not included in the last sync, it is new, so it will be synced from the side where it was added.
First, we find out, how each of the two sides have changed.
If the database was already included, check if it was changed using the function \code{textChanged} to compare the md5 checksum of the current text on disk with the checksum of the last sync (stored in kpilot's config). The handheld side is a bit trickier: A PalmDOC on the handheld contains of a header record, several text records, and finally several bookmark records. Each of these records can have the dirty flag set, so we first get the number of text records from the header record. Then we search for the index of the first changed record (i.e. dirty flag set) after the header record. If no text record (but a bookmark record) was changed, a config setting determines if the PalmDOC should still be considered as changed.
Finally, from the status of the two sides, determine the sync direction:
If you work with record-based conduits (e.g. addressbook, calendar conduits etc.), you might proceed similar to a document-based conduit (where records correspond to documents, of course), although you probably want to do the synchronization step immediately instead of storing all information about the states in a big list and only later sync the data. To dig deeper into the structure of such conduits (which I admit are the most common ones, but also the most complex), take a look at the addressbook conduit of KDE 3.1 (e.g. using KDE's webcvs at \htmladdnormallink{http://webcvs.kde.org/cgi-bin/cvsweb.cgi/tdepim/kpilot/conduits/abbrowserconduit/}{http://webcvs.kde.org/cgi-bin/cvsweb.cgi/tdepim/kpilot/conduits/abbrowserconduit/}).
\section{Using KitchenSync in your conduit}\label{SectionKitchenSync}
Currently none of the KPilot conduits use the KitchenSync framework, which
is intended to be KDE's synchronization application of the future. Thus I cannot and will not
describe how to use it for your conduits. In the future, however,
KPilot will probably 's conduits will be ported to KitchenSync. For a quick overview over
KitchenSync see Cornelius Schumacher's original proposal at \htmladdnormallink{http://pim.kde.org/development/ksync.php}{http://pim.kde.org/development/ksync.php}.
Currently, the KitchenSync framework doesn't seem to be mature enough to replace KPilot's already very sophisticated conduits and their conflict resolution.
Some time ago I mailed Holger Freyther (one of the KitchenSync authors) about the current state of KitschenSync, and asked several important questions. His answer is printed below. In my eyes, KitchenSync is just lacking too many important features to be able to replace KPilot soon (or rather port KPilot to KitchenSync):
{\footnotesize
\begin{verbatim}
Re: [Kitchensync] [Kde-pim] Some Kitchensync questions
From: Holger Freyther <XXXXXX@XXXXX.XXX> (Opie)
To: Reinhold Kainhofer <reinhold@kainhofer.com>
Date: 05.10.2002 00:01
On Wednesday 18 September 2002 19:57, Reinhold Kainhofer wrote:
> Hello,
> Yesterday night I took a quick look at the kitchensync sources to find out
> how kpilot might use it or how kpilot might need to be changed. Please
> forgive me if some questions might be obvious or stupid, but I this is my
> first look at kitchensync/ksync, and I didn't have too much time. Just give
> me a hint on where to look. Additionally, it was very late at night, so my
> brain wasn't working properly any more ;-))
>
> My questions will of course be from a Palm/handheld syncing view, so they
> are quite specific concerning the current functionality of the conduits. I
> wouldn't like switching to something that - in theory - has a better and
> cleaner architecture, but for the next few years will provide less features
> than we currently have.
Thats good! so then we find problems because you look at it from a different
point of view
> 1) Do I understand it right from the ksync proposal on pim.kde.org that
> kitchensync is the application that does all the communication with the
