You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
273 lines
11 KiB
273 lines
11 KiB
15 years ago
|
DESIGN:
|
||
|
=======
|
||
|
|
||
12 years ago
|
libtdeio uses tdeioslaves (separate processes) that handle a given protocol.
|
||
|
Launching those slaves is taken care of by the tdeinit/tdelauncher tandem,
|
||
15 years ago
|
which are notified by DCOP.
|
||
|
|
||
|
Connection is the most low-level class, the one that encapsulates the pipe.
|
||
|
|
||
|
SlaveInterface is the main class for transferring anything to the slave
|
||
|
and Slave, which inherits SlaveInterface, is the sub class that Job should handle.
|
||
|
|
||
|
A slave inherits SlaveBase, which is the other half of SlaveInterface.
|
||
|
|
||
|
The scheduling is supposed to be on a two level basis. One is in the daemon
|
||
|
and one is in the application. The daemon one (as opposite to the holy one? :)
|
||
|
will determine how many slaves are ok for this app to be opened and it will
|
||
|
also assign tasks to actually existing slaves.
|
||
|
The application will still have some kind of a scheduler, but it should be
|
||
|
a lot simpler as it doesn't have to decide anything besides which
|
||
|
task goes to which pool of slaves (related to the protocol/host/user/port)
|
||
|
and move tasks around.
|
||
|
Currently a design study to name it cool is in scheduler.cpp but in the
|
||
|
application side. This is just to test other things like recursive jobs
|
||
|
and signals/slots within SlaveInterface. If someone feels brave, the scheduler
|
||
|
is yours!
|
||
|
On a second thought: at the daemon side there is no real scheduler, but a
|
||
|
pool of slaves. So what we need is some kind of load calculation of the
|
||
|
scheduler in the application and load balancing in the daemon.
|
||
|
|
||
|
A third thought: Maybe the daemon can just take care of a number of 'unused'
|
||
|
slaves. When an application needs a slave, it can request it from the daemon.
|
||
|
The application will get one, either from the pool of unused slaves,
|
||
|
or a new one will be created. This keeps things simple at the daemon level.
|
||
|
It is up to the application to give the slaves back to the daemon.
|
||
|
The scheduler in the application must take care not to request too many
|
||
|
slaves and could implement priorities.
|
||
|
|
||
|
Thought on usage:
|
||
|
* Typically a single slave-type is used exclusively in one application. E.g.
|
||
|
http slaves are used in a web-browser. POP3 slaves used in a mail program.
|
||
|
|
||
|
* Sometimes a single program can have multiple roles. E.g. konqueror is
|
||
|
both a web-browser and a file-manager. As a web-browser it primarily uses
|
||
|
http-slaves as a file-manager file-slaves.
|
||
|
|
||
|
* Selecting a link in konqueror: konqueror does a partial download of
|
||
|
the file to check the mimetype (right??) then the application is
|
||
|
started which downloads the complete file. In this case it should
|
||
|
be able to pass the slave which does the partial download from konqueror
|
||
|
to the application where it can do the complete download.
|
||
|
|
||
|
Do we need to have a hard limit on the number of slaves/host?
|
||
|
It seems so, because some protocols are about to fail if you
|
||
|
have two slaves running in parralel (e.g. POP3)
|
||
|
This has to be implemented in the daemon because only at daemon
|
||
|
level all the slaves are known. As a consequence slaves must
|
||
|
be returned to the daemon before connecting to another host.
|
||
|
(Returning the slaves back to the daemon after every job is not
|
||
|
strictly needed and only causes extra overhead)
|
||
|
|
||
|
Instead of actually returning the slave to the daemon, it could
|
||
|
be enough to ask 'recycling permission' from the daemon: the
|
||
|
application asks the daemon whether it is ok to use a slave for
|
||
|
another host. The daemon can then update its administration of
|
||
|
which slave is connected to which host.
|
||
|
|
||
|
The above does of course not apply to hostless protocols (like file).
|
||
|
(They will never change host).
|
||
|
|
||
|
Apart from a 'hard limit' on the number of slaves/host we can have
|
||
|
a 'soft limit'. E.g. upon connection to a HTTP 1.1 server, the web-
|
||
|
server tells the slave the number of parallel connections allowed.
|
||
|
THe simplest solution seems to be to treat 'soft limits' the same
|
||
|
as 'hard limits'. This means that the slave has to communicate the
|
||
|
'soft limit' to the daemon.
|
||
|
|
||
|
Jobs using multiple slaves.
|
||
|
|
||
|
If a job needs multiple slaves in parallel (e.g. copying a file from
|
||
|
a web-server to a ftp-server or browsing a tar-file on a ftp-site)
|
||
|
we must make sure to request the daemon for all slaves together since
|
||
|
otherwise there is a risk of deadlock.
|
||
|
|
||
|
(If two applications both need a 'pop3' and a 'ftp' slave for a single
|
||
|
job and only a single slave/host is allowed for pop3 and ftp, we must
|
||
|
prevent giving the single pop3 slave to application #1 and the single
|
||
|
ftp slave to application #2. Both applications will then wait till the
|
||
|
end of times till they get the other slave so that they can start the
|
||
|
job. (This is a quite unlikely situation, but nevertheless possible))
|
||
|
|
||
|
|
||
|
File Operations:
|
||
14 years ago
|
listRecursive is implemented as listDir and finding out if in the result
|
||
15 years ago
|
is a directory. If there is, another listDir job is issued. As listDir
|
||
|
is a readonly operation it fails when a directory isn't readable
|
||
|
.. but the main job goes on and discards the error, because
|
||
|
bIgnoreSubJobsError is true, which is what we want (David)
|
||
|
|
||
|
del is implemented as listRecursive, removing all files and removing all
|
||
|
empty directories. This basically means if one directory isn't readable
|
||
14 years ago
|
we don't remove it as listRecursive didn't find it. But the del will later
|
||
15 years ago
|
on try to remove it's parent directory and fail. But there are cases when
|
||
|
it would be possible to delete the dir in chmod the dir before. On the
|
||
|
other hand del("/") shouldn't list the whole file system and remove all
|
||
14 years ago
|
user owned files just to find out it can't remove everything else (this
|
||
15 years ago
|
basically means we have to take care of things we can remove before we try)
|
||
|
|
||
|
... Well, rm -rf / refuses to do anything, so we should just do the same:
|
||
|
use a listRecursive with bIgnoreSubJobsError = false. If anything can't
|
||
|
be removed, we just abort. (David)
|
||
|
|
||
|
... My concern was more that the fact we can list / doesn't mean we can
|
||
|
remove it. So we shouldn't remove everything we could list without checking
|
||
|
we can. But then the question arises how do we check whether we can remove it?
|
||
|
(Stephan)
|
||
|
|
||
|
... I was wrong, rm -rf /, even as a user, lists everything and removes
|
||
|
everything it can (don't try this at home!). I don't think we can do
|
||
|
better, unless we add a protocol-dependent "canDelete(path)", which is
|
||
|
_really_ not easy to implement, whatever protocol. (David)
|
||
|
|
||
|
|
||
|
Lib docu
|
||
|
========
|
||
|
|
||
|
mkdir: ...
|
||
|
|
||
|
rmdir: ...
|
||
|
|
||
|
chmod: ...
|
||
|
|
||
|
special: ...
|
||
|
|
||
|
stat: ...
|
||
|
|
||
|
get is implemented as TransferJob. Clients get 'data' signals with the data.
|
||
|
A data block of zero size indicates end of data (EOD)
|
||
|
|
||
|
put is implemented as TransferJob. Clients have to connect to the
|
||
|
'dataReq' signal. The slave will call you when it needs your data.
|
||
|
|
||
|
mimetype: ...
|
||
|
|
||
|
file_copy: copies a single file, either using CMD_COPY if the slave
|
||
|
supports that or get & put otherwise.
|
||
|
|
||
|
file_move: moves a single file, either using CMD_RENAME if the slave
|
||
|
supports that, CMD_COPY + del otherwise, or eventually
|
||
|
get & put & del.
|
||
|
|
||
|
file_delete: delete a single file.
|
||
|
|
||
|
copy: copies a file or directory, recursively if the latter
|
||
|
|
||
|
move: moves a file or directory, recursively if the latter
|
||
|
|
||
|
del: deletes a file or directory, recursively if the latter
|
||
|
|
||
|
PROGRESS DISPLAYING :
|
||
|
=====================
|
||
|
Taj brought up the idea of deligating all progress informations to an extern
|
||
|
GUI daemon which could be provided in several implementations - examples
|
||
|
are popup dialogs (most are annoyed by them, like me :) or a kicker applet
|
||
|
or something completely different. This would also remove the dependency on
|
||
13 years ago
|
libtdeui (I hope).
|
||
12 years ago
|
Conclusion: tdeio_uiserver is this single GUI daemon, but the dependency on
|
||
13 years ago
|
libtdeui couldn't be removed (for many reasons, including Job::showErrorDialog())
|
||
15 years ago
|
|
||
|
A. progress handling
|
||
|
---------------------
|
||
|
There will be two ways how the application can display progress :
|
||
|
|
||
|
1. regular apps will use NetAccess for all kio operations and will not care
|
||
|
about progress handling :
|
||
|
- NetAccess creates Job
|
||
|
- NetAccess creates JobObserver that will connect to the Job's signals and
|
||
|
pass them via dcop to the running GUI Progress Server
|
||
|
|
||
|
2. apps that want to do some handling with progress dialogs like Caitoo or
|
||
|
KMail :
|
||
|
- app creates Job
|
||
|
- app creates a progress dialog : this should be a ProgressBase descendant
|
||
14 years ago
|
e.g. StatusProgress or custom progress dialog
|
||
15 years ago
|
- app calls progress->setJob( job ) in order to connect job's signals with
|
||
|
progress dialog slots
|
||
|
|
||
|
B. customized progress dialogs
|
||
|
-------------------------------
|
||
|
This will be similar to what we had before.
|
||
|
|
||
|
- ProgressBase class that all other dialogs will inherit.
|
||
12 years ago
|
will contain an initialization method setJob( TDEIO::Job*) for apps of the
|
||
15 years ago
|
second class (see A.2 above), that will connect job's signals to dialog's
|
||
|
slots
|
||
|
|
||
|
- DefaultProgress ( former KIOSimpleProgressDialog ) that will be used for
|
||
|
regular progress dialogs created by GUI Progress Server
|
||
|
|
||
14 years ago
|
- StatusProgress ( former KIOLittleProgressDialog ) that can be used for
|
||
15 years ago
|
embedding in status bar
|
||
|
|
||
|
C. GUI Progress Server
|
||
|
-----------------------
|
||
|
This is a special progress server.
|
||
|
- createProgress() will either create a DefaultProgress dialog or add new entry
|
||
|
in a ListProgress ( an all-jobs-in-one progress dialog )
|
||
|
- after receiving signals from the JobObserver via DCOP it will call
|
||
|
appropriate method of progress dialog ( either in DefaultProgress or ListProgress )
|
||
|
- ListProgres can be a Caitoo style dialog, kicker applet or both in one.
|
||
|
|
||
|
D. Some notes
|
||
|
--------------
|
||
|
1. most of the apps will not care at all about the progress display
|
||
|
2. user will be able to choose whether he wants to see separate progress
|
||
|
dialogs or all-in-one ListProgress dialog
|
||
|
3. developers can create their custom progress dialogs that inherit
|
||
|
ProgressBase and do any manipulation with a dialog if they use a second
|
||
|
approach ( see A.2 above )
|
||
|
|
||
|
|
||
|
Streaming
|
||
|
---------
|
||
|
|
||
|
1. We currently support a streaming "GET": e.g. file:/tmp/test.gz#gzip:/
|
||
|
works. The following should also work: file:/tmp/test.gz.gz#gzip:/#gzip:/
|
||
|
The current approach makes a TrasnferJob for gzip:/ and then adds a
|
||
|
subjob for "file:/tmp/test.gz.gz#gzip:/" which itself adds a subjob
|
||
|
for "file:/tmp/test.gz.gz".
|
||
|
2. This doesn't extend very well to PUT, because there the order should
|
||
|
basically be the other way around, but the "input" to the job as a whole
|
||
|
should go to the "gzip:/" job, not to the "file:/tmp/test.gz.gz."
|
||
|
It would probably be easier to implement such a job in the way the
|
||
|
current "CopyJob" is done. Have a Job and make all sub-urls sub-jobs of
|
||
|
this Job.
|
||
|
3. As a result of 1. COPY FROM an url like file:/tmp/test.gz#gzip:/ should
|
||
|
work. COPY TO does not, because that would require PUT.
|
||
|
|
||
|
|
||
|
Resuming
|
||
|
--------
|
||
|
|
||
|
A rough note for now, just to have this somewhere :
|
||
|
(PJ=put-job, GJ=get-job)
|
||
|
|
||
|
PJ can't resume:
|
||
|
PJ-->app: canResume(0) (emitted by dataReq)
|
||
|
GJ-->app: data()
|
||
|
PJ-->app: dataReq()
|
||
|
app->PJ: data()
|
||
|
|
||
|
PJ can resume but GJ can't resume:
|
||
|
PJ-->app: canResume(xx)
|
||
|
app->GJ: start job with "resume=xxx" metadata.
|
||
|
GJ-->app: data()
|
||
|
PJ-->app: dataReq()
|
||
|
app->PJ: data()
|
||
|
|
||
|
PJ can resume and GJ can resume:
|
||
|
PJ-->app: canResume(xx)
|
||
|
app->GJ: start job with "resume=xxx" metadata.
|
||
|
GJ-->app: canResume(xx)
|
||
|
GJ-->app: data()
|
||
|
PJ-->app: dataReq()
|
||
|
app->PJ: canResume(xx)
|
||
|
app->PJ: data()
|
||
|
|
||
|
So when the slave supports resume for "put" it has to check after the first
|
||
|
dataRequest() whether it has got a canResume() back from the app. If it did
|
||
|
it must resume. Otherwise it must start from 0.
|
||
|
|
||
|
|