amaroK VIS_PLAN - Created: 22-11-2003 Abstract ======== In order to create a visualisation framework that is superior to anything popular available for Linux today we should plan amaroK's thoroughly before implementation. This document exists to map out our ideas and is available for anyone to edit, please, if you have some good suggestion, or experience with this kind of work, add some comments! Thanks, Max Howell Formatting ========== The document has no regimented formatting, but do try to keep the wordwrap set the same! Feel free to recommend formatting if the document becomes unmanageable. Plan ==== My initial thoughts were to make visualisations a separate process rather than separate thread. The reasoning being, a crash in the vis won't crash amaroK, threads can be unmanageable, *nix is good for new processes, we can use DCOP to control the vis and presumable we just need to pass a handle to the relevant arts server to the vis and leave it. Since the music playing bit is already a separate process it makes sense to make the visualisation separate too! Also this may mean that these visuals could be separate from amaroK and just depend on KDE. I was also thinking that this means it may be possible to not use Qt to do rendering. There are probably better libraries for high speed graphics and if there isn't a way to make DCOP not depend on a QApplication, then maybe we could write something. It would be sorted if we could make something that can react to arts sound output in general. we could use SDL for 2d and OpenGL for 3d rendering. I was also thinking it would be neat to embed the visuals somehow into amaroK, so as the background in the playlist perhaps (I know, this would probably not work well due to the need to re-render all the AA text all the time, but still the idea is neat) This is similar to Sonique on Windows if anyone has ever used it. If we make the visualisations not amaroK specific it may be possible to embed visualisations into the Kicker, onto the desktop, etc. Do we want more than an FFT of the audio data? Would other data be useful to visuals at all? Beat detection certainly would be a useful thing to offer the visualisations, and it would certainly stop every visual implementing its own detection. Vis Process Architecture ======================== I suggest having a separate process for each visualization. Each vis process would only load one visualization plugin. This way we have good crash safety and it's also possible to develop and ship plugins separately from amaroK. So we have one "vis loader skeleton" in amaroK, which will be instanciated whenever we need a vis. This loader could get started with the vis plugin as cli argument. It would also contain all the socket and interface needed to communicate with amaroK. We should take care to keep the design as simple as possible. Why make visualisations plugins as well as binaries? It seems sensible to have visualisations as separate processes and then communicate with amK through the socket. Why have the extra layer of plugins if you're going to force each process to only load one plugin anyway? Merging Analyzer and Vis ======================== If we decide on unifying vis and analyzer, the question arises how to get the rendered bitmap graphic back into amaroK's playerwidget. It seems we have the following options: * Writing it into a shared memory buffer. amaroK must then read the buffer and bitBlt() the pixmap into the framebuffer * Transferring the pixmap over a Unix socket (or UDP network socket). This can amount to several megabytes per second. After the transfer amaroK must bitBlt() it to the framebuffer. * Writing it directly into the framebuffer at the desired position, like a video overlay. There might be technical problems doing so, and it will be difficult when the user moves the window. [Comment from muesli, to be integrated into the paragraphs above] imho, it would be cool to offer this service via sockets. udp would fit per- fectly, since it has to be realtime, anyways (better loose a packet and show nothing, instead of a out-of-sync beat). e.g., you would be able to have various of visualizations-clients connected to amarok, showing different animations, at the same time. this would also remove the DCOP-dependency-chains for the OpenGL clients. right now, i can imagine two implementation methods: A) transmit the frequency vector of the analyzer to the vis-client. beat-detection and all that stuff gets done by the vis-client. B) basic beat-detection and music analysis gets done by amarok itself. the results of that process are sent to the vis-clients. IMO it would be good to offer services like beat detection. This way vis-writers have less to do and are more likely to write a vis since beat-detection is already done, also it means beat-detection is likely to be better as we will over time offer a really good implementation, and it means that if the user has many visuals running, the beat-detection is only done once for all visualisations. [/Comment by muesli] //FIXME beat detection code temporarily moved here /* // Muesli's Beat Detection - A Night's Oddysee // shift old elements for ( uint x = 0; x < 18; ++x ) for ( uint y = 42; y > 0; --y ) m_beatEnergy[x][y] = m_beatEnergy[x][y - 1]; // get current energy values for ( uint x = 0; x < pScopeVector->size(); ++x ) m_beatEnergy[x][0] = pScopeVector->at(x); // compare to old elements and get averages double beatAvg[18]; // double beatVariance[18]; // double beatMood[18]; for ( uint x = 0; x < 18; ++x ) { beatAvg[x] = 0; for ( uint y = 1; y < 44; ++y ) beatAvg[x] += m_beatEnergy[x][y]; beatAvg[x] = beatAvg[x] / 43; } */ /* for ( uint x = 0; x < 18; ++x ) { beatVariance[x] = 0; for ( uint y = 0; y < 42; ++y ) beatVariance[x] += (pow((m_beatEnergy[x][y] - beatAvg[x]), 2) / 43); } for ( uint x = 0; x < 18; ++x ) beatMood[x] = (-0.0025714 * beatVariance[x]) + 1.5142857; */ // do we have a beat? let's dance! /* int total_hits = 0; for ( uint x = 0; x < 18; ++x ) { double factor = cos( x * 4 ) * 18; factor = beatAvg[x] * factor; if ( m_beatEnergy[x][0] > factor ) { total_hits++; kdDebug() << "*CLAP* factor: " << factor << " - x: " << x << " - average energy: " << beatAvg[x] << " - current peak: " << m_beatEnergy[x][0] << endl; } } if ( total_hits > 3 ) kdDebug() << "***CLAPCLAPCLAP***" << endl; */