This marks the very final part of Semester 1, 2009 for me. We were given the option of completing our documentation for Multimedia Authoring 1 with either a video or website, which were due today. I chose the latter because of the non-action required for my project. It wouldn’t make for a particularly exciting video experience: “Wow, look at that guy, not doing anything!” Exciting stuff.
So as my projects have a tendency to do, this one got a bit out of hand. So I turned it into a minisite. This is the first of a couple that I have in the works for the mid-year break (the second is a far more exciting return of Concrète. More on that soon) and doing this prevents me from making any more bloody WordPress databases – it’s like they’ve begun breeding on the angryPixel server.
For anyone who’s read past posts about my Multimedia Authoring 1 process, there’s not a lot new here, but I’m hoping that it’s helpful to someone out there at some point, maybe coming across Processing for the very first time.
Well, against all odds, it would seem my final coding project is complete (or as complete as I’m willing to make it – adding more code is now making it all very unstable). In the end, there was a bit of a struggling and wrangling disparate sketches, trying to get them to play well together.
Before I go into my process over the last week, I think that it’s important to point out that whilst this sketch ‘works’, it’s likely that I haven’t put it together in the most efficient manner possible. My knowledge of which code puts pressure on CPU is totally non-existent, so I don’t doubt that this project could be cleaned up a great deal to run better. My work around for this project was to decrease the size of the display window to just 320×240.
Sketch 01
My first step was to bring the main elements together: the RSS feed and particles. I ended up quickly ditching my original idea of using the traer.physics balls in favor of using Andy Best’s bubble popping sketch. To keep things simple to begin with, everything is triggered by holding down the mouse button for a few seconds…
To create some movement from different sides of the sketch, I shifted the falling bubbles to come from the left of screen. But instead of them simply moving from left to right, I wanted them to move toward the face. Introduction of face detection suddenly made the sketch run very slowly. A bit of digging through the Processing forums led me to discover that other people have had similar experiences: video feed at 640×480 in Processing can be very clunky…
I was struggling to get an x and y coordinate out of the face recognition rectangle, so instead I started by using the mouseX and mouseY coordinates. I also switched to using the traer.physics cloud sketch, which gave the particles more movement and randomisation, as they head towards the mouse cursor…
Finally, I found the face detection coordinates and replaced the red bubbles with mini newspaper articles. However, the events were still being triggered by the mouse button being held down for a few seconds, not through movement of the user…
The next iteration of the sketch refined the movement of the particles and added a sound loop with the Ess library. As part of my pitch, I wanted to have an array of sound samples playing in a completely randomised and generative way, almost like a John Cage piece. Because of the time constraints I found myself with, I instead used a single sample of tuning between radio channels…
The final few versions of the sketch simply moved through refinement of the major elements: the sound would slowly lower its volume as the particles covered the face; the video feed would decrease in brightness the longer the user did not move; the RSS feeds would repeat and cover previous feeds to create a nonsensical version of bars across the image; and finally, the loop would restart once the user moves their head a certain amount…
Note: the movie maker in Processing has exported at a fairly quick frame rate and I’ve added the sound in afterwards.
There are a couple of problems with the final sketch. As you can see in the video, occasionally the action loop will restart, even though there has been minimal movement. This seems to be when the face detection thinks there is another face in the image. The jump in coordinates re-triggers the loop.
I also think that even though this is very close to what I planned as my concept pitch, in the end it was not successful as an interactive work. Although I had planned it to be the antithesis of interactive: it’s triggered by non-action, there needed to be another element to encourage users to explore this non-action. Whether there was a secondary event that happened when users move, or something that happened at the end of the loop, I think that the end result was a bit weak.
Having said that though, I am encouraged that I managed to pull this together in a short amount of time. Over the break, I plan on trying to get this piece to work more effectively and through that, learn a little more before I set foot in Multimedia Authoring 2 next semester.
In the midst of three different final assessments, my code has only been taking baby steps this week. I’m a long way from getting it complete and if it wasn’t for the saving grace of a few extra days with nothing else to distract me, I think I would be having a minor meltdown.
Face detection and traer.physics were combined this week, with some positive results…
Whilst it isn’t really the behavior I plan for the final sketch, it is at least a proof of concept that I can combine these elements for the newspaper particles of my original brief.
So the remaining steps for my final project are now:
change traer.physics balls into newspaper particles
detect small vs large movement of face detection
add sound samples, which change once small movement loop is triggered
combine all of the above with RSS feed
find time to write more comprehensive journal entries
It looks like I’ll have my work cut out for me in this last week. Time to immerse myself in some code.
Inspiration
As a complete distraction from my end-of-semester-stress-out, I’ve been enjoying Moderat’s new stuff lately. Spotting on CDM that they’re doing a live show with visuals made me super excited: Modeselektor’s (one half of Moderat) live show is ridiculously good and live visuals would seal the deal for me. Here’s hoping that all this study will lead me down this road someday soon…
Finally, the first element of my sketch is behaving itself. The RSS feeds are now appearing one beneath the other, and moving steadily down the page. This was a lot more difficult than I had first expected, and completed in no small part with the help of my good friend, Paul Jebanasam (coding extraordinaire)…
With the first hurdle out of the way, I was hoping that the rest would come a little easier. Not so.
As part of my pitch, I wanted the user to be an abstracted image of themselves: a silhouette. The plan was that this silhouette would attract the particles once there wasn’t a certain level of movement. Pretty quickly, I realised that my project is attempting to combine a lot of different libraries and therefore will likely run into a lot of different problems down the track.
So instead of using a silhouette, I have started trying to combine openCV’s face recognition with the traer.physics bouncy balls. I think this will be a better combination, because the face recognition should already give me an x and y coordinate, rather than trying to find the silhouette pixels from the entire screen.
As a final note, because the final assessment due for this subject one week after everything else, I’ve found I really put this off a lot more than I should. Certainly as a novice coder, it’s important that I continue chipping away at the project and getting feedback from the tutor. Much like learning a spoken language, it’s important that I regularly put myself into the environment of code, so that it starts to become second nature. Particularly as I want to continue down this path, I need to spend as much time with Processing as I can, and hopefully in the same way that HTML and CSS began to ‘make sense’, I expect that Processing’s version of Java will (soon?) do the same.
It’s going to be a recurring opening statement for my (late) journal entries this week that I’m officially now behind. A week of being sick, running a music stage at The Finders Keepers markets and an overly ambitious freelance schedule have all ganged up against my uni work and at this point, they look like they’re winning the battle. Fingers crossed they don’t win the war…
After breaking my sketch up into more managable sections (RSS feed/silhouette/particles/sound), I decided to begin from the top of that list; with the RSS feed. Using a combination of romeFeeder and NextText, the example sydicated feed sketch on the romeFeeder site actually suited me quite well. However, I want to have the feeds arriving one after the other, from the top to bottom of screen and not disappear until there is input from the participant. Which is where things got a little bit trickier.
Fortunately, the NextText library puts a bounding box around the feeds, so that makes it easy to figure out the height of each feed, regardless of its dynamic length. However, it’s noted on the NextText documentation that this figure isn’t totally reliable, so I’m just hoping that I can still make this sketch function reasonable well in spite of this shortcoming. The point I found myself stuck at was actually getting those heights to play with each other.
As I’ve found to be the way with Processing, often you know how the code should come together, you just can’t find the right syntax. Kind of like learning a new language: you know what you’re supposed to say, you just can’t find the words. And so I found myself trying to not only calculate the height of each feed, but then take that figure and add it to the previous height, so that the latest feed would sit below the one before.
(Yes, I’m still having troubles getting Processing sketches to display properly. I’ll update these posts as soon as I can figure out what the problem is.)
Instead of progressing down the screen, the feeds are laying one in front of the other. I have managed to keep them on screen – instead of disappearing – and starting at the top of the screen, but there’s still a lot of work to do.
We were introduced to sound within Processing this week. Using primarily the Minim library, we took a look at how to get sound both in and out of our sketches.
I’m quite interested in how sound can be used in programming and installation work, although I’m aware of Processing’s limitations with this topic, so it’s not something I’ve pursued. I think a more powerful approach would be to link Processing with either a sequencer like Ableton, or to try and learn Pure Data or Max/MSP. Any of these options are fairly time consuming, so it’ll stay on the back-burner until I have a little bit of time up my sleeve.
Code Research
Create Digital Music, along with Create Digital Motion, are sites that I frequent almost every day, to check up on AV nerdery. The primary writer, Peter Kirn, is across all kinds of technology and has a real DIY aesthetic, which I love. More often than not, the posts on both sites give you links to everything you need to get started with interesting video/sound/code projects.
I came across this post on CDM, which shows a unique glitchy sequencer, totally built with Processing…
The creator, Bret Truchan, uses the Game of Life as a type of randomising step sequencer and arrangement tool. I’m a big fan of the results you can get using the Game of Life (and I’m not alone – it’s hugely popular amongst nerds everywhere) as a means for generating random results. As a concept, it has some more meaning than a simple number generator.
In an attempt to edit this weeks’ tutorial sketch into something useful for my project, I added code which recognises sound coming into the computer (microphone, line-in etc) with one of the RSS sketches I have outlined below. This resulted in an interesting problem: the visuals associated with both sketches did not appear together. I didn’t have a chance to really work through the code and get to the bottom of the problem, but led me down a tangent of searching for the use of ‘layers’ within Processing. Nothing to report as yet.
Concept Research
So now that we’re really just a few weeks out from handing in our finished project, there are definitely some alarm bells ringing. Speaking to other people in the course, I’m feeling fortunate that whilst my concept is reasonably strong, the execution for my program is relatively simple. And there are a bunch of great resources out there to get me well on my way. I can break my project down into four main components: the text/RSS feed; the particle elements; silhouette/reflection; and sound. This is where I’m at as far as sourcing code…
Text/RSS Feed
romeFeeder is one that I’m really glad I came across. This sketch takes the RSS feed idea and pushes it a step further: it searches Twitter for tweets which include a particular word, or phrase. This could be a powerful way to contain what is displayed within some kind of context.
Another great library in the text world is NextText. It allows Processing to use TrueType fonts, which means I could get a chance to use Trade Gothic, or something equally sexy, rather than Times New Roman, or (God forbid) Comic Sans.
Particle Elements
traer.physics is where all the impressive real world stuff takes place. I imagine that this is where things will get tricky. I still feel as though my coding skills are lacking, and whilst I would love to be able to spend a good amount of time getting this right, I know only too well that as things get more and more hectic toward the end of semester, time won’t something I’ll know much of.
Silhouette/Reflection
So that really only leaves the silhouette. The Golan Levin article, Computer vision for artists and designers: pedagogic tools and techniques for novice programmers (just as an aside: what’s going on with titles for academic papers? Why do they feel the need to impress how intelligent they are before you’ve even reached the first paragraph?) has some excellent Processing sketches in the Appendix that whilst being written for older incarnations of Processing, will probably be quite simply reworked into a current, functioning sketch.
Sound
There are a slew of libraries out there that will do what I’m after with sound: basic, randomising array of samples, with a simple low-pass filter. Because it’s a relatively simple area of my sketch, it will no doubt be left until the end.
With the presentations held in last week’s tutorial, we had something of a respite from the weekly code editing and ‘hacking’ tasks. I probably should have been a little bit more proactive and done my own code work, but following the homework rush over the holiday ‘break’, I was feeling a little flat and took some time out. Some trolling… er research on the Internet was stressful enough for my weary head this week.
The first thing I came across, whilst not totally related to this project, was quite interesting. It seems that every other week, there is a new release of another open source (or at least freely available) software, ready to take on the established giant. Whilst I don’t put Processing in the category of multinationals like Adobe or Microsoft, it certainly does have something of a following now and there aren’t a lot of other packages that I can think of which enjoy the community following that Processing does. So it’s interesting to see Field come along (also interesting that it came from the same institution that bore Processing – MIT).
I have to say, at first glance it looks like Processing could be under siege here. Whilst each environment uses different languages (Processing is based on Java, whilst Field is based on Python), they are both designed to help you get ideas down as quickly as possible. The area that Processing falls down in here is the need to refresh the applet window every time you update the code (particularly annoying in OS X, seeing as Apple don’t believe in supporting Java fully). However, Field not only looks like it updates changes in real time, but allows you to edit certain parameters in the graphic window itself. Nerdariffic.
I can’t say I have a spare week or so to learn a new code language right now, so Field seems to have been built with lazy people like me in mind: you can run Processing inside Field! I think I’ve prattled on about this long enough, so if you’re interested, here’s the video…
In my presentation last week, I linked to Memo Akten’s ofxMSAPhysics library. It’s not really for Processing, but the traer.physics library that it’s based on is. Taking a look at the examples on the project page, the Bouncy Balls sketch looks promising. It both attracts and repels elements to one another, so if I’m able to add the silhouette of the participant as an element and choose at which time to attract and repel the other elements, this sketch could be particularly useful.
To tell the program when to attract and when to repel the elements, I need a trigger for the change. I think the simplest way to do this will be something along the lines of counting a pixel difference between frames, but I did stumble across something that could be put to interesting use: Gesture Recognition. This video shows handwriting being recognised, but I can’t think of a reason that this couldn’t also recognise body gestures as well…
I think this is something I should leave until I can see how much time I’m going to have left to actually get this work completed. It looks like something I could end up spending a lot of time playing around with, rather than getting the rest of the project underway.
Searching the Discourse forum on the Processing site yielded some interesting results for the RSS feed section of my project. Not only have people been adding RSS feeds into sketches for some time, but there are also people working on feeds to work with the Twitter andlast.fm API, much like my header on this page. This is an interesting idea, because particularly with last.fm, there is a lot more information that can be drawn from than simply headings: statistics; artist bios; relationships between users etc are all stored on last.fm.
Clearly, there’s a lot of information out there that will be useful for my project. Now I’ve just got to start experimenting with it.
Launching into more chapters of Learning Processing over the break, I think I’ve come to the first point of really struggling with the concepts presented in the book. Making my way through ‘Functions’, ‘Objects’ and ‘Arrays’, I feel like we’re starting to get into territory that whilst more difficult to understand, is capable of some really powerful end results.
The Circle Follow example we were given during class showed how motion tracking can actually be relatively simple in Processing. Using the JMyron library though, I’ve found that the results can be a little problematic: the library causes some crashes in OS X. I have briefly looked at some more promising OpenCV tutorials, which I think will be the way forward once I start working on my major project for the semester.
As for the Circle Follow sketch, I made some relatively minor changes this week (I’ve realised that sketches trying to use an attached camera = applet death, so I’ll just link for now). I introduced an ‘if’ statement, whereby if the participant gets the ball in the hole, they are greeted with a (tacky) Win Screen and the loop finishes. I also attempted to introduce some animation to my Win Screen, but had some strange errors that I couldn’t get to the bottom of, so I moved on.
Concept Research
This week being the presentation due date, all my research went into a Keynote presentation and resulting PDF. So there’s probably not much point repeating myself here.
However, I am quite happy with where this project is headed:
This project aims to look at the issues surrounding the ways which we interact with technologies. This includes not just how we try to express ourselves through different media, but also the often hidden methods that these media change both how we are represented and how we perceive the world around us.
I hope to achieve this through showing the participant that you must engage to really have an effect on your environment. That if you choose to remain static, your real self will eventually be replaced by what outside forces wish you to become.
…I’m looking forward to getting down to work with it.
"a reasonable person would find it offensive" - I haven't seen a lot of reason in the art vs child pornography debate... http://j.mp/bxMO2B 23 hours ago
No Comments »
